Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins?

On 26.05.2015 09:06, Neel Natu wrote:
> On Mon, May 25, 2015 at 1:40 PM, Garrett Cooper <yaneurabeya_at_gmail.com> wrote:
>> On Apr 28, 2015, at 0:54, Alexander Motin <mav_at_FreeBSD.org> wrote:
>>> On 27.04.2015 21:17, Garrett Cooper wrote:
>>>> On Apr 27, 2015, at 11:16, Garrett Cooper <yaneurabeya_at_gmail.com>
>>>> wrote:
>>>>
>>>>> I was looking at the console log for the latest kyua run and I’ve
>>>>> noticed that it’s timing out a bit more [1] than it was
>>>>> previously [2]. I’ve seen some of your commits recently to cam(4)
>>>>> dealing with bhyve — has there been a performance regression
>>>>> there? Thanks! -NGie
>>>>>
>>>>> 1.
>>>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console
>>>>> 2.
>>>>> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console
>>>>
>>>> (Sorry for not being more explicit for the archives) These are the
>>>> timeouts I’m referring to:
>>>>
>>>> ahcich0: is 00000000 cs 00000000 ss 1f000000 rs 1f000000 tfd 50
>>>> serr 00000000 cmd 1000dc17 (ada0:ahcich0:0:0:0):
>>>> WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00
>>>> (ada0:ahcich0:0:0:0): CAM status: Command timeout
>>>> (ada0:ahcich0:0:0:0): Retrying command
> 
> Can you try to reproduce with a virtio-blk instead of ahci-hd?
> 
> It will help narrow down the problem because both ahci-hd and
> virtio-blk emulations share a common backend to read/write to the
> disk.

virtio-blk has no command timeouts, so it may be problematic to notice
delayed commands there, unless they stuck forever. This won't be fair
comparison. Though if not use TRIM, virtio-blk should be more efficient
now, as it is supposed to.

>>> Last time I was more working on bhyve host disk emulation, rather then
>>> on cam(4) running on guest. Considering that, what guest and what host
>>> versions are you running? Is there any other load on host except this VM
>>> that could cause I/O delays high enough to trigger timeouts? What are
>>> you using to back the virtual disk (file, zvol, ...)?
>>
>> I have no idea what the Jenkins slaves are running in terms of configuration/version/etc. You’ll have to ask jenkins-admin_at_…
>> Thanks!
>> -NGie


-- 
Alexander Motin