Re: PMC Sierra SPC SAS-SATA Card + ESXi

> On Mar 9, 2019, at 18:50, Evgheni Melman <mevgheni_at_gmail.com> wrote:
> 
> So I got this weird setup, where I need to passthrough a PMC-Sierra PM8003
> card (same as PM8001, but with external QSFP afaik) to FreeBSD 11-STABLE
> running in ESXi.
> The card works fine in baremetal FreeBSD 11. Boot is ok, all disks
> detected. When done in ESXi with passthrough though:
> 1. First try, host crashed with PSOD. After tweaking passthrough.map in
> esxi to do d3d0 reset, host does not psod anymore.
> 2. FreeBSD kernel panics at pciAttach. After analyzing the driver
> (sys\dev\pms), turns out it only works in MSI-X mode. Disabling
> msi_blacklist in loader.conf makes FreeBSD not panic.
> 3. FreeBSD deadlocks on boot. After analyzing the driver more (turning on
> all debug features, and adding a few extra), I got this log (pastebin link:
> https://pastebin.com/cEGwFZZ2). The code tries to initialize SGPIO, but for
> whatever reason does not get an interrupt reply when run on ESXi which
> makes it loop for infinity waiting for said interrupt.
> 
> The deadlock happens in the
> 
>    while(!sgpioResponseSet)
>    {
> tiCOMDelayedInterruptHandler(tiRoot, 0, 1, tiNonInterruptContext);
>    }
> 
> section of SendSgpioRequest in
> sys\dev\pms\RefTisa\tisa\sassata\common\tdport.c
> 
> Sgpi seems not-so-critical feature to me, so I just limited the while loop
> to 100 iterations, which lets SendSgpioRequest return an error that is not
> considered fatal, and lets the code execute further which can be seen in
> the log: https://pastebin.com/Rk6MD5ci (disregard the final fatal trap, I
> think it is caused by me and/or my debugging code, same modifications
> without debugging features do not trap and I get same trap on baremetal
> with my debug kernel).
> 
> With those modifications, pciAttach is successful, I finally get a boot,
> the card exists in lspci, BUT no drives connected to it are detected.
> 
> At this stage, I am out of ideas, looking for suggestions.

If you’re trying to get the host production ready, I recommend using an LSI/Avago based card (or emulate it in ESXi), or avoid FreeBSD if you can. The driver in FreeBSD is based on a Linux driver that’s over a decade old and had a number of stability bugs the last time I used it [2 years ago].

-Enji