Re: CURRENT crashes at early boot on Lenovo T540p: rtsx to blame - 13.0-RELEASE crashes same way!

From: Lev Serebryakov <lev_at_FreeBSD.org>
Date: Wed, 12 May 2021 18:17:33 +0300
On 12.05.2021 16:35, Henri Hennebert wrote:

>>> It would be fine if you can test the driver with dev.rtsx.0.inversion=1 in loader.conf and see it it solve the problem.
>>> the output of
>>> sysctl dev.rtsx  and
>>> kenv | grep smbios.system
>>> would be useful
>>>
>>> BTW is a dummy card inserted in the SD slot?
>>>
>>> Thank for your time
>>
>>   My T540p:
>>
>>   (1) rtsx in the kernel, enabled in BIOS, no settings in loader.conf, EMPTY SLOT — panic on boot, typically WITHOUT rtsx in stack trace.
>>
>>   (2) rtsx in the kernel, enabled in BIOS, no settings in loader.conf, SD CARD IN THE SLOT — no panic on boot, but WiFi card detected
>>       too late for startup scripts to run wpa_supplicant properly.
>>
>>   (3) rtsx in the kernel, enabled in BIOS, "dev.rtsx.0.inversion=1", SD CARD IN THE SLOT — no panic on boot, but WiFi card detected
>>       too late for startup scripts to run wpa_supplicant properly. And card reporting is inverted related to real deal: rtsx0
>>       reports "Card present" when I remove card and vice versa.
>>
>>   (4) rtsx in the kernel, disabled in BIOS — device not found, everything (but SD reader) boots & works (as expected!)
>>
>>   (5) rtsx in the kernel, enabled in BIOS, "dev.rtsx.0.inversion=1", EMPTY SLOT — boots, but prints out "timeout for CMD8/55/1" for very long time and *console*is*not*accessible* till "no compatible cards found on bus".
>>       ALSO (!) wifi card is found only AFTER all these timeouts, when startup scripts are FAILED to attach to wireless network (!!!).
>>       ALSO (!) it says "Card Absent" when I *INSERT* card after boot, and "Card present" (+ a lot of timeouts again) when I *REMOVE* card, looks like this "inversion" is wrong for my hardware.
>>       So, it boots, but practically unusable.
>>
>>   (6) rtsx in the kernel, enabled in BIOS, "dev.rtsx.0.inversion=1", SD CARD IN THE SLOT
>>
>>   (7) rtsx is loaded as module after boot (manually, from console), enabled in BIOS, no setting in loader.conf — looks to work properly.
>>       Card could be mounted, read/write, it works.
>>
>>   (9) rtsx is loaded as module after boot (manually, from console), enabled in BIOS, "dev.rtsx.0.inversion=1" — don't panic, but thinks wrong about card state (Card Present/Absent is inverted, as instructed).
>>
>>   My theory: rtsx detect/attach code has some race conditions / incompatibilities for multi-core boot when card is not present, and it thrash kernel memory and/or block something on boot. "inversion" removes this bad behavior due to waiting for card commands (as with "inversion" it thinks card is here and try to access it).
> 
> Firts, thank you for your exhaustive testing!  Your analysis seems pertinent for me.
> 
> What seems strange: case (1) and (3) follow the same path in the driver but (1) produce a panic.

  Indeed. But boot panics is not 100%, it is about 95%, and I've tested each case only one time...

> 
>>
>>   See (2), (3) and (4) — when rtsx doesn't cause panic on boot, it still mangle other devices detection & initialization.
> 
> booting in case (2) can you show the output of
> 
> pciconf -lvbc

hostb0_at_pci0:0:0:0:	class=0x060000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x0c04 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = 'Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller'
     class      = bridge
     subclass   = HOST-PCI
     cap 09[e0] = vendor (length 12) Intel cap 0 version 1
pcib1_at_pci0:0:1:0:	class=0x060400 rev=0x06 hdr=0x01 vendor=0x8086 device=0x0c01 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = 'Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller'
     class      = bridge
     subclass   = PCI-PCI
     cap 0d[88] = PCI Bridge subvendor=0x17aa subdevice=0x2210
     cap 01[80] = powerspec 3  supports D0 D3  current D0
     cap 05[90] = MSI supports 1 message
     cap 10[a0] = PCI-Express 2 root port max data 256(256)
                  max read 128
                  link x8(x8) speed 2.5(8.0) ASPM L0s/L1(L0s/L1)
                  slot 1 power limit 75000 mW
     ecap 0002[100] = VC 1 max VC0
     ecap 0005[140] = Root Complex Link Declaration 1
     ecap 0019[d94] = PCIe Sec 1 lane errors 0xfe
vgapci1_at_pci0:0:2:0:	class=0x030000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x0416 subvendor=0x17aa subdevice=0x221e
     vendor     = 'Intel Corporation'
     device     = '4th Gen Core Processor Integrated Graphics Controller'
     class      = display
     subclass   = VGA
     bar   [10] = type Memory, range 64, base 0xf3000000, size 4194304, enabled
     bar   [18] = type Prefetchable Memory, range 64, base 0xc0000000, size 536870912, enabled
     bar   [20] = type I/O Port, range 32, base 0x5000, size 64, enabled
     cap 05[90] = MSI supports 1 message enabled with 1 message
     cap 01[d0] = powerspec 2  supports D0 D3  current D0
     cap 13[a4] = PCI Advanced Features: FLR TP
hdac0_at_pci0:0:3:0:	class=0x040300 rev=0x06 hdr=0x00 vendor=0x8086 device=0x0c0c subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = 'Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller'
     class      = multimedia
     subclass   = HDA
     bar   [10] = type Memory, range 64, base 0xf4630000, size 16384, enabled
     cap 01[50] = powerspec 2  supports D0 D3  current D0
     cap 05[60] = MSI supports 1 message enabled with 1 message
     cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR NS
                  max read 128
xhci0_at_pci0:0:20:0:	class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c31 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family USB xHCI'
     class      = serial bus
     subclass   = USB
     bar   [10] = type Memory, range 64, base 0xf4620000, size 65536, enabled
     cap 01[70] = powerspec 2  supports D0 D3  current D0
     cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
none0_at_pci0:0:22:0:	class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3a subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family MEI Controller'
     class      = simple comms
     bar   [10] = type Memory, range 64, base 0xf4639000, size 16, enabled
     cap 01[50] = powerspec 3  supports D0 D3  current D0
     cap 05[8c] = MSI supports 1 message, 64 bit
em0_at_pci0:0:25:0:	class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x153a subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = 'Ethernet Connection I217-LM'
     class      = network
     subclass   = ethernet
     bar   [10] = type Memory, range 32, base 0xf4600000, size 131072, enabled
     bar   [14] = type Memory, range 32, base 0xf463f000, size 4096, enabled
     bar   [18] = type I/O Port, range 32, base 0x5080, size 32, enabled
     cap 01[c8] = powerspec 2  supports D0 D3  current D0
     cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
     cap 13[e0] = PCI Advanced Features: FLR TP
ehci0_at_pci0:0:26:0:	class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c2d subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family USB EHCI'
     class      = serial bus
     subclass   = USB
     bar   [10] = type Memory, range 32, base 0xf463e000, size 1024, enabled
     cap 01[50] = powerspec 2  supports D0 D3  current D0
     cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
     cap 13[98] = PCI Advanced Features: FLR TP
hdac1_at_pci0:0:27:0:	class=0x040300 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c20 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset High Definition Audio Controller'
     class      = multimedia
     subclass   = HDA
     bar   [10] = type Memory, range 64, base 0xf4634000, size 16384, enabled
     cap 01[50] = powerspec 2  supports D0 D3  current D0
     cap 05[60] = MSI supports 1 message, 64 bit enabled with 1 message
     cap 10[70] = PCI-Express 1 root endpoint max data 128(128) FLR
                  max read 128
     ecap 0002[100] = VC 1 max VC1
pcib2_at_pci0:0:28:0:	class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x8c10 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
     class      = bridge
     subclass   = PCI-PCI
     cap 10[40] = PCI-Express 2 root port max data 128(128)
                  max read 128
                  link x1(x1) speed 2.5(5.0) ASPM L0s/L1(L0s/L1)
                  slot 0 power limit 100 mW
     cap 05[80] = MSI supports 1 message
     cap 0d[90] = PCI Bridge subvendor=0x17aa subdevice=0x2210
     cap 01[a0] = powerspec 3  supports D0 D3  current D0
pcib3_at_pci0:0:28:1:	class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x8c12 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
     class      = bridge
     subclass   = PCI-PCI
     cap 10[40] = PCI-Express 2 root port max data 128(128)
                  max read 128
                  link x1(x1) speed 2.5(5.0) ASPM L1(L0s/L1)
                  slot 1 power limit 100 mW
     cap 05[80] = MSI supports 1 message
     cap 0d[90] = PCI Bridge subvendor=0x17aa subdevice=0x2210
     cap 01[a0] = powerspec 3  supports D0 D3  current D0
pcib4_at_pci0:0:28:2:	class=0x060400 rev=0xd4 hdr=0x01 vendor=0x8086 device=0x8c14 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
     class      = bridge
     subclass   = PCI-PCI
     cap 10[40] = PCI-Express 2 root port max data 128(128)
                  max read 128
                  link x0(x1) speed 0.0(5.0) ASPM L0s/L1(L0s/L1)
                  slot 2 power limit 100 mW HotPlug(empty) surprise
     cap 05[80] = MSI supports 1 message enabled with 1 message
     cap 0d[90] = PCI Bridge subvendor=0x17aa subdevice=0x2210
     cap 01[a0] = powerspec 3  supports D0 D3  current D0
ehci1_at_pci0:0:29:0:	class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c26 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family USB EHCI'
     class      = serial bus
     subclass   = USB
     bar   [10] = type Memory, range 32, base 0xf463d000, size 1024, enabled
     cap 01[50] = powerspec 2  supports D0 D3  current D0
     cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
     cap 13[98] = PCI Advanced Features: FLR TP
isab0_at_pci0:0:31:0:	class=0x060100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c4f subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = 'QM87 Express LPC Controller'
     class      = bridge
     subclass   = PCI-ISA
     cap 09[e0] = vendor (length 12) Intel cap 1 version 0
		 features: AMT, 4 PCI-e x1 slots
ahci0_at_pci0:0:31:2:	class=0x010601 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c03 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]'
     class      = mass storage
     subclass   = SATA
     bar   [10] = type I/O Port, range 32, base 0x50a8, size 8, enabled
     bar   [14] = type I/O Port, range 32, base 0x50b4, size 4, enabled
     bar   [18] = type I/O Port, range 32, base 0x50a0, size 8, enabled
     bar   [1c] = type I/O Port, range 32, base 0x50b0, size 4, enabled
     bar   [20] = type I/O Port, range 32, base 0x5060, size 32, enabled
     bar   [24] = type Memory, range 32, base 0xf463c000, size 2048, enabled
     cap 05[80] = MSI supports 1 message enabled with 1 message
     cap 01[70] = powerspec 3  supports D0 D3  current D0
     cap 12[a8] = SATA Index-Data Pair
ichsmb0_at_pci0:0:31:3:	class=0x0c0500 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c22 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Intel Corporation'
     device     = '8 Series/C220 Series Chipset Family SMBus Controller'
     class      = serial bus
     subclass   = SMBus
     bar   [10] = type Memory, range 64, base 0xf4638000, size 256, enabled
     bar   [20] = type I/O Port, range 32, base 0xefa0, size 32, enabled
vgapci0_at_pci0:1:0:0:	class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1290 subvendor=0x17aa subdevice=0x221e
     vendor     = 'NVIDIA Corporation'
     device     = 'GK208M [GeForce GT 730M]'
     class      = display
     subclass   = VGA
     bar   [10] = type Memory, range 32, base 0xf2000000, size 16777216, enabled
     bar   [14] = type Prefetchable Memory, range 64, base 0xe0000000, size 268435456, enabled
     bar   [1c] = type Prefetchable Memory, range 64, base 0xf0000000, size 33554432, enabled
     bar   [24] = type I/O Port, range 32, base 0x4000, size 128, enabled
     cap 01[60] = powerspec 3  supports D0 D3  current D0
     cap 05[68] = MSI supports 1 message, 64 bit
     cap 10[78] = PCI-Express 2 endpoint max data 256(256) RO NS
                  max read 512
                  link x8(x8) speed 2.5(8.0) ASPM L0s/L1(L0s/L1) ClockPM disabled
     ecap 0002[100] = VC 1 max VC0
     ecap 0004[128] = Power Budgeting 1
     ecap 000b[600] = Vendor [1] ID 0001 Rev 1 Length 36
     ecap 0019[900] = PCIe Sec 1 lane errors 0
rtsx0_at_pci0:3:0:0:	class=0xff0000 rev=0x01 hdr=0x00 vendor=0x10ec device=0x5227 subvendor=0x17aa subdevice=0x2210
     vendor     = 'Realtek Semiconductor Co., Ltd.'
     device     = 'RTS5227 PCI Express Card Reader'
     bar   [10] = type Memory, range 32, base 0xf4500000, size 4096, enabled
     cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
     cap 05[50] = MSI supports 1 message, 64 bit enabled with 1 message
     cap 10[70] = PCI-Express 2 endpoint max data 128(128) RO
                  max read 512
                  link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM enabled
     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
     ecap 0003[140] = Serial 1 00000001004ce000
     ecap 0018[150] = LTR 1
     ecap 001e[158] = L1 PM Substates 1
iwm0_at_pci0:4:0:0:	class=0x028000 rev=0x6b hdr=0x00 vendor=0x8086 device=0x08b2 subvendor=0x8086 subdevice=0xc262
     vendor     = 'Intel Corporation'
     device     = 'Wireless 7260'
     class      = network
     bar   [10] = type Memory, range 64, base 0xf4400000, size 8192, enabled
     cap 01[c8] = powerspec 3  supports D0 D3  current D0
     cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
     cap 10[40] = PCI-Express 2 endpoint max data 128(128) FLR RO NS
                  max read 128
                  link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ClockPM enabled
     ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
     ecap 0003[140] = Serial 1 7c7a91ffff988046
     ecap 0018[14c] = LTR 1
     ecap 000b[154] = Vendor [1] ID cafe Rev 1 Length 20


> and
> 
> vmstat -i

interrupt                          total       rate
irq1: atkbd0                         278          3
irq9: acpi0                          232          3
irq16: ehci0                         171          2
irq23: ehci1                         226          3
cpu0:timer                          3636         43
cpu1:timer                          2033         24
cpu2:timer                          1839         22
cpu3:timer                          1610         19
cpu4:timer                          2363         28
cpu5:timer                          3004         36
cpu6:timer                          2990         36
cpu7:timer                          2894         34
irq32: hdac0                           6          0
irq33: xhci0                      109175       1299
irq35: hdac1                          45          1
irq36: rtsx0                         359          4
irq38: ahci0                        3098         37
irq39: iwm0                          550          7
irq40: vgapci1                        82          1
Total                             134591       1602



-- 
// Lev Serebryakov
Received on Wed May 12 2021 - 13:17:39 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:28 UTC