Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

From: Steve Kargl <sgk_at_troutmask.apl.washington.edu>
Date: Fri, 22 Feb 2019 19:26:44 -0800
On Thu, Feb 21, 2019 at 10:04:10PM -0800, Steve Kargl wrote:
> On Thu, Feb 21, 2019 at 07:39:25PM -0800, Steve Kargl wrote:
> > r343567 merges the PAE vs non-PAE pmap headers for i386
> > freebsd.  After bisection and dealing with the drm-legacy-kmod
> > fallout, I bisected /usr/src to r343567.  Building world and
> > a GENERIC kernel and the minimum set of ports to start Xorg
> > on my Dell Latitude D530 laptop, results in a black screen
> > of death and a locked up laptop (no keyboard, mouse, or video).
> > 
> > A comparison of /etc/log/Xorg.0.log for r343566 (Xorg loads
> > and functions) and r353467 (Xorg black screen of death) shows
> > that /boot/modules/i915kms.ko loads correctly as the log
> > files are identical.
> > 
> > Comparing dmesg for r343566 to r343567 shows the following
> >  
> > --- dmesg.343566	2019-02-20 08:13:07.727202000 -0800
> > +++ dmesg.343567	2019-02-21 19:02:24.469562000 -0800
> > _at__at_ -3,11 +3,11 _at__at_
> >  Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> >  	The Regents of the University of California. All rights reserved.
> >  FreeBSD is a registered trademark of The FreeBSD Foundation.
> > -FreeBSD 13.0-CURRENT r343566 GENERIC i386
> > +FreeBSD 13.0-CURRENT r343567 GENERIC i386
> >  FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1)
> >  WARNING: WITNESS option enabled, expect reduced performance.
> >  VT(vga): resolution 640x480
> > -CPU: Intel(R) Core(TM)2 Duo CPU     T7250  _at_ 2.00GHz (1995.05-MHz 686-class CPU)
> > +CPU: Intel(R) Core(TM)2 Duo CPU     T7250  _at_ 2.00GHz (1995.04-MHz 686-class CPU)
> >    Origin="GenuineIntel"  Id=0x6fd  Family=0x6  Model=0xf  Stepping=13
> >    Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >    Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
> > _at__at_ -16,7 +16,7 _at__at_
> >    VT-x: (disabled in BIOS) HLT,PAUSE
> >    TSC: P-state invariant, performance statistics
> >  real memory  = 4294967296 (4096 MB)
> > -avail memory = 3639914496 (3471 MB)
> > +avail memory = 4154175488 (3961 MB)
> > 
> > Somehow the r343567 kernel found an addition 490 MB of memory,
> > which leads me to believe the after loading i915kms.ko there
> > is some serious memory stomping issues.
> > 
> > I willing to do whatever is necessary to fix this issue (shorter
> > of mailing the laptop to someone).  Is it possible to revert
> > r343567 and move forward? 
> > 
> 
> More info from sysctl.  With the "good" r343566, I see
> 
> vm.kmem_map_free: 1187033088
> vm.kmem_map_size: 27234304
> vm.kmem_size_scale: 3
> vm.kmem_size_max: 1715470336
> vm.kmem_size_min: 12582912
> vm.kmem_zmax: 65536
> vm.kmem_size: 1214267392
> hw.physmem: 3714269184
> hw.usermem: 3650867200
> hw.realmem: 4294963200
> 
> With the problematic r343567, I see
> 
> vm.kmem_map_free: 1683152896
> vm.kmem_map_size: 28123136
> vm.kmem_size_scale: 1
> vm.kmem_size_max: 1711276032
> vm.kmem_size_min: 12582912
> vm.kmem_zmax: 65536
> vm.kmem_size: 1711276032
> hw.physmem: 4252360704
> hw.usermem: 4146999296
> hw.realmem: 4294963200
> 
> Ideas?
> 

Here's the 'diff -uw' between a verbose dmesg boot of r343566
and dmesg boot of r343567.  The memory size looks rather puzzling.
Can the people responsible for the i386 pmap.h merging take a
look?

--- dmesg.343566.verbose	2019-02-22 19:08:33.458559000 -0800
+++ dmesg.343567.verbose	2019-02-22 08:55:21.623310000 -0800
_at__at_ -8,25 +8,25 _at__at_
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
-FreeBSD 13.0-CURRENT r343566 GENERIC i386
+FreeBSD 13.0-CURRENT r343567 GENERIC i386
 FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1)
 WARNING: WITNESS option enabled, expect reduced performance.
 VT(vga): resolution 640x480
-Preloaded elf kernel "/boot/kernel/kernel" at 0x2501000.
-Preloaded boot_entropy_cache "/boot/entropy" at 0x2509b20.
-Preloaded elf module "/boot/kernel/cpuctl.ko" at 0x2509b6c.
-Preloaded elf module "/boot/kernel/if_ath.ko" at 0x2509f68.
-Preloaded elf module "/boot/kernel/ath_dfs.ko" at 0x250a314.
-Preloaded elf module "/boot/kernel/ath_rate.ko" at 0x250a6c0.
-Preloaded elf module "/boot/kernel/ath_hal_ar9300.ko" at 0x250aa70.
-Preloaded elf module "/boot/kernel/ath_hal_ar5416.ko" at 0x250ae9c.
-Preloaded elf module "/boot/kernel/ath_hal_ar5212.ko" at 0x250b2c8.
-Preloaded elf module "/boot/kernel/ath_hal_ar5211.ko" at 0x250b6f4.
-Preloaded elf module "/boot/kernel/ath_hal_ar5210.ko" at 0x250baf8.
+Preloaded elf kernel "/boot/kernel/kernel" at 0x2528000.
+Preloaded elf module "/boot/kernel/cpuctl.ko" at 0x2530b20.
+Preloaded boot_entropy_cache "/boot/entropy" at 0x2530f1c.
+Preloaded elf module "/boot/kernel/if_ath.ko" at 0x2530f68.
+Preloaded elf module "/boot/kernel/ath_dfs.ko" at 0x2531314.
+Preloaded elf module "/boot/kernel/ath_rate.ko" at 0x25316c0.
+Preloaded elf module "/boot/kernel/ath_hal_ar9300.ko" at 0x2531a70.
+Preloaded elf module "/boot/kernel/ath_hal_ar5416.ko" at 0x2531e9c.
+Preloaded elf module "/boot/kernel/ath_hal_ar5212.ko" at 0x25322c8.
+Preloaded elf module "/boot/kernel/ath_hal_ar5211.ko" at 0x25326f4.
+Preloaded elf module "/boot/kernel/ath_hal_ar5210.ko" at 0x2532af8.
 Table 'FACP' at 0xdf6c1c9c
 FACP: Found table at 0xdf6c1c9c
-Calibrating TSC clock ... TSC clock: 1995051200 Hz
-CPU: Intel(R) Core(TM)2 Duo CPU     T7250  _at_ 2.00GHz (1995.05-MHz 686-class CPU)
+Calibrating TSC clock ... TSC clock: 1995040840 Hz
+CPU: Intel(R) Core(TM)2 Duo CPU     T7250  _at_ 2.00GHz (1995.04-MHz 686-class CPU)
   Origin="GenuineIntel"  Id=0x6fd  Family=0x6  Model=0xf  Stepping=13
   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
_at__at_ -53,14 +53,15 _at__at_
 Physical memory chunk(s):
 0x0000000000001000 - 0x000000000009dfff, 643072 bytes (157 pages)
 0x0000000000100000 - 0x00000000007fffff, 7340032 bytes (1792 pages)
-0x0000000002829000 - 0x00000000db998fff, 3642163200 bytes (889200 pages)
-avail memory = 3639914496 (3471 MB)
+0x00000000026ff000 - 0x00000000da2bcfff, 3619414016 bytes (883646 pages)
+0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages)
+avail memory = 4154175488 (3961 MB)
 MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
 SMP: Added CPU 0 (AP)
 MADT: Found CPU APIC ID 1 ACPI ID 1: enabled
 SMP: Added CPU 1 (AP)
 Event timer "LAPIC" quality 100
-LAPIC: ipi_wait() us multiplier 45 (r 4401300 tsc 1995051200)
+LAPIC: ipi_wait() us multiplier 45 (r 4401350 tsc 1995040840)
 ACPI APIC Table: <DELL   M08    >
 Package ID shift: 1
 L2 cache ID shift: 1
_at__at_ -76,7 +77,6 _at__at_
 		CPU1 (AP): APIC ID: 1
 APIC: CPU 0 has ACPI ID 0
 APIC: CPU 1 has ACPI ID 1
-Pentium Pro MTRR support enabled
 bios32: Found BIOS32 Service Directory header at 0x4ffa00
 bios32: Entry = 0xffa10 (4ffa10)  Rev = 0  Len = 1
 pcibios: PCI BIOS entry at 0xf0000+0xab56
_at__at_ -84,6 +84,7 _at__at_
 pnpbios: Entry = f0000:e2f4  Rev = 1.0
 pnpbios: Event flag at 4b4
 Other BIOS signatures found:
+Pentium Pro MTRR support enabled
 random: read 3840 bytes from preloaded cache
 random: unblocking device.
 arc4random: read 32 bytes from preloaded cache
_at__at_ -117,7 +118,7 _at__at_
 lapic1: LINT1 trigger: edge
 lapic1: LINT1 polarity: high
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
-lapic: Divisor 2, Frequency 99752593 Hz
+lapic: Divisor 2, Frequency 99752070 Hz
 cpu0 BSP:
      ID: 0x00000000   VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
_at__at_ -129,7 +130,7 _at__at_
   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
 TSC timecounter disables C2 and C3.
 SMP: passed TSC synchronization test
-Timecounter "TSC" frequency 1995051200 Hz quality 1000
+Timecounter "TSC" frequency 1995040840 Hz quality 1000
 random: entropy device external interface
 wlan: <802.11 Link Layer>
 snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
_at__at_ -138,7 +139,7 _at__at_
 [ath_hal] loaded
 nfslock: pseudo-device
 crypto: <crypto core>
-module_register_init: MOD_LOAD (vesa, 0x1669cc0, 0) error 19
+module_register_init: MOD_LOAD (vesa, 0x166cb90, 0) error 19
 io: <I/O>
 kbd: new array size 4
 kbd1 at kbdmux0
_at__at_ -152,9 +153,9 _at__at_
 [ar5211] loaded
 [ar5210] loaded
 [ath] loaded
-hpt27xx: RocketRAID 27xx controller driver v1.2.8
 hptnr: R750/DC7280 controller driver v1.1.5
 hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
+hpt27xx: RocketRAID 27xx controller driver v1.2.8
 nexus0
 vtvga0: <VT VGA driver> on motherboard
 cryptosoft0: <software crypto> on motherboard
_at__at_ -204,8 +205,8 _at__at_
 PCIe: Memory Mapped configuration base _at_ 0xf8000000
 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
 ACPI: Dynamic OEM Table Load:
-ACPI: SSDT 0x0000000009FFEA00 000043 (v01 LMPWR  DELLLOM  00001001 INTL 20050624)
-acpi0: wakeup code va 0xa1d1000 pa 0x1000
+ACPI: SSDT 0x000000001A404A80 000043 (v01 LMPWR  DELLLOM  00001001 INTL 20050624)
+acpi0: wakeup code va 0x1a452000 pa 0x1000
 Table 'FACP' at 0xdf6c1c9c
 FACP: Found table at 0xdf6c1c9c
 Table 'FACP' at 0xdf6c1c9c
_at__at_ -234,15 +235,15 _at__at_
 cpu0: Processor \134_PR_.CPU0 (ACPI ID 0) -> APIC ID 0
 cpu0: <ACPI CPU> on acpi0
 ACPI: Dynamic OEM Table Load:
-ACPI: SSDT 0x000000000A21C000 000286 (v01 PmRef  Cpu0Ist  00003000 INTL 20050624)
+ACPI: SSDT 0x000000001A422C00 000286 (v01 PmRef  Cpu0Ist  00003000 INTL 20050624)
 ACPI: Dynamic OEM Table Load:
-ACPI: SSDT 0x000000000A22D800 0004B6 (v01 PmRef  Cpu0Cst  00003001 INTL 20050624)
+ACPI: SSDT 0x000000001A44B800 0004B6 (v01 PmRef  Cpu0Cst  00003001 INTL 20050624)
 cpu1: Processor \134_PR_.CPU1 (ACPI ID 1) -> APIC ID 1
 cpu1: <ACPI CPU> on acpi0
 ACPI: Dynamic OEM Table Load:
-ACPI: SSDT 0x000000000A235100 0000C4 (v01 PmRef  Cpu1Ist  00003000 INTL 20050624)
+ACPI: SSDT 0x000000001A428D00 0000C4 (v01 PmRef  Cpu1Ist  00003000 INTL 20050624)
 ACPI: Dynamic OEM Table Load:
-ACPI: SSDT 0x000000000A235000 000085 (v01 PmRef  Cpu1Cst  00003000 INTL 20050624)
+ACPI: SSDT 0x000000001A428C00 000085 (v01 PmRef  Cpu1Cst  00003000 INTL 20050624)
 Table 'FACP' at 0xdf6c1c9c
 FACP: Found table at 0xdf6c1c9c
 atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
_at__at_ -526,7 +527,7 _at__at_
 vgapci0: <VGA-compatible display> port 0xeff8-0xefff mem 0xfea00000-0xfeafffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
 agp0: <Intel GM965 SVGA controller> on vgapci0
 pcib0: allocated type 3 (0xf0000000-0xf0000fff) for rid 64 of agp0
-agp0: Allocated flush page phys 0xf0000000 virt 0x18fba000
+agp0: Allocated flush page phys 0xf0000000 virt 0xd522000
 agp0: aperture size is 256M, detected 7676k stolen memory
 agp0: AGP_I965_PGTBL_CTL2: 00000000
 agp0: AGP_I855_GCC1: 0x30
_at__at_ -933,9 +934,9 _at__at_
 IPsec: Initialized Security Association Processing.
 tcp_init: net.inet.tcp.tcbhashsize auto tuned to 32768
 AcpiOsExecute: enqueue 6 pending tasks
-hpt27xx: no controller detected.
 hptnr: no controller detected.
 hptrr: no controller detected.
+hpt27xx: no controller detected.
 hdacc0: <Sigmatel STAC9205X HDA CODEC> at cad 0 on hdac0
 hdaa0: <Sigmatel STAC9205X Audio Function Group> at nid 1 on hdacc0
 hdaa0: Subsystem ID: 0x10280200
_at__at_ -1057,27 +1058,27 _at__at_
 pcm0: Automatically set rec source to: monitor
 pcm0: Recording channel set is: Front Left, Front Right, 
 pcm0: Recording channel matrix is: 2.0 (unknown)
-ugen1.1: <Intel UHCI root HUB> at usbus1
+ugen5.1: <Intel UHCI root HUB> at usbus5
 ugen6.1: <Intel EHCI root HUB> at usbus6
-uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
+uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
-ugen2.1: <Intel EHCI root HUB> at usbus2
-ugen5.1: <Intel UHCI root HUB> at usbus5
-uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
-ugen4.1: <Intel UHCI root HUB> at usbus4
 ugen0.1: <Intel UHCI root HUB> at usbus0
-uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
+ugen4.1: <Intel UHCI root HUB> at usbus4
+uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen3.1: <Intel UHCI root HUB> at usbus3
+ugen1.1: <Intel UHCI root HUB> at usbus1
+uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
+uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
+ugen2.1: <Intel EHCI root HUB> at usbus2
 ata0: reset tp1 mask=03 ostat0=50 ostat1=01
-uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
+uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 acpi_acad0: acline initialization start
 acpi_acad0: On Line
 acpi_acad0: acline initialization done, tried 1 times
 battery0: battery initialization start
-battery1: battery initialization start
 battery0: battery initialization done, tried 1 times
-uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
-uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
+battery1: battery initialization start
+uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
 ata0: stat0=0x00 err=0x01 lsb=0x14 msb=0xeb
 ata0: stat1=0x01 err=0x04 lsb=0x00 msb=0x00
 ata0: reset tp2 stat0=00 stat1=01 devices=0x10000
_at__at_ -1092,10 +1093,6 _at__at_
 ses0 at ahciem0 bus 0 scbus3 target 0 lun 0
 ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
 ses0: SEMB SES Device
-cd0 at ata0 bus 0 scbus0 target 0 lun 0
-cd0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device
-cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
-cd0: 680MB (348277 2048 byte sectors)
 ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
 ada0: <Patriot Burst SBFM91.0> ACS-4 ATA SATA 3.x device
 ada0: Serial Number 1B0607771A0800257271
_at__at_ -1104,11 +1101,15 _at__at_
 ada0: 114473MB (234441648 512 byte sectors)
 ses0: Generation Code 0x0 has 1 SubEnclosures
 pass0 at ata0 bus 0 scbus0 target 0 lun 0
-ses0: pass0:  SubEnclosure ID 0, 1 Types With this ID, Descriptor Length 36, offset 8
-<SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device
-ses0: pass0: 33.300MB/s transfersWWN: 0
-ses0:  Type Desc[0]: Type 0x17, MaxElt 3, In Subenc 0, Text Length 0: 
+ses0:  SubEnclosure ID 0, 1 Types With this ID, Descriptor Length 36, offset 8
+pass0: ses0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device
+WWN: 0
+ses0: pass0: 33.300MB/s transfers Type Desc[0]: Type 0x17, MaxElt 3, In Subenc 0, Text Length 0: 
  (UDMA2, ATAPI 12bytes, PIO 65534bytes)
+cd0 at ata0 bus 0 scbus0 target 0 lun 0
+cd0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device
+cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
+cd0: 680MB (348277 2048 byte sectors)
 pass1 at ahcich0 bus 0 scbus1 target 0 lun 0
 pass1: <Patriot Burst SBFM91.0> ACS-4 ATA SATA 3.x device
 pass1: Serial Number 1B0607771A0800257271
_at__at_ -1119,15 +1120,15 _at__at_
 Trying to mount root from ufs:/dev/ada0s1a [rw]...
 WARNING: WITNESS option enabled, expect reduced performance.
 uhub0: 2 ports with 2 removable, self powered
+uhub2: 2 ports with 2 removable, self powered
+uhub5: 2 ports with 2 removable, self powered
 uhub3: 2 ports with 2 removable, self powered
 uhub4: 2 ports with 2 removable, self powered
-uhub5: 2 ports with 2 removable, self powered
-uhub6: 2 ports with 2 removable, self powered
-uhub2: 4 ports with 4 removable, self powered
+uhub6: 4 ports with 4 removable, self powered
+uhub1: 6 ports with 6 removable, self powered
 GEOM: new disk ada0
 atrtc0: providing initial system time
 start_init: trying /sbin/init
-uhub1: 6 ports with 6 removable, self powered
 ugen6.2: <Toshiba External USB HDD> at usbus6
 umass0 on uhub1
 umass0: <Bulk Only Interface> on usbus6
_at__at_ -1146,7 +1147,7 _at__at_
 da0: quirks=0x2<NO_6_BYTE>
 da0: Delete methods: <NONE(*),ZERO>
 ugen4.2: <Logitech USB Receiver> at usbus4
-ukbd0 on uhub3
+ukbd0 on uhub4
 ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4
 kbd2 at ukbd0
 kbd2: ukbd0, generic (0), config:0x0, flags:0x3d0000
_at__at_ -1154,10 +1155,10 _at__at_
 wlan0: bpf attached
 wlan0: Ethernet address: 00:1c:bf:90:ab:44
 lo0: link state changed to UP
-firmware: 'wpifw' version 153229: 150100 bytes loaded at 0x2f809000
+firmware: 'wpifw' version 153229: 150100 bytes loaded at 0x34009000
 wlan0: link state changed to UP
-ums0 on uhub3
+ums0 on uhub4
 ums0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4
 ums0: 16 buttons and [XYZT] coordinates ID=2
-uhid0 on uhub3
+uhid0 on uhub4
 uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4


-- 
Steve
20170425 https://www.youtube.com/watch?v=VWUpyCsUKR4
20161221 https://www.youtube.com/watch?v=IbCHE-hONow
Received on Sat Feb 23 2019 - 02:26:49 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:20 UTC