On Fri, Feb 22, 2019 at 07:26:44PM -0800, Steve Kargl wrote: > On Thu, Feb 21, 2019 at 10:04:10PM -0800, Steve Kargl wrote: > > On Thu, Feb 21, 2019 at 07:39:25PM -0800, Steve Kargl wrote: > > > r343567 merges the PAE vs non-PAE pmap headers for i386 > > > freebsd. After bisection and dealing with the drm-legacy-kmod > > > fallout, I bisected /usr/src to r343567. Building world and > > > a GENERIC kernel and the minimum set of ports to start Xorg > > > on my Dell Latitude D530 laptop, results in a black screen > > > of death and a locked up laptop (no keyboard, mouse, or video). > > > > > > A comparison of /etc/log/Xorg.0.log for r343566 (Xorg loads > > > and functions) and r353467 (Xorg black screen of death) shows > > > that /boot/modules/i915kms.ko loads correctly as the log > > > files are identical. > > > > > > Comparing dmesg for r343566 to r343567 shows the following > > > > > > --- dmesg.343566 2019-02-20 08:13:07.727202000 -0800 > > > +++ dmesg.343567 2019-02-21 19:02:24.469562000 -0800 > > > _at__at_ -3,11 +3,11 _at__at_ > > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > > > The Regents of the University of California. All rights reserved. > > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > > -FreeBSD 13.0-CURRENT r343566 GENERIC i386 > > > +FreeBSD 13.0-CURRENT r343567 GENERIC i386 > > > FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) > > > WARNING: WITNESS option enabled, expect reduced performance. > > > VT(vga): resolution 640x480 > > > -CPU: Intel(R) Core(TM)2 Duo CPU T7250 _at_ 2.00GHz (1995.05-MHz 686-class CPU) > > > +CPU: Intel(R) Core(TM)2 Duo CPU T7250 _at_ 2.00GHz (1995.04-MHz 686-class CPU) > > > Origin="GenuineIntel" Id=0x6fd Family=0x6 Model=0xf Stepping=13 > > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > > > _at__at_ -16,7 +16,7 _at__at_ > > > VT-x: (disabled in BIOS) HLT,PAUSE > > > TSC: P-state invariant, performance statistics > > > real memory = 4294967296 (4096 MB) > > > -avail memory = 3639914496 (3471 MB) > > > +avail memory = 4154175488 (3961 MB) > > > > > > Somehow the r343567 kernel found an addition 490 MB of memory, > > > which leads me to believe the after loading i915kms.ko there > > > is some serious memory stomping issues. > > > > > > I willing to do whatever is necessary to fix this issue (shorter > > > of mailing the laptop to someone). Is it possible to revert > > > r343567 and move forward? > > > > > > > More info from sysctl. With the "good" r343566, I see > > > > vm.kmem_map_free: 1187033088 > > vm.kmem_map_size: 27234304 > > vm.kmem_size_scale: 3 > > vm.kmem_size_max: 1715470336 > > vm.kmem_size_min: 12582912 > > vm.kmem_zmax: 65536 > > vm.kmem_size: 1214267392 > > hw.physmem: 3714269184 > > hw.usermem: 3650867200 > > hw.realmem: 4294963200 > > > > With the problematic r343567, I see > > > > vm.kmem_map_free: 1683152896 > > vm.kmem_map_size: 28123136 > > vm.kmem_size_scale: 1 > > vm.kmem_size_max: 1711276032 > > vm.kmem_size_min: 12582912 > > vm.kmem_zmax: 65536 > > vm.kmem_size: 1711276032 > > hw.physmem: 4252360704 > > hw.usermem: 4146999296 > > hw.realmem: 4294963200 > > > > Ideas? > > > > Here's the 'diff -uw' between a verbose dmesg boot of r343566 > and dmesg boot of r343567. The memory size looks rather puzzling. > Can the people responsible for the i386 pmap.h merging take a > look? What is puzzling ? When kernel boots in PAE mode, it can (and will) get a use for physical memory mapped above 4G. I highlighted the SMAP entry which represents such memory, below. kmem_scale was changed in the PAE commit, see the commit message for explanation. > > --- dmesg.343566.verbose 2019-02-22 19:08:33.458559000 -0800 > +++ dmesg.343567.verbose 2019-02-22 08:55:21.623310000 -0800 > _at__at_ -8,25 +8,25 _at__at_ > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > -FreeBSD 13.0-CURRENT r343566 GENERIC i386 > +FreeBSD 13.0-CURRENT r343567 GENERIC i386 > FreeBSD clang version 7.0.1 (tags/RELEASE_701/final 349250) (based on LLVM 7.0.1) > WARNING: WITNESS option enabled, expect reduced performance. > VT(vga): resolution 640x480 > -Preloaded elf kernel "/boot/kernel/kernel" at 0x2501000. > -Preloaded boot_entropy_cache "/boot/entropy" at 0x2509b20. > -Preloaded elf module "/boot/kernel/cpuctl.ko" at 0x2509b6c. > -Preloaded elf module "/boot/kernel/if_ath.ko" at 0x2509f68. > -Preloaded elf module "/boot/kernel/ath_dfs.ko" at 0x250a314. > -Preloaded elf module "/boot/kernel/ath_rate.ko" at 0x250a6c0. > -Preloaded elf module "/boot/kernel/ath_hal_ar9300.ko" at 0x250aa70. > -Preloaded elf module "/boot/kernel/ath_hal_ar5416.ko" at 0x250ae9c. > -Preloaded elf module "/boot/kernel/ath_hal_ar5212.ko" at 0x250b2c8. > -Preloaded elf module "/boot/kernel/ath_hal_ar5211.ko" at 0x250b6f4. > -Preloaded elf module "/boot/kernel/ath_hal_ar5210.ko" at 0x250baf8. > +Preloaded elf kernel "/boot/kernel/kernel" at 0x2528000. > +Preloaded elf module "/boot/kernel/cpuctl.ko" at 0x2530b20. > +Preloaded boot_entropy_cache "/boot/entropy" at 0x2530f1c. > +Preloaded elf module "/boot/kernel/if_ath.ko" at 0x2530f68. > +Preloaded elf module "/boot/kernel/ath_dfs.ko" at 0x2531314. > +Preloaded elf module "/boot/kernel/ath_rate.ko" at 0x25316c0. > +Preloaded elf module "/boot/kernel/ath_hal_ar9300.ko" at 0x2531a70. > +Preloaded elf module "/boot/kernel/ath_hal_ar5416.ko" at 0x2531e9c. > +Preloaded elf module "/boot/kernel/ath_hal_ar5212.ko" at 0x25322c8. > +Preloaded elf module "/boot/kernel/ath_hal_ar5211.ko" at 0x25326f4. > +Preloaded elf module "/boot/kernel/ath_hal_ar5210.ko" at 0x2532af8. > Table 'FACP' at 0xdf6c1c9c > FACP: Found table at 0xdf6c1c9c > -Calibrating TSC clock ... TSC clock: 1995051200 Hz > -CPU: Intel(R) Core(TM)2 Duo CPU T7250 _at_ 2.00GHz (1995.05-MHz 686-class CPU) > +Calibrating TSC clock ... TSC clock: 1995040840 Hz > +CPU: Intel(R) Core(TM)2 Duo CPU T7250 _at_ 2.00GHz (1995.04-MHz 686-class CPU) > Origin="GenuineIntel" Id=0x6fd Family=0x6 Model=0xf Stepping=13 > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > _at__at_ -53,14 +53,15 _at__at_ > Physical memory chunk(s): > 0x0000000000001000 - 0x000000000009dfff, 643072 bytes (157 pages) > 0x0000000000100000 - 0x00000000007fffff, 7340032 bytes (1792 pages) > -0x0000000002829000 - 0x00000000db998fff, 3642163200 bytes (889200 pages) > -avail memory = 3639914496 (3471 MB) > +0x00000000026ff000 - 0x00000000da2bcfff, 3619414016 bytes (883646 pages) This change is most likely due to some early data shuffled around because initial VM structures sizes were different due to larger amount of total memory to describe. We allocate memory early by reducing the size of SMAP entries. > +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages) This is your memory above 4G, now found and used. > +avail memory = 4154175488 (3961 MB) > MADT: Found CPU APIC ID 0 ACPI ID 0: enabled > SMP: Added CPU 0 (AP) > MADT: Found CPU APIC ID 1 ACPI ID 1: enabled > SMP: Added CPU 1 (AP) > Event timer "LAPIC" quality 100 > -LAPIC: ipi_wait() us multiplier 45 (r 4401300 tsc 1995051200) > +LAPIC: ipi_wait() us multiplier 45 (r 4401350 tsc 1995040840) > ACPI APIC Table: <DELL M08 > > Package ID shift: 1 > L2 cache ID shift: 1 > _at__at_ -76,7 +77,6 _at__at_ > CPU1 (AP): APIC ID: 1 > APIC: CPU 0 has ACPI ID 0 > APIC: CPU 1 has ACPI ID 1 > -Pentium Pro MTRR support enabled > bios32: Found BIOS32 Service Directory header at 0x4ffa00 > bios32: Entry = 0xffa10 (4ffa10) Rev = 0 Len = 1 > pcibios: PCI BIOS entry at 0xf0000+0xab56 > _at__at_ -84,6 +84,7 _at__at_ > pnpbios: Entry = f0000:e2f4 Rev = 1.0 > pnpbios: Event flag at 4b4 > Other BIOS signatures found: > +Pentium Pro MTRR support enabled > random: read 3840 bytes from preloaded cache > random: unblocking device. > arc4random: read 32 bytes from preloaded cache > _at__at_ -117,7 +118,7 _at__at_ > lapic1: LINT1 trigger: edge > lapic1: LINT1 polarity: high > ioapic0 <Version 2.0> irqs 0-23 on motherboard > -lapic: Divisor 2, Frequency 99752593 Hz > +lapic: Divisor 2, Frequency 99752070 Hz > cpu0 BSP: > ID: 0x00000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff > lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff > _at__at_ -129,7 +130,7 _at__at_ > timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 > TSC timecounter disables C2 and C3. > SMP: passed TSC synchronization test > -Timecounter "TSC" frequency 1995051200 Hz quality 1000 > +Timecounter "TSC" frequency 1995040840 Hz quality 1000 > random: entropy device external interface > wlan: <802.11 Link Layer> > snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024] > _at__at_ -138,7 +139,7 _at__at_ > [ath_hal] loaded > nfslock: pseudo-device > crypto: <crypto core> > -module_register_init: MOD_LOAD (vesa, 0x1669cc0, 0) error 19 > +module_register_init: MOD_LOAD (vesa, 0x166cb90, 0) error 19 > io: <I/O> > kbd: new array size 4 > kbd1 at kbdmux0 > _at__at_ -152,9 +153,9 _at__at_ > [ar5211] loaded > [ar5210] loaded > [ath] loaded > -hpt27xx: RocketRAID 27xx controller driver v1.2.8 > hptnr: R750/DC7280 controller driver v1.1.5 > hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2 > +hpt27xx: RocketRAID 27xx controller driver v1.2.8 > nexus0 > vtvga0: <VT VGA driver> on motherboard > cryptosoft0: <software crypto> on motherboard > _at__at_ -204,8 +205,8 _at__at_ > PCIe: Memory Mapped configuration base _at_ 0xf8000000 > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 > ACPI: Dynamic OEM Table Load: > -ACPI: SSDT 0x0000000009FFEA00 000043 (v01 LMPWR DELLLOM 00001001 INTL 20050624) > -acpi0: wakeup code va 0xa1d1000 pa 0x1000 > +ACPI: SSDT 0x000000001A404A80 000043 (v01 LMPWR DELLLOM 00001001 INTL 20050624) > +acpi0: wakeup code va 0x1a452000 pa 0x1000 > Table 'FACP' at 0xdf6c1c9c > FACP: Found table at 0xdf6c1c9c > Table 'FACP' at 0xdf6c1c9c > _at__at_ -234,15 +235,15 _at__at_ > cpu0: Processor \134_PR_.CPU0 (ACPI ID 0) -> APIC ID 0 > cpu0: <ACPI CPU> on acpi0 > ACPI: Dynamic OEM Table Load: > -ACPI: SSDT 0x000000000A21C000 000286 (v01 PmRef Cpu0Ist 00003000 INTL 20050624) > +ACPI: SSDT 0x000000001A422C00 000286 (v01 PmRef Cpu0Ist 00003000 INTL 20050624) > ACPI: Dynamic OEM Table Load: > -ACPI: SSDT 0x000000000A22D800 0004B6 (v01 PmRef Cpu0Cst 00003001 INTL 20050624) > +ACPI: SSDT 0x000000001A44B800 0004B6 (v01 PmRef Cpu0Cst 00003001 INTL 20050624) > cpu1: Processor \134_PR_.CPU1 (ACPI ID 1) -> APIC ID 1 > cpu1: <ACPI CPU> on acpi0 > ACPI: Dynamic OEM Table Load: > -ACPI: SSDT 0x000000000A235100 0000C4 (v01 PmRef Cpu1Ist 00003000 INTL 20050624) > +ACPI: SSDT 0x000000001A428D00 0000C4 (v01 PmRef Cpu1Ist 00003000 INTL 20050624) > ACPI: Dynamic OEM Table Load: > -ACPI: SSDT 0x000000000A235000 000085 (v01 PmRef Cpu1Cst 00003000 INTL 20050624) > +ACPI: SSDT 0x000000001A428C00 000085 (v01 PmRef Cpu1Cst 00003000 INTL 20050624) > Table 'FACP' at 0xdf6c1c9c > FACP: Found table at 0xdf6c1c9c > atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0 > _at__at_ -526,7 +527,7 _at__at_ > vgapci0: <VGA-compatible display> port 0xeff8-0xefff mem 0xfea00000-0xfeafffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 > agp0: <Intel GM965 SVGA controller> on vgapci0 > pcib0: allocated type 3 (0xf0000000-0xf0000fff) for rid 64 of agp0 > -agp0: Allocated flush page phys 0xf0000000 virt 0x18fba000 > +agp0: Allocated flush page phys 0xf0000000 virt 0xd522000 > agp0: aperture size is 256M, detected 7676k stolen memory > agp0: AGP_I965_PGTBL_CTL2: 00000000 > agp0: AGP_I855_GCC1: 0x30 > _at__at_ -933,9 +934,9 _at__at_ > IPsec: Initialized Security Association Processing. > tcp_init: net.inet.tcp.tcbhashsize auto tuned to 32768 > AcpiOsExecute: enqueue 6 pending tasks > -hpt27xx: no controller detected. > hptnr: no controller detected. > hptrr: no controller detected. > +hpt27xx: no controller detected. > hdacc0: <Sigmatel STAC9205X HDA CODEC> at cad 0 on hdac0 > hdaa0: <Sigmatel STAC9205X Audio Function Group> at nid 1 on hdacc0 > hdaa0: Subsystem ID: 0x10280200 > _at__at_ -1057,27 +1058,27 _at__at_ > pcm0: Automatically set rec source to: monitor > pcm0: Recording channel set is: Front Left, Front Right, > pcm0: Recording channel matrix is: 2.0 (unknown) > -ugen1.1: <Intel UHCI root HUB> at usbus1 > +ugen5.1: <Intel UHCI root HUB> at usbus5 > ugen6.1: <Intel EHCI root HUB> at usbus6 > -uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 > +uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 > uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6 > -ugen2.1: <Intel EHCI root HUB> at usbus2 > -ugen5.1: <Intel UHCI root HUB> at usbus5 > -uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 > -ugen4.1: <Intel UHCI root HUB> at usbus4 > ugen0.1: <Intel UHCI root HUB> at usbus0 > -uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 > +ugen4.1: <Intel UHCI root HUB> at usbus4 > +uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > ugen3.1: <Intel UHCI root HUB> at usbus3 > +ugen1.1: <Intel UHCI root HUB> at usbus1 > +uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 > +uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 > +ugen2.1: <Intel EHCI root HUB> at usbus2 > ata0: reset tp1 mask=03 ostat0=50 ostat1=01 > -uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 > +uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 > acpi_acad0: acline initialization start > acpi_acad0: On Line > acpi_acad0: acline initialization done, tried 1 times > battery0: battery initialization start > -battery1: battery initialization start > battery0: battery initialization done, tried 1 times > -uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 > -uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 > +battery1: battery initialization start > +uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 > ata0: stat0=0x00 err=0x01 lsb=0x14 msb=0xeb > ata0: stat1=0x01 err=0x04 lsb=0x00 msb=0x00 > ata0: reset tp2 stat0=00 stat1=01 devices=0x10000 > _at__at_ -1092,10 +1093,6 _at__at_ > ses0 at ahciem0 bus 0 scbus3 target 0 lun 0 > ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device > ses0: SEMB SES Device > -cd0 at ata0 bus 0 scbus0 target 0 lun 0 > -cd0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device > -cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) > -cd0: 680MB (348277 2048 byte sectors) > ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 > ada0: <Patriot Burst SBFM91.0> ACS-4 ATA SATA 3.x device > ada0: Serial Number 1B0607771A0800257271 > _at__at_ -1104,11 +1101,15 _at__at_ > ada0: 114473MB (234441648 512 byte sectors) > ses0: Generation Code 0x0 has 1 SubEnclosures > pass0 at ata0 bus 0 scbus0 target 0 lun 0 > -ses0: pass0: SubEnclosure ID 0, 1 Types With this ID, Descriptor Length 36, offset 8 > -<SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device > -ses0: pass0: 33.300MB/s transfersWWN: 0 > -ses0: Type Desc[0]: Type 0x17, MaxElt 3, In Subenc 0, Text Length 0: > +ses0: SubEnclosure ID 0, 1 Types With this ID, Descriptor Length 36, offset 8 > +pass0: ses0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device > +WWN: 0 > +ses0: pass0: 33.300MB/s transfers Type Desc[0]: Type 0x17, MaxElt 3, In Subenc 0, Text Length 0: > (UDMA2, ATAPI 12bytes, PIO 65534bytes) > +cd0 at ata0 bus 0 scbus0 target 0 lun 0 > +cd0: <SONY CDRWDVD CRX880A KD09> Removable CD-ROM SCSI device > +cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) > +cd0: 680MB (348277 2048 byte sectors) > pass1 at ahcich0 bus 0 scbus1 target 0 lun 0 > pass1: <Patriot Burst SBFM91.0> ACS-4 ATA SATA 3.x device > pass1: Serial Number 1B0607771A0800257271 > _at__at_ -1119,15 +1120,15 _at__at_ > Trying to mount root from ufs:/dev/ada0s1a [rw]... > WARNING: WITNESS option enabled, expect reduced performance. > uhub0: 2 ports with 2 removable, self powered > +uhub2: 2 ports with 2 removable, self powered > +uhub5: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > -uhub5: 2 ports with 2 removable, self powered > -uhub6: 2 ports with 2 removable, self powered > -uhub2: 4 ports with 4 removable, self powered > +uhub6: 4 ports with 4 removable, self powered > +uhub1: 6 ports with 6 removable, self powered > GEOM: new disk ada0 > atrtc0: providing initial system time > start_init: trying /sbin/init > -uhub1: 6 ports with 6 removable, self powered > ugen6.2: <Toshiba External USB HDD> at usbus6 > umass0 on uhub1 > umass0: <Bulk Only Interface> on usbus6 > _at__at_ -1146,7 +1147,7 _at__at_ > da0: quirks=0x2<NO_6_BYTE> > da0: Delete methods: <NONE(*),ZERO> > ugen4.2: <Logitech USB Receiver> at usbus4 > -ukbd0 on uhub3 > +ukbd0 on uhub4 > ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4 > kbd2 at ukbd0 > kbd2: ukbd0, generic (0), config:0x0, flags:0x3d0000 > _at__at_ -1154,10 +1155,10 _at__at_ > wlan0: bpf attached > wlan0: Ethernet address: 00:1c:bf:90:ab:44 > lo0: link state changed to UP > -firmware: 'wpifw' version 153229: 150100 bytes loaded at 0x2f809000 > +firmware: 'wpifw' version 153229: 150100 bytes loaded at 0x34009000 > wlan0: link state changed to UP > -ums0 on uhub3 > +ums0 on uhub4 > ums0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4 > ums0: 16 buttons and [XYZT] coordinates ID=2 > -uhid0 on uhub3 > +uhid0 on uhub4 > uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 2> on usbus4 > > > -- > Steve > 20170425 https://www.youtube.com/watch?v=VWUpyCsUKR4 > 20161221 https://www.youtube.com/watch?v=IbCHE-hONowReceived on Sat Feb 23 2019 - 08:19:42 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:20 UTC