Re: Page fault in udp_usrreq.c:823

From: Matthew Macy <mat.macy_at_gmail.com>
Date: Thu, 21 Jun 2018 18:16:57 -0700
Try updating. It should be fixed.

On Thu, Jun 21, 2018 at 6:14 PM, Cy Schubert <Cy.Schubert_at_cschubert.com> wrote:
> In message <20180620090957.GA130_at_x2.osted.lan>, Peter Holm writes:
>> 20180620 10:32:47 all (1/1): udp.sh
>> Kernel page fault with the following non-sleepable locks held:
>> shared rw udpinp (udpinp) r = 0 (0xfffff80bbc808d78) locked _at_ netinet/in_pcb.
>> c:2398
>> stack backtrace:
>> #0 0xffffffff80c00733 at witness_debugger+0x73
>> #1 0xffffffff80c01b11 at witness_warn+0x461
>> #2 0xffffffff81075763 at trap_pfault+0x53
>> #3 0xffffffff81074d7a at trap+0x2ba
>> #4 0xffffffff8105076c at calltrap+0x8
>> #5 0xffffffff80dd21b0 at udp_ctlinput+0x50
>> #6 0xffffffff80d3081d at icmp_input+0x96d
>> #7 0xffffffff80d316d7 at ip_input+0x3f7
>> #8 0xffffffff80cc0a92 at netisr_dispatch_src+0xa2
>> #9 0xffffffff80ca3ebe at ether_demux+0x16e
>> #10 0xffffffff80ca5377 at ether_nh_input+0x427
>> #11 0xffffffff80cc0a92 at netisr_dispatch_src+0xa2
>> #12 0xffffffff80ca437f at ether_input+0x8f
>> #13 0xffffffff80cbc500 at iflib_rxeof+0xc90
>> #14 0xffffffff80cb6b6f at _task_fn_rx+0x7f
>> #15 0xffffffff80bdd209 at gtaskqueue_run_locked+0x139
>> #16 0xffffffff80bdcf88 at gtaskqueue_thread_loop+0x88
>> #17 0xffffffff80b54514 at fork_exit+0x84
>>
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 10; apic id = 0a
>> fault virtual address = 0x8
>> fault code  = supervisor read data, page not present
>> instruction pointer = 0x20:0xffffffff80dd2423
>> stack pointer         = 0x0:0xfffffe00004a5500
>> frame pointer         = 0x0:0xfffffe00004a55a0
>> code segment  = base 0x0, limit 0xfffff, type 0x1b
>>    = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags = interrupt enabled, resume, IOPL = 0
>> current process  = 0 (if_io_tqg_10)
>> [ thread pid 0 tid 100069 ]
>> Stopped at      udp_common_ctlinput+0x263:      cmpq    $0,0x8(%rax)
>> db>
>>
>> Details _at_ https://people.freebsd.org/~pho/stress/log/udp_usrreq.txt
>>
>> --
>> Peter
>> _______________________________________________
>> freebsd-current_at_freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>>
>
> This is surprisingly similar to my panic. Twice since June 19.
>
> slippy# kgdb /boot/kernel/kernel vmcore.3
> GNU gdb (GDB) 8.1 [GDB v8.1 for FreeBSD]
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
> html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-portbld-freebsd12.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /boot/kernel/kernel...Reading symbols from
> /usr/lib/debug//boot/kernel/kernel.debug...done.
> done.
>
> Unread portion of the kernel message buffer:
> Copyright (c) 1992-2018 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 12.0-CURRENT #355 r335477M: Thu Jun 21 05:26:35 PDT 2018
>     root_at_slippy:/export/obj/opt/src/svn-current/amd64.amd64/sys/BREAK
> amd64
> FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on
> LLVM 6.0.0)
> VT(vga): text 80x25
> module_register: cannot register mmc/mmcsd from kernel; already loaded
> from mmcsd.ko
> Module mmc/mmcsd failed to register: 17
> CPU: Intel(R) Core(TM) i3-2350M CPU _at_ 2.30GHz (2294.83-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
>   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,P
> GE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>   Features2=0x1dbae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE
> 3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,A
> VX>
>   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>   AMD Features2=0x1<LAHF>
>   XSAVE Features=0x1<XSAVEOPT>
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>   TSC: P-state invariant, performance statistics
> real memory  = 8589934592 (8192 MB)
> avail memory = 8080965632 (7706 MB)
> ACPI APIC Table: <ACRSYS ACRPRDCT>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
> random: unblocking device.
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> random: entropy device external interface
> module_register_init: MOD_LOAD (vesa, 0xffffffff8097b700, 0) error 19
> kbd1 at kbdmux0
> nexus0
> vtvga0: <VT VGA driver> on motherboard
> cryptosoft0: <software crypto> on motherboard
> acpi0: <ACRSYS ACRPRDCT> on motherboard
> acpi0: Power Button (fixed)
> cpu0: <ACPI CPU> on acpi0
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 950
> Event timer "HPET" frequency 14318180 Hz quality 550
> Event timer "HPET1" frequency 14318180 Hz quality 440
> Event timer "HPET2" frequency 14318180 Hz quality 440
> Event timer "HPET3" frequency 14318180 Hz quality 440
> Event timer "HPET4" frequency 14318180 Hz quality 440
> atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
> atrtc0: Warning: Couldn't map I/O.
> atrtc0: registered as a time-of-day clock, resolution 1.000000s
> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> acpi_ec0: <Embedded Controller: GPE 0x17> port 0x62,0x66 on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> vgapci0: <VGA-compatible display> port 0x2000-0x203f mem
> 0xf0000000-0xf03fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
> vgapci0: Boot video device
> pci0: <simple comms> at device 22.0 (no driver attached)
> ehci0: <Intel Cougar Point USB 2.0 controller> mem
> 0xf060a000-0xf060a3ff irq 16 at device 26.0 on pci0
> usbus0: EHCI version 1.0
> usbus0 on ehci0
> hdac0: <Intel Cougar Point HDA Controller> mem 0xf0600000-0xf0603fff
> irq 22 at device 27.0 on pci0
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
> pci2: <ACPI PCI bus> on pcib2
> iwn0: <Intel Centrino Advanced-N 6205> mem 0xf0500000-0xf0501fff irq 17
> at device 0.0 on pci2
> pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
> pci3: <ACPI PCI bus> on pcib3
> bge0: <Broadcom BCM57765 B0, ASIC rev. 0x57785100> mem
> 0xf0400000-0xf040ffff,0xf0410000-0xf041ffff irq 19 at device 0.0 on pci3
> bge0: CHIP ID 0x57785100; ASIC REV 0x57785; CHIP REV 0x577851; PCI-E
> miibus0: <MII bus> on bge0
> brgphy0: <BCM57765 1000BASE-T media interface> PHY 1 on miibus0
> brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
> 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> <6>bge0: Using defaults for TSO: 65518/35/2048
> <6>bge0: Ethernet address: 20:6a:8a:72:03:17
> pci3: <base peripheral, SD host controller> at device 0.1 (no driver
> attached)
> ehci1: <Intel Cougar Point USB 2.0 controller> mem
> 0xf0609000-0xf06093ff irq 23 at device 29.0 on pci0
> usbus1: EHCI version 1.0
> usbus1 on ehci1
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> ahci0: <Intel Cougar Point AHCI SATA controller> port
> 0x2088-0x208f,0x2094-0x2097,0x2080-0x2087,0x2090-0x2093,0x2060-0x207f
> mem 0xf0608000-0xf06087ff irq 19 at device 31.2 on pci0
> ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
> ahcich0: <AHCI channel> at channel 0 on ahci0
> ahcich1: <AHCI channel> at channel 1 on ahci0
> ahcich5: <AHCI channel> at channel 5 on ahci0
> ahciem0: <AHCI enclosure management bridge> on ahci0
> ichsmb0: <Intel Cougar Point SMBus controller> port 0xefa0-0xefbf mem
> 0xf0604000-0xf06040ff irq 18 at device 31.3 on pci0
> smbus0: <System Management Bus> on ichsmb0
> smb0: <SMBus generic I/O> on smbus0
> acpi_lid0: <Control Method Lid Switch> on acpi0
> acpi_button0: <Sleep Button> on acpi0
> acpi_tz0: <Thermal Zone> on acpi0
> acpi_tz1: <Thermal Zone> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: model Synaptics Touchpad, device ID 0
> acpi_acad0: <AC Adapter> on acpi0
> battery0: <ACPI Control Method Battery> on acpi0
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid
> PNP0900 on isa0
> ata0: <ATA channel> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
> ata1: <ATA channel> at port 0x170-0x177,0x376 irq 15 on isa0
> acpi_perf0: <ACPI CPU Frequency Control> on cpu0
> coretemp0: <CPU On-Die Thermal Sensors> on cpu0
> ZFS filesystem version: 5
> ZFS storage pool version: features support (5000)
> Timecounters tick every 10.000 msec
> IP Filter: v5.1.2 initialized.  Default = pass all, Logging = enabled
> hdacc0: <Realtek ALC269 HDA CODEC> at cad 0 on hdac0
> hdaa0: <Realtek ALC269 Audio Function Group> at nid 1 on hdacc0
> pcm0: <Realtek ALC269 (Analog 2.0+HP/2.0)> at nid 20,33 and 27 on hdaa0
> pcm1: <Realtek ALC269 (Left Analog Mic)> at nid 24 on hdaa0
> hdacc1: <Intel Cougar Point HDA CODEC> at cad 3 on hdac0
> hdaa1: <Intel Cougar Point Audio Function Group> at nid 1 on hdacc1
> pcm2: <Intel Cougar Point (HDMI/DP 8ch)> at nid 5 on hdaa1
> usbus0: 480Mbps High Speed USB v2.0
> usbus1: 480Mbps High Speed USB v2.0
> ugen1.1: <Intel EHCI root HUB> at usbus1
> uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
> ugen0.1: <Intel EHCI root HUB> at usbus0
> uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
> ses0 at ahciem0 bus 0 scbus3 target 0 lun 0
> ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
> ses0: SEMB SES Device
> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
> ada0: <HGST HTS721010A9E630 JB0OA3U0> ATA8-ACS SATA 3.x device
> ada0: Serial Number JR1000D33969RE
> ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
> ada0: Command Queueing enabled
> ada0: 953869MB (1953525168 512 byte sectors)
> cd0 at ahcich1 bus 0 scbus1 target 0 lun 0
> cd0: <PIONEER DVD-RW DVRTD11RS 1.01> Removable CD-ROM SCSI device
> cd0: Serial Number SBB5103801
> cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO
> 8192bytes)
> cd0: Attempt to query device size failed: NOT READY, Medium not present
> - tray closed
> Launching APs: 1 3 2
> Trying to mount root from ufs:/dev/ufs/Sroot [rw]...
> Timecounter "TSC-low" frequency 1147416846 Hz quality 1000
> <118>Setting hostuuid: 34f5ed40-8938-11da-b265-efe316da850d.
> <118>Setting hostid: 0x7f5a03b9.
> uhub0: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
> uhub2 on uhub1
> uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2>
> on usbus0
> ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
> uhub3 on uhub0
> uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2>
> on usbus1
> uhub2: 6 ports with 6 removable, self powered
> uhub3: 6 ports with 6 removable, self powered
> ugen0.3: <Logitech USB Receiver> at usbus0
> ukbd0 on uhub2
> ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 3> on
> usbus0
> kbd2 at ukbd0
> ums0 on uhub2
> ums0: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 3> on
> usbus0
> ums0: 16 buttons and [XYZT] coordinates ID=2
> ugen1.3: <vendor 0x05e3 USB2.0 Hub> at usbus1
> uhub4 on uhub3
> uhub4: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 3> on
> usbus1
> uhub4: 4 ports with 4 removable, self powered
> ugen0.4: <AABBX0E6S 1.3M HD WebCam> at usbus0
> ugen1.4: <vendor 0x05e3 USB2.0 Hub> at usbus1
> uhub5 on uhub4
> uhub5: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 4> on
> usbus1
> uhub5: 4 ports with 4 removable, self powered
> ugen1.5: <ALCOR Generic USB Hub> at usbus1
> uhub6 on uhub5
> uhub6: <ALCOR Generic USB Hub, class 9/0, rev 1.10/3.12, addr 5> on
> usbus1
> uhub6: 4 ports with 4 removable, self powered
> ugen1.6: <Logitech USB Receiver> at usbus1
> ukbd1 on uhub6
> ukbd1: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 6> on
> usbus1
> kbd3 at ukbd1
> ums1 on uhub6
> ums1: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 6> on
> usbus1
> ums1: 16 buttons and [XYZT] coordinates ID=2
> ugen1.7: <Device 2Port KVMSwitcher> at usbus1
> ukbd2 on uhub6
> ukbd2: <Device 2Port KVMSwitcher, class 0/0, rev 1.10/0.01, addr 7> on
> usbus1
> kbd4 at ukbd2
> ugen1.8: <vendor 0x05e3 USB TO IDE> at usbus1
> umass0 on uhub5
> umass0: <vendor 0x05e3 USB TO IDE, class 0/0, rev 2.00/0.33, addr 8> on
> usbus1
> umass0:  SCSI over Bulk-Only; quirks = 0x408c
> umass0:6:0: Attached to scbus6
> da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
> da0: <TOSHIBA MK8026GAX 0811> Fixed Direct Access SCSI device
> da0: 40.000MB/s transfers
> da0: 76319MB (156301488 512 byte sectors)
> da0: quirks=0x2<NO_6_BYTE>
> <118>Starting file system checks:
> <118>/dev/ufs/Sroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/Sroot: clean, 413829 free (909 frags, 51615 blocks, 0.2%
> fragmentation)
> <118>/dev/ufs/Susr: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/Susr: clean, 459287 free (4847 frags, 56805 blocks, 0.6%
> fragmentation)
> <118>/dev/ufs/Svtmp: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/Svtmp: clean, 503642 free (50 frags, 62949 blocks, 0.0%
> fragmentation)
> <118>/dev/ufs/SCusr: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/Svar: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SSusr: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SCusr: clean, 469826 free (4578 frags, 58156 blocks, 0.6%
> fragmentation)
> <118>/dev/ufs/Svar: clean, 453726 free (574 frags, 56644 blocks, 0.1%
> fragmentation)
> <118>/dev/ufs/SCroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SSusr: clean, 569245 free (3653 frags, 70699 blocks, 0.5%
> fragmentation)
> <118>/dev/ufs/SCroot: clean, 422625 free (617 frags, 52751 blocks, 0.1%
> fragmentation)
> <118>/dev/ufs/SCvar: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SSvar: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SCvar: clean, 400011 free (99 frags, 49989 blocks, 0.0%
> fragmentation)
> <118>/dev/ufs/SSvar: clean, 457058 free (138 frags, 57115 blocks, 0.0%
> fragmentation)
> <118>/dev/ufs/SSroot: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/ufs/SSroot: clean, 412995 free (219 frags, 51597 blocks, 0.0%
> fragmentation)
> <118>/dev/msdosfs/SHARED: FILESYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1a: clean, 254465 free (337
> frags, 31766 blocks, 0.1% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1a: clean, 335508 free
> (1348 frags, 41770 blocks, 0.1% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1a: clean, 566833 free
> (2297 frags, 70567 blocks, 0.2% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1a: clean, 510529 free
> (1185 frags, 63668 blocks, 0.1% fragmentation)
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1a: clean, 354656 free (336
> frags, 44290 blocks, 0.1% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1a: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1a: clean, 633994 free
> (2106 frags, 78986 blocks, 0.2% fragmentation)
> <118>/dev/zvol/tank/VMs/current/amd64/disk3p1: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/amd64/disk3p1: clean, 2535564 free (564
> frags, 316875 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1d: clean, 897924 free
> (188 frags, 112217 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1d: clean, 897896 free
> (152 frags, 112218 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1d: clean, 889422 free (214
> frags, 111151 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1d: clean, 889419 free (219
> frags, 111150 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1d: clean, 408410 free (90
> frags, 51040 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/amd64/disk0s1e: clean, 1230068 free
> (7300 frags, 152846 blocks, 0.5% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/amd64/disk0s1e: clean, 726361 free
> (6297 frags, 90008 blocks, 0.4% fragmentation)
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable10/i386/disk0s1e: clean, 1299097 free
> (6385 frags, 161589 blocks, 0.4% fragmentation)
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/amd64/disk0s1e: clean, 822303 free
> (1047 frags, 102657 blocks, 0.1% fragmentation)
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1d: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1d: clean, 399855 free (15
> frags, 49980 blocks, 0.0% fragmentation)
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/stable11/i386/disk0s1e: clean, 937324 free
> (7524 frags, 116225 blocks, 0.5% fragmentation)
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1e: FILE SYSTEM CLEAN;
> SKIPPING CHECKS
> <118>/dev/zvol/tank/VMs/current/i386/disk0s1e: clean, 1077125 free (845
> frags, 134535 blocks, 0.1% fragmentation)
> <118>Mounting local filesystems:.
> <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
> /usr/local/lib/compat/pkg /usr/local/krb5/lib /usr/local/kde4/lib
> /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/dovecot
> /usr/local/lib/e2fsprogs /usr/local/lib/ffmpeg0 /usr/local/lib/freeradiu
> s-3.0.15 /usr/local/lib/gcc48 /usr/local/lib/gcc5 /usr/local/lib/gcc6
> /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/gcc9
> /usr/local/lib/gegl-0.2 /usr/local/lib/graphviz /usr/local/lib/httrack
> /usr/local/lib/itcl3.4 /usr/local/lib/jitsi/lib/native
> /usr/local/lib/mysql /usr/local/lib/mysql/plugin /usr/local/lib/nss
> /usr/local/lib/opencollada /usr/local/lib/perl5/5.24/mach/CORE
> /usr/local/lib/perl5/5.26/mach/CORE /usr/local/lib/perl5/5.28/mach/CORE
> /usr/local/lib/pgtcl /usr/local/lib/pidgin /usr/local/lib/pth
> /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/lib/samba4
> /usr/local/lib/xrdp /usr/local/libexec/openldap /usr/local/llvm-devel/li
> b /usr/local/llvm40/lib /usr/local/llvm50/lib /usr/local/llvm60/lib
> /usr/local/share/chromium
> <118>32-bit compatibility ldconfig path: /usr/lib32
> /alt/i386/root/usr/local/lib /usr/local/lib32/compat
> <118>Setting hostname: slippy.
> <118>Additional TCP/IP options: rfc1323 extensions=NO.
> <118>Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ET
> HER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
> <118>Feeding entropy: .
> <118>Loading IP Pools.
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>70018:add pool node(252.0.0.1/255.255.255.255: node entry already
> present in pool
> <118>Enabling ipfilter.
> <118>32:1:ioctl(add/insert rule): rule already exists
> <118>Installing NAT rules.
> <118>0 entries flushed from NAT table
> <118>0 entries flushed from NAT list
> <6>wlan0: Ethernet address: 20:6a:8a:72:03:17
> <118>Created wlan(4) interfaces: wlan0.
> <118>Created clone interfaces: lagg0.
> <6>lo0: link state changed to UP
> <6>bge0: link state changed to DOWN
> iwn0: iwn_read_firmware: ucode rev=0x12a80601
> <118>Starting wpa_supplicant.
> <6>lagg0: link state changed to DOWN
> iwn0: iwn_read_firmware: ucode rev=0x12a80601
> <118>Starting dhclient.
> <118>lagg0: no link ...
> <6>wlan0: link state changed to UP
> <6>lagg0: link state changed to UP
> <118> got link
> <6>bge0: link state changed to UP
> <118>Starting Network: lo0 bge0 wlan0 lagg0.
> <118>lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> <118>   options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> <118>   inet 127.0.0.1 netmask 0xff000000
> <118>   inet6 ::1 prefixlen 128
> <118>   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> <118>   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> <118>   groups: lo
> <118>bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> <118>   options=80080<VLAN_HWCSUM,LINKSTATE>
> <118>   ether 20:6a:8a:72:03:17
> <118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> <118>   media: Ethernet autoselect (1000baseT <full-duplex>)
> <118>   status: active
> <118>wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> <118>   ether 20:6a:8a:72:03:17
> <118>   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> <118>   media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
> <118>   status: associated
> <118>   ssid Walden_Pond channel 1 (2412 MHz 11g ht/20) bssid
> 4c:60:de:31:3c:17
> <118>   regdomain FCC country US authmode WPA2/802.11i privacy ON
> <118>   deftxkey UNDEF AES-CCM 2:128-bit txpower 30 bmiss 120 scanvalid
> 16959
> <118>   protmode CTS ampdulimit 64k ampdudensity 8 -amsdutx amsdurx
> shortgi
> <118>   -stbc -ldpc wme roaming MANUAL
> <118>   groups: wlan
> <118>lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> <118>   ether 20:6a:8a:72:03:17
> <118>   inet6 fe80::226a:8aff:fe72:317%lagg0 prefixlen 64 scopeid 0x4
> <118>   inet 0.0.0.0 netmask 0x0 broadcast 255.255.255.255
> <118>   nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> <118>   media: Ethernet autoselect
> <118>   status: active
> <118>   groups: lagg
> <118>   laggproto failover lagghash l2,l3,l4
> <118>   laggport: bge0 flags=5<MASTER,ACTIVE>
> <118>   laggport: wlan0 flags=0<>
> <118>filter sync'd
> <118>Starting devd.
> <118>Starting ums0 moused.
> <118>Autoloading module: uhid.ko
> uhid0 on uhub2
> uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 3> on
> usbus0
> uhid1 on uhub6
> uhid1: <Logitech USB Receiver, class 0/0, rev 2.00/12.07, addr 6> on
> usbus1
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Cannot 'start' webcamd. Set webcamd_enable to YES in /etc/rc.conf
> or use 'onestart' instead of 'start'.
> uhid2 on uhub6
> uhid2: <Device 2Port KVMSwitcher, class 0/0, rev 1.10/0.01, addr 7> on
> usbus1
> device_attach: uhid2 attach returned 12
> <118>Cannot 'start' webcamd. Set webcamd_enable to YES in /etc/rc.conf
> or use 'onestart' instead of 'start'.
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Starting ums1 moused.
> <118>Autoloading module: uhid.ko
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>Autoloading module: uhid.ko
> <118>Cannot 'start' uhidd. Set uhidd_enable to YES in /etc/rc.conf or
> use 'onestart' instead of 'start'.
> <118>local: Not in a function
> <118>local: Not in a function
> <118>dhclient not running? (check /var/run/dhclient.bge0.pid).
> <118>Stopping dhclient.
> <118>Waiting for PIDS: 471.
> <118>Starting dhclient.
> <118>add host 127.0.0.1: gateway lo0 fib 0: route already in table
> <118>Additional inet routing options: ignore ICMP redirect=YES log ICMP
> redirect=YES.
> <118>add host ::1: gateway lo0 fib 0: route already in table
> <118>add net fe80::: gateway ::1
> <118>add net ff02::: gateway ::1
> <118>add net ::ffff:0.0.0.0: gateway ::1
> <118>add net ::0.0.0.0: gateway ::1
> <118>Starting rtsold.
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 3; apic id = 03
> fault virtual address   = 0x0
> fault code              = supervisor read data, page not present
> instruction pointer     = 0x20:0xffffffff80806966
> stack pointer           = 0x28:0xfffffe00004667d0
> frame pointer           = 0x28:0xfffffe0000466840
> code segment            = base 0x0, limit 0xfffff, type 0x1b
>                         = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags        = interrupt enabled, resume, IOPL = 0
> current process         = 12 (swi1: netisr 0)
> trap number             = 12
> panic: page fault
> cpuid = 3
> time = 1529587144
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe0000466480
> vpanic() at vpanic+0x1a3/frame 0xfffffe00004664e0
> panic() at panic+0x43/frame 0xfffffe0000466540
> trap_fatal() at trap_fatal+0x35f/frame 0xfffffe0000466590
> trap_pfault() at trap_pfault+0x49/frame 0xfffffe00004665f0
> trap() at trap+0x29e/frame 0xfffffe0000466700
> calltrap() at calltrap+0x8/frame 0xfffffe0000466700
> --- trap 0xc, rip = 0xffffffff80806966, rsp = 0xfffffe00004667d0, rbp =
> 0xfffffe0000466840 ---
> udp_append() at udp_append+0x26/frame 0xfffffe0000466840
> udp_input() at udp_input+0x574/frame 0xfffffe0000466910
> ip_input() at ip_input+0x145/frame 0xfffffe0000466970
> swi_net() at swi_net+0x17b/frame 0xfffffe00004669e0
> intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame
> 0xfffffe0000466a20
> ithread_loop() at ithread_loop+0xe7/frame 0xfffffe0000466a70
> fork_exit() at fork_exit+0x83/frame 0xfffffe0000466ab0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000466ab0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> Uptime: 51s
> Dumping 697 out of 7975 MB:..3%..12%..21%..33%..42%..51%..62%..72%..81%.
> .92%
>
> __curthread () at ./machine/pcpu.h:231
> 231             __asm("movq %%gs:%1,%0" : "=r" (td)
> (kgdb) bt
> #0  __curthread () at ./machine/pcpu.h:231
> #1  doadump (textdump=1) at /opt/src/svn-current/sys/kern/kern_shutdown.
> c:366
> #2  0xffffffff8061b330 in kern_reboot (howto=260)
>     at /opt/src/svn-current/sys/kern/kern_shutdown.c:446
> #3  0xffffffff8061b7c3 in vpanic (fmt=<optimized out>,
> ap=0xfffffe0000466520)
>     at /opt/src/svn-current/sys/kern/kern_shutdown.c:863
> #4  0xffffffff8061b5b3 in panic (fmt=<unavailable>)
>     at /opt/src/svn-current/sys/kern/kern_shutdown.c:790
> #5  0xffffffff8095e76f in trap_fatal (frame=0xfffffe0000466710, eva=0)
>     at /opt/src/svn-current/sys/amd64/amd64/trap.c:892
> #6  0xffffffff8095e7c9 in trap_pfault (frame=0xfffffe0000466710,
> usermode=0)
>     at /opt/src/svn-current/sys/amd64/amd64/trap.c:728
> #7  0xffffffff8095ddee in trap (frame=0xfffffe0000466710)
>     at /opt/src/svn-current/sys/amd64/amd64/trap.c:427
> #8  <signal handler called>
> #9  udp_append (inp=0xfffff80031d2ed58, ip=0xfffff800039333a0,
>     n=0xfffff80003933300, off=20, udp_in=0xfffffe0000466850)
>     at /opt/src/svn-current/sys/netinet/udp_usrreq.c:314
> #10 0xffffffff80806464 in udp_input (mp=<optimized out>,
> offp=<optimized out>,
>     proto=17) at /opt/src/svn-current/sys/netinet/udp_usrreq.c:722
> #11 0xffffffff8075f645 in ip_input (m=0x0)
>     at /opt/src/svn-current/sys/netinet/ip_input.c:825
> #12 0xffffffff8073959b in netisr_process_workstream_proto (
> ---Type <return> to continue, or q <return> to quit---q
> nwsp=<optimized out>,Quit
> (kgdb) frame 9
> #9  udp_append (inp=0xfffff80031d2ed58, ip=0xfffff800039333a0,
>     n=0xfffff80003933300, off=20, udp_in=0xfffffe0000466850)
>     at /opt/src/svn-current/sys/netinet/udp_usrreq.c:314
> 314             if (up->u_tun_func != NULL) {
> (kgdb) l
> 309
> 310             /*
> 311              * Engage the tunneling protocol.
> 312              */
> 313             up = intoudpcb(inp);
> 314             if (up->u_tun_func != NULL) {
> 315                     in_pcbref(inp);
> 316                     INP_RUNLOCK(inp);
> 317                     (*up->u_tun_func)(n, off, inp, (struct sockaddr *)&udp_in[0],
> 318                         up->u_tun_ctx);
> (kgdb) p up
> $1 = (struct udpcb *) 0x0
> (kgdb) p *inp
> $2 = {inp_hash = {cle_next = 0x0, cle_prev = 0xfffff800023abf70},
>   inp_pcbgrouphash = {cle_next = 0x0, cle_prev = 0x0}, inp_lock = {
>     lock_object = {lo_name = 0xffffffff80a02fc7 "udpinp", lo_flags =
> 90898432,
>       lo_data = 0, lo_witness = 0x0}, rw_lock = 33}, inp_hpts = {
>     tqe_next = 0x0, tqe_prev = 0x0}, inp_hpts_request = 0,
>   inp_in_hpts = 0 '\000', inp_in_input = 0 '\000', inp_hpts_cpu = 0,
>   inp_refcount = 1, inp_flags = 8405056, inp_flags2 = 16, inp_input_cpu
> = 0,
>   inp_hpts_cpu_set = 0 '\000', inp_input_cpu_set = 0 '\000',
>   inp_hpts_calls = 0 '\000', inp_input_calls = 0 '\000',
>   inp_spare_bits2 = 0 '\000', inp_spare_byte = 0 '\000', inp_ppcb =
> 0x0,
>   inp_socket = 0x0, inp_hptsslot = 0, inp_hpts_drop_reas = 0, inp_input
> = {
>     tqe_next = 0x0, tqe_prev = 0x0}, inp_pcbinfo = 0xfffffe0048566738,
>   inp_pcbgroup = 0x0, inp_pcbgroup_wild = {cle_next = 0x0, cle_prev =
> 0x0},
>   inp_cred = 0xfffff80031366600, inp_flow = 0, inp_vflag = 1 '\001',
>   inp_ip_ttl = 64 '_at_', inp_ip_p = 0 '\000', inp_ip_minttl = 0 '\000',
>   inp_flowid = 60, inp_snd_tag = 0x0, inp_flowtype = 63,
>   inp_rss_listen_bucket = 0, inp_inc = {inc_flags = 0 '\000',
>     inc_len = 0 '\000', inc_fibnum = 0, inc_ie = {ie_fport = 0,
>       ie_lport = 28353, ie_dependfaddr = {id46_addr = {ia46_pad32 = {0,
> 0, 0},
>           ia46_addr4 = {s_addr = 0}}, id6_addr = {__u6_addr = {
>             __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
> 0, 0, 0,
>               0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}},
> ie_dependladdr = {
>         id46_addr = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = {
> ---Type <return> to continue, or q <return> to quit---
>             s_addr = 16777343}}, id6_addr = {__u6_addr = {
>             __u6_addr8 = '\000' <repeats 12 times>, "\177\000\000\001",
>             __u6_addr16 = {0, 0, 0, 0, 0, 0, 127, 256}, __u6_addr32 =
> {0, 0,
>               0, 16777343}}}}, ie6_zoneid = 0}}, inp_label = 0x0,
>   inp_sp = 0xfffff8000725dee0, {inp_ip_tos = 0 '\000', inp_options =
> 0x0,
>     inp_moptions = 0x0}, {in6p_options = 0x0, in6p_outputopts = 0x0,
>     in6p_moptions = 0x0, in6p_icmp6filt = 0x0, in6p_cksum = 0,
> in6p_hops = 0},
>   inp_portlist = {cle_next = 0x0, cle_prev = 0xfffff80002334d20},
>   inp_phd = 0xfffff80002334d00, inp_gencnt = 169, inp_lle = 0x0,
>   inp_rt_cookie = 7, {inp_route = {ro_rt = 0x0, ro_lle = 0x0,
>       ro_prepend = 0x0, ro_plen = 0, ro_flags = 256, ro_mtu = 0, spare
> = 0,
>       ro_dst = {sa_len = 0 '\000', sa_family = 0 '\000',
>         sa_data = '\000' <repeats 13 times>}}, inp_route6 = {ro_rt =
> 0x0,
>       ro_lle = 0x0, ro_prepend = 0x0, ro_plen = 0, ro_flags = 256,
> ro_mtu = 0,
>       spare = 0, ro_dst = {sin6_len = 0 '\000', sin6_family = 0 '\000',
>         sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__u6_addr = {
>             __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
> 0, 0, 0,
>               0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id
> = 0}}},
>   inp_list = {cle_next = 0xfffff80031d4e000, cle_prev =
> 0xfffffe0048566828},
>   inp_epoch_ctx = {data = {0xffffffff80759b10 <in_pcbfree_deferred>,
>       0xfffff80002334d08}}}
> (kgdb) up
> #10 0xffffffff80806464 in udp_input (mp=<optimized out>,
> offp=<optimized out>,
>     proto=17) at /opt/src/svn-current/sys/netinet/udp_usrreq.c:722
> 722             if (udp_append(inp, ip, m, iphlen, udp_in) == 0)
> (kgdb) l
> 717                             return (IPPROTO_DONE);
> 718                     }
> 719             }
> 720
> 721             UDP_PROBE(receive, NULL, inp, ip, inp, uh);
> 722             if (udp_append(inp, ip, m, iphlen, udp_in) == 0)
> 723                     INP_RUNLOCK(inp);
> 724             return (IPPROTO_DONE);
> 725
> 726     badunlocked:
> (kgdb) up
> #11 0xffffffff8075f645 in ip_input (m=0x0)
>     at /opt/src/svn-current/sys/netinet/ip_input.c:825
> 825             (*inetsw[ip_protox[ip->ip_p]].pr_input)(&m, &hlen, ip->ip_p);
> (kgdb) l
> 820             /*
> 821              * Switch out to protocol's input routine.
> 822              */
> 823             IPSTAT_INC(ips_delivered);
> 824
> 825             (*inetsw[ip_protox[ip->ip_p]].pr_input)(&m, &hlen, ip->ip_p);
> 826             return;
> 827     bad:
> 828             m_freem(m);
> 829     }
> (kgdb)
>
>
> In both cases rtsold was starting.
>
>
>
> --
> Cheers,
> Cy Schubert <Cy.Schubert_at_cschubert.com>
> FreeBSD UNIX:  <cy_at_FreeBSD.org>   Web:  http://www.FreeBSD.org
>
>         The need of the many outweighs the greed of the few.
>
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
Received on Thu Jun 21 2018 - 23:17:01 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:16 UTC