Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

On Sun, Feb 24, 2019 at 10:27:26AM +0100, Tijl Coosemans wrote:
> On Sat, 23 Feb 2019 17:28:51 -0800 Steve Kargl
> <sgk_at_troutmask.apl.washington.edu> wrote:
> > On Sat, Feb 23, 2019 at 12:03:58PM -0700, Warner Losh wrote:
> >> On Sat, Feb 23, 2019 at 10:57 AM Steve Kargl
> >> <sgk_at_troutmask.apl.washington.edu> wrote:
> >>> Supposely, the laptop only has 4 GB of memory.  Not sure how
> >>> it finds memory above 4 GB.
> >> 
> >> Some older chipsets had a 'hole' in memory that they mapped the PCI bus
> >> into and then remapped RAM in that range up above the 4GB boundary. That's
> >> how it can find memory above 4GB when you have only 4GB of RAM. I hit it
> >> with the PC Card stuff I did back in the day since it broke certain
> >> heuristics I had in the code that turned out to be unwise for many reasons
> >> (not just this one). I don't recall all the details, since it's been so
> >> long ago.
> >> 
> >> So I think kib_at_ is right when he highlights
> >>> +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages)
> >> 
> >> as the memory, since this is indeed above the 4GB limit.  It's about 128k
> >> of 4k pages (just shy of the 131072 I'd expect), which is a surprisingly
> >> round number. Also one that's easy to implement in hardware. So it
> >> certainly "smells" the same...
> >> 
> >> That's why I agree with others that hw.above4g_allow=0 is worth a shot, for
> >> at least diagnostic purposes. This memory wasn't used before and if it's
> >> used now by the drm drivers, and those aren't PAE safe (meaning they cope
> >> with allocations beyond 4GB), then that's quite useful to know. Or maybe
> >> it's a different driver hating things and stomping on video memory due to
> >> wrap around.
> > 
> > Thanks for the explanation.  Here's an update.  TL;DR: xorg is
> > up and running; drm-legacy-kmod seems to be unsafe/unaware of
> > PAE.
> > 
> > Build world/kernel, drm-legacy-kmod, minimum needed ports for xorg.
> > Kernel is unmodified GENERIC.
> > 
> > Reboot without setting anything in /boot/loader.conf
> > 
> > % sysctl -a | grep above
> > % sysctl -a | grep pae
> > vm.pmap.pae_mode: 1
> > % kldload /boot/modules/i915kms.ko
> > 
> > Black screen of death. Did not even get to running xinit.
> > 
> > Hard reset to single user mode.
> > 
> > # fsck -y
> > # mount -a
> > # vi /boot/loader.conf.
> > (Add hw.above4g_allow=0)
> > # sync
> > # shutdown -r now
> > 
> > % sysctl -a | grep above
> > % sysctl -a | grep pae
> > vm.pmap.pae_mode: 1
> > % cat /boot/loader.conf
> > if_ath_load="YES"
> > if_ath_pci_load="YES"
> > cpuctl_load="YES"
> > hw.above4g_allow=0
> > % kldload /boot/modules/i915kms.ko
> > 
> > Switch to vt3, login as normal user.
> > 
> > % startx -- -depth 24 >& ~/tmp/.x.out
> > 
> > Xorg is up and running.  Not sure why my first attempt at using
> > hw.above4g_allow=0 did not work.  Perhaps, mismatch between the xorg
> > bits and kernel/world bits.
> > 
> > % sysctl -a | grep mem
> > vm.lowmem_period: 10
> > vm.kmem_map_free: 1669365760
> > vm.kmem_map_size: 41910272
> > vm.kmem_size_scale: 1
> > vm.kmem_size_max: 1711276032
> > vm.kmem_size_min: 12582912
> > vm.kmem_zmax: 65536
> > vm.kmem_size: 1711276032
> > hw.physmem: 3715489792
> > hw.usermem: 3592175616
> > hw.realmem: 4294963200
> > 
> > % dmesg | grep memory
> > real memory  = 4294967296 (4096 MB)
> > avail memory = 3637673984 (3469 MB)
> > agp0: aperture size is 256M, detected 7676k stolen memory
> > 
> > The pre-r343567 dmesg has
> > 
> > real memory  = 4294967296 (4096 MB)
> > avail memory = 3639914496 (3471 MB)
> > 
> > I can live with 2 MB loss.
> > 
> > Conclusion, drm-legacy-kmod is not PAE safe/aware.
> > 
> > Probably want to put something in /usr/src about possible
> > problems with new pmap.h on i386 FreeBSD.
> 
> Now it would be interesting to do the same tests with drm-current-kmod.

Caveats: old laptop and I only rebuilt drm-current-kmod and the
gpu-firmware-kmod.

drm-current-kmod does not work.  The setting of hw.above4g_allow
is immaterial.

-- 
Steve