Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

On Sun, Feb 24, 2019 at 2:27 AM Tijl Coosemans <tijl_at_freebsd.org> wrote:

> On Sat, 23 Feb 2019 17:28:51 -0800 Steve Kargl
> <sgk_at_troutmask.apl.washington.edu> wrote:
> > On Sat, Feb 23, 2019 at 12:03:58PM -0700, Warner Losh wrote:
> >> On Sat, Feb 23, 2019 at 10:57 AM Steve Kargl
> >> <sgk_at_troutmask.apl.washington.edu> wrote:
> >>> Supposely, the laptop only has 4 GB of memory.  Not sure how
> >>> it finds memory above 4 GB.
> >>
> >> Some older chipsets had a 'hole' in memory that they mapped the PCI bus
> >> into and then remapped RAM in that range up above the 4GB boundary.
> That's
> >> how it can find memory above 4GB when you have only 4GB of RAM. I hit it
> >> with the PC Card stuff I did back in the day since it broke certain
> >> heuristics I had in the code that turned out to be unwise for many
> reasons
> >> (not just this one). I don't recall all the details, since it's been so
> >> long ago.
> >>
> >> So I think kib_at_ is right when he highlights
> >>> +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048
> pages)
> >>
> >> as the memory, since this is indeed above the 4GB limit.  It's about
> 128k
> >> of 4k pages (just shy of the 131072 I'd expect), which is a surprisingly
> >> round number. Also one that's easy to implement in hardware. So it
> >> certainly "smells" the same...
> >>
> >> That's why I agree with others that hw.above4g_allow=0 is worth a shot,
> for
> >> at least diagnostic purposes. This memory wasn't used before and if it's
> >> used now by the drm drivers, and those aren't PAE safe (meaning they
> cope
> >> with allocations beyond 4GB), then that's quite useful to know. Or maybe
> >> it's a different driver hating things and stomping on video memory due
> to
> >> wrap around.
> >
> > Thanks for the explanation.  Here's an update.  TL;DR: xorg is
> > up and running; drm-legacy-kmod seems to be unsafe/unaware of
> > PAE.
> >
> > Build world/kernel, drm-legacy-kmod, minimum needed ports for xorg.
> > Kernel is unmodified GENERIC.
> >
> > Reboot without setting anything in /boot/loader.conf
> >
> > % sysctl -a | grep above
> > % sysctl -a | grep pae
> > vm.pmap.pae_mode: 1
> > % kldload /boot/modules/i915kms.ko
> >
> > Black screen of death. Did not even get to running xinit.
> >
> > Hard reset to single user mode.
> >
> > # fsck -y
> > # mount -a
> > # vi /boot/loader.conf.
> > (Add hw.above4g_allow=0)
> > # sync
> > # shutdown -r now
> >
> > % sysctl -a | grep above
> > % sysctl -a | grep pae
> > vm.pmap.pae_mode: 1
> > % cat /boot/loader.conf
> > if_ath_load="YES"
> > if_ath_pci_load="YES"
> > cpuctl_load="YES"
> > hw.above4g_allow=0
> > % kldload /boot/modules/i915kms.ko
> >
> > Switch to vt3, login as normal user.
> >
> > % startx -- -depth 24 >& ~/tmp/.x.out
> >
> > Xorg is up and running.  Not sure why my first attempt at using
> > hw.above4g_allow=0 did not work.  Perhaps, mismatch between the xorg
> > bits and kernel/world bits.
> >
> > % sysctl -a | grep mem
> > vm.lowmem_period: 10
> > vm.kmem_map_free: 1669365760
> > vm.kmem_map_size: 41910272
> > vm.kmem_size_scale: 1
> > vm.kmem_size_max: 1711276032
> > vm.kmem_size_min: 12582912
> > vm.kmem_zmax: 65536
> > vm.kmem_size: 1711276032
> > hw.physmem: 3715489792
> > hw.usermem: 3592175616
> > hw.realmem: 4294963200
> >
> > % dmesg | grep memory
> > real memory  = 4294967296 (4096 MB)
> > avail memory = 3637673984 (3469 MB)
> > agp0: aperture size is 256M, detected 7676k stolen memory
> >
> > The pre-r343567 dmesg has
> >
> > real memory  = 4294967296 (4096 MB)
> > avail memory = 3639914496 (3471 MB)
> >
> > I can live with 2 MB loss.
> >
> > Conclusion, drm-legacy-kmod is not PAE safe/aware.
> >
> > Probably want to put something in /usr/src about possible
> > problems with new pmap.h on i386 FreeBSD.
>
> Now it would be interesting to do the same tests with drm-current-kmod.
>

Maybe I missed it, but Steve, did you run the patched in a different way
tests that I suggested? Replacing the limits with 0xffffffff for testing
purposes to ensure that drm isn't saying it can cope with larger addresses?
That might help narrow down what the problem here one more level than "It's
PAE".

Warner