Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

On Sun, Feb 24, 2019 at 10:06:53AM -0700, Warner Losh wrote:
> On Sun, Feb 24, 2019 at 2:27 AM Tijl Coosemans <tijl_at_freebsd.org> wrote:
> 
> > On Sat, 23 Feb 2019 17:28:51 -0800 Steve Kargl
> > <sgk_at_troutmask.apl.washington.edu> wrote:
> > > On Sat, Feb 23, 2019 at 12:03:58PM -0700, Warner Losh wrote:
> > >> On Sat, Feb 23, 2019 at 10:57 AM Steve Kargl
> > >> <sgk_at_troutmask.apl.washington.edu> wrote:
> > >>> Supposely, the laptop only has 4 GB of memory.  Not sure how
> > >>> it finds memory above 4 GB.
> > >>
> > >> Some older chipsets had a 'hole' in memory that they mapped the PCI bus
> > >> into and then remapped RAM in that range up above the 4GB boundary.
> > That's
> > >> how it can find memory above 4GB when you have only 4GB of RAM. I hit it
> > >> with the PC Card stuff I did back in the day since it broke certain
> > >> heuristics I had in the code that turned out to be unwise for many
> > reasons
> > >> (not just this one). I don't recall all the details, since it's been so
> > >> long ago.
> > >>
> > >> So I think kib_at_ is right when he highlights
> > >>> +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048
> > pages)
> > >>
> > >> as the memory, since this is indeed above the 4GB limit.  It's about
> > 128k
> > >> of 4k pages (just shy of the 131072 I'd expect), which is a surprisingly
> > >> round number. Also one that's easy to implement in hardware. So it
> > >> certainly "smells" the same...
> > >>
> > >> That's why I agree with others that hw.above4g_allow=0 is worth a shot,
> > for
> > >> at least diagnostic purposes. This memory wasn't used before and if it's
> > >> used now by the drm drivers, and those aren't PAE safe (meaning they
> > cope
> > >> with allocations beyond 4GB), then that's quite useful to know. Or maybe
> > >> it's a different driver hating things and stomping on video memory due
> > to
> > >> wrap around.
> > >
> > > Thanks for the explanation.  Here's an update.  TL;DR: xorg is
> > > up and running; drm-legacy-kmod seems to be unsafe/unaware of
> > > PAE.
> > >
> > > Build world/kernel, drm-legacy-kmod, minimum needed ports for xorg.
> > > Kernel is unmodified GENERIC.
> > >
> > > Reboot without setting anything in /boot/loader.conf
> > >
> > > % sysctl -a | grep above
> > > % sysctl -a | grep pae
> > > vm.pmap.pae_mode: 1
> > > % kldload /boot/modules/i915kms.ko
> > >
> > > Black screen of death. Did not even get to running xinit.
> > >
> > > Hard reset to single user mode.
> > >
> > > # fsck -y
> > > # mount -a
> > > # vi /boot/loader.conf.
> > > (Add hw.above4g_allow=0)
> > > # sync
> > > # shutdown -r now
> > >
> > > % sysctl -a | grep above
> > > % sysctl -a | grep pae
> > > vm.pmap.pae_mode: 1
> > > % cat /boot/loader.conf
> > > if_ath_load="YES"
> > > if_ath_pci_load="YES"
> > > cpuctl_load="YES"
> > > hw.above4g_allow=0
> > > % kldload /boot/modules/i915kms.ko
> > >
> > > Switch to vt3, login as normal user.
> > >
> > > % startx -- -depth 24 >& ~/tmp/.x.out
> > >
> > > Xorg is up and running.  Not sure why my first attempt at using
> > > hw.above4g_allow=0 did not work.  Perhaps, mismatch between the xorg
> > > bits and kernel/world bits.
> > >
> > > % sysctl -a | grep mem
> > > vm.lowmem_period: 10
> > > vm.kmem_map_free: 1669365760
> > > vm.kmem_map_size: 41910272
> > > vm.kmem_size_scale: 1
> > > vm.kmem_size_max: 1711276032
> > > vm.kmem_size_min: 12582912
> > > vm.kmem_zmax: 65536
> > > vm.kmem_size: 1711276032
> > > hw.physmem: 3715489792
> > > hw.usermem: 3592175616
> > > hw.realmem: 4294963200
> > >
> > > % dmesg | grep memory
> > > real memory  = 4294967296 (4096 MB)
> > > avail memory = 3637673984 (3469 MB)
> > > agp0: aperture size is 256M, detected 7676k stolen memory
> > >
> > > The pre-r343567 dmesg has
> > >
> > > real memory  = 4294967296 (4096 MB)
> > > avail memory = 3639914496 (3471 MB)
> > >
> > > I can live with 2 MB loss.
> > >
> > > Conclusion, drm-legacy-kmod is not PAE safe/aware.
> > >
> > > Probably want to put something in /usr/src about possible
> > > problems with new pmap.h on i386 FreeBSD.
> >
> > Now it would be interesting to do the same tests with drm-current-kmod.
> >
> 
> Maybe I missed it, but Steve, did you run the patched in a different way
> tests that I suggested? Replacing the limits with 0xffffffff for testing
> purposes to ensure that drm isn't saying it can cope with larger addresses?
> That might help narrow down what the problem here one more level than "It's
> PAE".
> 

I did try a few patches to the drm-legacy-kmod port when I first 
found that it would not build, but I honestly don't remember if
your suggestion was one of them.  I'm testing with the stock port
at 

  r492863 | zeising | 2019-02-13 12:52:03 -0800 (Wed, 13 Feb 2019) | 8 lines

  graphics/drm-legacy-kmod: Update snapshot

  Update the graphics/drm-legacy-kmod drivers to the latest snapshot.  This
  includes fixes to make the driver build on CURRENT after base r343567.

  Reported by:    Steve Kargl
  Approved by:    jmd (maintainer, implicit)

If I do not set hw.above4g_allow to 0, the above port will lock up
the system.  I will need to find your email with the 0xffffffff 
sugestion and compare that to what Niclas did to fix the build 
of the port.

-- 
Steve