Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

On Sat, Feb 23, 2019 at 12:03:58PM -0700, Warner Losh wrote:
> On Sat, Feb 23, 2019 at 10:57 AM Steve Kargl <
> sgk_at_troutmask.apl.washington.edu> wrote:
> 
> > Supposely, the laptop only has 4 GB of memory.  Not sure how
> > it finds memory above 4 GB.
> >
> 
> Some older chipsets had a 'hole' in memory that they mapped the PCI bus
> into and then remapped RAM in that range up above the 4GB boundary. That's
> how it can find memory above 4GB when you have only 4GB of RAM. I hit it
> with the PC Card stuff I did back in the day since it broke certain
> heuristics I had in the code that turned out to be unwise for many reasons
> (not just this one). I don't recall all the details, since it's been so
> long ago.
> 
> So I think kib_at_ is right when he highlights
> > +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages)
> 
> as the memory, since this is indeed above the 4GB limit.  It's about 128k
> of 4k pages (just shy of the 131072 I'd expect), which is a surprisingly
> round number. Also one that's easy to implement in hardware. So it
> certainly "smells" the same...
> 
> That's why I agree with others that hw.above4g_allow=0 is worth a shot, for
> at least diagnostic purposes. This memory wasn't used before and if it's
> used now by the drm drivers, and those aren't PAE safe (meaning they cope
> with allocations beyond 4GB), then that's quite useful to know. Or maybe
> it's a different driver hating things and stomping on video memory due to
> wrap around.
> 
> Warner

Thanks for the explanation.  Here's an update.  TL;DR: xorg is
up and running; drm-legacy-kmod seems to be unsafe/unaware of
PAE.

Build world/kernel, drm-legacy-kmod, minimum needed ports for xorg.
Kernel is unmodified GENERIC.

Reboot without setting anything in /boot/loader.conf

% sysctl -a | grep above
% sysctl -a | grep pae
vm.pmap.pae_mode: 1
% kldload /boot/modules/i915kms.ko

Black screen of death. Did not even get to running xinit.

Hard reset to single user mode.

# fsck -y
# mount -a
# vi /boot/loader.conf.
(Add hw.above4g_allow=0)
# sync
# shutdown -r now

% sysctl -a | grep above
% sysctl -a | grep pae
vm.pmap.pae_mode: 1
% cat /boot/loader.conf
if_ath_load="YES"
if_ath_pci_load="YES"
cpuctl_load="YES"
hw.above4g_allow=0
% kldload /boot/modules/i915kms.ko

Switch to vt3, login as normal user.

% startx -- -depth 24 >& ~/tmp/.x.out

Xorg is up and running.  Not sure why my first attempt at using
hw.above4g_allow=0 did not work.  Perhaps, mismatch between the xorg
bits and kernel/world bits.

% sysctl -a | grep mem
vm.lowmem_period: 10
vm.kmem_map_free: 1669365760
vm.kmem_map_size: 41910272
vm.kmem_size_scale: 1
vm.kmem_size_max: 1711276032
vm.kmem_size_min: 12582912
vm.kmem_zmax: 65536
vm.kmem_size: 1711276032
hw.physmem: 3715489792
hw.usermem: 3592175616
hw.realmem: 4294963200

% dmesg | grep memory
real memory  = 4294967296 (4096 MB)
avail memory = 3637673984 (3469 MB)
agp0: aperture size is 256M, detected 7676k stolen memory

The pre-r343567 dmesg has

real memory  = 4294967296 (4096 MB)
avail memory = 3639914496 (3471 MB)

I can live with 2 MB loss.

Conclusion, drm-legacy-kmod is not PAE safe/aware.

Probably want to put something in /usr/src about possible
problems with new pmap.h on i386 FreeBSD.

-- 
Steve