Re: r343567 aka PAE vs non-PAE merge breaks i386 freebsd

On Sat, 23 Feb 2019 17:28:51 -0800 Steve Kargl
<sgk_at_troutmask.apl.washington.edu> wrote:
> On Sat, Feb 23, 2019 at 12:03:58PM -0700, Warner Losh wrote:
>> On Sat, Feb 23, 2019 at 10:57 AM Steve Kargl
>> <sgk_at_troutmask.apl.washington.edu> wrote:
>>> Supposely, the laptop only has 4 GB of memory.  Not sure how
>>> it finds memory above 4 GB.
>> 
>> Some older chipsets had a 'hole' in memory that they mapped the PCI bus
>> into and then remapped RAM in that range up above the 4GB boundary. That's
>> how it can find memory above 4GB when you have only 4GB of RAM. I hit it
>> with the PC Card stuff I did back in the day since it broke certain
>> heuristics I had in the code that turned out to be unwise for many reasons
>> (not just this one). I don't recall all the details, since it's been so
>> long ago.
>> 
>> So I think kib_at_ is right when he highlights
>>> +0x0000000100000000 - 0x000000011ffe7fff, 536772608 bytes (131048 pages)
>> 
>> as the memory, since this is indeed above the 4GB limit.  It's about 128k
>> of 4k pages (just shy of the 131072 I'd expect), which is a surprisingly
>> round number. Also one that's easy to implement in hardware. So it
>> certainly "smells" the same...
>> 
>> That's why I agree with others that hw.above4g_allow=0 is worth a shot, for
>> at least diagnostic purposes. This memory wasn't used before and if it's
>> used now by the drm drivers, and those aren't PAE safe (meaning they cope
>> with allocations beyond 4GB), then that's quite useful to know. Or maybe
>> it's a different driver hating things and stomping on video memory due to
>> wrap around.
> 
> Thanks for the explanation.  Here's an update.  TL;DR: xorg is
> up and running; drm-legacy-kmod seems to be unsafe/unaware of
> PAE.
> 
> Build world/kernel, drm-legacy-kmod, minimum needed ports for xorg.
> Kernel is unmodified GENERIC.
> 
> Reboot without setting anything in /boot/loader.conf
> 
> % sysctl -a | grep above
> % sysctl -a | grep pae
> vm.pmap.pae_mode: 1
> % kldload /boot/modules/i915kms.ko
> 
> Black screen of death. Did not even get to running xinit.
> 
> Hard reset to single user mode.
> 
> # fsck -y
> # mount -a
> # vi /boot/loader.conf.
> (Add hw.above4g_allow=0)
> # sync
> # shutdown -r now
> 
> % sysctl -a | grep above
> % sysctl -a | grep pae
> vm.pmap.pae_mode: 1
> % cat /boot/loader.conf
> if_ath_load="YES"
> if_ath_pci_load="YES"
> cpuctl_load="YES"
> hw.above4g_allow=0
> % kldload /boot/modules/i915kms.ko
> 
> Switch to vt3, login as normal user.
> 
> % startx -- -depth 24 >& ~/tmp/.x.out
> 
> Xorg is up and running.  Not sure why my first attempt at using
> hw.above4g_allow=0 did not work.  Perhaps, mismatch between the xorg
> bits and kernel/world bits.
> 
> % sysctl -a | grep mem
> vm.lowmem_period: 10
> vm.kmem_map_free: 1669365760
> vm.kmem_map_size: 41910272
> vm.kmem_size_scale: 1
> vm.kmem_size_max: 1711276032
> vm.kmem_size_min: 12582912
> vm.kmem_zmax: 65536
> vm.kmem_size: 1711276032
> hw.physmem: 3715489792
> hw.usermem: 3592175616
> hw.realmem: 4294963200
> 
> % dmesg | grep memory
> real memory  = 4294967296 (4096 MB)
> avail memory = 3637673984 (3469 MB)
> agp0: aperture size is 256M, detected 7676k stolen memory
> 
> The pre-r343567 dmesg has
> 
> real memory  = 4294967296 (4096 MB)
> avail memory = 3639914496 (3471 MB)
> 
> I can live with 2 MB loss.
> 
> Conclusion, drm-legacy-kmod is not PAE safe/aware.
> 
> Probably want to put something in /usr/src about possible
> problems with new pmap.h on i386 FreeBSD.

Now it would be interesting to do the same tests with drm-current-kmod.