Towards a working "wine". [long]

Y'all,

I wanted to run a substantial windows app using "wine", and failed
miserably. Looking around, it appears to be an acknowledged fact
that wine is borked on FreeBSD, so I did some hacking about.

The problems I found are centered around address space allocation:
Wine is inevitably somewhat sensitive to address space layout,
seeing as it has to present the Win32 binaries with a reasonably
familiar environment. (All this assumes standard kernel, etc. Fixed
addresses are the order of the day)

Problem 1:

When starting up, Wine needs to mmap stuff at address 0x80000000,
but does so without using MAP_FIXED: I think the intention is that
the code involved should be "best effort" to map to the hint address
supplied to mmap(), rather than an all-or-nothing thing.

FreeBSD's mmap sees this as an address in the data segment (see
problem 2 below), and shifts the hint along to an address past
there.  There appears to be a MAP_TRYFIXED flag that wine uses on
some systems that affords the hint more weight, which is pretty
trivial to implement (see wine_mmap.txt).  That was enough to get
my app running, but not for long.

Problem 2:

The "wine" executable itself is located strangely, so the text and
data go right up near the 2G point to help out Win32. (actually at
address 0x77f00000, or 2GB-129MB).  This is causes the heap to
appear at such a high address, as FreeBSD merges the data from the
executable image with the heap space (which malloc uses). Once the
mmap at 0x80000000 happens, that puts a nice hole in the data segment
that means sbrk starts to fail after 128MB is allocated.  (On linux,
it appears that the brk point and load address of the data segment
from the executable are unrelated, and malloc() pays little attention
to [s]brk() anyhow.)

End result is, that with the break point sitting where it does, and
the mapped Windows stuff 129MB above it, wine is very low on
malloc()able memory.

There is a disasterously ugly hack attached, wine_malloc.txt that
hacks on malloc(), and adds a "W" option to enable the hack. This
works by trading the brk()/sbrk() calls for an mmapping starting
at 0xa0000000, which should be able to grow towards the process
stack. (phkmalloc works with a large contiguous heap, rather than
a fragmented one, so a more "pure" mmap-based approach won't fit
into it too smoothly.)

WARNING: Don't put the malloc patch directly into libc unless you
are in the mood to deal with a hosed libc. I ran it like this:

> $ cd /tmp
> $ cp /usr/src/lib/libc/stdlib/malloc.c ./wine_malloc.c
> $ patch wine_malloc.c < /tmp/wine_malloc.txt
> $ cc -I /usr/src/lib/libc/include -o wine_malloc.so \
> -shared wine_malloc.c

Then invoke wine as
$ env LD_PRELOAD=/tmp/wine_malloc.so MALLOC_OPTIONS=W wine ...

With these patches, wine is much, much more stable for me.

I'd like to commit the kernel part at some stage if someone can
review it, but the malloc() part seems like a particularly ugly
hack. It could be cleaned up and added as a patch to the port I
suppose, but has anyone got any better ideas?