4GB RAM with an intel core 2 processor

I recently tried to upgrade my Dell workstation from 2GB to 4GB RAM.
It's an Optiplex 755 with an Intel Core 2 Quad Q6600 processor. I'm running 7.0-RC1
When I install the 2 extra memory modules, the kernel tells me on startup
that it's ignoring 640MB memory above 4GB, leaving me with a total of 3.375GB
of accessible memory. However when X starts (using a PCI-e Geforce with nvidia driver)
the system reboots.
Now, I took a quick look at PAE and discovered that the PAE kernel is
not meant for anything but headless systems, most of the drivers (including USB!) are out and no
module support either. 

Do I have any options for using my 4GB of RAM and still retain USB and the nvidia driver?

Baldur