Should we make WITHOUT_LIB32=yes the default for 11-RELEASE?

On most amd64 systems I run, I usually set WITHOUT_LIB32=yes in /etc/src.conf
because I don't need them.  This weekend I did a stock install on an older
AMD64 Core 2 Duo minipc and a buildworld of 10-STABLE took almost two hours 
with LIB32 and CLANG since much of it gets compiled twice.

Is it time to deprecate LIB32 in -current for 11-RELEASE?  I realize some
ports may need it, but I hope that need is waning and we are just spending
a lot of compile time by default for little gain.  We could save a lot of
compile time for a lot of users, and they could still opt-in if needed.
Putting it up for discussion, not insisting it should be done.  Thanks.