can buffer cache pages be used in ext_pgs mbufs?

Hi,

I've been at this game for a while and one of the axioms is...
"Everything is harder than it at first looks."

Currently, when the FreeBSD NFS client does a write, it does:
- VOP_WRITE() copies the data into buffer cache block(s).
--> An nfsiod thread (or sometimes the thread that called VOP_WRITE()),
   copies the data from the buffer cache block into a list of mbuf clusters,
   prepends the NFS and RPC headers, then passes it down to TCP via sosend().

   After the RPC reply is received (or the RPC fails due to timeout):
   - m_freem() is called on the mbuf list.
   - bufdone()/brelse() is called for the buffer cache block.

For TLS, the mbuf list passed into sosend() must be ext_pgs mbufs, so the
mbuf clusters get copied to ext_pgs mbufs with anonymous pages before
the sosend() call.

So, what if the pages associated with the buffer cache block (b_pages)
were entered in the m_epg_pa[] array for the ext_pgs mbufs, instead of
copying the data into mbuf clusters?
- At a glance, this just seems like it would work.
  It looks like the buffer cache pages are wired down until bufdone()/brelse(),
  which happens after m_freem() on the mbuf list.
- There would need to be a custom m__ext.ext_free, but it looks like a no-op.
  (ie. does nothing, since the buffer cache code deals with the pages later.)

The only thing I can think of (and I don't understand the vm/memory cache
parts of FreeBSD) is that, since the buffer cache pages are written via copying
into their kva addresses and then read via the direct map of their physical
pages, there might be some sort of memory cache flush needed to ensure the
physical pages are up to date (no data still working its way through write-back).
- Is this a problem and how is it handled?

In summary, what am I missing that makes this difficult/impossible to do?

If no one has an answer, I'll just code it up and see what happens.

Thanks for any comments, rick