Re: [HEADS-UP] mbuma is in the tree

Wes Peters wrote:
>It may also be worthwhile investigating eliminating clusters entirely.  This
>is the point Poul-Henning, Robert and I were trying to make at the end of
>you talk at BSDCan.

  I believe only Poul-Henning was actually suggesting something along
  these lines at the end of the talk.  As I explained then, this is not
  a good approach.

  First of all, it pessimizes the send case.  You no longer have 2K of
  space for payload, you have 2K - whatever the mbuf struct is.

  Secondly, even on the receiver side, it is probably not worth the
  complication, especially with mbuma now.  You should really both
  read the paper and read the code before you go ahead and toy around
  with an idea like this.  You will notice that when we need both a
  cluster and an mbuf, there is no longer a double-allocation in the
  common case.  It is a single object allocation as we maintain a
  secondary zone which caches mbufs with pre-allocated clusters.
  That means you're looking at one pcpu UMA lock in the common case
  right now and, if I have my way eventually, no locks whatsoever
  in the common UMA allocation path.

  By increasing the mbuf size, you will create a third type of object
  called a "large mbuf."  You'll still need clusters for the sender
  side, and regular mbufs for smaller packets (there are many of these,
  refer to the paper).  So in actual fact, you're not fixing anything,
  you're just introducing another object type that the mbuf code now
  needs to identify and free appropriately from the free routine.

  How about instead of this, you look into creating mini-mbufs,
  which are sort of like regular mbufs, but without the internal data
  region, and which are ONLY used for external storage.  They work
  for all types of external storage, waste less space, and can be
  cached within a UMA zone and thus allocated as effectively a single
  object in the common case; this is exactly what happens now already
  with m_getcl(), except that there is some additional wastage due to
  the internal mbuf data region not being used. 

>Since the double allocation required to create a cluster makes the locking
>(and cache slushing) requirements go up, it is probably worthwhile to
>investigate if raising the nominal mbuf size doesn't end up decreasing
>overall memory pressure.  If you allocate more memory, but the allocation
>takes less time due to the simpler locking, you may actually decrease the
>total memory need.

  No.  See above.

>This is worth investigating partly because it is such a simple change.  I
>propose investigating with mbuf size of 2K, large enough to fit standard
>ethernet frames, and a cluster size of 8K, which means a cluster mbuf is
>large enough to hold a 9K jumbo frame.

  LOL.

>Now that you've got mbuma in the tree, I can test this for you, unless this
>proposal catches your interest enough to give it a try.  I'll see if I
>can't get a couple of our beefier machines at work updated to -CURRENT in
>the next week.
>
>Thanks for the good work.

  Sure.  You can test whatever you like.

-Bosko