Re: Getting/Forcing Greater than 4KB Buffer Allocations

On Wed, Jul 18, 2007 at 11:50:02AM -0700, David Christensen wrote:
 > > On Tue, Jul 17, 2007 at 04:54:31PM -0700, David Christensen wrote:
 > >  > I'm investigating a problem with my bce driver which 
 > > occurs when I ask
 > >  > for a jumbo
 > >  > mbuf cluster (through m_cljget()).  When I map the memory for DMA I
 > >  > normally 
 > >  > get 3 memory segments (4KB + 4KB + 1KB) on my system, but 
 > > on another
 > >  > user's 
 > >  > system he's seeing 2 memory segments (8KB + 1KB).  Is there a
 > >  > configuration
 > >  > option that allows this or some other tuning variable 
 > > involved?  The
 > >  > system is a 
 > >  > Xeon dual-core processor and has 8GB of RAM, running an 
 > > AMD64 version of
 > >  > the kernel.
 > >  >  
 > > 
 > > I've briefly looked over bus_dma usage on bce(4). It seems that you
 > > told bus_dma the the dma map could be made up of BCE_MAX_SEGMENTS
 > > segments, where a dma segment could be MJUM9BYTES bytes. If you want
 > > just two segments you may have to use 2 instead of BCE_MAX_SEGMENTS.
 > > If the hardware can support up to BCE_MAX_SEGMENTS dma segments on Rx
 > > descriptors you should be prepared to handle that number of dma
 > > segments too(e.g. You don't know how may dma segments would be
 > > returned by bus_dma, you just know the upper bound as you specified
 > > in bus_dma_tag_create()).
 > > If the hardware can handle just up to 4KB for a dma segment you
 > > should tell bus_dma the restriction of the dma segment.
 > > If you have to get a single dma segment that covers MJUM9BYTES bytes
 > > due to the limitation of the hardware you may have to use local
 > > allocator.
 > > 
 > 
 > Thanks Pyun but I'm really just looking for a way to test that I can
 > handle the number of segments I've advertised that I can support.  I 
 > believe my code is correct but when all I see are allocations of 3 
 > segments I just can't prove it.  I was hoping that running a utility
 > such as "stress" would help fragment memory and force more variable
 > responses but that hasn't happened yet.
 > 

It seems you've used the following code to create jumbo dma tag.
        /*
         * Create a DMA tag for RX mbufs.
         */
        if (bus_dma_tag_create(sc->parent_tag,
                        1,
                        BCE_DMA_BOUNDARY,
                        sc->max_bus_addr,
                        BUS_SPACE_MAXADDR,
                        NULL, NULL,
                        MJUM9BYTES,
                        BCE_MAX_SEGMENTS,
                        MJUM9BYTES,
                        ^^^^^^^^^^
                        0,
                        NULL, NULL,
                &sc->rx_mbuf_tag)) {
                BCE_PRINTF("%s(%d): Could not allocate RX mbuf DMA tag!\n",
                        __FILE__, __LINE__);
                rc = ENOMEM;
                goto bce_dma_alloc_exit;
        }
If you want to have > 9 dma segements change maxsegsz(MJUM9BYTES) to
1024. bus_dma honors maxsegsz argument so you wouldn't get a dma
segments larger than maxsegsz. With MJUM9BYTES maxsegsz you would get
up to 4 dma segments on systems with 4K PAGE_SIZE.(You would have
got up to 3 dma segements if you used PAGE_SIZE alignment argument.)

 > Dave
 > 
 > 
-- 
Regards,
Pyun YongHyeon