Re: About the "USB Cache and busdma usage in USB" thread

On Tuesday 04 August 2009 16:41:36 Grzegorz Bernacki wrote:
> Hans Petter Selasky wrote:
> > CC'ed current: We have a case on ARM where bus_dmamap_sync() is not
> > suffient to update the CPU cache. One reason for this is that USB needs
> > to invalidate the same memory area multiple times. Busdma sync expects
> > paired operation when using the PRE and POST flags, from what I
> > understand. I do not consider this an USB issue, hence Semihalf has got
> > the USB stack working by manually inserting CPU flush/invalidate calls
> > into usb_pc_cpu_invalidate() and usb_pc_cpu_flush(). Their other solution
> > however which modifies the bus_dmamap_sync() flags will break on
> > platforms with more than 4 GByte of memory.
> >
> > Maybe Rafal can give a quick summar to new people at the -current list,
> > or see previous thread on the ARM mailing list.
> >
> > USB needs a solution where it can call a function given a busdma mapping,
> > preferably with an offset and length, which handles the cache sync issue
> > and works with bounce pages on +4GB systems.
>
> Hi Hans,
>
> New USB stack uses busdma in a little unconventional way. As you
> mentioned in one of previous mails your assumptions are:
>
> XXX_PREXXX functions should be used prior to read/write device access.
> In other words, PRE has to be a flush operation.
>
> XXX_POSTXXX functions should be used after read/write device access.
> In other words, POST has to be an invalidate operation.
>
> Generally it is true, but if you look at ARM code you will find out that
>   it is not that simple. You assumed that after
> bus_dmamap_sync(..,BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD) there
> will be no data in cache, but it that's not true.
>
> Cache operation are performed on cache lines (32 bytes on our ARM
> device). Let's say you want to invalidate buffer with size 10 bytes. In
> this case first whole cache line is invalidated ( and now all
> requirements related to busdma synchronization are fulfilled, old
> contents of cache is gone). The second step is to restore back into
> cache 22 bytes of data which were not a part of buffer. After this
> second step data are loaded into cache line (it is because our device
> uses write allocate feature).
> So busdma on ARM "Perform any synchronization required after an update
> of host memory by the device", but we still end up with not invalidated
> flush.
> It is hard to fix it. We cannot just invalidate whole cache line. We
> cannot also use cpu_dcache_wbinv, because this function is called after
> buffer was used by device so we dont want to overwrite those data with
> old cache contents.
>
> One possible solution is to call first
> bus_dmamap_sync(..,BUS_DMASYNC_POSTREAD) and then
> bus_dmamap_sync(..,BUS_DMASYNC_PREREAD) in usb_pc_cpu_invalidate(), but
> this is ugly workaround which applies probably only to ARM case.
>
> The second problem is that you cannot use cpu_dcache_wb(inv) function
> directly because you need to handle bounce pages in USB code. I think
> that  duplication of busdma code makes no sense. Probably it takes less
> work to add bus_dmamap_sync() before/after each transaction.
>
> Could you give us a quick overview of buffer handling in USB stack? I
> want to understand what is the relation between
> usb_pc_cpu_invalidate/flush() functions and reading/writing to USB
> device? From yours previous mail I understand that invalidate is called
> *before* reading and flush *before* writing. Is that true? Can we add a
> functions which will be called *after* reading/writing?

Hi,

There are two kinds of DMA memory in USB regard:

1) Transfer descriptors are allocated in coherent DMA memory.
Operation logic:

1.a) Write to descriptor.
1.b.0) Call usb_pc_cpu_flush() to write data to RAM.
1.b.1) Write more fields to descriptor.
1.b.2) Call usb_pc_cpu_flush() to write data to RAM.
1.c) Call usb_pc_cpu_invalidate() to clear cache.
1.d) Read status field. If not complete goto 1.c)

2) Any kernel virtual memory (which might not be coherent)

2.a.0) CPU read case:
2.a.1) Before transfer start usb_pc_cpu_invalidate() is called to clear any 
data in cache for this buffer.
2.a.2) After transfer completion usb_pc_cpu_invalidate() is called again.

2.b.0) CPU write case:
2.b.1) Before transfer start usb_pc_cpu_flush() is called to to flush any data 
in cache to RAM for this buffer.
2.b.2) After transfer completion there is no cache operation.

Anything unclear?

--HPS

>
> If you have any questions regarding cache operation on ARM.  please let
> me know, I will try to answer them.
>
> regards,
> Grzesiek