Re: About the "USB Cache and busdma usage in USB" thread

Hans Petter Selasky wrote:
> 
> CC'ed current: We have a case on ARM where bus_dmamap_sync() is not suffient 
> to update the CPU cache. One reason for this is that USB needs to invalidate 
> the same memory area multiple times. Busdma sync expects paired operation when 
> using the PRE and POST flags, from what I understand. I do not consider this 
> an USB issue, hence Semihalf has got the USB stack working by manually 
> inserting CPU flush/invalidate calls into usb_pc_cpu_invalidate() and 
> usb_pc_cpu_flush(). Their other solution however which modifies the 
> bus_dmamap_sync() flags will break on platforms with more than 4 GByte of 
> memory.
> 
> Maybe Rafal can give a quick summar to new people at the -current list, or see 
> previous thread on the ARM mailing list.
> 
> USB needs a solution where it can call a function given a busdma mapping, 
> preferably with an offset and length, which handles the cache sync issue and 
> works with bounce pages on +4GB systems.
> 

Hi Hans,

New USB stack uses busdma in a little unconventional way. As you 
mentioned in one of previous mails your assumptions are:

XXX_PREXXX functions should be used prior to read/write device access.
In other words, PRE has to be a flush operation.

XXX_POSTXXX functions should be used after read/write device access.
In other words, POST has to be an invalidate operation.

Generally it is true, but if you look at ARM code you will find out that 
  it is not that simple. You assumed that after 
bus_dmamap_sync(..,BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD) there 
will be no data in cache, but it that's not true.

Cache operation are performed on cache lines (32 bytes on our ARM 
device). Let's say you want to invalidate buffer with size 10 bytes. In 
this case first whole cache line is invalidated ( and now all 
requirements related to busdma synchronization are fulfilled, old 
contents of cache is gone). The second step is to restore back into 
cache 22 bytes of data which were not a part of buffer. After this 
second step data are loaded into cache line (it is because our device 
uses write allocate feature).
So busdma on ARM "Perform any synchronization required after an update 
of host memory by the device", but we still end up with not invalidated 
flush.
It is hard to fix it. We cannot just invalidate whole cache line. We 
cannot also use cpu_dcache_wbinv, because this function is called after 
buffer was used by device so we dont want to overwrite those data with 
old cache contents.

One possible solution is to call first 
bus_dmamap_sync(..,BUS_DMASYNC_POSTREAD) and then 
bus_dmamap_sync(..,BUS_DMASYNC_PREREAD) in usb_pc_cpu_invalidate(), but 
this is ugly workaround which applies probably only to ARM case.

The second problem is that you cannot use cpu_dcache_wb(inv) function 
directly because you need to handle bounce pages in USB code. I think 
that  duplication of busdma code makes no sense. Probably it takes less 
work to add bus_dmamap_sync() before/after each transaction.

Could you give us a quick overview of buffer handling in USB stack? I 
want to understand what is the relation between 
usb_pc_cpu_invalidate/flush() functions and reading/writing to USB 
device? From yours previous mail I understand that invalidate is called 
*before* reading and flush *before* writing. Is that true? Can we add a 
functions which will be called *after* reading/writing?

If you have any questions regarding cache operation on ARM.  please let 
me know, I will try to answer them.

regards,
Grzesiek