Re: cp(1) of large files is causing 100% CPU utilization and poor transfer

It seems pretty clear to me, based on my experiments, that FIOSEEKHOLE is
O(n) in filesize on UFS (though not on ZFS).  And
vn_generic_copy_file_range calls VOP_IOCTL(..., FIOSEEKHOLE) once for every
block it copies, where blocks can range from 4KB to 1 MB.  So I think there
are three required actions:

1) Fix vn_generic_copy_file_range to remember the hole locations across
different iterations of its loop.
2) Increase the block size that cp uses with copy_file_range.  Right now
it's 2MB, but it should be much larger.
3) Optionally, improve UFS's FIOSEEKHOLE performance.  But it probably
won't be necessary if we fix 1 and 2.

-Alan

On Sat, Jan 2, 2021 at 10:06 AM Rick Macklem <rmacklem_at_uoguelph.ca> wrote:

> Just fyi, I've reproduced the problem.
> All I did was create a 20Gbyte file
> on UFS on a slow (4Gbyte or RAM,
> slow spinning disk) laptop.
> (The UFS file system is just what the installer creates these days.)
>
> cp still hasn't finished and is definitely
> taking a looott longer than dd did.
>
> I'll start drilling down later to-day.
>
> I'll admit doing lots of testing of copy_file_range(2)
> with large sparse files, but I may have missed testing
> a large non-sparse file.
>
> rick
> ps: I've added Kostik and Kirk to the cc.
>
>
> ________________________________________
> From: owner-freebsd-current_at_freebsd.org <owner-freebsd-current_at_freebsd.org>
> on behalf of Alan Somers <asomers_at_freebsd.org>
> Sent: Saturday, January 2, 2021 11:30 AM
> To: Matthias Apitz; FreeBSD CURRENT
> Subject: Re: cp(1) of large files is causing 100% CPU utilization and poor
> transfer
>
> CAUTION: This email originated from outside of the University of Guelph.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe. If in doubt, forward suspicious emails to
> IThelp_at_uoguelph.ca
>
>
> On Sat, Jan 2, 2021 at 9:12 AM Matthias Apitz <guru_at_unixarea.de> wrote:
>
> > El día sábado, enero 02, 2021 a las 09:06:24a. m. -0700, Alan Somers
> > escribió:
> >
> > > > As I said, it can be reproduced using only the local file system.
> This
> > > > was setup recently on a SSD:
> > > >
> > > > # dmesg | grep ada0
> > > > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
> > > > ada0: <TS512GMTS430S R0906A> ACS-2 ATA SATA 3.x device
> > > > ada0: Serial Number F995890846
> > > > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
> > > > ada0: Command Queueing enabled
> > > > ada0: 488386MB (1000215216 512 byte sectors)
> > > >
> > > > and by this procedure:
> > > >
> > > > # gpart create -s gpt ada0
> > > > # gpart add -t freebsd-boot -s 512k -a4k -l ssdboot ada0
> > > > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i1 ada0
> > > > # gpart add -t freebsd-ufs -l ssdrootfs -b 1m -s 2g ada0
> > > > # gpart add -t freebsd-ufs -l ssdvarfs -a 1m -s 2g ada0
> > > > # gpart add -t freebsd-ufs -l ssdusrfs -a 1m ada0
> > > > # newfs -U -t /dev/gpt/ssdrootfs
> > > > # newfs -U -t /dev/gpt/ssdvarfs
> > > > # newfs -U -t /dev/gpt/ssdusrfs
> > > >
> > > > # gpart show -l ada0
> > > > =>        40  1000215136  ada0  GPT  (477G)
> > > >           40        1024     1  ssdboot  (512K)
> > > >         1064         984        - free -  (492K)
> > > >         2048     4194304     2  ssdrootfs  (2.0G)
> > > >      4196352     4194304     3  ssdvarfs  (2.0G)
> > > >      8390656    16777216     4  ssdswap  (8.0G)
> > > >     25167872   975046656     5  ssdusrfs  (465G)
> > > >   1000214528         648        - free -  (324K)
> > > >
> > > > # mount -t ufs
> > > > /dev/gpt/ssdrootfs on / (ufs, local, soft-updates)
> > > > /dev/gpt/ssdvarfs on /var (ufs, local, soft-updates)
> > > > /dev/gpt/ssdusrfs on /usr (ufs, local, soft-updates)
> > > >
> > > > When I run in the /usr fs the command
> > > >
> > > > # cp -p guru-20210102.tar.gz xxx
> > > >
> > > > it copies around 168M per minute.
> > > >
> >
> > > Is that copying from /usr to /usr, or from /usr to /var or /?
> >
> > # cd /home/backups
> > # cp -p guru-20210102.tar.gz xxx
> >
> > i.e. from /usr to /usr.
> >
> >         matthias
> >
>
> Ok, let's narrow this down.  Could you please run the command with the
> attached D script ?
> sudo dtrace -s copy_file_range.d -c "cp -p guru-20210102.tar.gz xxx"
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>