Re: libfetch ftp patch for less latency

On Thursday 29 March 2007 01:53:33 pm Rui Paulo wrote:
> Nate Lawson wrote:
> > To speed up pkg_add and other ftp command channel performance over slow
> > links, change this:
> > [...]
> > All ftp servers I've ever seen support a full path when changing down
> > dirs. This might be a DOS ftp server thing however.  In any case, if
> > there is an error to the all-in-one CWD, the code reverts back to legacy
> > behavior of multiple CWDs.
> 
> Yes, I think this behavior is acceptable. If the pathname fails it only 
> adds one more CWD, but reduces the time on the successful cases (which 
> are many if you consider the Ports collection).
> 
> Dag-Erling Smørgrav wrote:
>  > No.  This is a violation of the FTP protocol.
>  > [...]
>  > When the all-in-one CWD fails, you're SOL.  You have no idea what
>  > state the server is in, and you have to start over.
> 
> About the possible violation of the RFC, I quote:
> 
>        pathname
> 
>           Pathname is defined to be the character string which must be
>           input to a file system by a user in order to identify a file.
>           Pathname normally contains device and/or directory names, and
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>           file name specification.  FTP does not yet specify a standard
>           ^^^^^^^^^^^^^^^^^^^^^^^
>           pathname convention.  Each user must follow the file naming
>           conventions of the file systems involved in the transfer.
> 
> 
>           CHANGE WORKING DIRECTORY (CWD)
> 
>              This command allows the user to work with a different
>              directory or dataset for file storage or retrieval without
>              altering his login or accounting information.  Transfer
>              parameters are similarly unchanged.  The argument is a
>              pathname specifying a directory or other system dependent
>              ^^^^^^^^
>              file group designator.
> 
> 
> I'm not sure why you consider this a violation of the protocol.
> Also, considering the reply codes to the CWD command, only 421 seems to 
> change the server state.
> 
> Have you seen servers that change their state by issuing giving invalid 
> arguments to the CWD command ?

http://www.freebsd.org/cgi/query-pr.cgi?pr=83278

In this case IIRC, the problem was the leading / as the path fetched was 
supposed to be relative to the user's home directory, but instead fetch sent 
an absolute path and failed.

See also http://www.ietf.org/rfc/rfc1738.txt which is the RFC for URLs and 
explains exactly how the <url-path> for ftp URLs should be interpreted:

<quote from="rfc1738">
3.2.2. FTP url-path

   The url-path of a FTP URL has the following syntax:

        <cwd1>/<cwd2>/.../<cwdN>/<name>;type=<typecode>

   Where <cwd1> through <cwdN> and <name> are (possibly encoded) strings
   and <typecode> is one of the characters "a", "i", or "d".  The part
   ";type=<typecode>" may be omitted. The <cwdx> and <name> parts may be
   empty. The whole url-path may be omitted, including the "/"
   delimiting it from the prefix containing user, password, host, and
   port.

   The url-path is interpreted as a series of FTP commands as follows:

      Each of the <cwd> elements is to be supplied, sequentially, as the
      argument to a CWD (change working directory) command.

      If the typecode is "d", perform a NLST (name list) command with
      <name> as the argument, and interpret the results as a file
      directory listing.

      Otherwise, perform a TYPE command with <typecode> as the argument,
      and then access the file whose name is <name> (for example, using
      the RETR command.)

   Within a name or CWD component, the characters "/" and ";" are
   reserved and must be encoded. The components are decoded prior to
   their use in the FTP protocol.  In particular, if the appropriate FTP
   sequence to access a particular file requires supplying a string
   containing a "/" as an argument to a CWD or RETR command, it is
   necessary to encode each "/".

   For example, the URL <URL:ftp://myname_at_host.dom/%2Fetc/motd> is
   interpreted by FTP-ing to "host.dom", logging in as "myname"
   (prompting for a password if it is asked for), and then executing
   "CWD /etc" and then "RETR motd". This has a different meaning from
   <URL:ftp://myname_at_host.dom/etc/motd> which would "CWD etc" and then
   "RETR motd"; the initial "CWD" might be executed relative to the
   default directory for "myname". On the other hand,
   <URL:ftp://myname_at_host.dom//etc/motd>, would "CWD " with a null
   argument, then "CWD etc", and then "RETR motd".
</quote>

-- 
John Baldwin