Re: Import of DragonFly Mail Agent

On 2014-02-25 16:31, RW wrote:
> On Mon, 24 Feb 2014 19:24:02 -0500 (EST)
> Benjamin Kaduk wrote:
> 
>> On Mon, 24 Feb 2014, Lyndon Nerenberg wrote:
>>
>>>
>>> What would really help is if the ports fetch-recursive-list target
>>> could extend to reliably include the distfiles for the runtime
>>> dependencies as well.  But I'm not even sure that's possible.  We
>>> tried a few different things, but in the end we had to brute force
>>> it by running 'make fetch' in every one of the ports directories in
>>> order to get all the distfiles onto an external system, which we
>>> then rsynced to a USB drive, marched inside, and rsynced to the
>>> fileserver.  Not pretty ... but with all the distfiles at hand we
>>> knew the inside ports builds wouldn't fail due to missing
>>> dependencies.
>>
>> I'm rather confused by why it isn't working for you. 
>> http://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk?revision=345884&view=markup#l5187 
>> is quite clearly looking in ALL-DEPENDS-LIST, which includes runtime 
>> dependencies.  The only thing I can think of is that non-default 
>> configurations are in play, so that 'make config && make
>> config-recursive' should be (re-)run until it does not prompt, and
>> only then fetch-recursive-list be used.  
> 
> 
> One oddity is that fetch-recursive-list generates a script that
> downloads all the files into the current directory. It doesn't take
> account of the fact that some ports look for their files are in a
> sub-directory. 


Some snippets from a script that is used to manage updates,
tinderboxe builds, poudriere builds ...


I collected all ports that are required to build my environments
from tinderbox (./tc listPorts) and others in a plain txt file.
in the format $cat/$port.

...
databases/php5-pdo
databases/php5-pdo_mysql
databases/php5-pdo_pgsql
databases/php5-pdo_sqlite
databases/php5-pgsql
databases/postgresql92-client
databases/postgresql92-server
databases/postgresql93-client
databases/postgresql93-server
databases/py-gdbm
databases/rrdtool
databases/rrdtool12
databases/sqlite3
...


Reading this file in a loop with a command like the following
will fetch all required distfiles.

while read port; do
  env -i WRKDIRPREFIX=/tmp/rbtrash PKG_DBDIR=/var/empty \
   LOCALBASE=/var/empty make fetch -DBATCH -C /usr/ports/${port} \
   -DCLEAN_FETCH_ENV -DDISABLE_CONFLICTS
done < $path/to/interesting/port/list


A list of all required dependency's can be generated with this command
(for a single port or in the sample loop (s/fetch/all-depends-list/)

$> make all-depends-list /usr/ports/$cat/${port}


Ports tree updates (portsnap or svn up) are written to a log which is used
to generate a list of ports where the distfile is maybe missing, the loop
reads then only this new list.

The directory with all distfiles is distributed via httpd to all build
systems (make.conf: MASTER_SITE_OVERRIDE=$central/fetch/server/url )


Hope this gives some ideas ;)

-- 
olli