Re: tar/libarchive gzip problem [was: portsnap corrupted]

On 2008.12.31 18:31:17 -0800, Tim Kientzle wrote:
> I just committed a fix for this (r186670).  Please let
> me know if you see any other problems.

This seems to fix it, thanks!

Max, could you see if you have portsnap problems with the latest
libarchive?

> Tim Kientzle wrote:
> > I think I know what this is.  I tried recently to add
> > support for concatenated gzip files.  This involves looking
> > ahead for another GZip header after the end of the compressed
> > data.  The current version screws this up and ends up trying
> > to match the file CRC as a Gzip header.  About 1 in 256 files
> > will match the first byte, which triggers the subsequent meltdown.
> > This also explains why neither of us saw it in testing.
> > (I knew the code didn't actually work for concatenated gzip
> > files but didn't realize it would break decode of some
> > regular non-concatenated files.)
> > 
> > The attached patch simply disables this additional
> > header check, which should fix the immediate problem.
> > Please try it and let me know.
> > 
> > Apologies,
> > 
> > Tim
> > 
> > Simon L. Nielsen wrote:
> > 
> >> Hey Tim,
> >>
> >> I think one of the recent changes to tar or libarchive broke gzip
> >> handling in some cases.  See more below.
> >>
> >> [portsnap extract fails with gzip error]
> >>
> >> I'm not sure why I didn't run into it in my tests, but I think the
> >> problem is in tar / libarchive's handling of gzip files.  Taking one
> >> random "broken" file [1] it fails with tar's build in decompression,
> >> but works using external zcat.
> >>
> >> [1] 
> >> http://portsnap1.freebsd.org/f/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz 
> >>
> >> http://people.freebsd.org/~simon/tmp/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz 
> >>
> >>
> >>
> >> Broken system:
> >>
> >> [simon_at_eddie:/tmp] tar --version
> >> bsdtar 2.5.903a - libarchive 2.5.903a
> >> [simon_at_eddie:/tmp] uname -a
> >> FreeBSD eddie.nitro.dk 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Tue Dec 30 
> >> 22:28:33 CET 2008     
> >> simon_at_eddie.nitro.dk:/FreeBSD/obj/FreeBSD/system-CURRENT/sys/EDDIE  i386
> >> [simon_at_eddie:/tmp] tar tvf 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
> >> /dev/null
> >> tar: Error opening archive: Invalid GZip header (saw 99 at offset 1)
> >> [simon_at_eddie:/tmp] zcat 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
> >> tar tf - > /dev/null [simon_at_eddie:/tmp] zcat 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
> >> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
> >>
> >>
> >> OK system:
> >>
> >> [simon_at_benji:/tmp] tar --version
> >> bsdtar 2.5.5 - libarchive 2.5.5
> >> [simon_at_benji:/tmp] uname -a
> >> FreeBSD benji.s 7.1-RC2 FreeBSD 7.1-RC2 #0: Tue Dec 23 15:18:30 UTC 
> >> 2008     root_at_logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
> >> [simon_at_benji:/tmp] tar tvf 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
> >> /dev/null [simon_at_benji:/tmp] zcat 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
> >> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
> >>
> >>
> >> Another OK system:
> >>
> >> [simon_at_ref8-amd64:files] tar --version
> >> bsdtar 2.5.5 - libarchive 2.5.5
> >> [simon_at_ref8-amd64:files] uname -a
> >> FreeBSD ref8-amd64.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #2 
> >> r184542:185402: Fri Nov 28 19:14:40 UTC 2008     
> >> peter_at_ref8-amd64.freebsd.org:/scratch/src/sys/amd64/compile/REF8-AMD64  
> >> amd64
> >> [simon_at_ref8-amd64:files] tar tvf 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
> >> /dev/null
> >> [simon_at_ref8-amd64:files] zcat 
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
> >> sha256
> >> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
> >>
> > 
> > ------------------------------------------------------------------------
> > 
> > Index: archive_read_support_compression_gzip.c
> > ===================================================================
> > --- archive_read_support_compression_gzip.c	(revision 185679)
> > +++ archive_read_support_compression_gzip.c	(working copy)
> > _at__at_ -428,8 +428,7 _at__at_
> >  						"Failed to clean up gzip decompressor");
> >  					return (ARCHIVE_FATAL);
> >  				}
> > -				/* Restart header parser with the next block. */
> > -				state->header_state = state->header_done = 0;
> > +				state->eof = 1;
> >  				/* FALL THROUGH */
> >  			case Z_OK: /* Decompressor made some progress. */
> >  				/* If we filled our buffer, update stats and return. */
> 

-- 
Simon L. Nielsen