Re: tar/libarchive gzip problem [was: portsnap corrupted]

I just committed a fix for this (r186670).  Please let
me know if you see any other problems.

Tim

Tim Kientzle wrote:
> I think I know what this is.  I tried recently to add
> support for concatenated gzip files.  This involves looking
> ahead for another GZip header after the end of the compressed
> data.  The current version screws this up and ends up trying
> to match the file CRC as a Gzip header.  About 1 in 256 files
> will match the first byte, which triggers the subsequent meltdown.
> This also explains why neither of us saw it in testing.
> (I knew the code didn't actually work for concatenated gzip
> files but didn't realize it would break decode of some
> regular non-concatenated files.)
> 
> The attached patch simply disables this additional
> header check, which should fix the immediate problem.
> Please try it and let me know.
> 
> Apologies,
> 
> Tim
> 
> Simon L. Nielsen wrote:
> 
>> Hey Tim,
>>
>> I think one of the recent changes to tar or libarchive broke gzip
>> handling in some cases.  See more below.
>>
>> [portsnap extract fails with gzip error]
>>
>> I'm not sure why I didn't run into it in my tests, but I think the
>> problem is in tar / libarchive's handling of gzip files.  Taking one
>> random "broken" file [1] it fails with tar's build in decompression,
>> but works using external zcat.
>>
>> [1] 
>> http://portsnap1.freebsd.org/f/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz 
>>
>> http://people.freebsd.org/~simon/tmp/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz 
>>
>>
>>
>> Broken system:
>>
>> [simon_at_eddie:/tmp] tar --version
>> bsdtar 2.5.903a - libarchive 2.5.903a
>> [simon_at_eddie:/tmp] uname -a
>> FreeBSD eddie.nitro.dk 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Tue Dec 30 
>> 22:28:33 CET 2008     
>> simon_at_eddie.nitro.dk:/FreeBSD/obj/FreeBSD/system-CURRENT/sys/EDDIE  i386
>> [simon_at_eddie:/tmp] tar tvf 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
>> /dev/null
>> tar: Error opening archive: Invalid GZip header (saw 99 at offset 1)
>> [simon_at_eddie:/tmp] zcat 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
>> tar tf - > /dev/null [simon_at_eddie:/tmp] zcat 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
>> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
>>
>>
>> OK system:
>>
>> [simon_at_benji:/tmp] tar --version
>> bsdtar 2.5.5 - libarchive 2.5.5
>> [simon_at_benji:/tmp] uname -a
>> FreeBSD benji.s 7.1-RC2 FreeBSD 7.1-RC2 #0: Tue Dec 23 15:18:30 UTC 
>> 2008     root_at_logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>> [simon_at_benji:/tmp] tar tvf 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
>> /dev/null [simon_at_benji:/tmp] zcat 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
>> sha256 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
>>
>>
>> Another OK system:
>>
>> [simon_at_ref8-amd64:files] tar --version
>> bsdtar 2.5.5 - libarchive 2.5.5
>> [simon_at_ref8-amd64:files] uname -a
>> FreeBSD ref8-amd64.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #2 
>> r184542:185402: Fri Nov 28 19:14:40 UTC 2008     
>> peter_at_ref8-amd64.freebsd.org:/scratch/src/sys/amd64/compile/REF8-AMD64  
>> amd64
>> [simon_at_ref8-amd64:files] tar tvf 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > 
>> /dev/null
>> [simon_at_ref8-amd64:files] zcat 
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | 
>> sha256
>> 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
>>
> 
> ------------------------------------------------------------------------
> 
> Index: archive_read_support_compression_gzip.c
> ===================================================================
> --- archive_read_support_compression_gzip.c	(revision 185679)
> +++ archive_read_support_compression_gzip.c	(working copy)
> _at__at_ -428,8 +428,7 _at__at_
>  						"Failed to clean up gzip decompressor");
>  					return (ARCHIVE_FATAL);
>  				}
> -				/* Restart header parser with the next block. */
> -				state->header_state = state->header_done = 0;
> +				state->eof = 1;
>  				/* FALL THROUGH */
>  			case Z_OK: /* Decompressor made some progress. */
>  				/* If we filled our buffer, update stats and return. */