hashtar and zip_sha256 utility

Example usage:

codebird@excession:/media/spinny/data/weeklybackup/code/hashtar$ md5sum ~/.steam/bin/*.so 70adba6c667013b229b57640c0eb08a7 /home/codebird/.steam/bin/chromehtml.so d4023af7260e9ebeb0f5aa95323fbc77 /home/codebird/.steam/bin/crashhandler.so ceb1aeecc720ea5ed2bd0ab19ba63f78 /home/codebird/.steam/bin/filesystem_stdio.so 5be7650775851986ce9c94545d34507c /home/codebird/.steam/bin/friendsui.so 1e8a4121f81786248620de6693dcd67d /home/codebird/.steam/bin/gameoverlayrenderer.so 36765e2c7170d2e3608e0fc2a57618fb /home/codebird/.steam/bin/gameoverlayui.so 204cc13835f750c27937d07089cdf062 /home/codebird/.steam/bin/libaudio.so c54f023a551bc32614d334828adcadc3 /home/codebird/.steam/bin/libcef.so 8cdce5fcbb7257699e580ee9ce6887a7 /home/codebird/.steam/bin/libffmpegsumo.so d6a4abc0516567ca7c851d05c8771fc6 /home/codebird/.steam/bin/libmiles.so 50ef494c404b55932dfefdd29e8fd9c0 /home/codebird/.steam/bin/liboverride.so 796c45d3645d8e4b95f5a88e57f545b8 /home/codebird/.steam/bin/libsteam.so 5f86fc52856c1269f1a6e8d77a901e41 /home/codebird/.steam/bin/libtier0_s.so 8faaded3ba84825948dcdf27964bcfcb /home/codebird/.steam/bin/libvstdlib_s.so 1a81b4b5bd4fda72d171eed1219f0700 /home/codebird/.steam/bin/serverbrowser.so 6c243a333a4e0c03ec1a451046294a10 /home/codebird/.steam/bin/steamclient.so f5c8f172cf3bd3c5be0a74728d3d267f /home/codebird/.steam/bin/steamservice.so 6c65cdd39c9a5d12bd93282896e7dee8 /home/codebird/.steam/bin/steamui.so 03fb71f53e5df287c8b1e20b6c8d2472 /home/codebird/.steam/bin/vgui2_s.so codebird@excession:/media/spinny/data/weeklybackup/code/hashtar$ tar -cf test.tar ~/.steam/bin/*.so tar: Removing leading `/' from member names codebird@excession:/media/spinny/data/weeklybackup/code/hashtar$ ./hashtar md5sum test.tar 70adba6c667013b229b57640c0eb08a7 - d4023af7260e9ebeb0f5aa95323fbc77 - ceb1aeecc720ea5ed2bd0ab19ba63f78 - 5be7650775851986ce9c94545d34507c - 1e8a4121f81786248620de6693dcd67d - 36765e2c7170d2e3608e0fc2a57618fb - 204cc13835f750c27937d07089cdf062 - c54f023a551bc32614d334828adcadc3 - 8cdce5fcbb7257699e580ee9ce6887a7 - d6a4abc0516567ca7c851d05c8771fc6 - 50ef494c404b55932dfefdd29e8fd9c0 - 796c45d3645d8e4b95f5a88e57f545b8 - 5f86fc52856c1269f1a6e8d77a901e41 - 8faaded3ba84825948dcdf27964bcfcb - 1a81b4b5bd4fda72d171eed1219f0700 - 6c243a333a4e0c03ec1a451046294a10 - f5c8f172cf3bd3c5be0a74728d3d267f - 6c65cdd39c9a5d12bd93282896e7dee8 - 03fb71f53e5df287c8b1e20b6c8d2472 -

Hashtar and zip_sha256 are utility programs for working with tar and zip files, particually in identifying ZIP files that are content-identical but binary-distinct.

Hashtar is a very small program. It performs only one simple task. Execute it, passing a tar file (Or tar on stdin) and the name of an executable. It will then, for each file within the tar, execute the specified program and feed it that file on stdin, passing output to stdout. Most often the specified program will be a hashing function - md5sum, sha256sum, something of similar nature. It's handy for verifying tape backups, as it can be easily used to produce a list of hashes for every file stored on a tape.

It won't take tar.gz/bz2/xz directly, but you can easily pipe them through their respective decompressors first. Symlinks in tar files and other such non-file contents are ignored. The 'longlink' extension is correctly handled.

The program is just one small source file - the tar format is quite simple, so only the bare minimum of code is required to parse it and pass the data to an external program.

zip_sha256 is a similar program, but limited to the sha256 hash. It has three modes of operation. The first outputs a list of sha256 hashes, one for each file contained within the ZIP, sorted by filename. The second outputs the SHA256 of all files concatenated, again sorted by filename. The third outputs the SHA256 of all files plus their filenames concatenated, again sorted by filename.

I created this small utiltity as an aid in identifying duplicated files: It generates fingerprints. The hashes output by zip_sha256 are unique not to the ZIP file, but to the contents of the ZIP file - regardless of the compression settings, the utility used to compress the file, or the order in in which the files are stored within. A zip which has been optimised with advzip, for example, will generate the same fingerprint as the original file. As will the the same contents after having been extracted and re-compressed into a new ZIP (As zip_sha256 ignores all modification times).