Hashdeep

It is useful to calculate md5, sha, any kind of CRC / hash deep / recursively in a folder.

Create hash file

cd folder1
hashdeep -r -e -l . > hashdeep_$(date +%y%m%d).txt

To avoid following links:

hashdeep -r -e -o f -l . > hashdeep_$(date +%y%m%d).txt

• -r Recursive mode
• -l relative file paths
• -e Estimate time (optional)
• -o f Only files. Other options:
  • f - Regular files
  • b - Block Devices
  • c - Character Devices
  • p - Named Pipes
  • l - Symbolic Links
  • s - Sockets
  • d - Solaris Doors

To get a deterministic order, use -j0 which disables multithreading (see the man page)

Sort two hash files and compare them

It's faster to create again another hashfile.txt from the second folder and then compare the two hashfiles.txt

To compare the hashfiles.txt we need to sort them (unless they were created with the -j0 option), so for each of them:

Sort the hashfiles by the 4th field (filename)

sort -k 4 -t, hashdeep_$(date +%y%m%d).txt > hashdeep_$(date +%y%m%d).txt.sorted

List files only in the first folder

diff first.sorted second.sorted |grep "<" |cut -d, -f4 |less

List files only in the second folder

diff first.sorted second.sorted |grep ">" |cut -d, -f4 |less

Verify hash on the same or another folder

List new files or changed (not missing files)

cd folder2
hashdeep -r -l -X -k hashdeep_$(date +%y%m%d).txt .

• -r Recursive mode
• -l relative file paths
• -X Negative matching (only list differences): Only those files NOT in the list of known hashes are displayed. Display each failed hash that does not match the list of known hashes
• (-v Verbose mode, it does nothing?)
• -k Load list of known hashes
• hashdeep.txt File containing hashes
• dir Name of the directory in question

List how the files have been moved, new files and changed (slower)

cd folder2
hashdeep -r -l -avv -k hashdeep_$(date +%y%m%d).txt .

• -a audit mode: Each input file is compared against the set of knowns. An audit is said to pass if each input file is matched against exactly one file in set of knowns. Any collisions, new files, or missing files will make the audit fail
• -vv More verbose mode (display list of files missing, new files and changed files)
  • -vvv More verbose mode (also list files that are OK)

• "Known file not used": file that disappeared
• "No match": new file added
• "Moved from": file renamed/moved from folder

MAN: http://md5deep.sourceforge.net/hashdeep.html More info: https://linhost.info/2010/05/using-hashdeep-to-ensure-data-integrity/