Getting Started with md5deep

This document provides an introduction to using md5deep and was last updated on The current version of this document can be found on the md5deep web site at http://md5deep.sourceforge.net/. sha1deep, sha256deep, tigerdeep, and whirlpooldeep work just like md5deep; all of the command line options are the same.

Installing md5deep

Microsoft Windows

Users running Microsoft Windows are strongly encouraged to download the precompiled binaries from http://md5deep.sourceforge.net/. Please note that these binaries are created using a Mingw cross compiler. Compiling the programs directly from Windows is not supported.

Automatic Installation

Before you try to install md5deep manually, see if your operating system supports the programs via an automatic installation method. Some operating systems that provide this feature for md5deep are:


Mac OS X: DarwinPorts, Fink
Linux: Ubuntu, Fedora, Gentoo, Slackware
BSD: FreeBSD

Manual installation

If your operating system does not support the automatic installation methods described above, you will have to download the source code and compile the programs yourself. First download the latest tarball of the program from http://md5deep.sourceforge.net/. This file should be named something like md5deep-3.7.tar.gz. Uncompress the file with the following command:

    $ tar zxvf md5deep-4.1.tar.gz 

Change into the decompressed directory

    $ cd md5deep-4.1 
and configure the program.
    $ ./configure 
The configure script can accept lots of options. Run ./configure --help for the complete list. The most common option used is the prefix option which installs the program in a location other than the default, /usr/local/bin. If you wanted to install the program elsewhere, for example, /tmp/md5deep, you would run ./configure --prefix=/tmp/md5deep instead.

You can now compile the program using the make command:

    $ make 
and install it:
    $ make install 
Note that you must be root on most operating systems to install the program to its default location, /usr/local/bin. The tool sudo may help:
    $ sudo make install 

Basic Operation

Opening a command prompt

md5deep is a command line program. You cannot run the program by double clicking on it! On Microsoft Windows, click on the Start button and choose "Run..." from the menu. In this dialog box, type cmd.exe and hit enter. A command prompt should appear. In this window, type the full path to md5deep.exe and then the files you want to hash. For example:

 c:\Documents and Settings\jessek\Desktop\md5deep.exe c:\Windows\*
Note that you can drag the md5deep icon into this window and the operating system will fill in the path information for you. You can also install the programs in a directory that is included in your PATH environment variable.

Computing hashes

By default, md5deep generates MD5 hashes for each input file, displaying the full path information for each file:
$ md5deep config.h INSTALL README
b08b18e0a3d2440feb0b321ea8080b36  /home/jessek/coreutils-5.2.0/config.h
9f3e20fdff9c78aa8e3f9b42be166769  /home/jessek/coreutils-5.2.0/INSTALL
4aa18483f73fc56c907f0ae9025e2582  /home/jessek/coreutils-5.2.0/README
If no input files are specified, standard input is hashed. You can either pipe the output of other programs into md5deep or type manually at the command line. To end input from the command line, most shells use Control-D, except for Microsoft Windows, which uses Control-Z.
$ uname -a | md5deep
f54c5e12c13791b67d299221424a5d80

$ md5deep
This is a test  
[Enter and Control-D hit]
ff22941336956098ae9a564289d1bf1b
You can also have md5deep print relative filenames instead of absolute ones. That is, omit all of the path information except that specified on the command line. To enable relative paths, use the -l flag. Repeating our first example with the -l flag:
$ md5deep -l ../*
b08b18e0a3d2440feb0b321ea8080b36  ../config.h
9f3e20fdff9c78aa8e3f9b42be166769  ../INSTALL
4aa18483f73fc56c907f0ae9025e2582  ../README
You can have md5deep only print out the basename of each file it processes. That is, all directory information will be stripped off. To enable basename mode, use the -b flag:
$ md5deep -b config.h INSTALL README
b08b18e0a3d2440feb0b321ea8080b36  config.h
9f3e20fdff9c78aa8e3f9b42be166769  INSTALL
4aa18483f73fc56c907f0ae9025e2582  README
Finally, if you need md5deep to insert an asterisk before the filename, similar to how md5sum does when you use the -b flag with that program, you can use the -k flag on md5deep.
$ md5deep -k config.h INSTALL README
b08b18e0a3d2440feb0b321ea8080b36 */home/jessek/coreutils-5.2.0/config.h
9f3e20fdff9c78aa8e3f9b42be166769 */home/jessek/coreutils-5.2.0/INSTALL
4aa18483f73fc56c907f0ae9025e2582 */home/jessek/coreutils-5.2.0/README

Error messages

If an input file can't be found, an error message is normally printed. These, and all other error messages, can be surpressed by using the -s flag.
$ md5deep doesnotexist.txt
md5deep: /home/jessek/doesnotexist.txt: No such file or directory

$ md5deep -s doesnotexist.txt
$

Recursive Mode

Normally, attempting to process a directory will generate an error message. Under recursive mode, md5deep will hash files in the current directory and file in subdirectories. Recursive mode is activated by using the -r flag.
$ md5deep *
md5deep: /home/jessek/archives: Is a directory
md5deep: /home/jessek/bin: Is a directory

$ md5deep -r *
ea024eaf04ee7a2a4270655d584445d2  /home/jessek/archives/coreutils-5.2.0.tar.bz2
a07715c3344524da1270e9eb39f9b9e1  /home/jessek/archives/md5deep-0.16.tar.gz
ef62d2a26c266df6151ae35447bfd2ad  /home/jessek/archives/dcfldd-1.0.tar.gz
fb3dab239dd0baa0f4a1ccb10c3a5b0a  /home/jessek/bin/hex2dec

Time Estimation Mode

When processing large files, it is sometimes helpful to have an estimate of the time remaining in the operation. md5deep can generate an estimate of how long it will take to finish processing the current file. The -e flag prints this estimate to standard error, like this:
$ md5deep -e /dev/hda1
hda1: 1MB of 47MB done, 00:00:46 left
When the file is completed, the last time estimate is removed and the hash is displayed:
$ md5deep -e /dev/hda1
ca1b8297dbceaa14682d889483320a1a  /dev/hda1

File Size Mode

md5deep can optionally display a ten digit representation of each file's size along with the hash. The file size comes before the hash on the line. This mode is activated with the -z flag, like this:
$ md5deep -z *.h
      1666  3a5353527e28b1cb5b844d602094c25a  /home/jessek/md5deep/src/algorithms.h
      1232  a27d2799c36c7c64e370bf480dd463c0  /home/jessek/md5deep/src/hashTable.h
      1880  d8defb61898fe255c6d66c4d880a8536  /home/jessek/md5deep/src/md5.h
Note that ten spaces are used, even if the file size doesn't require that much space. This is done to make sorting easier.

If the file size is larger than 9,999,999,999 bytes (about 9.3 GB), the program will display the size as 9999999999.


Matching mode

One of the more powerful features of md5deep is the ability to match the hashes of input files against a list of known hashes. You can do both postive matching, which displays those files that do match the list of known hashes, or negative matching, which displays those files that do not match the list of known hashes.

Positive Matching

Let's say that we have a text file known-hashes.txt which contains a few hashes:
65e7d67f9dbc831d4334f23c7fb9cfb1  foo.doc
406f25e49e9e08bb859b574a96746177  bar.gif
You can use hash files generated by md5deep, md5sum, md5 (found on *BSD systems), Hashkeeper, iLook, and the National Software Reference Library. We can specify to use this file for positive matching by using the -m flag. Then, any input files that match either of these hashes will be displayed.
$ md5deep -m known-hashes.txt *
/home/jessek/tmp/a-matching-file.doc
/home/jessek/tmp/some-other-file.gif
If you want to see the hashes along with the filenames, you can use the -M flag instead.
$ md5deep -M known-hashes.txt *
65e7d67f9dbc831d4334f23c7fb9cfb1 /home/jessek/tmp/a-matching-file.doc
406f25e49e9e08bb859b574a96746177 /home/jessek/tmp/some-other-file.gif
If you need to match against a single hash, or would like to add a single hash to the set of known hashes, you can use the -a flag. This flag enables matching mode and add a single hash.
$ md5deep -a 65e7d67f9dbc831d4334f23c7fb9cfb1 *
/home/jessek/tmp/a-matching-file.doc
If you would like to see filename of the known file that generated the match, use the -w flag. Continuing our example:
$ md5deep -wM known-hashes.txt *
65e7d67f9dbc831d4334f23c7fb9cfb1 /home/jessek/tmp/a-matching-file.doc matched foo.doc
406f25e49e9e08bb859b574a96746177 /home/jessek/tmp/some-other-file.gif matched bar.gif

Negative Matching

Negative matching is the same as positive matching, above, but displays those files that are not in the list of known hashes. Negative matching can be enabled using the -x flag, or the -X flag if you want to see the hashes along with the filenames. The -A flag will add a single hash to the set of hashes.


Advanced Matching Modes

Which File Matched

First, the -w mode can be used to see which file an input matched. For example, let's say we have a file of known hashes, known.txt that we're matching against. Normally we see a list of input files that matched any known file:
$ md5deep -m known.txt *
/home/jessek/an-input-file.txt
/home/jessek/another-matching-file.txt
When we add the -w mode to the command line, we see the filename in the list of known files that matched.
$ md5deep -wm known.txt *
/home/jessek/an-input-file.txt matched EVILEVIL.EXE
/home/jessek/another-matching-file.txt matched san-dimas-high-school.jpg
The -w flag only works with positive matching, or -m mode. Attempting to use -w mode with negative matching produces no more information that negative matching does normally:
$ md5deep -wx known.txt *
/home/jessek/unknown-file does NOT match

Unused Hashes

The second advanced matching mode, -n, indicates which known hashes were not matched by any of the input files. Using our example from above, let's say our file of known hashes, known.txt looked like this:

3668422d5b728776cb8720de8496abc9  san-dimas-high-school.jpg
b85444c8b82f208ad24e41f20c2b1831  EVILEVIL.EXE
6e43e96bde5c9f8abdbc47661cf4975b  never-seen-again.dat
If we run md5deep with the -n flag, we will get a listing of any hashes that were not matched to an input file.
$ md5deep -nm known.txt *
never-seen-again.dat


Expert Mode

md5deep's expert mode allows you to specify which and only which types of files are processed. The available file types are: To use expert mode, use the -o flag followed by the letter or letters corresponding to the types of files you want to process.

File type Letter
Regular f
Block b
Character c
Named Pipe p
Symbolic Link l
Socket s
Solaris Door d

Let's say that in the current directory there are files hda (a block device), my-link, a symbolic link to a block device, and data.txt, a regular file.
$ md5deep -o f *
2e1582d37db445402faee0fc331e9032  /home/jessek/data.txt
$
Note that only the regular files are hashed. Conversely:
$ md5deep -o lb *
ee1615f01210aa22c27058db161d2160  /home/jessek/hda
7fa749a2ea2cc635c77eb0a5370b93b0  /home/jessek/my-link
$
Note that the recursive mode can be used in conjunction with the expert mode. Directories are ignored without the recursive flag.
SourceForge.net Logo