Hashing and Hidden Characters

I was sitting in a training class the other day, learning about hashing. The instructor ran a demo on the screen of the below command.

C:\Users\student\Desktop\md5deep-4.4>echo koala | md5deep64
340fa0664ec760eadddf2216bb8bc1d5

One eagle-eyed student asked an inisghtful question, absolutely showing off his critical thinking skills (ok, ok, it was me, and I only asked it in my brain).

“In windows, does the space between koala and | matter?”.

Being a naturally lazy lad, I can’t be bothered downloading and installing md5deep64, so I crack open the amazing CyberChef. I type koala into the input box, and add the MD5 recipe, as shown below.

Success! Those hashes don’t match, so the space between koala and pipe must have been included in the hash input. Heck, we’ve already got cyberchef open, let’s do a sanity check while we’re here (the pipe looking character after koala is actually a flashing cursor, showing there’s a space there)

Hoo- wait, what? That’s a different hash again. Fine, damn our laziness, let’s go find md5deep and do our own testing. First, we replicate the condition we’re testing for:

C:\Users\student\Desktop\md5deep-4.4>echo koala | md5deep64
340fa0664ec760eadddf2216bb8bc1d5

Hooray! It matches. Aight, let’s get rid of the space, see if our hash changes

C:\Users\student\Desktop\md5deep-4.4>echo koala|md5deep64
fa453c56d2590383aac34119a707ad35

Starting to feel like a pokemon master collecting all these hashes. Fine, let’s see what’s actually getting passed to md5deep64 when we pipe it through. I know in the below command I’m also replacing pipe (|) with a redirect to disk (>), but we’ll test out to make sure that doesn’t break it later.

C:\Users\student\Desktop\md5deep-4.4>echo koala > myfile.txt

C:\Users\student\Desktop\md5deep-4.4>type myfile.txt
koala

So far so good. Let’s crack open myfile.txt in a hex editor (HxD for me) and see exactly what made it to disk:

Referencing an ASCII table, we can see the characters 6B 6F 61 6C 61 map to k o a l a, so we’re on the right track. 20 maps to space, so boom! we’ve answered our initial question (is the space included to be hashed). But what are these other crazy characters - 0D 0A. Looking at the ASCII table, they map to carriage return and line feed respectively. This is basically what computers record when you hit enter in a document.

History time! Think of an old school typewriter, first the “carriage” has to return to it’s starting position (when you type, it moves right along the page, so it’s returns to it’s leftmost position). Then we feed in a new line of paper (because otherwise we’d type over what we just typed). So, 0D, carriage return, 0A, line feed, new line.

We solved it! … or did we? We’ve still made some changes, and need to prove they didn’t introduce these changes separate from the hashing issue. Validation time!

This recipe is effectively taking the hex values we enter, converting them back into ASCII, then hashing the output. And we can see our original hash! Looks like echo in Windows adds a new line (a 0D 0A) after it’s finished printing to make it look prettier on the command line. Thanks Windows!