Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.
Part I 路 Apr 2, 2004
[ Sorry for the delay everyone, but to say that I’m busy right now is the understatement of the year. Please enjoy the new casefile! ~ Chief]
“Man I hate moving.” I mumbled one fine morning. I was moving into new digs, and hadn’t realized all the stuff that I’d never thrown away.
“Byte magazine, circa 1995. Hmmm. Probably don’t need this.” I mumbled as I threw the magazine accross the room into the trash.
Seconds later my cell phone was vibrating on my belt. Time to answer the Chief phone.
“Chief.” I said while balancing a pile of laundry on one shoulder, yet cleverly supporting the phone with my chin in my best Jay Leno impersonation yet.
“Hey, Chief? Detective Soandso from the highway patrol. How are things?” asked a familiar voice. Detective Soandso was the lead computer forensic investigator in a newly created computer crime lab established by the highway patrol.
“I’m moving, but unfortunately it’s not into bed. What’s new man?” I asked.
“I’m way out of my league on this new case. Busted a 15 year-old hacker after he broke into the Sheriff’s office systems via an unmonitored modem connection. From what we can tell, he grabbed a lot of information - but we’re not sure what exactly. Got a minute to pay us a visit down here in doughnut land?” he asked.
“Sure - filing my underwear in the dresser can wait until tomorrow I suppose.” I replied. A stop at Starbucks would be required of course - and a white chocolate mocha would take the sting out of my back.
Time to hit the road.The highway patrol’s HQ was just off a busy highway (coincidence - I think not!). I made the drive in record time, although the irony of getting a speeding ticket while on the way to visit the highway patrol was quite amusing.
After a quick security check-in and having my private parts checked for explosives, I was attached to a visitor’s badge and escorted to Detective Soandso’s desk.
“Chief! That was fast. Have a seat. We have much to discuss.” he said as he refilled his coffee mug.
“Ok, here’s the deal. The kid has a two gigabyte encrypted file of somesort sitting on the harddrive of his linux machine over there on the counter. We’ve imaged the drive and have a few forensic copies for you to play with. " Soandso said with a wrinkled brow.
“Lemme guess - kiddo isn’t giving up the passphrase, and you’re trying a brute force attack using the 5 machines stacked up over there in the corner, eh?” I asked.
“You’re good. Yes, in all honestly it’s been running for days - we’re not going to get anyway fast this way. We need some help.” Soandso said as he slammed back the rest of his coffee.
“Where’s the kid?” I asked as I removed my tired rearend from the safety of the folding chair in front of Soandso’s desk.
“He’s over there - room 2.” Soandso replied as he escorted me to the door.
Upon entering the room, I started laughing inside and I’m sure a grin was splattered all over my face.
The kid looked about twleve years old, pony tail, and ice-cold eyes. As my eyes focused on his, I glanced down at his black T-shirt that read “I r00ted your Mom.”
This was going to be a long afternoon.
Source: archived original
Part II 路 Apr 13, 2004
“Can you ask somebody to grab me a fresh coffee?” I asked Soandso as I sat down at the interview table. I glanced around the room to see another detective which I had not yet met, a gentleman that just had to be an attorney, and the kid.
I noticed some recording equipment on the table in front of the other detective. The kid and his attorney were playing the whisper game in each other’s ears.
I couldn’t take my eyes off the kid - and it appeared to make him a bit uncomfortable.
Soandso closed the door behind him, fresh coffee in hand.
“Thanks.” I said taking a few sips. Oh man, this must be a pre-ground columbian bean. Ewww.
The interview began with the detectives and the lawyer exchanging some terms and understandings, the recording equipment was switched on, and everyone in the room identified themselves.
The kid’s name was Nick, and his lapdog attorney was John Chaser. How appropriate.
I took a moment to read through some notes that Soandso had passed me on our way into the room, while at the same time trying to listen to the questioning going on in the room. Soandso was walking through the events leading up to Nick’s apprehension. Most of the information was in greater detail in the notes I was reading.
I was reading about a situation that I had seen all too many times, although this had an interesting twist to it.Nick was the stepson of one of the officers: Officer Lopez. This particular highway patrolman had been injured in the line of duty and was now working towards retirement behind a desk doing research and data entry work.
Nick had a fascination with computers, and was very much a loner in the real world. Most of his friends existed online in Internet Relay Chat(IRC) or instant messaging (IM) communities.
Soandso wrote that during a recent security review of the systems at the highway patrol, a consultant found a modem attached to a PC that was connected to the internal/secure LAN. The consultant asked Lopez what the modem was for, and Lopez replied that the modem was used as a backup fax line because they had reliability issues with the fax machine in the same room.
The consultant did a desktop application review of the Windows 2000 workstation, and found that a remote-access application was running in the background and was configured to answer the modem line.
When the consultant asked Lopez about the remote access application, Lopez replied that he believed that the application was on the PC when he received it and he never used it.
The consultant pulled the asset tag information for the PC, and found that the PC had previously been used in a dispatch area to dial into a computer system of a state agency. Apparently the machine was moved to Lopez’s area with the software load and hardware configuration intact.
In a written opinion attached to the notes, the consultant indicated that he could find no evidence to suggest that Lopez was not telling the truth on this issue.
I flipped back to the beginning of the investigative notes to find out what the trigger was in this case.
The consultant expressed concern that the modem was in auto-answer mode, and that log files indicated long connection times to the machine from a remote host that exceeded one hour.
“That’s a pretty big fax job.” I sarcastically thought to myself.
A few pages later, the consultant expressed dire concerns that access logs to the highway patrol’s CMS (case management system) showed access from Lopez’s machine that did not corrolate to Lopez’s work schedule. The user ID in use at the time was Lopez’s. A functional review by the consultant showed that the remote access application was configured to operate while the console screen was “locked out”. The consultant also noticed that Lopez had the only user ID on the system.
“No administrator account, eh?” I mumbled to myself.
The machine was removed from service immediately, and the phone records were analyzed to see what phone number was dialing in to the modem. The number was Lopez’s home fax number.
“This guys sends a lot of faxes to himself!” I almost said aloud as I read through the report. Soandso’s team began the forensic process on the machine shortly after the machine was removed from service. The team concluded that the consultant’s findings were valid - someone had been dialing into the machine from Lopez’s home fax number and accessing the CMS during the time period of 2300 to 0400 hours.
A search warrant was executed on Lopez’s home, and two computers were confiscated. One machine was Lopez’s personal machine running Windows 98SE, the other machine was Nick’s - a Redhat Linux machine.
Lopez’s machine was forensically analyzed and the team found nothing conclusive. Nick’s machine posed a greater challenge, because none of the team members had a lot of experience with Linux and the ext2 file system. The team borrowed a forensic examiner from another state agency who held an RHCE (Redhat Certified Engineer) certification and did a complete examination of the machine.
In the examiner’s notes, the examiner was quoted as finding “a large 1.8GB file: /home/haX0r/dump” that “… appeared to be a PGP-encrypted filesystem image.”
Soandso’s notes indicate that they had brought Nick in for questioning and placed Lopez on leave. Nick was pleading ignorance and had no idea what the 1.8GB file was, or how it got on his machine.
I flipped back to the examiner’s notes, and apparently all user accounts on Nick’s machine (including the haX0r account) had their ~/.bash_history file symbolically linked to /dev/nul.
Way to cover those tracks Nick.
I closed the folder that Soandso had given me when I heard Soandso wrap up his same line of questioning from before.
It was my turn now.
Source: archived original
Part III 路 Apr 16, 2004
I had to keep reminding myself during my questioning of Nick that I was there strictly as a specialist/consultant, and it was my job to give Soandso some guidance on how to verify if there was indeed evidence on the confiscated workstation.
The weak coffee I was sipping was going down like water. The craving for Starbucks was increasing. Somebody was going to have to make a coffee run soon.
“Nice to meet you Nick.” I said, staring a hole through him. “You like linux?”
“Yeah.” Nick replied back. “It’s pretty cool.”
“Not into Windows? Mac?” I asked, trying to draw his interest.
“Hell no. I like linux, it does everything I need. Better programs too. " Nick said staring at the table.
“You like encryption?” I asked.“Doesn’t everyone? I don’t want people looking at my stuff.” Nick replied back, tapping his hand on the underside of the table.
“People? You live with your stepdad, right? You worried about your stepdad looking at your stuff?” I inquired.
Nick looked up from the table.
“Maybe. Maybe I don’t want anyone looking at my stuff, ok?” Nick said, slightly irritated.
“That’s cool Nick - what kind of stuff are we talking about here?” I asked, folding my arms in front of me.
“Just stuff man. Whatever I’m working on.” Nick replied, returning my stare.
“Stuff that requires an encrypted PGP file that you’re mounting as a filesystem?” I asked, unsheathing my questioning sword.
“Hey, it’s my stuff. My business. What the hell do you all care?” Nick said nervously.
“Look Nick,” I began, “I understand the privacy thing. But you’re in possession of a machine that contained diald logs showing that someone on that machine dialed into a machine here at the highway patrol. That resources on that machine were manipulated in an unauthorized manner, and you have a 1.8G PGP-encrypted file on your machine that the nice folks here at the HP would like to examine. How about you tell them what the passphrase is?”
“Passphrase? I don’t remember. Look, there’s nothing but homework and some porn pictures on there, ok? I don’t want my stepdad looking through my porn, alright?” Nick said as put his head on the table.
“Your stepdad won’t see any of the files. The HP wants to make sure that none of it’s information is in that file. Just give them the passphrase.” I pleaded.
“Look man, my work is picture perfect. I taught myself everything I know. Whatever I keep around there is my business, and I don’t believe anything you’re saying anyway. I told you, I don’t remember the damn passphrase.” Nick yelled.
Time for a break.
Soandso and I stepped out of the room and back into his office.
“Well, now you see what we’re up against!” Soandso said. “All we need to finish this puzzle is the contents of that file. There has to be a way to get into it. What do you think Chief? I’m going to owe you a lot of Starbucks after this one.”
I was pouring myself another cup of wanna-be-coffee and thinking out loud.
“I’m not going to get anywhere talking to him either.” I said. “Give me some space in the lab. I want to look through his system myself.”
A few hours later I was in the HP’s forensics lab performing an autopsy on Nick’s ext2 filesystem. I wasn’t seeing anything too exciting. Starting with a cursory examination of the applications Nick used, I was hoping to get lucky.
Nick had his .xinitrc set to start the KDE desktop environment. Scanning through the applications links in his menu bar, I saw the typical collection of apps that I would expect a teenaged-wannabe hacker to own. But one application caught my eye - Quanta. Quanta is a WYSIWYG html editor, similar to frontpage (except it writes pretty clean HTML code!). I dug into his Quanta config files and noticed that he was working on two websites. One appeared to be a site dedicated to him and his collection of Star Wars action figures, the other was a strange hax0r-group site. The site contained only one page, a single image, and a paragraph of meaningless hax0r-speak bragging about the group’s skills and accomplishments.
To make things even more interesting, the HTML in the page referenced the single image as “images\yoface.jpg”, yet the image yoface.jpg did not exist on the hard drive. Maybe in the encrypted file “dump”? I located the actual URL of the site, and sure enough it was still online and broadcasting the hax0r message loud and clear. A view of the HTML source showed that it matched the HTML source on Nick’s machine perfectly.
I took a deep breath, and shook my head. Soandso’s team didn’t mention this in their case notes - not good.
I picked up my cell phone and dialed Soandso, who was working another case outside of the HP HQ.
“Soandso, this is Chief. " I said.
“Hey - find something?” he asked in a rather excitable voice.
“Get that Starbucks gift card ready.” I replied.
(Aside)
I am summarizing the hell out of this case due to the depth and detail involved. Some things to think about: Nick got sloppy in his interview. How? Why did the website draw my interest? Where am I going with this?
Grab a hot cup of java goodness and discuss. (/Aside)
Source: archived original
Part IV 路 Apr 30, 2004
It’s a good thing that I didn’t have my trusty clue banana holstered and ready when Soandso returned to the HP HQ later that day. I would have unleashed a banana-bashing that would make any monkey proud.
One of the officers that was fascinated with what I was doing (and looking over my shoulder throughout my examination of Nick’s machine) was kind enough to run down the freeway and grab me a venti cup of goodness from Starbucks. It went down smooth and satisfying.
The caffeine was ten seconds from permeating my stomach lining when Soandso popped his head into the forensic lab.
“Chief! What’s the scoop?” Soandso said as he pulled up some quality folding-chair space next to me.
“You like Star Wars?” I asked in between burps of coffee. “Sure, who doesn’t. What’s your point?” Soandso said with a wrinkled brow.
“Do you remember in the original ‘Star Wars’ movie how Princess Leia had hid a message in R2D2 to prevent the Empire from finding the Death Star plans?” I asked.
“Yeah, sure. Where are you going with this?” Soandso replied, his eyes a bit bewildered.
“As much as I like your lab guys, they need to apply for jobs as Stormtroopers. They missed some critical clues on this machine.” I said as I clicked away from my Autopsy screen and when to an rxvt window.
I explained to Soandso about the Quanta sites, and how they led me to find the actual web sites on the internet.
I had used wget to grab both websites to my linux laptop, and I was showing the sites to Soandso after configuring a local Apache installation.
“He has two sites, one dedicated to Star Wars memorabilia, the other to a kiddie hacker site. See anything strange here?” I asked as handed the mouse to Soandso.
“I’m not following you Chief. I don’t see anything in the HTML source that’s weird either.” Soandso said.
“Exactly, and I didn’t either until I ran some tools. Watch this.” I said as I typed a few commands in the rxvt window.
_ $ cd /tmp/starwarssite/images $ stegdetect .jpg r2d2.jpg : jsteg(**) chewie.jpg : negative
{List of images truncated} _
“There it is.” I said as I pointed to the stegdetect output. On a hunch I had taken my existing dictionary word list and added a list of common Star Wars words, names and phrases. I ran a few quick scripts to modify the phrases (inserting/removing spaces, underscores, etc) to increase the chances of finding a match.
_ $ stegbreak -tj r2d2.jpg Loaded 1 files… r2d2.jpg : jsteg(helpmeobiwan) Processed 1 files, found 1 embeddings. Time: 58 seconds: Cracks: 673201, 11607 c/s _
“What does that mean?” Soandso asked. “Is THAT the passphrase to the PGP file we have?
“We can try it, but I doubt it.” I replied.
“Why?” Soandso said in a puzzled tone.
“Because we just turned the key in the lock. Now let’s open the door…”
(Aside)
Discussion points:
- Nick was pretty slick. He knew steg pretty well. What’s “behind the door”?
- The HP was clueless on steg. Why?
- I got really lucky on my dictionary attack. Is it surprising that the steg passphrase was so easy to guess?
(/Aside)
Grab some coffee (put down that instant crap) and use your heads.
~ Chief
Source: archived original
Part V 路 May 6, 2004
“Well, so much for that idea.” Soandso mumbled as he tried to use the key extracted by stegdetect on the PGP file. Denied!
“I expected as much - this was just the passphrase to place another file within the image.” I replied, not paying attention to what Soandso was typing.
“What were you saying about door? Where are you going with this?” Soandso said as he leaned back over to get a view of my laptop screen.
“Take a look.” I said as I swung my laptop around and opened yet another rxvt window._ $ outguess -k “helpusobiwan” -r r2d2.jpg hax0r.txt Reading r2d2.jpg…. Extracting usable bits: XXXXX bits Steg retrieve: seed: XXXX, len: XXXX _
“What the…” Soandso said as he held his chin in his right hand. “You mean the SOB really did hide a text file INSIDE of that picture? How the…”
“Hold on.” I said as my fingers eagerly started a ‘vi’ session for ‘hax0r.txt’.
Up came ‘vi’ in no time flat (because it’s better than emacs - kidding!) and we were staring at a single phrase:
“d0ntb3l00k1ngAtMystuffa55h0l3s”
“Is THAT the passphrase Chief?” Soandso said. He was nearly trembling with excitement at this point.
“Let’s find out.” I said as I turned around in my chair to the forensic workstation that contained the forensic image of Nick’s linux machine.
_ $pgp –decrypt /home/haX0r/dump pgp: no valid PGP data found. pgp: decrypt_message failed: eof _
The passphrase wasn’t working. Well, NOTHING was working.
“Big surprise - because that’s what it was doing for me too.” Soandso said.
I blinked. Now he tells me.
Using the notes from the HP’s forensics team, I read them carefully this time:
_ The examiner was quoted as finding “a large 1.8GB file: /home/haX0r/dump” that “… appeared to be a PGP-encrypted filesystem image.” _
Emphasis on APPEARED. Grrrr. I need more coffee.
I was removing my arse from the chair and explaining to Soandso what was going on when I sat right back down.
“Hold the phone - he’s smart but lazy. I have a few other tools to try real quick.” I said as I let my fingers fly accross the keyboard.
I tried the typical tools and configuration options, but I saved the obvious one for last: bestcrypt. I had seen this used in several other cases, and it was pretty powerful for thwarting unauthorized access to files, considering the encryption algorithms that it supports.
_ $ su - Password:
#mkdir /root/haX0r
bctool mount dump /root/haX0r/
Enter password: [d0ntb3l00k1ngAtMystuffa55h0l3s] #ls -la /root/haX0r/ drwxrwxr-x 5 root root 4096 Dec 7 11:01 . drwx—— 63 root root 4096 Dec 7 11:34 .. drwx—— 1 root root 858 Dec 7 storez _
We’re in.
(Aside) Talk about pins and needles. The events in this entry seem to go by very fast, however this was one long afternoon.
Questions to consider:
- Why did the HP and their consultants automatically assume that the file was a pgp file?
- Where was my slip-up early in the case? It goes to show you that no matter how many cases you’ve worked on, _________________ (fill in the blank grin). Should I have been peering over shoulders?
- Why do you think Nick used bestcrypt? Security through obscure toolsets?
- Would you have handled anything differently?
Note: NONE of the tools that I used here were found on Nick’s / (root) filesystem. How was Nick doing what he was doing with no tools on the system? Time for a latte.
(/Aside)
Source: archived original
Part VI 路 May 13, 2004
I stared at the blinking cursor, and couldn’t help but be doubtful at what we were about to find.
_ #ls -la /root/haX0r/ drwxrwxr-x 5 root root 4096 Dec 7 11:01 . drwx—— 63 root root 4096 Dec 7 11:34 .. drwx—— 1 root root 858 Dec 7 storez _
I started a cursory examination of the directory:
_ #cd storez
ls -la
rwxrwxr-x 5 root root 4096 Dec 7 11:01 . drwx—— 63 root root 4096 Dec 7 11:34 .. drwx—— 63 root root 4096 Dec 7 11:49 …
_ “Wait a tick.” I said to myself. That directory listing appeared to be empty, but only to someone that had never seen a few hundred cracked ftp servers.
One of the most common tricks that warez kiddies use is to create a directory called “…”. It’s meant to be easily overlooked by the other listings in an empty directory - namely the current directory (” . “) and the parent directory (” .. “).
Nick had utilized this trick here. But why?
_ #cd …
ls -la
rwxrwxr-x 5 root root 4096 Dec 7 11:01 . drwx—— 63 root root 4096 Dec 7 11:34 .. -rwx—— 63 root root 13444096 Dec 7 11:59 master.mdb _
Paydirt.
I burned the master.mdb file onto a CDR disc, and examined it using one of the forensic workstations running Windows 2000. I opened the master.mdb file with Access.
“Holy Mary of Magnolia” Soandso whispered. “He grabbed the entire database. He had access to every arrest we’ve made in the past year!”
My watch showed that it was 30 minutes past coffee time, so I excused myself to the local Starbucks for a relaxing cup of java and some quiet time to enter my case notes into my laptop.
Ah, the glories of information security work. I was one tired monkey.
(Summary)
What have we learned from this casefile?
Nick was a young, brash, reckless haX0r that knew a few tricks, but he was lazy. He tried way too hard to hide something that he knew any law enforcement agency would want - their own data. He went to an awful lot of trouble to protect this data, but he made a series of mistakes. What were those mistakes?
It was later learned by the HP that Nick was intending on selling the information that he had gathered to anyone that wanted it. Nice business model there, eh? What would some hurdles be to Nick successfully selling this information?
Nick was convicted and placed on probation. Big surprise.
(/Summary)
New casefile next thursday - stay tuned.
~ Chief
Source: archived original