Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.

Part I · Dec 3, 2004

“Is it even scientifically possible for coffee to freeze within a few minutes?” I asked myself as I stared into my coffee cup. What was once an interesting house blend had turned into an iced mocha in a matter of minutes. Then again, I was in Canada.

During the winter time.

Even my blood was freezing.I was perched inside a small café, enjoying free wireless connectivity and, apparently, an iced mocha. I was meeting a friend of mine, Nelson. Nelson was a fellow geek and had one of the coolest jobs to be had: he worked at a video game company. Nelson was a top-notch systems administrator and always had a plethora of interesting geekish side jobs going on at any time.

“Chief! Your teeth are chattering - welcome to Canada!” Nelson joked as he sat next to me at my table. “Nice Powerbook man - looks a lot like mine.” He said as he pulled his own Powerbook from his backpack. “I need some coffee, be right back.”

“My teeth aren’t chatter-er-ering!” I thought to myself as I browsed a few auctions I was watching on eBay.

Nelson returned with a steaming cup of joe (why was his coffee hot? Is there a trick to ordering coffee in this country?) and cracked open the top to his powerbook. “So, how you liking Canada?”

“It’s, uh, nice… and pretty close to the north pole from the temperature.” I quipped as I scanned the weather applet on my powerbook’s desktop. “Jeezus man, it’s 7 degrees outside!”

“Yeah, well it’s a warm winter this year. So how’s business? You working a gig up here?” Nelson asked with a loud slurp.

“Finishing one up, actually. It was mostly a due diligence job - you know, two companies merging that don’t trust each other. As the law of hilarity would dictate, neither one of them had adequate security controls in my opinion.”

“Hey, speaking of security controls - I just had a case of my own that I solved.” Nelson beamed.

“Oh, do tell Sherlock. I have to hear this.” I said as I made myself comfortable in my chair.

Nelson pushed back his chair and crossed his legs. This story MUST be good.

“Well, as you might remember I run a small web hosting company on the side from my normal day job as a SysAdmin, mostly so I can host my personal sites and retain some control over the machine. It doesn’t make money, but it can be fun. Most of my clients are friends or friends of friends.”

I stopped Nelson immediately. “Wait a tick, I don’t remember an offer to host any of my stuff!”

Nelson grinned. “Anyway, I get charged based on the amount of traffic I use, so I try to keep tabs on it. For individual sites, I use Webalizer and generate stats every night from cron. But, that doesn’t track non-http traffic, so I thought I needed something better.”

“Oh man, now I know where all that Canadian pr0n is hosted.” I said with a quick ducking motion.

“I won’t ask how you know about Canadian pr0n - but I digress. The first attempt was with a nice little app called IPFM (IP flow monitor). It just sits there listening on the network interface and logs the amount of traffic you use to a file. I had it configured to log hourly, then I used another application called bw-ipfm to make nicer looking reports based on it. They worked really nice, and I’ve been using them for years.”

I smiled and joked “Sounds like a good use of open source tools, my man. So when does this start getting good?”

Nelson raised both eyebrows at me, clearly forgetting how much I enjoyed giving him a hard time.

“I gradually started using more and more bandwidth. I could fairly easily figure out the web traffic, and I wrote a perl script that grep’d the mail logs and figured out how much bandwidth incoming mail was using, but there was a large gap. Since those are the two primary services I provide, I was kind of concerned. And I had no idea what I was about to find.”

I ordered another coffee - hopefully hot this time. Nelson was on a roll.

Source: archived original

Part II · Dec 7, 2004

My heart was doing a pitter-patter dance. Not only had the coffee I ordered come hot, but apparently it was “high octane”. Wow - I’m not going to sleep for days!

“Alright, so let me make sure I’m following you here. You have these hosting boxes, and by setting up some cool tools you’ve found that your bandwidth usage numbers are all out of whack.” I said in between sips.“You got it. So, I’m thinking at this point that I might have been compromised. There’s quite a bit of script kiddy and wanna-be hackers up around these parts - so I ran chkrootkit and thoroughly inspected the machine (even using trusted sources for binaries) to make sure that it wasn’t a hacker . Now I have to admit that I was kind of concerned, because it was a 4 year old install of Debian, but I was always pretty good at keeping it up to date. But, what if it was an exploit in the kernel - I might have missed it! As luck would have it, my box was fine and there were no signs that the box had been compromised. The only other thing I could think of was secure shell.” Nelson postulated, raising a hand into the air.

“So these hosting customers have shell access to your box as well? The world is their oyster.” I said with outstretched arms. “So I have to ask, how were you able to track the shell traffic?”

“Well, I looked into trying to log scp and sftp traffic, but wasn’t able to find anything immediately. Later I found that you can get patches for OpenSSH. But anyway, I checked the logs to see who it could be, and spoke to a couple of prime suspects, but they all denied it. " Nelson said.

“They always do.”

“Duh - but I figured it wasn’t someone being malicious, and they were probably just using it as a way to transfer files back and forth from home or something.” Nelson responded as he finished off his coffee.

“That’s a pretty good theory. After a clue banana or too, I’m betting that you did more research and found something… interesting?” I said, signaling the barkeep for another round of coffees.

“Finally I read an article in SysAdmin magazine by Randal Schwartz where he used OpenBSD’s pf - it has this ability to generate totals for traffic based on a rule using labels. He then inserted it into a sqllite database and generated fancy graphs with Perl.

I figured I would do the same, with a few twists. He was running it on the machine he was monitoring, I decided to use a transparent bridging firewall in front of the machine. I also decided to insert it into PostgreSQL instead. Oh, and I did it all in Python.”

Now I was really captivated. “I can understand PostgreSQL for the speed, but Python? Are you wielding the holy banana of coders too, now? Impressive!!”

“Hey, Python is cool. Finally I set it up and had it logging for a few weeks (I was busy), then I managed to take some time to go through the data I had collected and find out who it was. First, I did some queries to figure out which days had spikes in traffic. Then I took those days and figured out what type of traffic it was - secure shell! Then I narrowed down when the traffic spikes occurred.” Nelson was really into this story.

“So where did the rabbit hole lead?” I asked. “And how did you correlate the traffic to a user?”

“How did I know you were going to interrupt with that? Okay - after I had a one hour time window from the previous day that showed a large amount of traffic sent, I started going through the authlogs on the server itself. I was quite lucky in a way, because there was only one ssh login during that time, and it was exactly when I had seen the traffic in the database.” Nelson said.

“And did you know this person?” I asked on the edge of my seat.

“Did I ever.” Nelson replied.

Source: archived original

Part III · Dec 10, 2004

“Ok, ok - enough with the suspense. Out with it man, details!” I demanded.

“You know that we deal with large files at work. Well, this person apparently had a lot of large files too.”

“Well, couldn’t they have been iso images out of someone’s warez collection, pirated movie files or something? There are so many good Canadian films to pirate nowadays.” I said, poking Nelson’s Canadian button a few more times.

“The user was a customer of my hosting services - and a coworker.” Nelson said, his voice growing strangely softer.

“Holy sh…. You serious?” I said, eyes wide open.

“Honestly, I wasn’t surprised that this person would be tranfering large files. We have some pretty beefy machines in the office, so I figured they were working on a personal project at work and was just scp’ing it up to my machine and then scp’ing it down once they got home.” Nelson replied.

I pondered his words for a moment. “Ok, that sounds pretty legit. So where does this go wrong?”

“I started writing them an email telling them that I paid for the bandwidth they were using, and asked if they could pick up a USB drive or something to carry their files back and forth. Just on a fluke, I checked to see if they had already downloaded the files that they had uploaded.

That’s when everything changed. I realized that it was files from work - yes, actual intellectual property of work - that they were not allowed to remove from the facility. Now I had a dilemma, this person was a coworker, and I am sure they weren’t taking them home for a malicious reason. They probably just wanted to show them to friends or something, right? But alas, it was my job to try to make sure things didn’t leave the facility.” Nelson said.

“Ah, now that’s quite the quagmire to find one’s self in. You find out that a coworker is doing something naughty, and the esprit d’duty kicks in. Quite common nowadays from my experience. What did you do?” I said with a puzzled look on my face.

“I did something that I really didn’t want to do - something that I think about every day now.” Nelson said, staring a whole through me.

Nelson was in neck deep - what DID Nelson do?

(Aside)

Put yourself in Nelson’s shoes (in this case, big snow shoes).

What plan of action would you follow?

(/Aside)

Source: archived original

Part IV · Dec 17, 2004

I could tell during Nelson’s accounting of this story that it bothered him quite a bit. I was feeling rather privileged that he was wanting to share this story with me. I felt a little off balance though, because it’s usually me recanting my stories to others in hopes of educating the masses.

Nelson moved a bit in his chair, and with a satisfying gulp of joe he continued.“Chief, I figured I had two options. I could go to them and explain that I had found the files, and then deal with it myself, or I could go the management and they would reprimand them in some form or fashion. It is their responsibility to do so, right?

I wrestled with this for a while - it made me ill.

I thought about it for a few hours, and I decided to go to management, but proposed that I talk to them, let them know that they weren’t allowed to take the files, and have them promise to delete them. Then our management folks could send out a reminder to the other employees.” Nelson shared.

I gave Nelson a reassured look across the table.

“There’s nothing wrong with that decision man. If you managed that company, I bet you’d really appreciate a trusted employee coming to you with similar findings. So how did management react? In the States they send people to the electric chair for that now.” I said without missing a beat.

“Seriously? Isn’t that a bit rigid for intellectual property theft?” Nelson wondered.

“Obviously you’ve never been subpoenaed by the MPAA or the RIAA.” I fired back. “Ok, I’m actually kidding. Sorry to interrupt again - what happened next?”

“Great Gods of Gilder, they took it much more seriously than I expected. I had a conference call with the management team where I had to explain exactly how I figured it out, and how confident I was that it was in fact this employee. I imaged the drive of their machine using dd just in case.”

I smiled. “You’re going to be a forensic investigator yet Nelson. I’m guessing management was freaking out at this point?”

“Well, they scheduled a management meeting the next morning to decide what to do. I was pretty stunned at how they took it, and I couldn’t sleep that night. I should add that she was a friend of mine.”

Now that was a new bit of information. It was a female coworker!

“Crickies! A fellow employee, a female AND a friend of yours? Nelson, have I ever told you the story about Scrappy and…” I trailed off.

Ok, I wasn’t going to go there.

(Aside)

What will management’s decision be?

How will this affect the relationship between Nelson and his co-worker?

How would you approach this?

Have you had to approach this? Share!

(/Aside)

Source: archived original

Part V · Dec 24, 2004

Nelson was getting more upset as this case was unwound. I completely understood.

“Seriously man, were you getting raked over the coals at this meeting?” I asked.

“I had to go over everything again, and explain things and thankfully my management is fairly technical. They also had a lawyer present just to cover the bases. He asked me a million questions about my business and what kind of contract I had with my hosting clients. I realized that this might not stand up in court, so I was pretty nervous.” Nelson said as quickly downed the rest of his second coffee.“Well, I can’t speak for Judge Judy up here, but if you documented everything and maintained some sort of chain of custody of the evidence, you might have a chance in the States.” I said. “But I agree, this is very shaky Nelson.”

“I was feeling like total crap, Chief! After the meeting her fate was pretty much sealed. She was going to be terminated. Did I mention that she had a B1 visa, so that meant she would probably have to go back to her home country only a few months after her husband had finally moved to Canada!!!”

“Good grief.” I lamented. Now I was getting depressed.

“So a few hours later they called her into the conference room and had a meeting. While she was in there, I had to do my job and disable all of her accounts on the systems and remove her machine from the floor.

I never actually saw her after the meeting, which is probably a good thing. This was probably the most difficult thing that I have ever had to do in my career. I have worked 120 hour weeks, I have stayed up for three days trying to recover critical data from a file server, and I have dealt with hackers and virii. Let me tell you Chief, they are nothing compared to having to carry away a friend’s machine after she is fired based on information that I provided.” Nelson said, clearly drained from our conversation.

Nelson and I spent the rest of the afternoon discussing happier geek topics and putting a hurt on the café’s bandwidth bill.

Neither of us transferred any large files.

(Aside)

This was a bit different that my past case files, as it was relayed to me by a friend of mine. I hope that while reading this case file, the message was loud and clear.

Doing the right thing is almost always the right thing to do - but it’s never easy.

What would you have done in Nelson’s place?

(/Aside)

Source: archived original