Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.

Part I · Sep 23, 2009

“I’d love to know what he’s staring at.” I thought to myself as I glanced up from the screen of my powerbook and met the gaze of a gruff looking man, complete with trucker cap and overalls. He quickly looked back down at his cafe menu and took another drag off of his cigarette.

I suppose that I probably stood out in this rather small cafe in the middle of nowhere, Arizona. I was dressed business casual with a warm and rather uncomfortable matching sports jacket. Ordinarily I’d never be caught dead in a desert climate with sheep clippings covering my body, but business protocol dictated that I dress in the most uncomfortable fashion.

The trucker glanced up from his menu again and held my stare this time for a rather uncomfortable amount of time. I was half tempted to sip my coffee and return to reviewing my notes on my laptop, but curiosity was getting the better of me. Who was this guy? Did I know him? Did he know me?

The ring of my cellphone stirred my trance-like state. I pawed randomly for it on the table in front of me, nearly knocking my coffee off of the table.

“Chief.” I mumbled into the phone while covering the receiver with my other paw.

“Hey sunshine, how’s it going?” Scrap asked from the comfort of my couch, many unfathomable miles away.

“I’ve been better. No call from Maria. I’m camped out at Cafe Deerhunter trying to, uh, fit in.” I replied.

“Good luck with that. Don’t sweat the call from Maria, I’m sure she’s up to her pits in attorney-foo right now. I do have some good news for you.” Scrap bragged.

“I’m not going to have to do my best ‘pig’ impersonation before the day is through?” I joked.

“No guarantees there, boss. I examined all four million lines of the proxy log data you sent back here to the lab, and…”

Scrap didn’t get to finish his sentence, because my phone was suddenly removed from my ear and dropped on the floor.

“I know you. I KNOW YOU.” came a deep, gravely voice from beside me. I instinctively slide to the opposite side of my seat and glanced up to see the same trucker cap-wearing individual glaring at me.

“Excuse me? Can I help you?” I politely responded as I eyed the condition of my beloved cellphone on the ground.

“You’re that lawyer from the city. You’re putting Randy away!”

I glanced around the room to see everyone else in the cafe enjoying a free episode of “Redneck Rampage: The Series” starring yours truly and trucker cap man.

“Hold on a second, buddy. I think you have me mixed up with someone else.” I said as I stood up and inched closer to my cellphone.

“Naw, I’m sure of it. You and the rest of them guys are down at the meeting hall. Randy told me all about it. He didn’t do it. If Randy said he didn’t, then he didn’t!” bellowed my new country nightmare as he eyed me bending down to pick up my still flipped-open cellphone.

The pieces of this chess game suddenly appeared on my board. He was mistaking me for my client’s attorney, George. I was the assigned investigator for the defense of Tom, a worker at the nearby smelter plant that had been accused of browsing pornography on his computer during his shift. George and his paralegal Maria were utilizing my investigative expertise to exonerate their client and place the blame where Tom knew it belonged: with his coworker, Randy.

Trucker cap man took another step towards me and was breathing rather loudly out of his mouth, a sure sign that I wasn’t going to get a chance to order any pie to accompany my coffee.

“Scrap…” I said into my phone. “I’m going to have to call you back.”

Source: archived original

Part II · Sep 23, 2009

“Hold on a second, you really do have me confused with someone else!” I said as I raised my hands in front of me, constructing a nice two foot gap between myself and the growing threat of physical violence against me and my phone.

“Yeah, well I saw you down at the meeting hall, and so did my buddy over there!” trucker cap man announced as he pointed to another unshaven man sitting at the cafe counter.

I slowly dipped my right hand into my sport jacket’s inner pocket and removed a business card. I placed it on the table in front of my new adversary.

“No, really. I’m not an attorney. I’m an investigator. I don’t want any trouble, and I’m sure you don’t either.” I said as I moved my right hand above my right hip and then safely under my sport jacket. I took a classic law enforcement self defense stance and raised my left hand higher to distract his focus. Next to running out the door faster than Jesse Owens, this was my only alternative.

“I think you should turn around, pay your tab to Becky over there and leave.”

My very risky broadway-quality performance had peaked the interest of the cafe onlookers enough to prod one of them into talking some sense into my newly found foe.

“Morris! Come on man, you don’t want to be messin’ with them types. He’s probably a cop or one them military spooks from down south. Let’s go.” came a not-so-melodic request from Morris’ previously mentioned friend at the counter.

Morris took a few seconds to size up the possible trouble he could be in, walked slowly back towards his friend, tossed a few dollar bills on the counter and left. I maintained my stance until I was sure the he had made his way down the drive towards the parking area, and then casually returned to my seat. If only I had brought a change of underwear.

I immediately re-dialed Scrap and filled him in on my taste of country kindness.

“Holy crap, Chief! Wait, you were packing heat? Even with a concealed carry permit, that’s just not your style.” Scrap exclaimed.

“Not to worry,” I said with a grin as I patted my belt just above my right hip, “but he almost got a taste of my cellphone holster.”

Scrap was apparently lost in laughter and coughing up a lung while I tried to inquire about his previous success with proxy log files.

“Oh, right, the files.” Scrap said. “I was going through your case notes that you sent over, and analyzed the files with your hunch in mind.”

Scrap’s voice grew quiet in my ear as my mind drifted back to the events of the past few days.

===

“Thanks for joining us Chief. It’s a pleasure to meet you.” said George through a mouth of newly-bleached white teeth. George was tall and slim, and his rather expensive suit hung to his frame like a freshly arranged set of curtains.

He introduced me to his paralegal, a rather exotic-looking Latin woman named Maria. We had exchanged no more than a few words before George was dragging me around his new legal office on the tenth floor of a beautiful building in Phoenix, Arizona.

“I hear you like coffee. You aren’t going to believe the coffee machine that I picked out. You can pick your bean, grind preference, everything from these controls!” George said excitedly as he brewed me a cup of coffee and invited Maria and I to join him at a comfortable break room table.

“Chief, George and I have been retained by the family of a gentleman named Tom that works at the smelter plant up north. Tom has been with Ore, Inc. nearly his entire adult life. He has a clean record both at the company and in the eyes of the law. He’s been accused of a rather nasty breach of company policy regarding usage of a computer located in the plant.” Maria said after a quick swig from a water bottle.

“Interesting.” I replied. “What kind of a breach of policy are we talking about?”

“Viewing pornography on a frequent basis.” George stated in a very official tone.

I took a sip of the caffeinated beverage in front of me and slowly placed the cup back on the table.

“This seems like a pretty big legal response to viewing naked women on the job?” I pondered out loud.

Maria and George exchanged a glance.

“He wasn’t looking at women.”

Source: archived original

Part III · Sep 29, 2009

George read the expression on my face well. He prompted Maria for a manilla folder that she had been carrying. George opened the folder carefully and slid several sheets of paper across the table. I immediately noticed that the pages were face down, and probably contained images due to the massive color ink bleed through of the paper.

“I should caution you that these prints are rather graphic and were pulled from websites that were supposedly accessed by our client.” Maria said in a voice just above a whisper.

I slammed the rest of my coffee down as if it were laced with bourbon and turned the pages over one by one like a large deck of playing cards.

At first glance I knew two things. I was cognizant of the fact that both my cup of coffee and breakfast that I had eaten earlier were considering coming back to visit me via their method of entry into my stomach. I was also fully aware that this case had implications that were far above looking at naked women on ones lunch hour.

The pages were screenshots of a website that catered to consumers of a cornucopia of activities that ranged from video taped (simulated?) male rapes, “water sports”, humiliation, bondage and… animals.

I placed the pages back on the table and exhaled loudly.

“Just when you think that you’ve seen it all.” I moaned as I opened my notebook and began jotting down notes.

George placed the pages back into the folder and took a moment to compose his thoughts.

“You can clearly see the ramifications of being found guilty of viewing these types of things in a workplace, and the double damnation that would come from living in such a small town.” George stated.

“I’m totally following you. I’m guessing that this surfing activity was recorded somehow, the employer viewed these sites, and here we are?” I asked.

“You are quite correct.” George replied. “Through the discovery process we were able to determine that the company logged all internet access through a proxy server that is located at their corporate office in California. There is no Internet point-of-presence at the plant. Ore provided us with the logs that were used in Tom’s termination hearing, and they are very much leaning towards solving this ‘issue’ through mediation rather than a court appearance.”

“Mediation? Interesting. Did you also request a certified copy of the hard drive of the workstation? Was chain of custody preserved? Do we have any additional technical details on how their network is put together? Who did the initial investigation on Ore’s side?” I asked without even looking up from my notebook.

My questions were met with silence. I looked up to find George and Maria raising their eyebrows at each other again, which I had now learned was never a good sign.

“Well, we’ve run into a stumbling block with our e-discovery requests.” George responded.

“Really?” I asked in a much too nervous voice. “Oh don’t tell me…”

“That’s why we called you.”

Source: archived original

Part IV · Oct 5, 2009

“So let me get this straight.” I said as I peered out of my window in the backseat of George’s car as we rounded a dusty dirt road towards the headquarters of Ore, Incorporated. “You sent over an e-discovery request for all items pertaining to this matter ‘including but not limited to the hard drive of the PC with chain of custody attached, certified Internet access logs, worker time sheets including the badge entry system…’”

“Yes, yes we did. We received a box of log files, and a printout from their time & attendance system.” George replied as he motioned for Maria, who was sitting quietly in the shotgun position, to hand him a bottle of water. “It’s been one delay after another. I swear, if we were going to court I’d have a judge so far up their…”

“But, quite magically, all of these items are available for pickup now?” I queried as I scratched my head with my paw.

“Like we said back in the office over coffee, Ore’s legal representative called this morning and asked that we pick up the items and verify that everything is there. Your participation in this is, well, important.” Maria said with a flirty smile.

“So it is.” I responded as our car approached the aged and copper dust-stained security gate for the plant. The guard was apparently on the lookout for three strangers in a BMW, or he was angry that we were arriving during his late lunch time, because he waved us through the gate with two quick motions of his left arm and then disappeared into the confines of the guard station.

George masterfully whisked the BMW around several sharp curves in the road and came to sudden halt in front of the plant’s main office.

“This is right out of the Flintstones.” I mumbled as I exited the car and took in the view. The office was aged and stained copper and green like the front gate. The sound of heavy trucks driving around us at a slow but steady pace was deafening. The heat, even for Arizona, was amazing. It was like opening the oven on Thanksgiving day to check on the turkey and having a wave of dry, hot, temperate air assault your face and not apologize.

“Chief, this way.” Maria yelled as she guided me up the wooden steps to the front door of the office.

I eagerly climbed the steps and thought about how great a shower would feel considering that I now had copper-colored dust in places that I didn’t even know I had places.

George held the door open for Maria, and she politely held the door for me. Within seconds of entering the office I could feel the cool breeze of an overworked air conditioner, and hear the unusually high voice of the receptionist exchange words with George and Maria.

“Someone is coming to get us.” Maria said in my direction, seconds before a new figure appeared in the doorway behind the receptionist.

“Welcome to Ore.” the figure said as he approached our group. “I’m Chris, the HR manager here at Ore. I was instructed by a… oh what was her name…”

“Somebody in legal?” Maria blurted out.

“Yes! From our legal department. I suppose her name is unnecessary at this point. If you’ll come this way I have the items that you requested. I’ll need you to sign for custody and receipt of these items.” Chris requested as he motioned us into a small room, furnished with a single metal table that supported a banker-style cardboard box.

“Here you go.” Chris said as he popped the top of the box and placed the contents in front of us on the table. “A tagged and bagged hard drive thing… here’s the chain of custody form… sign there please… here’s a ‘write-protected floppy disk’ that contains the badge entry logs, and a printed copy directly from the same system for your perusal…”

George meticulously signed off on each form as Chris read from the inventory sheet.

“Oh, and here’s the bundle from HR corporate. This has both Tom and Randy’s personnel files.” Chris announced as he placed a large sealed envelope on the table.

“I hope you don’t mind if I open this.” George said as he put down his pen and cracked open the envelope. Inside the envelope were two folders, each with an orange sticker on the tab next to the employees’ names.

“What gives with the orange stickers?” I asked out of morbid curiosity.

Chris pondered my question for a few seconds. “Orange? That means they both work in the plant on the control side.”

I pondered Chris’ response. “In other words, they are desk jockeys? They watch gauges and screens and stuff?”

“Bingo.” Chris said excitedly. “It’s shift work. I helped write the brief that our legal team sent you. Both employees have the same job code, they just work on different shift teams.”

I made several notes on my notepad, and was surprised that Chris was as talkative as he was. I had always found human resources managers to be less than friendly at times like these.

“Tom and Randy got along at work? No workplace issues?”

“It’s in the brief.” Chris said after a sudden realization about lips being loose and ships sinking.

“Chris, I really appreciate you meeting us and handing this over personally. Would you mind showing Chief the control room that you showed me and Maria on our initial visit here? He’s our assigned investigator on this case.” George asked Chris with a showing of his freshly bleached teeth.

Chris looked at his watch. “Well, it has to be quick. I have another appointment.”

Before words of thanks could escape my lips, Chris had left the room and was quickly escorting us to the far side of the office. After a few minutes of power walking, we arrived at a door with a badge reader. Chris slid his card on the side of the reader and with a loud “pop” the door opened into another hallway that ended with a more secure-looking door. With another swipe of his badge, Chris silently welcomed us to one of the control rooms of the plant.

“Can you give us a minute?” Chris asked the lone worker in the room. The worker was writing down readings onto a clipboard. Judging from the number of gauges and boxes on the form he was using, the worker had been standing there quite a long time. He said something under his breath, handed Chris the clipboard, shrugged his shoulders and then exited through the same door that we had just entered.

“Friendly fellow.” I said under my breath. The control room wasn’t much bigger than a large boardroom. Rows of mechanical gauges lined the wall where the worker had been standing. I took note of a single desk at the far end of the room. As I walked closer to the desk, I could see a single PC with a blank screen, and a coffee cup occupying the top of the desk.

George and Maria engaged Chris in smalltalk while I craned my neck to see underneath the desk. The PC had a single ethernet connection in the back that hosted a blue ethernet cable. I walked counter-clockwise around the desk and noticed that the ethernet cable terminated somewhere underneath the floor. I stared at the blank screen.

“Chris, can I ask you a question?” I asked from across the room. “Can you come here for a minute?”

Chris walked over to my location with George and Maria closely behind him.

“I’m assuming that this is the PC that both Tom and Randy used, right?” I asked.

“Correct. Chief, I need to be running to my next appointment. If you all don’t mind?” Chris asked as he motioned us to move towards the entry door.

I walked back around the desk towards the door and bumped the desk with my thigh.

“Chief, you okay back there?” George asked after turning towards me.

I peered over at the PC and was treated to a Windows desktop, complete with an assortment of icons.

“I’m just fine.”

Source: archived original

Part V · Nov 24, 2009

We began our walk back to the reception area and struggled to hold his brisk pace.

As we walked through the corridors, my attention was diverted by a picture hanging on the corridor wall in a sterile stainless steel frame. It was a picture of “start up day”, and it featured the first employees to participate in “day one” operations at the plant. I could clearly make out both Tom and Randy’s faces.

My mind wandered back to a conversation that George and I had in his office. According to his client, someone else had clearly viewed those websites. Tom was adamant that it must have been his longtime coworker Randy because Randy was into “that porn stuff”. George had noted that Tom made the same accusation at the time of his termination. Ore had followed up on this accusation, but was relying solely on the proxy server logs and staff interviews in their termination decision.

A painful grunt from Maria snapped me back to reality. She was clenching the handles of the banker box tightly as we made our way back to the reception area. She insisted on carrying the box even after numerous offers to assist.

“Can we sit down for a moment in the lobby? I’d like to make sure that nothing was overlooked in their evidence collection process.” I asked as we rounded the corner to the melodic voice of the receptionist.

Chris appeared to be in a hurry and a tad annoyed at my request. “Yes, of course.”

Maria sat the banker box down on a small formica-covered table near an uncomfortable row of chairs. I removed the top of the box and gazed inside again. I removed my notepad from my bag and flipped through a few pages.

“I really have to be going to my next appointment,” Chris stammered. “If you need anything, the receptionist…”

“What about the logs?” I queried.

“I beg your pardon?” Chris asked.

“The proxy server logs. I see that snippets were printed and included in the personnel file, and I remember that George indicated that those same snippets were included in the brief. We’ll need the entirety if you don’t mind.” I said without my eyes leaving my notepad.

Chris looked like a child that had been scolded for not finishing his chores. He reluctantly removed his cell phone from his pocket and excused himself to the corner of the room.

“Good call. I assumed those were on that floppy disk,” George whispered. “But I’m guessing that you know better.”

I flipped back through a few more pages in my notepad. “Your original e-discovery request clearly stated that you wanted digital copies of the proxy server logs for a 3-day period; one day before the incident, the day of the incident, and one day after. Considering the facts that multiple shifts are involved, your request was right on the money – and nearly impossible to fit on a floppy disk.”

Maria shifted nervously in her seat. “We totally missed that. We assumed that the printouts they sent us contained everything.”

“Judging from the number of pages that they sent you, I very much doubt that Maria.” I said quietly as Chris ended his phone call on the other side of the room.

The human resources manager returned with a smile on his face. “I must apologize. We’re very new at this. I’m going to retrieve a compact disc from our IT staff here that was sent by our corporate IT staff in California and bring it to you right now. They shipped us two CDs and we completely overlooked the fact that one was for you.”

“We’ll wait,” George responded, “we certainly don’t want to make you any later.”

Chris forced a smile and darted back through the entrance that we had just exited.

I decided to make use of our newly found free time with a bit of reading.

I scanned both personnel records and noted that both employees had a reasonably clean employment record, similar education and training backgrounds, and nearly identical salaries. They were hired within a week of each other when the plant opened years ago.

I then moved my attention to the time and attendance records, followed by the security badge system reports. I repeatedly flipped back and forth between the pages with an increased pace, and scrawled notes in my notebook as fast as I could write them.

George and Maria watched me intently and were hesitant to say anything.

Chris returned moments later with another manilla envelope.

“I’ll take that,” George said as took the envelope from Chris and peered inside. “One CD, and one letter of attestation. Perfect. Thank you.”

Chris nodded. “It was a pleasure meeting you all. Have a safe trip back to Phoenix.”

I raised a hand and motioned for Chris to come near.

“I think you might want to cancel that next appointment. According to these records, we’re missing a personnel folder.” I announced.

“What?” Chris asked as he moved into position to look at the documents I was holding.

“Both Tom and Randy had company in that control room.”

Source: archived original

Part VI · Sep 29, 2010

(Since I’ve been incredibly busy and haven’t updated this case file in forever, please go back to Part I and get caught up again!)

(Edit: I realize I haven’t finished this case in several years.  Unfortunately, life has gotten in the way and the real life “George” has asked that I not pursue the rest of this story.  I am still negotiating with him on this.  To all of my loyal readers, I greatly apologize and hope to make it up to you soon.)

Chris was pissed. Considering that he was a human resources representative of Ore, it was evident that he clearly lacked interpersonal skills. It took a few seconds for my statement to register with Chris. He stood in front of us, motionless, and staring into a part of the wall just above my head. He was… thinking.

“What do you mean they had company?” Chris asked nervously, still staring at the wall.

“Tom owns badge number 1025. Randy owns badge number 1041. During the time period in question, a third badge - badge number 1069 - entered the room about four hours into Tom’s shift, and finally badged out about 1 hour into Randy’s shift. Who owns badge number 1069?” I asked.

Chris finally lost the staring contest with the wall, and asked to see the badge reader logs. I handed the papers to him, and pointed out the entries in question.

“I don’t have an answer for you right now. I need to call corporate. Please have a seat and I’ll, uh, be right back.”

We retired to the semi-comfortable chairs in the reception area and watched Chris disappear through a door down the hall.

George and Maria were staring at me with eyes wider than a Japanese manga hero.

“What?” I pondered out loud.

“How… how the… how does a company just ‘miss’ something critical like that? This is the first time we’ve heard about a third badge!” George said excitedly. “This ought to be good!”

A mere five minutes had passed before Chris returned to the reception area, sweating profusely, and clearly troubled.

“Our legal team would like you to send your request for additional information directly to them. They’ll look at your request, and send anything that is appropriate. I’m sorry, but I can’t do anything more. I’m literally just a messenger and tour guide today.”

Almost by cue, the three of us stood up in front of Chris.

“Chris, we thank you for your time. Please let your legal team know we’ll be contacting them shortly.” George said as he shook Chris’ hand.

We exited the building the way we had came, and were blasted in the face with Arizona’s cruel hot wind.

“Back to the office?” I queried as I climbed into the backseat of George’s BMW.

“And air conditioning!” Maria added with a smile.

Inappropriate use cases like this one often revolve around a few different types of casework.

The first group revolves around doing zero-hour (“this just happened yesterday”) investigative work for an employer regarding “BDM” (boobs, dice, or money). After some log analysis, traffic captures, and desktop examinations I usually find that the accused is guilty and is terminated. Most of these cases are reasonably well developed by the employer.

The second group includes companies that are pretty darned sure that something stinky is going on, but they aren’t sure who the perpetrator is. These employers are usually the victim of poor network security design, and I usually have to revert to video surveillance to build a strong enough case for the employer. Defense lawyers are very skilled nowadays in the art of netsec-fu, and can usually tear apart cases based on logs and analysis software, and sometimes forensic work. It’s pretty hard to argue with a videotape of your employee of the month watching porn during his lunch hour.

The third group is what I call the “nightmare magnets”. These are employers that know just enough about network security to be dangerous, and don’t have the common sense to know that, well, they have no idea what they are doing. My gut instinct was telling me that Ore, Inc. fit my third group profile to the “T”.

My favorite thing about occupying workspace at a successful attorney’s office is that I seem to always have plenty of room to work. George had graciously given me an unused conference room equipped with a generously sized conference room table to spread out my work papers and set up my laptop.

After a few billable minutes of learning how to operate George’s completely awesome coffee machine, I was sitting at my conference room table building my “vertical view” of this case.

Vertical views are timelines placed next to each other (vertically, of course). There’s a timeline for the physical clock-in and clock-out of the employee. A second slidable timeline contains the time marks for the alleged behavior (surfing porn). A third (and now fourth) slidable timeline is developed for the other employees in the same confined physical space. This incredibly simple tool is fantastic for lining up events and doing old-school correlation. I still use this method to this day, much to the embarrassing gasps of coworkers.

The first, third and fourth timelines were quick to develop - the information was readily available from the badge system logs that Ore provided. The second timeline was going to be tricky, since Ore provided the raw proxy logs for the requested 72-hour period. Ore had a single proxy server for the entire western region. I was looking at ALL of their traffic, not just traffic for the Arizona plant. This was going to take me forever, unless I multitasked.

I grabbed my cell phone and called the best log processor I knew.

“Scrap here - what’s up boss?” came a familiar voice from my phone.

“I’m back from my Ore trip, and I’m going to need your help with some analysis.” I said.

I could hear Scrap taking a long drag off of a cigarette.

“I need you to look at some logs. You up for it?” I asked jokingly.

“No problem. How many logs are we talking?” Scrap asked.

“A little over four million lines. Scrap? You still there?”

** Now what are the odds that you’ll get case file updates tomorrow… and maybe the day after… speculate! ** Chief

Source: archived original