Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.
Part I 路 Sep 12, 2006
“We’re not dating, dude.” I said as I shuffled uncomfortably in my plane seat and stared out the tiny window to my right.
We were on a full flight back home, a mere two days after wrapping up our work at Buhlchet and closing The Case Of The Teetering Timeframe. Our flight was preparing to taxi out to the runway, and I was dodging questions from my unyielding apprentice.
Scrap punched me in the arm and laughed.
“Uh huh. That sure was a long hug goodbye she gave you at the airport.”
I shook my head and glared back at Scrap, letting him know that he was way off base.
“Scrap, Inez and I have known each other for a long time. We don’t work that often together. I said nothing to make her think that our relationship is anything more than a professional one.”
“Whatever you say, boss.” Scrap said as he popped his iPod earbuds into his ears.
“She is pretty damn cute though, and she was totally checking me out. Maybe I need to give her a call.”
“SCRAP!” I said as I whacked him in the face with a barf bag which was unfortunately empty.
Scrap enjoys torturing me on every level.
“Ladies and gentleman, this is the Captain. There seems to be a backup on the runway. We might be sitting here for a bit. Please remain seated with your seat belts on, and we thank you for your patience.” came the mysterious voice overhead.
“That’s just great.” I mumbled to myself. We were due back in town for a potential gig that my good friend Hugo the P.I. had set us up with. This was a ‘fat cow’ gig, meaning that the client had a fat wallet and believed in liberally applying cash to the right people to get problems solved.
“I guess I should call Hugo and check in. We don’t want to anger the fat cow!” I said to Scrap as I turned my trusty Treo back on. Scrap was jamming to what sounded like Iron Maiden and was completely oblivious to my words.
“Hugo? It’s Chief! We’re sitting on a runway and I’m going to guess that we’re going to arrive home a bit late. Can you move our introductory meeting with the fat cow back an hour or so?” I begged.
“I’ll do my best. After the referral speech I gave them about you, I’m pretty sure they’d wait a few days to get you on this case.” Hugo said with a laugh. “You want the details?”
“Well, it’s either get the details from you now, or watch Scrap play some air guitar next to me. Lay it on me.” I said.
Hugo cleared his throat and sucked some water out of a very noisy water bottle.
“Here’s the deal. This thing is hush-hush and highly sensitive, so remind Scrap not to talk about this with whatever girl he’s with this week.” Hugo said.
“Got it covered. Who’s the client?” I asked.
“A school.” Hugo replied.
“Gee, that narrows it down big guy. Maybe you could be a bit more specific.” I joked.
“Your Alma Mater.”
漏 2006 All Rights Reserved.
Source: archived original
Part II 路 Sep 14, 2006
“Ladies and gentlemen, this is the captain. Welcome to your destination, and thanks for flying Jungle Airlines.” came a crackled voice over the plane’s intercom system.
We had landed a mere minute ago and I was wiping an hour or so of sleep from my eyes. It was at that moment that something felt very wrong.
“Dude! DUDE!” I said as I pushed Scrap’s head away from my shoulder. “You drooled on me!”
“She said she wasn’t married! She said she wasn’t… huh?” Scrap blurted out as he woke up. “Why are you yelling at me?”
“Forget it. Grab your stuff, we need to make a beeline for the parking garage to get to our appointment on time.” I said as I contemplated telling the gentleman in front of me about a recent scientific hygiene discovery called deodorant.
“What’s the gig? Did Hugo call you?” Scrap asked.
“Affirmative. The gig is over at Jungle University in the medical sciences department.” I responded in between trying to hold my breath and not letting my eyes water from the body aroma of Mr. Stinky.
“Jungle U! Woot! I haven’t been back on campus in years. So what’s the case? Lots of illegally downloaded music? Booby-trapped servers?” Scrap said referencing The Case Of The Stealthy Spike.
“Hugo didn’t have all the details, but from what he told me this is going to be one technically challenging case.” I replied.
“Thank goodness. Yummy network monitoring? Freaky forensics?” Scrap asked melodically.
I shuffled my way to the exit door of the airplane and picked up the pace as I climbed the boarding ramp.
“It sounds like it could be a little bit of everything. The case is surrounding the termination of a prominent professor named John Dunnit. Dunnit was performing some research on a drug of some sort and had great successes for the university. The school was in negotiations with a pharmaceutical company on possibly licensing this experimental drug when Dunnit abruptly quit and disappeared.” I said as I noticed a Starbucks out of the corner of my eye as we passed through the airport’s food court.
“Uh, Chief? We don’t do missing persons man. That’s Hugo’s specialty. Where’s the technology come into this?” Scrap asked as we both rolled our travel luggage up to the ordering counter at Starbucks.
“Venti coffee, stat!” I exclaimed to the attractive young girl wearing the holy green apron.
“Two! TWO!” Scrap chimed in.
“Scrap, you are going to love this. Two days after they determine that Dunnit isn’t coming back, guess what they find out?” I queried.
“He didn’t wash his hands after using the lab’s bathroom?” Scrap replied sarcastically.
I sighed.
“Dunnit’s research materials disappeared with him. The school had recently implemented a storage area network, but hadn’t quite come up with a backup strategy. They can’t seem to find anything.” I explained.
“Woah. So we have a missing dude who’s known for not washing his hands and some missing lab work. Sweet. So I’m still missing where we come in?” Scrap asked with a loud slurp of his coffee.
“The school purchased a SAN solution called ‘Haystack’. After they noticed that the research was missing, they immediately locked it down and haven’t allowed anyone access to it. They believe that the data is recoverable.” I said.
“We’re going on another goose chase, aren’t we boss?” Scrap lamented.
“More like looking for a needle.”
漏 2006 All Rights Reserved.
Source: archived original
Part III 路 Sep 19, 2006
“Scrap, what the hell are you doing?” I asked as I spied Scrap on his hands and knees performing a strange worship ritual.
Scrap peered up at me from his prayer position. “Chief, that is the biggest storage array I’ve ever seen. Bow down and worship the bytes!”
I rolled my eyes. We were standing (well, I was standing) in a glass-encased hallway that wrapped around Jungle University’s datacenter in their medical sciences building. It was a glorious site. Rows and rows of perfectly racked storage array shelves blinked their LEDs in what could only be described as computing harmony.
“Like what you see?” came a voice from my left side. Dr. Rheinhold Kiltlifter, the department chairman, had escorted us from his office to gaze upon their technological trophy room.
“It’s amazing.” I said through the huge smile that had crept across my envious face. “I don’t really want to know what that much storage cost you, but I won’t mind if you tell me.”
Dr. Kiltlifter laughed. “Let’s just say that we could have almost built another building for what all the technology in that room cost us.”
“Holy crap! Are those penguins I see?” Scrap exclaimed as he walked towards the entry door.
“Yes. The servers that you’re looking at in that row are all Linux machines running Gentoo Linux.” the good doctor replied as he badged us into the datacenter. “We do a lot of simulation work, and these machines are typically busy all of the time.”
“Great gods of bogomips!” Scrap said as he approached the first rack of Linux goodness. “These are fricking awesome! Xeon processors?”
“All of them are Xeons. All are maxed out configurations and netboot from a master machine located over on the other end of this row.” Dr. Kiltlifter said as he escorted us around the room. “I wanted you to have an idea of what we’re dealing with in this case, and validate why we took a friend’s advice and brought you in to help us with our little ‘problem’.”
“We need to find more of these kind of problems.” Scrap whispered in my ear.
We exited the datacenter through a second door and walked down the other side of the building towards the research laboratories. Within a few minutes we were badged into what appeared to be an unused laboratory with medical analyzer equipment that I’d never seen before.
“This is, errr was, Dunnit’s lab. Towards the rear of the room is Dunnit’s workstation and file cabinets full of whatever paper records remained.” said Kiltlifter. “We locked this room down shortly after Dunnit walked out, so it’s pretty much in the condition that he left it.”
“Excellent. Now that we’ve had the formal tour of the facilities, I guess we need to get into additional details. Hugo mentioned that Dunnit was working on a drug that the university was looking to license to a pharmaceutical company.” I said after a quick glance at my notepad.
“Correct. I’m going to remind you that everything we’re talking about is extremely confidential and protected under the non-disclosure agreements you both signed.” Dr. Kiltlifter said after glancing at us both for nods of comprehension.
“Dr. Dunnit was a one man research machine, and very secretive. We put up with his introvert behavior because, quite simply, his results were sensational. The drug that he was working on was code-named ‘b00b7’, and it was designed to specifically target breast cancer cells. You can imagine what kind of market that type of drug would have.” Dr. Kiltlifter said as he picked up a file folder from Dunnit’s desk. “I won’t bore you with the medical mumbo-jumbo, but let’s just say that he was sitting on a gold mine for the university.”
“That is mind blowing.” I responded. “I know that Hugo is in the process of finding Dunnit using his unique methods, and he has shared a few bits about your suspicions. Can you elaborate on those?”
“Dunnit worked very late one night, which wasn’t unusual. He practically lived here. The next day he failed to show up at the lab. We figured that he was burnt out and taking a much-needed day off. When he failed to answer E-mails, phone calls, and show up for work a few days later we became worried. He lived by himself in a rented house not too far from here. I drove over to his house myself, and found the place completely empty. His landlord told me that Dunnit left no forwarding address. I immediately called a few other staff members and we locked down the lab just to be safe. I went through Dunnit’s personnel file and there were no contacts listed.” Kiltlifter lamented. “I have to admit that the first thing I thought of was that perhaps Dunnit had just finally lost it and left town to live in a shanty in somewhere. He was weird like that. The second thought that crossed my mind was that he grabbed all of his proprietary research and left the country to work with a foreign pharmaceutical company.”
“May I ask why that thought crossed your mind? I mean, working for a pharmaceutical company?” I asked as I turned to a fresh page in my note pad.
Dr. Kiltlifter reached into the folder that he had taken from Dunnit’s desk and removed a white envelope. “This is why.”
I carefully examined the envelope and noticed that there was no return address. Dr. Dunnit’s name and home address were handwritten on the front of the envelope. The envelope was stamped, and the post office cancel mark was from a city a few states away. I removed a single piece of folded white letter-sized paper from the envelope and opened it to reveal a piece of correspondence.
It was an offer of employment letter.
漏 2006 All Rights Reserved.
Source: archived original
Part IV 路 Sep 27, 2006
“There are a lot of things that are consistent in life, my friend.” I said as I adjusted myself into a patio chair at Jungle University’s Starbucks. Scrap and I had voted to retreat to neutral territory to strategize.
“Yeah, like what? Hot women on university campuses…” Scrap said as his head turned in a turret-like fashion to watch a few students walk by.
“Uh, no. I mean human behavior and the probability that our next case is always going to be more interesting and challenging than the last.” I retorted.
“Yeah, that’s great boss.” Scrap responded with the back of his head facing me.
“Scrap. SCRAP!” I yelled. “We need to discuss our plan of action. Pay attention, or we’re moving inside!”
“Right. Sorry. Okay, I took an inventory of the electronic assets that we’ll be interested in. I vote that we first do a standard forensic grab of the good doctor’s workstation. This would give us some clues as to where his most important data was stashed. Secondly, we’ll want to tap the sweet goodness of that SAN. The LUNs, Chief! THE LUNS!” Scrap said with a wide-eyed grin on his face.
“Good ideas. We’re also going to want to dig through those filing cabinets and eyeball the work area. Remember, it’s always the stuff in plain sight that gets overlooked.” I replied.
“But I really want to check out that SAN!” Scrap cried.
I rolled my eyes.
Moments later, we badged our way back into the medical sciences building and proceeded to Dunnit’s work area.
“Alright. Scrap, you grab Roscoe and do your magic on the workstation. It’s powered off, just the way you like it.” I said as I threw him a pair of latex gloves. “Watch the work area. I’m going to sift through these file cabinets.”
“Roger that.” Scrap said he snapped the gloves onto his paws with medical-like prowess.
RING
“That had better be Kiltlifter telling us that lunch is on him.” Scrap joked as he carefully snapped photographs of Dunnit’s work area.
“Chief.”
“Hay-o, amigo!” came the voice from the receiver. It was Hugo!
“Hey stranger. We’re at the lab getting busy with some forensics 101. How goes the hunt?” I asked.
“Not bad at all. I just had a nice long talk with Dunnit’s sister, and I’m headed out to Research Park in Big City.” Hugo replied.
I paused for a moment.
“Sister? Kiltlifter didn’t sound too positive on us locating anyone that knew Dunnit real well.” I thought out loud. “How the hell did you find his sister?”
Hugo chuckled.
“Dude, I’m a private investigator. This is what I DO. Dunnit apparently visited his sister a few days ago and didn’t give any impression that he had walked off his job, or that he was headed any place in particular.” Hugo said over his bad cell phone connection.
“Uh, so how do you know he’s headed towards Research Park? That’s far away from here, at least 500 miles or so.” I stated.
“Try 700 miles. Dunnit has been talking to Amalgamar Research Laboratories, and I have reason to believe that he’s going to meet someone there tomorrow.” Hugo said. “I have a great gut feeling on this.”
I paused again. I wasn’t sure if I was just confused, not paying attention, or in awe of Hugo’s investigative abilities.
“I’m missing something, man. Fill in the blanks.” I requested.
“Well, it was easy to track down his sister since he called her frequently. She was the most called number on his cell phone. Amalgamar was the second most called number.” Hugo stated.
“Cell phone? You have his cell phone bill? How the hell did you get his cell phone bill?” I asked.
There was a long pause on the phone.
“Wait.” I said. “You pre-texted his cell phone carrier?” I asked in horror. “Hugo, that is illegal as hell.”
“I can not confirm nor deny anything you just said.” Hugo said with a laugh. “Look, I have to go. My signal is dying. I’ll call you again shortly. Let me know if you glean anything exciting from the electronic stuff.”
“Will do.” I said as I hung up. I didn’t approve of the practice of pre-texting at all. Pre-texting involved calling a cell phone carrier and pretending to be the customer. After passing through rudimentary security screening questions, one was able to make changes to the cell phone service or, as Hugo had apparently done, gain access to billing records.
“Did I hear the words ‘pre-texting’ from your mouth?” Scrap asked, narrowly missing bumping his head on the underside of Dunnit’s desk.
“Yeah, something like that.” I replied as I carefully placed my Treo back into its carrying case located conveniently on my belt.
“OW!” Scrap cried from under Dunnit’s desk.
“Watch your head.” I said with a chuckle as I returned to sifting through the filing cabinets.
“It’s not my head man. I just got jabbed in the knee with this.” Scrap said holding up a tiny piece of plastic. I turned my gaze to Scrap’s hand. “Marker cap?”
Scrap tossed the object towards me, and I caught it with a gloved paw.
“Nope. USB drive.”
漏 2006 All Rights Reserved.
Source: archived original
Part V 路 Oct 2, 2006
Seven hundred miles away, Hugo was pushing his rental car to the limits. After some brilliant (but highly illegal) intelligence gathering, he was eager to position himself outside of Amalgamar headquarters in the hopes of catching a glimpse of Dunnit.
“What is going through your head, Dr. Dunnit?” Hugo pondered as he made his last highway transition with a smooth right turn.
_“Why did you just leave a research position and take all of your data with you?
… why haven’t you talked to anyone?
… why are you going to Amalgamar?
… what is your motive? Money? Fame? Better cafeteria food? What?” _
It was maddening. Hugo had worked the private investigator scene for over twenty years. He had seen everything from wives cheating on their husbands with circus midgets to wealthy retirees killing each other off to increase their retirement funds. However, in each of his previous assignments, the motive had been pretty clear.
Not this time.
Hugo glanced at his watch with tired eyes. With any luck, he had arrived in time to book a room at a nearby motel and catch some sleep before his Amalgamar reconnaissance mission tomorrow.
RING
“I really hope that’s not the wife again.” Hugo thought as he glanced down at his phone. It was Chief!
“Hey monkey boy.” Hugo said. “I’m in Big City and looking for a cheap pair of pillows and sheets to press for a few hours. You must have some news, because you never call me otherwise.”
I laughed on the other end of the phone. Hugo knew me all too well.
“You’re going to love this man.” I said with excitement. “Scrap was pawing around under Dunnit’s desk and found a USB pen drive with his knee.”
Hugo grimaced. “Did it survive?”
“Yes.” I replied.
“As a matter of fact, it’s completely readable. Scrap manually imaged it with his linux laptop and ‘dd’ cleverness. Dunnit must have been one paranoid guy, because he stored everything from his address book to zebra wallpapers on this thing. A few of the files are PGP-encrypted, but we’ve got Roscoe taking a shot at those with some dictionary pass phrase attacks.” I said, remembering that I had used a similar method in The Case Of The Hushed Haxor.
“No kidding.” Hugo said as most of the techno-drabble sailed clear over his head. “What’s the punch line?”
“Well, I was going to compliment you on your fine detective work. According to an excel spreadsheet that we just cleared of a password, Dunnit was definitely involved with Amalgamar at some point.” I said.
“How involved are we talking, and am I in the right place?” Hugo queried in a tired voice.
“I’d say that you’re right on target. The spreadsheet was a comma-seperated file dump of his checking account at NorthEastern Bank from a few weeks ago, and you’ll never guess who was dumping several thousands of dollars into Dunnit’s account?” I asked.
Hugo knew the answer before I had finished my statement.
Amalgamar.
漏 2006 All Rights Reserved.
Source: archived original
Part VI 路 Oct 10, 2006
“Done.” Scrap announced after glancing at the Encase prompt on Roscoe. “I’m pretty impressed with Roscoe. He’s banging away on those PGP files and still managed to acquire that drive image in an acceptable amount of time.”
“No kidding?” I asked as I wandered over to Roscoe and shoved Scrap out of the way. Sure enough, the image had completed and I was getting anxious to explore it.
“Excuse you!” Scrap whined as he gave up his comfy office chair.
I started a new case in FTK and added the freshly acquired image as evidence.
“Coffee. Now.” I said in a monotone voice as I pointed at the door.
Hundreds of miles away and several hours later, Hugo awoke from a poor excuse for a slumber. He was more exhausted than when he collapsed on the hotel bed.
“Time to get my game face on.” Hugo said as he rubbed the sleep from his eyes. A quick glance at the alarm clock with the missing LEDs teased him with it being six o’clock in the morning.
After a quick shower and a changed shirt, Hugo glanced at his watch with disgust.
“Cripes. It’s eight o’clock!” Hugo yelled as he made a quick dash for the motel door and cursed the broken alarm clock.
Hugo pushed his car to the limit in an attempt to navigate towards Amalgamar’s headquarters building located in the busy center of Big City. He had a surveillance spot all picked out in a parking lot across the street that faced the front door of the facility.
Traffic was heavy, yet Hugo maneuvered his car around several major delays and parked in the lot with a clear view of the front entrance to Amalgamar.
“Coming or going, I’m going to see you Mr. Dunnit.” Hugo said to himself as he took a few swigs out of a warm water bottle, his eyes firmly locked on his target.
“Man, this guy is boring as hell.” I thought to myself as I started my second day of evidence examination. The image of Dunnit’s workstation contained thousands of files, and any one of them could prove to be a significant digital artifact.
“Sorry I took so long.” Scrap said apologetically. “The line at the ‘bucks was wrapped around the food court. These kids want their caffeine.”
I grabbed a piping hot venti coffee from Scrap and took a moment to rest my eyes and reset my brain bus.
“Anything?” Scrap asked as sucked a layer of whipped topping from his white chocolate mocha.
“Well, it depends on what you call ‘anything’. This guy is a straight-shooter science geek all the way around. I’ve never seen so many academic papers, letters and research materials on one workstation. What the hell was he storing on the SAN?” I asked.
“I think it’s cool that they use linux on their servers and their workstations.” Scrap commented. “None of that winders crap.”
I laughed. Scrap was pretty biased about everything in life, the obvious exception being women.
Half of my Starbucks was down the hatch, and I returned to my examination. I pointed Scrap and his ‘magic knee’ to the filing cabinets behind us.
Somebody had to do it.
“I’ve officially heard this song three times now.” Hugo lamented as he changed the radio station again. He liked Elvis, but too much ‘blue suede shoes’ in the morning was making him cranky.
Hugo looked up from his radio for a second and glanced at the front door of Amalgamar. Two gentlemen were exiting the front doors with several other business suits. He blindly reached for his binoculars, knocking his water bottle off of his center console and making a miniature pool in his drink holders.
“Damn.” Hugo swore as he brought the binoculars to his eyes. The first gentleman was well dressed and smiled as he shook the other gentleman’s hand.
“Move. MOVE!” Hugo begged as he adjusted the magnification on his binoculars. GQ boy was blocking Hugo’s view of the second man.
Exercising the patience developed over years as a private investigator, Hugo waited several minutes for his window of opportunity to appear.
“Yes!” Hugo said quietly as the first gentlemen stepped away to re-enter the building. Hugo focused his gaze on the second gentleman’s profile.
It was Dunnit.
漏 2006 All Rights Reserved.
Source: archived original
Part VII 路 Oct 19, 2006
“Everybody say ‘HEY HO!’” I said accompanied by my own made up beat box sound effect.
Scrap laughed. “Hey ho, you’re such a white boy.”
Scrap always knew how to pee on my parade.
I sat back in my chair and pointed Scrap’s attention at my blinking cursor.
“Check out this little ditty.”
Scrap placed the pile of paper he was clutching in his paws on a corner of Dunnit’s desk and wandered over to gaze upon my discovery.
“Holy crap. HEY HO!” Scrap responded as he moved the cursor to scroll down the artifact represented on the screen.
** To: [email protected] From: [email protected] Su: Result set 16679
Yes, we are very pleased. This combined with the projects alpha and beta is truly some incredible work. We look forward to the next result set in project charlie, and I have arranged for appropriate compensation for alpha and beta. Please keep me abreast of the situation.
D. Richter **
“Hold on to your date book, Scrappy-san. Read the next one.” I said as I pointed to another artifact that I had already carved out of the image using FTK.
Scrap opened the artifact and dropped his jaw appropriately.
** From: [email protected] To: [email protected] Su: Result set 16694
You are not understanding or interpreting the results correctly. The entire charlie project needs to be scrapped and the work re-done. Affective rate is nowhere near the 62% as thought. Kiltlifter does not understand this project, nor can he speak about it in an authoritative manner. **
“Trouble in paradise?” Scrap asked as he scratched his head.
“It would appear so.” I replied as I continued to dig through the workstation image.
“Not again!” Hugo complained as his water bottle spilled into his lap as he removed the keys from the ignition without taking his eyes from his binoculars.
“Gotta move. Gotta move!”
Hugo exited his car and made a slow yet direct beeline towards Dunnit who appeared to be busy scrutinizing the cars driving by looking for an available taxi.
Dunnit had his back turned towards Hugo and fidgeted nervously with something in one of his pockets. As Hugo closed in on Dunnit, Dunnit removed the object from his pocket and placed it to his mouth. With a quick applied pressure of his right hand, Dunnit took a deep breath and returned the object to his pocket.
“James Dunnit?” Hugo asked as stood behind Dunnit.
Dunnit turned around quickly, his eyes as big as saucers.
“Um, you are?” Dunnit asked as he removed his inhaler from his pocket once again.
“A friend.” Hugo replied as he removed his identification from his pocket and allowed Dunnit to gaze upon his private investigator credentials. “You’re missed back at the university.”
“You don’t understand!” Dunnit said with a twitch of his shoulder. “You don’t understand!”
Dunnit backed away slowly. Hugo placed his wallet back into his pocket and raised his hands on either side of his body.
“James. Calm down. I was asked to find you. People are concerned about you just disappearing. Dr. Kiltlifter asked me to find you and provide you with a safe return to the university.”
Dunnit sat down on a nearby bus bench and shook his head violently.
“NO! NO! Kiltlifter won’t listen to me! He has no idea what he’s talking about! This has got to stop. Stop!!!”
Hugo was feeling a bit overwhelmed by his tweaking target. Was Dunnit on drugs? Crazy? Not hugged enough as a child?
“James, I can’t help you unless you talk to me. What doesn’t Kiltlifter understand?” Hugo pleaded.
Dunnit moved several pieces of folded notebook paper from his left pocket and unfolded them like a loving parent. He displayed them to Hugo using both of his shaking hands.
“This! This! You see this? This is years of my life right here. This is one of the drugs I’ve been working on, but you probably already know that if Kiltlifter hired you. I’m not going back. I’m ending this right here!” Dunnit said, blowing a bit of spittle out of his mouth with every word.
“Why? Why won’t you go back? What’s so bad about going back and talking to Kiltlifter? We’re just going to sit down and have a nice adult conversation. What’s so wrong with that?” Hugo said empathetically.
“Because Kiltlifter is intent on killing a lot of people!”
漏 2006 All Rights Reserved.
Source: archived original
Part VIII 路 Oct 31, 2006
“I have a big zero, boss.” Scrap said as he closed the bottom drawer of the last filing cabinet. “Nothing in here but magazines, reports, blah blah.”
I straightened up in my chair and tapped my pen on the desk in front of me. I had just completed my examination of Dunnit’s workstation.
“It’s time to move on to the SAN.” I said.
“Aroo?” Scrap said excitedly as he magically appeared next to me as if he were powered by some sort of hidden warp drive.
“I hate it when you do that.” I whined as I turned my attention back to the workstation.
Scrap scratched the back of his head. “So, where was Dunnit working on the SAN? What did you find in /etc/fstab?”
I tapped on the keyboard and brought up the /etc/fstab file from the workstation.
**[ Note to readers who aren’t too UNIX-savy - The ‘man’ page for fstab describes fstab as: “The file fstab contains descriptive information about the various file systems. fstab is only read by programs, and not written; it is the duty of the system administrator to properly create and maintain this file. Each filesystem is described on a separate line; fields on each line are separated by tabs or spaces.”
In other words, this file tells the computer which filesystems are mapped to what devices, the filesystem used, what options are to be used when mounting (like read-only), and how the system file checker should treat the filesystem.
Example:
/dev/md0 /home ext3 defaults,usrquota,grpquota,errors=remount-ro 0 1
In our example, /dev/md0 is a RAID device. /home is a directory (mount point) where we want the system to put the contents of /dev/md0. Ext3 is the journaled filesystem of choice for this example, and we want the system to mount the filesystem with several options like quota. The fifth field is used to tell the system if the ‘dump’ utility should be used (man ‘dump’ for more information on dump), and the sixth field tells the system which order the filesystems should be checked on boot.
Knowing this, we return to our story! ]**
Dunnit’s fstab file looked very normal for a linux workstation:
/dev/hda3 / ext3 defaults 1 1 /dev/hda1 /boot ext2 defaults 1 2 /dev/hda2 /home ext3 defaults 1 2 /dev/hda4 swap swap defaults 0 0 /dev/fd0 /media/floppy auto pamconsole,exec,noauto,managed 0 0
“This is interesting. None of the SAN volumes are mounted at boot time. That’s a bit unusual for someone that stores all their precious data on a SAN, right?” I queried.
“Hmmm. I agree, boss. So the fun part is going to be how do we locate what SAN resources homeboy was using?” Scrap responded.
“Well, if he was manually mounting and unmounting the stuff as needed, then it would be in a shell history file somewhere. I just realized that I haven’t looked at those just yet.” I muttered. I quickly scanned the /etc/passwd file to see what login shells were used by any active accounts.
“Looks like this is a bash house.” I joked as I scanned the file. Every account (including root) used /bin/bash as their login shell.
I immediately scanned the Encase image for .bash_history files and located two. One was located inside /root, root’s home directory - and the other was in /home/jdunnit.
A quick scan of /home/jdunnit/.bash_history showed nothing regarding the SAN. When I dumped the /root/.bash_history file, both Scrap and I giggled like school girls.
Filesystem mount commands.
Lots of them.
漏 2006 All Rights Reserved.
Source: archived original
Part IX 路 Nov 7, 2006
“Look at all those glorious mount commands.” Scrap said as his eyes scanned the .bash_history file. “I have a theory…”
Scrap pushed me aside and looked at the /var/log/lastlog entry.
“Hmmm.” he mumbled.
“We are on the same page.” I said with a smile. “You’re thinking that Dunnit shuts his workstation down every day when he’s done. After a fresh boot in the morning, he’s manually mounting the SAN volume… probably following directions written down on a napkin somewhere, right?”
“Bingo boss.” Scrap replied, touching an index finger to his lips. “You know, after working in a number of academic environments I’ve come to expect stuff like this. It’s obvious that the vendor came in here, set the equipment up, gave them a book and took off.”
I scratched the back of my monkey neck in agreement. “Not to mention that campus I.T. probably doesn’t know about this SAN, and I doubt they have access to it at all. So, Scrappy-San… what’s your theory behind why the SAN volumes aren’t mounting at boot?”
“Hmmm.” he mumbled again. “It could be a number of things. Security through obscurity? Problem with kernel modules identifying devices at boot? It could be a number of things. I wonder if Kiltlifter would know.”
I chuckled. “Scrap, Kiltlifter told me on the phone that he can barely login to his workstation. He swears up and down that he can see the ‘SAN’ on his workstation and Dunnit’s stuff is gone, hence our spectacular appearance here. Wanna hear my plan?”
Scrap’s eyes widened into bloodshot marbles. “Uhhh, let me think. YES.”
“Here’s the plan. I need you to wander over to Kiltlifter’s office and find out what he’s mounting on the SAN. I’m going to finish looking through the system logs and see if I can’t spot some kernel messages from when he actually mounted his filesystems. With this knowledge, we’ll need to make a command decision on imaging the appropriate SAN volume or perhaps mounting it read only from a forensically sterile linux boot CD. Sound good so far?” I queried.
“Spot on.” Scrap said as he wandered towards the door. “And no, I’m not getting you more coffee.”
Several minutes later I was perusing the /var/log/syslog file on Dunnit’s workstation. I noticed that the HBA was being found at boot:
** Sep 6 20:19:58 localhost kernel: QLogic Fibre Channel HBA DriverOct 6 22:19:58 localhost kernel: ACPI: PCI Interrupt 0000:02:0b.0[A] -> GSI 37 (level, low) -> IRQ 3 Sep 6 20:19:58 localhost kernel: qla2300 0000:02:0b.0: Found an ISP2312, irq 37, iobase 0xe0002000 Sep 6 20:19:58 localhost kernel: scsi0 : qla2xxx Sep 6 20:19:58 localhost kernel: qla2300 0000:02:0b.0: Configuring PCI space… Sep 6 20:19:58 localhost kernel: qla2300 0000:02:0b.0: Configure NVRAM\ parameters… Sep 6 20:19:58 localhost kernel: qla2300 0000:02:0b.0: Verifying loaded RISC code… Sep 6 20:19:58 localhost kernel: qla2300 0000:02:0b.0: Waiting for LIP to complete… **
That’s refreshing. Farther down in the log was a treasure trove of information on what volumes were found.
** kernel: Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0 […] kernel: Attached scsi disk sdo at scsi1, channel 0, id 1, lun 2 **
The kernel had found resources from /dev/sda (the first volume) to /dev/sdo (the last volume). That’s quite a few volumes to look through.
I scribbled this information down on my notepad.
ring
I glanced down at the screen of my Treo. It was Scrap!
“Chief, I’m here with Kiltlifter. It would appear that his workstation is mounting /dev/sdb as /mnt/san. It’s mounting it automatically at boot.” Scrap said with Kiltlifter mumbling something in the background.
“What is Kiltlifter saying in the background?” I asked.
There was a brief pause on the phone as Scrap attempted to relay what he was hearing.
“Kiltlifter was just rehashing what he told you over the phone about the contents of /mnt/san/dunnit being empty.”
I raised an eyebrow.
“Scrap, dump Kiltlifter’s /etc/fstab file and /var/log directory onto a thumb drive and get back over here.” I barked.
“Huh? Uh, okay. Any message for Kiltlifter?” Scrap asked.
“Not a word.”
漏 2006 All Rights Reserved.
Source: archived original
Part X 路 Nov 13, 2006
“So much for a sunny day.” Hugo thought as he felt raindrops fall onto his face from the darkening sky above.
Dunnit was perched on a nearby cement planter and staring directly into the ground with a pouty look on his face.
“James, just come back to the university with me and we’ll have a private meeting with Kiltlifter and voice your concerns. He’ll listen, I swear.” Hugo begged.
Dunnit continued to stare into the ground as he shifted his weight from one side of his butt to the other.
“I don’t want to.”
Hugo sighed. This was becoming more frustrating than necessary. For a few seconds, he actually considered climbing back into his car and leaving Dunnit to the rain and frequent pigeon drippings.
Hugo pulled a small cigar from his coat pocket and searched his pockets aimlessly for a match. With a quick flick of his wrist, the match exploded into a small fireball and created a glowing red cherry on the end of his cigar.
“He’s just going to keep trying to kill me.” Dunnit mumbled under his breath.
“What?” Hugo said amidst a cloud of cigar smoke. “What’s that supposed to mean?”
“WHAT DO YOU THINK IT MEANS?” Dunnit screamed.
Hugo was no stranger to dealing with aggravated persons of interest. He walked a few paces away from Dunnit and opened his cell phone.
“Chief? Hugo. I’m staring at Dunnit. He keeps mumbling about Kiltlifter killing people and refuses to go back with me. This guy is not stable.” Hugo whispered.
“Great.” I replied. “Look, Scrap and I are making progress on our end. We may be able to complete the assignment without his help. I suggest calling Kiltlifter and letting him make the call.”
Hugo paused for a moment and glanced back at Dunnit, who was erratically scratching his face with both hands.
“That dude got some bad aftershave, or he’s freaking nuts.” Hugo said.
“Huh?” I asked.
“Nothing. I’ll call Kiltlifter and play babysitter.” Hugo said as he hung up and dialed Kiltlifter.
No answer. It was time for a command decision.
“James.” Hugo said as he approached Dunnit. “How about I take you back to your sister’s house?”
Dunnit lifted his head slowly and fixed his eyes on the cigar in Hugo’s hand.
“Okay. But I need to eat first.” Dunnit responded as he stood up.
Progress at last!
Hugo and Dunnit sat down at a nearby cafe where Dunnit ordered three breakfast entrees and drank nearly an entire pot of coffee. During the meal, Hugo questioned Dunnit on a few items that I had mentioned to him on the phone.
“James, are you working on some private projects for that pharmaceutical company?” Hugo inquired.
Dunnit looked up from his eggs.
“Yes. I’ve done them by myself. You know, I used to have a staff at one time. They would have been a big help on the moonlighting work.” Dunnit stated as he chugged another mug of coffee.
“Used to? So you and your staff were working on university work and private projects? Isn’t that a conflict of interest?” Hugo asked.
“Nah. It’s all legal as long as I don’t work on them at the university lab. The projects were ‘alpha’ and ‘beta’, both dealing with chemical therapy. Can you imagine going through chemical therapy for a cancer and not having your hair fall out? Not being sick all the time? Nevermind. Anyway, after my staff left, nobody wanted to work on the private projects with me so I was stuck with all the work.” Dunnit replied.
Hugo raised an eyebrow.
“What happened to your staff? Budget cut?” Hugo queried.
Dunnit let loose a hearty belly laugh.
“Hell no. Kiltlifter fired them from the university and made it look like it was my idea!” Dunnit said angrily.
“I don’t understand. Why would he do that?” Hugo asked.
“Because they didn’t want to play ball with project ‘charlie’. ‘Alpha’ and ‘beta’ were improvements to existing treatments. ‘Charlie’ is revolutionary. It’s a ‘smart’ treatment that knows how to find the right cancer cells and immobilize them. It’s freaking brilliant.” Dunnit bragged.
“Put the pieces together for me, James. I still don’t understand.” Hugo begged.
Dunnit looked annoyed as if he were trying to explain burping to a grade schooler. “Good grief. Kiltlifter has an obligation to the university to make as much money from research as possible. He had no control over ‘alpha’ or ‘beta’ and it drove him nuts because he saw them as competing treatment products. It’s all about cash. Kiltlifter came to the realization that the only way to make ‘charlie’ more valuable was to make ‘alpha’ and ‘beta’ irrelevant. In doing so, he wanted me to use portions of the research done for ‘alpha’ and ‘beta’ for ‘charlie’. However, there’s a huge problem.”
Hugo waited patiently for Dunnit to continue. “And?”
Dunnit wiped his mouth with his napkin and leaned forward in his seat.
“The new version of ‘charlie’? It doesn’t work.”
漏 2006 All Rights Reserved.
Source: archived original
Conclusion 路 Dec 4, 2006
“You want me to what?” Hugo asked for the third time during our phone conversation.
“Bring him here. Scrap and I just wrapped the technical side of this case up. It’s going to be up to Dunnit to wrap up the human side.” I said.
“Chief, he doesn’t want to come back. Well, unless…” Hugo said, drifting off into a whisper.
“Unless what?” I demanded.
“If you can get Kiltlifter to apologize to Dunnit, he might just come back. I know for a fact though that he’s not going to magically appear at the university before then.” Hugo replied.
“Okay, keep an eye on him and tell his sister what has happened. Maybe she can talk some sense into him.” I hypothesized.
“And tell her I said ‘hi’!” Scrap barked from a location behind me.
I turned my attention to Kiltlifter’s workstation. It was identical to Dunnit’s, and sat proudly on Kiltlifter’s desk amongst a pile of neatly organized papers.
“Shall I power it up, boss?” Scrap asked as he sat down in Kiltlifter’s chair.
“Exactly as you did before when you two were looking for Dunnit’s files on the SAN.” I said.
Scrap pressed the power button on the workstation and the power supply fans spun to life. In a few moments, the monitor glowed with the familiar boot messages of Gentoo Linux and eventually a login prompt.
“Here goes nothing!” Scrap joked as he logged in with Kiltlifter’s credentials.
“Great. Look at /mnt/storage please.” I asked.
Scrap changed the current directory to /mnt/storage and performed a ’ls’ command to list any available files.
There were none.
“See. This is exactly what happened last time, man.” Scrap whined. “What exactly am I supposed to be seeing, Obi Wan Chimp-nobi?”
“Open up /var/log/lastlog. Grep for ‘/dev/s??’ and tell me what you see.” I instructed.
Scrap performed the command and was greeted with a list of every hard drive device in the SAN.
“Okay, still not seeing it man. Maybe you should use your ‘force powers’ and stir me up a latte and a blonde named ‘Bambi’?” Scrap cracked. “Seriously, what am I supposed to be seeing?”
“Nothing yet, my impatient marsupial. Pop in the thumb drive that I copied Dunnit’s /var/log directory to. Do a ‘diff’ of Kiltlifter’s /var/log/lastlog and Dunnit’s /var/log/lastlog. Now what do you see?” I asked.
Scrap’s fingers whipped around on the keyboard and in a matter of seconds he was looking at the output of the ‘diff’ command.
“Holy needle in a haystack. No way! No fricking way!” Scrap cried, nearly falling out of the chair with laughter.
“Now you see it! Although both machines are identical in every way aside from the /etc/fstab entries, for some strange reason that can only be attributed to the Linux kernel and/or modules, Kiltlifter’s machine assigns /dev/sda to a different, empty volume on the SAN!” I exclaimed.
Scrap was laughing so hard he was turning red.
“You mean to tell me that ‘X’ out of 10 times that Kiltlifter boots his workstation, his /mnt/storage partition is going to be pointed at the wrong volume? And you’re telling me that because of this, ol’ Kiltlifter assumed that Dunnit had wiped the SAN volume and fled town? Oh man, this is rich!!!” Scrap wailed.
I grinned. “That’s exactly it. It didn’t hit me until I paid attention to the boot up kernel messages. As joyful of an occasion as this may be, we only have half of the equation solved. Now Hugo needs to get Dunnit back here and they need to do some serious talking.”
“Yeah, yeah. So when do we get to tell Kiltlifter? I can’t wait to see the look on his face!” Scrap said.
“No time better than the present.” I said as I picked up my cell phone and dialed Kiltlifter’s cell phone.
Conclusion:
Scrap and I spent a good deal of time with Kiltlifter explaining the technical details behind his ‘snafu’ with the SAN. After the facts were laid out, Kiltlifter agreed to do what was necessary to make Dunnit feel comfortable enough to return to the university. Hugo played mediator between Kiltlifter and Dunnit and talked through most of Dunnit’s issues.
After consulting with an attorney, Dunnit parted ways with the university and is currently exploring private research on what would have been ‘project charlie’. Projects ‘alpha’ and ‘beta’ are currently distributed by a large pharmaceutical company in the United States and Europe and play an active role in battling several types of cancers.
This case is a perfect example of how lack of technical expertise and poor investigative work on the client’s part turned into something much larger than needed. Scrap and I recommended to Dunnit that they work with Haystack to resolve the volume mounting problems and explore an adequate backup solution. We also performed a brief risk analysis on the network and advised Kiltlifter that the network access controls were quite lacking.
One of my earliest mentors used to say “No matter what the subject matter, the persons involved, the crime committed, or the resolution found, you can never predict human nature. It’s the most random element in the universe, and it was conveniently left off of the periodic table.”
He can say that again.
漏 2006 All Rights Reserved.
Source: archived original
Redux 路 Dec 7, 2006
I’ve had some interesting e-mail responses on this case, and I was surprised that none of the linux wonks that read my blog figured out how Kiltlifter’s workstation was incorrectly mounting volumes on the workstation.
I peppered the case file entries with clues. I’ll throw a few specifics out, and let you guys discuss them. Perhaps I should have gone into more detail in the case file entries, but part of my goal in writing these is to make everyone think very hard and apply your unique experiences.
-
The workstations were running Gentoo Linux. How is Gentoo different than other Linux distributions? (Think of how Gentoo is initially installed and configured.)
-
The vendor installed the SAN and literally ’left’. There was no IT support. Kiltlifter and Dunnit had basic linux administration knowledge.
-
The Linux kernel (and modules) have a tendency to assign device IDs “as they are found”. Depending on how the hardware responds to certain requests, identification can vary. How could this have played a part?
-
This was (relatively speaking) a small research lab (high security area) with few researchers, few workstations, and few clues. Could this have played a part in how the network and SAN were installed and integrated?
A few of you have made some good points which got me thinking that this case requires a little more frosting on the cupcake. I certainly don’t want to leave anyone confused.
Discuss!
Chief
漏 2006 All Rights Reserved.
Source: archived original