Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.

Part I · Apr 27, 2005

“And then she walks over to me and says ‘Let me guess Scrap, you forgot your wallet AGAIN!” Scrap belted into the other end of our cell phone conversation.

“Let me get this straight man, you’re seeing this girl for two months now, you take her to a nice restaurant, you have a nice meal, and then when the bill comes you forgot your wallet?” I asked in disbelief.“Yep, and then she reaches into her purse and pulls out my wallet and says ‘Not to worry babe, I remembered it for you’. She then pays the bill with my cash and tips the waiter 30%!” Scrap whined in between drags off of a cigarette.

“Exactly how many times did you think the ‘I forgot my wallet’ thing would work for you? I mean, REALLY?” I said as I let out a loud laugh. “And the funny part is that she figured out your scam and turned it around on you. Brilliant! I need to hire this girl.”

“Not funny Chief. Why should I have to pay every time?” Scrap asked.

“This is all very well and interesting man, but I’m driving down the freeway just waiting to hear about this job your guy at the school has for me?” I replied while trying to sound as annoyed as possible.

“Oh yeah, the job. Ok well you know Thom Fong, the IT dude at the school, right? He’s the guy you met when I gave you a tour of their new datacenter on campus? Well he has a curious problem with his internal servers - the ones that are only accessible from within the campus network.” Scrap said as he blew smoke into the mouthpiece of his phone.

“I’m listening.”

“Ok, ok… so somehow a person is getting r00t on all of his boxes and DoS’ing the boxes. They are completely unusable after this person does whatever they are doing.” Scrap said. “And I, of course, referred him to you.”

“Sounds good. I can be there tomorrow morning after my morning appointment with my doctor. Set it up and be there if you can when I show up.” I said, making an illegal U-turn upon spying a Starbucks on the opposite corner of the road.

“No probs man. Manana is my day off. And there’s a curve ball to this one.” Scrap announced.

“Isn’t there always? What kind of curve ball?” I asked.

“He thinks it’s an inside job of some sort.” Scrap replied.

Here we go again.

I better make my coffee a Venti.

Source: archived original

Part II · May 3, 2005

“You’re just going to have to cut back on the coffee, Chief. No buts about it.” Dr. Berry said as he read my test results. “Your caffeine addiction is playing havoc with your blood pressure and heart rate.”

“And I thought that was just my job.” I joked. The doc wasn’t laughing.

“Ok, ok. Decaf for me. How… criminal.” I said as unstuck myself from the paper on the examination table.

It was going to be a long day.I made my way down the freeway to the school, and fumbled with my cell phone to locate Scrappy.

“Yo Chief! What it is?” Scrap said as he answered his phone amongst a background serenade of noise.

“Where are you man?” I asked.

“I’m at the school. Some kind of pep rally or something going on out here. I’ll meet you in the faculty offices. Thom is in building C1 - office twelve.” Scrap said as he abruptly hung up on me.

“He must have spotted the next Mrs. Scrap.” I joked as the Starbucks cravings began.

This was going to be harder than I thought.

My footsteps echoed through the halls of building C1 as I paced about looking for Thom’s office. It wasn’t that hard to find as it was right next to the datacenter. How convenient!

“Hey Chief! I’m Thom! Thanks for making the drive.” Thom said as he ushered me into his office and located a chair for me to plop in.

“Yo! I brought the coffee man.” Scrap said, apparently 5 paces behind me.

“None for me thanks.” I said as passed on the java.

“EXCUSE ME?” Scrap said, his eyes as big as recordable CD discs.

“Doctor’s orders man. Thom - tell me about what you have going on here.” I asked as I changed the subject.

“Well, I prepared some network diagrams for you and a brief description of how the network is put together as reference materials.” Thom said.

“Groovy - I love diagrams. So what’s happening to your nameservers?” I asked, staring at Scrap’s cup of coffee.

“It is very strange indeed. At completely random times, our nameservers take on very heavy loads - so heavy that they stop responding to any traffic at all. We have to take them offline. We look around on them and it would appear that someone is getting in doing something nasty. We just can’t tell the how or what. " Thom said as he sat on his desk, narrowly missing a coffee mug full of pens.

“I’d like to examine one of these machines. Have you kept any machines intact after removing them from the network? Scrap tells me you are reformatting, reinstalling, reconfiguring and putting them back into service?” I queried.

“Hmmm. I think we have one machine that has not been put back into service. The attacks have slowed to around once every few weeks. My students don’t have the time or the expertise to investigate this kind of nonsense. Hence, the reason I asked you to look at this. I realize that this is probably a bit different and boring to some of the things that you have worked on - at least according to Scrap.” Thom replied as he slammed the rest of his coffee. I winced.

“I’ll have a look. Is the server still powered or did you shut it down?” I asked as we walked towards the datacenter.

“This machine is still powered on if my memory serves correctly. We just unplugged the network cable because we were in a hurry.” Thom said as he pointed to the server in a rack. It was a 1U Dell server, and it was missing it’s faceplate. A monitor and keyboard were plugged into the back of the server.

“Apparently you weren’t in too big of a hurry - looks like someone made an emergency console connection into this puppy.” I joked as I sat at the keyboard.

“That was me Chief. I knew you’d want to look at this baby.” Scrap said with a big toothy grin.

I just shook my head. What else had Scrap done? Perhaps it was time to send Scrap out for some cans of 7up.

“I really want to image this thing before we go poking around. However, I would like to run a few tools on the box before we image it.” I said.

“Scrap - get plugged in to that isolated switch there - let’s plug this baby in and see what responds.” I said as I pointed Scrap towards a standalone cisco switch.

Coffee or no coffee, this party was starting.

Source: archived original

Part III · May 9, 2005

I was quivering. The erratic motions weren’t from the low temperature in the school’s datacenter, but from a lack of artificial stimulants: I was going through Starbucks withdrawals.

tap tap tap I quickly looked up the words “withdrawal symptoms” on my PDA.

The definition wasn’t surprising:

withdrawal symptoms the unpleasant physical reaction that accompanies the process of ceasing to take an addictive drug.

Well, tell me something that I DON’T know there Webster.

Scrap was fumbling around with a Cisco 2924XL switch that was strategically placed on a table not too far from the target server. The switch was completely isolated - no network connections present. Scrap ran an ethernet connection from his laptop to the switch, and connected the ethernet cable from the server to the switch in a quick 30-second form of interpretive chimp dance. If only we had a crossover cable…

“Thom, can you verify the IP address and network settings for this machine? I’d like Scrap to configure his laptop with similar information so we can get some quality scans done.” I asked Thom as I sat down for a moment. The room was starting to spin a bit. I felt… sleepy.

“Sure, I have all that stuff written down. You look like you could use a break. Why don’t you go down to the lounge and get something to drink?” Thom suggested as he motioned for Scrap to grab the papers that he was being handed.

“Thanks, that’s a great idea. Scrap - when you’re locked and loaded, start a port scan. Don’t run any of the usual nessus plugins. I just want to look for listeners.” I said as I merged into the hallway.

Several minutes later I was seated in the faculty lounge admiring a fresh pot of what appeared to be Folger’s Classic. That smell…. it was so…. hypnotizing.

After determining that an 8 ounce cup of life wouldn’t hurt me, I quickly poured a mug full and applied liberal amounts of sugar and cream. I had to be 100% to work, and I wasn’t going to survive going cold turkey no matter what Dr. Berry thought.

I rested for a few minutes and let the magic java work it’s way into my digestive track, and then my circulatory system, and then the nervous system. My cylinders were firing, and my eyes brightened.

The Chief was back.

ring ring

I reached for my cellphone. Scrap must be done.

“Status?” I asked.

“I’m stumped.” Scrap replied, busily tapping on keys.

“And?” I asked more impatiently.

“I can ping all day long. But there’s not a single port responding on this box.” Scrap replied.

“Hmmm. Not unlike the symptoms described by Thom. " I thought.

“Wait.” Scrap cut in to my internal monologue. “What the…”

Perhaps this was going to be interesting after all…

Source: archived original

Part IV · May 14, 2005

“What! What?” I yelled as I walked back into the datacenter. Thom was anxiously peering over Scrap’s shoulder.

“I swear man, I just saw something weird. I increased the frequency of my port scan and reduced the number of ports to common services… and every few seconds I get a blip of… THERE! THERE!” Scrap said as he froze his console window. I squinted at the screen and moved in closer for a better look. I had to keep in mind that sometimes Scrap sees aliens in his bedroom and periodically accuses his socks of plotting against him.

“Ok, I see the results you’re talking about. And these only come up every once in a while?” I asked as I scratched my forehead.

_ Discovered open port 22/tcp on x.x.x.13 Discovered open port 25/tcp on x.x.x.13 Discovered open port 21/tcp on x.x.x.13 Discovered open port 53/tcp on x.x.x.13 _

This was a new one. Services that reveal themselves randomly. Now I had seen everything.

“Whatcha make of this, Chief?” Scrap said as he attempted to connect to a few of the services.

“I’m thinking that…” I trailed off as I watched Scrap in horror.

“Scrap, what the hell are you doing?” I demanded as he was logging in to the server via the console keyboard.

“Watch!” Scrap said as he typed “top” and hit enter.

The server worked for a bit, then returned to the command prompt.

“Huh?” Scrap said as he started to type the command again.

I quickly secured the keyboard and proceeded to scold Scrap like a 6-year old stealing ice cream.

“Dude, we do NOT work on live filesystems - and we do NOT run executables on those filesystems! You may have just screwed us big time.” I said as I proceeded to break my own rule and type “cd / && ls -la” on the keyboard.

I had a bad feeling about this.

_ dns13 #cd / && ls -la total 0 dns13# _

“Hey, where’s the filesystem?” Scrap said, almost knocking me out of the way to share my stare of disbelief at the prompt.

“Chief, what’s going on?” Scrap asked.

“I’m wondering if we’re ever going to know now - but I’ll bet a venti house coffee with cream and sugar that your top command was trojaned to erase the filesystem.

My doctor picked a HELL of a week to ask me to kick coffee.

Source: archived original

Part V · May 17, 2005

“Dammit Scrap.” I said as I sat down in a chair parked close to the desk. “You can’t do stuff like that when a live investigation is in progress. I’m glad you hooked me up with this gig, but you have to let me steer this spooky ship to port. Comprende?” I asked, gazing at the top of Scrap’s head. He was staring a hole into the floor. “Sorry man. Seriously. I had a gut feeling and just went with it.” he lamented.

“What now?” Thom said, removing his hands from his pockets. “We don’t have any other servers for you to look at that were exhibiting the same behavior.”

I collected my thoughts and rubbed my eyes.

“Strategy shift.” I blurted out.

“Eh?” Scrap whimpered.

“We need to think about this differently now. We could be dealing with several different grades of digital ‘ammunition’ here. We suspect the system was trojaned, so the probability that other dns servers are trojaned is pretty high in my betting book right now. We need to find a motive. We need to find some tracks. We need more coffee.” I said as I stood up for a painful stretch.

“I can handle the coffee for you guys.” Thom replied as he exited the datacenter door.

“Maybe I’ll just sit here.” Scrap said, still staring at the floor.

“Look, we all make mistakes. This was an expensive one. Learn from it. Let’s move on.” I said as I walked over to a buzzing rack of servers, all marked “DNSxx”. Why a university needed so many DNS servers was a puzzling question - but even better - why were they all located in one physical plant? Unusual from my experience.

“I see 7… 8… 9… 10 servers marked DNS here. They are all in the same network. They are all powered on. This one looks particularly interesting…” I trailed off, pointing to the server on the bottom of the rack marked “DNS PRIMARY”.

“Are we going to check it out, or take a picture?” Scrap said as he perked up.

“I want to take that box down.” I said under my breath.

“Eh?” Scrap responded.

“I said I want to take that box down - go get Thom. I want to look around on the box, but not while it’s running.” I said as I made my way out of the datacenter. “I’m going to the car to get my gear. Be back in a second.”

“Chief? What’s the plan man?” Scrap yelled into the hallway.

“We’re going to take a picture.” I responded back.

My kind of picture.

Source: archived original

Part VI · May 28, 2005

It was hot. Really hot. The walk out to the parking lot in 105F+ weather was brutal. A slight breeze blew by me as I paced through the parking lot with my equipment bag slung over my shoulder.

“So THIS is what a roast feels like in my convection oven.” I thought to myself a I wiped a large band of sweat from my forehead. I was twenty feet from the door of building C1 when I felt the cool breeze of air conditioning hit my face as a student exited the building. Pure paradise.“You’re going to take pictures of the servers? What good is that going to do us?” Scrap asked as I placed my equipment bag on the floor of the datacenter.

“Not so much a picture as it is a snapshot.” I said with a grin. “Let me put this into terms that you’ll understand. It’s like taking a CAT-scan of your head, then wandering around the images of your cranial cavity looking for brain matter. It’s a tough job, but we have hope - so we’re sure we’ll find something.”

“Oh I get it.” Scrap said with a serious look on his face. “You’re going to make a duplicate of the hard drive in the server, then look at the duplicate for evidence?” Scrap asked.

Apparently he didn’t get my joke.

“Precisely. Did Thom go to get his sysadmin kid? What’s his name?” I asked, unzipping the equipment bag and removing the contents.

“Uh, I think his name was Jack. Or Jim. No, Jared! Yeah his name was Jared.” Scrap said with a scratch of his scalp.

“Alright, when Jared gets here I’m going to ask him to double-check that we can take this server offline without harming the school’s network. Once we get the OK, we’re going to pull the cord and make an image of the drive.” I muttered. “However - I want to get a network traffic sample too. Do you think you can handle putting a network tap in and grab a few hundred megs of traffic here and there?” I asked Scrap as I handed him an ethernet tap.

“I am SO on that man. No worries dude!” Scrap said as he sauntered off to the patch panels located on the other side of the room. Surely Scrap couldn’t screw up a simple ethernet tap from this network switch to the core switch - could he?

“Uh, Scrap, don’t you want to know WHAT I want captured?” I yelled over to the other side of the room. Scrap tended to use the “shotgun” approach to everything he did, which was quite dangerous in an investigation.

Scrappy wandered back around to my side of the datacenter.

“Oh, you don’t want me to capture everything?” Scrap asked as he reached for his cup of cold coffee.

“We start small and work our way up. I want to get some samples of traffic from each of these machines. Are they communicating on ports that we don’t know about? Who are they talking to? What are they sending and/or receiving? This is going to be very important, so double-check your capture settings.” I demanded as I passed Scrap his notebook.

“Got it.” Scrap said while giving me a salute with his left hand.

Meanwhile, Thom had reappeared with his systems administrator Jared. I took a few minutes to introduce myself and to explain to Jared what we were trying to accomplish. Jared was a student in his early twenties, pale as an albino, and apparently still believed in the alluring look of the mullet.

“Uh, I don’t think we want to take any of those machines down.” Jared said, looking off into the distance. He then proceeded to wander around the rack of servers, carefully examining my equipment and peering at the back of Scrap’s head.

Interesting.

“Jared, I really need to image one of these machines as part of my investigative process. Perhaps we could take one of the others down?” I asked politely.

Jared thought for a few seconds and walked back towards Thom. With a quick shrug of his shoulders he addressed my concerns with a long list of technical reasons why each of the DNS servers was required.

I wasn’t buying it.

Source: archived original

Part VII · Jun 1, 2005

“Uh-oh.” Scrap mumbled as I walked towards him. “You have that look on your face.”

“It’s worse than you think. I have that attitude thing going as well.” I said as I motioned for Scrap to join me out in the hallway for some privacy.

“Oh man. Is this guy the suspect? You got it figured out already man?” Scrap asked as he nervously searched his pockets for a cigarette that he’d have to go outside in the 105F weather to enjoy.“This guy Jared is either trying to royally BS me, or he really has no idea what is going on around here. He just told me that he has one domain being served by each of the servers for DNS load distribution.” I spouted off as I nudged the wall with my shoe.

“He what? We both know that is a bunch of sh…” Scrap blurted out as I interrupted him.

“Wait. Wait, wait, wait. Scrap, why would you think you needed to dedicate an entire server to publishing a single domain name?” I quizzed Scrappy.

“Uh, cuz you’re a moron?” Scrap responded, sliding that cigarette behind his ear.

“Besides that. If your server is so damn busy doing other things, name resolution is going to be very slow, right? Even for one domain!” I said as I smacked my forehead.

“So Jared really has no idea what is going on, probably logs in every once in a while to edit a zone file, and someone else 0wnz his boxes.” Scrappy suggested.

“You’re not far off from my own theory. Go grab your laptop and let’s look over some traffic captures. I’ll go keep Jared occupied.” I said as I turned around to enter the datacenter.

“Chief, I have to apologize. I got to thinking about your request, and I completely overlooked the fact that we take… well not on purpose… one of these guys offline all the time when they stop responding. I go to login and the filesystems are destroyed. Weirdest thing ever. Anyways, I can image and prep that machine that you guys have on the table over there, slide it into the rack and we can down that server afterall.” Jared said, his eyes looking rather sincere.

“I’ll help Jared.” Thom volunteered. “We’re experts at this by now.”

“Deal. I’ll leave that up to you, I need to work with my… assistant.” I said, trailing off as I walked back towards Scrap’s location on the other side of the datacenter.

“Yo man, we got a real head-scratcher here.” Scrap announced as I sat down on the datacenter floor next to him.

“Eh? What’s your portscan show on that production box?” I inquired.

“No big surprises. I bet this looks familiar, man!” Scrap said. I leaned over his shoulder to read the laptop screen.

_ Discovered open port 22/tcp on x.x.x.1 Discovered open port 25/tcp on x.x.x.1 Discovered open port 21/tcp on x.x.x.1 Discovered open port 53/tcp on x.x.x.1 _

“That’s consistent with what we saw on the x.x.x.13 box.” I said.

“Yeah, but wait until you get a load of this!” Scrap said as he switched virtual desktops on his laptop and scrolled through some ethereal traffic dumps. “Now is that cool or what? Uhh, what is that exactly?” Scrap asked.

I took a deep breathe. I hate being right most of the time.

Source: archived original

Part VIII · Jun 3, 2005

Scrap was staring at the side of my head so hard I could actually feel my ear growing hotter with each passing second.

“Well? WHAT! What kind of traffic is that, man? I’ve been a systems administrator monkey for a while now and I’ve never seen a capture that looked like that.” Scrap practically screamed as he gave me a nudge.“It would appear that these servers are used for much more than enhancing the education of the students.” I cracked as I pointed to the Ethereal window on Scrap’s laptop. “Grab packet number 1080. Look at the ASCII view. Look CAREFULLY. What do you see?”

Scrap focused his eyes on the packet capture.

_** y.y.y.170,x.x.x.1,IP,337,….i……i..E..C..@[email protected]….D..w.g……………w……v=sQ….GET /uri-res/N2R?urn:sha1:4TSJQULHIUIGQZGWN2OZ4IYFNWBAR67G HTTP/1.1..HOST: z.z.z.119:6348..User-Agent: LimeWire/x.x.x.. _**

“Holy copyright violation on a stick! It’s a Limewire session! Peer to peer file sharing, man!” Scrap said with a large grin on his face.

“You got it. And from the looks of these captures and the sheer volume of traffic on this segment, I’d say there’s quite a few files being passed from this server.” I said as I watched the packet counter climb to a number nearing last month’s Starbucks tab.

“Chief, this would explain a lot. I mean, P2P traffic can suck the life out of any network. That dude Jared is either in on it, or completely clueless!” Scrap exclaimed as he peered towards the ceiling. “So what do we do now?”

“We act like a ziplock bag and say zilch to Jared. We need to have a pow-wow with Thom and see where he wants to go with this. Remember, Thom is the client - he’s the boss.” I said as I saved the packet captures to Scrap’s hard drive.

“But we have some serious info here, man. Let’s go ask him what’s up!” Scrap insisted as he motioned me towards Jared (who thank goodness had his back to us and was twenty feet away).

“Scrap. Take a breath. You’re sweating like Al Bundy in a strip club. We’re not going to get anywhere with some packet captures. This is raw evidence. We haven’t processed it yet. We sit down with the client and get some direction. If you’re going to do well in this type of profession, you need to learn patience and restraint.” I said as I placed Scrap’s laptop under my arm. “Now, let’s go!”

Scrap and I wandered towards the door of the datacenter. I waved at Thom to join us as we wandered into Thom’s office. Scrap closed the door behind us.

“You’re not going to turn on a heat lamp, are you?” Thom joked as he took a set behind his desk. “I’m wearing a good shirt. My wife ironed this baby this morning!”

“Thom, we have an unproven theory on why your servers are spiking and dying. Scrap has collected some interesting packet captures that lead us to believe that those DNS servers in there are actually P2P servers, and they are REAL busy.” I explained.

Thom thought for a few seconds and shifted in his chair.

“Somebody has rigged P2P software on these servers? And they are serving what?” Thom asked.

I opened Scrap’s laptop and brought up some sample packet captures from our recent capture session.

“Judging from these strings in the packets, I would say music files, movie files, picture files, and some documents.” I replied as my finger glided over the LCD screen.

“Dude! They are looking for Iron Maiden! Sweet!” Scrap said, obviously not able to contain his excitement any further.

Thom had a frown on his face that almost engulfed his entire chin. The man was not happy.

“Ok, this obviously can’t continue. We are putting the school at serious risk here. We need to take those servers down right now, whether Jared likes it or not. But I want to know how this happened! This is unacceptable! I am going to have a lot of explaining to do to the Dean.” Thom said as he leaned back in his chair.

I paused for a moment and made a suggestion to Thom. “Taking these servers offline may cause a service disruption to the campus network. We’re going to have to…”

“Chief.” Scrap said.

“Not now Scrap, wait a minute.” I fired back.

“Chief, you REALLY want to see this.” Scrap said as he passed his laptop to me.

Now what had he been working on while we were talking?

Scrap had performed a “whois” command on several of the campus domains.

He was pointing madly at the nameserver records.

“Check it out! There are two nameservers registered to answer for these domains. And you know what the best part is?” Scrap asked.

“Neither of them are in that rack.” I mumbled.

Cripes.

Source: archived original

Part IX · Jun 7, 2005

“Riddle me this, man!” Scrap said with a large grin on his face. “When is it that you have DNS servers that don’t really serve DNS?”

“When your DNS servers aren’t really DNS servers?” I replied, stating the obvious as we re-entered the datacenter. Jared was nowhere to be seen.“Seriously man, this is wild. Now look at the IP addresses of the registered master servers, now look at the network map. The master servers are in that rack right over… there!” Scrap said with a point of his finger to the rack on our right.

I wandered over to the rack and scanned the adhesive name labels on the face of each server. Nothing that appeared to be a DNS server by host name jumped out at me.

“This is like looking for a needle in a haystack.” I complained. “There’s at least ten servers in this rack, the host names aren’t useful, and there’s no IP addresses on any of these labels.”

I grasped my chin with my right hand and slowed down for a second. I walked around to the rear of the rack and examined the network connections. Above each connector was a factory-installed white sticker with a MAC address label in clear view. Jackpot.

“Scrap, nmap those IP addresses and read me the last few parts of the MAC addresses on both.” I said as I maneuvered myself completely behind the rack.

“Roger that, Chief. Okay - the first IP address is assigned to dc:69. The second IP address is assigned to bf:32.” Scrap announced proudly.

I scanned down the factory stickers and was not seeing a match until I came upon the last two servers. The last 2 fields matched exactly on both machines.

We had found the mysterious servers.

With Thom’s permission, a quick and dirty analysis of the two boxes ensued.

The analysis told me exactly what I expected to hear - these were two real DNS servers!

“Chief, what the hell is going on? This is like a Matrix movie gone terribly wrong.” Thom whined as he massaged his tired eyes with his forefingers.

“This is actually excellent news. We can safely remove the questionable servers from the network for examination and not cause a service disruption to the campus network. With your permission, I want to disconnect all the servers immediately and perform a more detailed analysis.” I stated as I waved to Scrap across the room.

“Yes! Yes! Let’s do this. I want answers to this puzzle now.” Thom replied.

Scrap had understood my monkey paw waving and started disconnecting power and ethernet cables from the backs of the suspect machines. Scrap had two cables to go when he suddenly stopped.

“Chief - dude, this machine is going insane!” Scrap yelled, now waving his monkey paws frantically about his head. “Look at the network activity light! It’s like a strobe light from the seventies. Listen to that hard drive chatter!”

I crouched down next to scrap and placed my left ear against the front of the server where the hard drives were mounted. I was inches from the machine and could hear the drive chatter over all the other noise in the datacenter. This machine was very busy.

“Uh, oh!” Thom said as he crouched down next to me.

“Oh man, what the heck IS that?” Scrap asked as he scratched his head.

“That, is our stealthy spike.” Thom replied.

Source: archived original

Part X · Jun 8, 2005

“Your call, Chief!” Scrap said, his monkey paw firmly grasping the network cable.

I peered over at Thom, raising one eyebrow.

“Thom, do you want answers or do you want us to just clean this up?” I asked.

“Well, we’re in this deep already. Let’s get some answers.” Thom responded with a grin.

I motioned for Scrap to back off from the cables and reached for the “Scraptop”. I handed Scrap his laptop and pointed a finger towards the network cable on the back of the spike-ridden server.

“Tap that. Full capture. Let’s grab a few gigabytes of traffic and play detective.” I said with a slight grin. This was going to be entertaining on many levels.

“Tapping!” Scrap said as he quickly put the tap in place, careful to only disconnect the network cable from the back of the server for a few seconds. After a few quick keystrokes on his laptop, the network traffic scrolled across his screen with blazing speed.

“Where’s Jared?” Thom asked as he looked around the datacenter.

“He wasn’t in here when we came in. Was he supposed to still be here?” I asked Thom, unaware if Jared was supposed to be in a class.

“I told him to keep his butt in here to answer questions. He has a very small attention span.” Thom responded.

“Alright, with the tap in place let’s get busy on imaging one of these servers. Scrap, would you mind assisting me with a ‘serverectomy’?” I asked.

“Your tonsils hurting, man?” Scrap asked.

I sighed. “Dude, just grab a screwdriver and let’s get the first server out of the rack. Don’t forget to pull the power cord first.”

“Alright, alright. Hey, you brought up your tonsils.” Scrap said as he started unscrewing the mounting screws holding the first server to the rack.

I was starting to wonder if Scrap dyed his hair.

A few minutes had gone by and the server was sitting safely on the table in the datacenter next to the server that Scrap had doomed to erasure by his over-eagerness to run shell commands.

“Hey Chief, hand me that bigger screwdriver.” Scrap barked as he pointed his paw at a red-handled screwdriver sticking out of my tool bag.

Scrap removed the screws that kept the lid of the server attached and proceeded to slowly pull the lid towards the rear of the server.

“Scrap! WAIT! WAIT DAMMIT!” I screamed as I grabbed the lid of the server.

A sliver of silver wire caught my eye that was somehow attached to the lid of the server. The other end couldn’t be seen. I reached into my tool bag to find my trusty penlight for a closer look.

“Dude, it’s just a wire…” Scrap joked as he clasped his hands around the sides of the lid.

“SCRAP! WAIT!” I screamed again.

White smoke drifted from the top of the server case.

“What the hell is that?” Thom asked, apparently startled like the rest of us.

“Scrap, I SWEAR TO GOD. If I tell you to do something, or NOT do something - you listen the first time. You do not EVER, EVER remove the case lid to a suspect machine without inspecting it first. I don’t know what that is, but if it had been explosives, we could be DEAD. Am I getting through to you?” I said with a stern look on my face.

I was furious.

“Yeah, yes. Yes. I’m sorry man.” Scrap replied as he took a seat next to Thom by the table. Both appeared rather stunned as they coughed from the smoke filling the room.

I grabbed my penlight and worked my way around the surface and lip of the lid, waving smoke out of my eyes with my other hand. The smell was very familiar from my days of trying to be a twelve year-old stunt bicycle rider.

It was a smoke mix - and a battery-powered trigger. You’d typically find this type of setup used on remote-controlled toy planes, helicopters, cars, you name it.

I scratched my head furiously.

Why would someone rig a smoke bomb in a server? Physical booby-traps like this are very, very rare in evidence equipment. This was only the second case I had ever seen.

“I’ll be right back. I’m going to go drag Jared out of the computer science lab down the hall. This is complete crap.” Thom said as he raced out the server room door.

“Chief, do you ever get the feeling that someone just doesn’t want anyone messing with these boxes?” Scrap asked as he reached into his shirt pocket for a cigarette.

Do I.

Source: archived original

Part XI · Jun 10, 2005

“It is a mistake to confound strangeness with mystery. The most commonplace crime is often the most mysterious, because it presents no new or special features from which deductions may be drawn.” – Sherlock Holmes

I needed a breather to clear all the white smoke that was lingering in the air out of my lungs. Scrap needed a cigarette. It was time to take a walk around campus.

Scrap took a few steps towards the door, his eyes looking straight at the ground. Learning lessons the hard way is a real ego killer sometimes.

“Scrap, what’s wrong with this picture right now?” I asked as I placed my hand on the exit door, blocking Scrap’s exit.

“Huh? Chief, what ISN’T wrong with this picture? Hey man, I don’t know where you’ve been but somebody just smoked us!” Scrap cried as he placed his hands in his pockets.

“No, I mean… look up at the ceiling. This room is equipped with Halon fire suppression. Why didn’t it fire off? Look at all the smoke that is still in this room!” I said as I trotted over to the wall where the Halon control panel resided.

I glanced at the control panel and squinted.

“Scrap, this system is keyed OFF. Look! The fire suppression system is turned off!” I said in disbelief. “Why the hell would it be off? Have you ever walked into a production datacenter and seen the fire suppression system off? I mean, when there’s not a test or some kind of maintenance going on?”

“Never, man. Never. Might want to bring that up to Thom.” Scrap said on his way out the door.

I followed scrap down the hall and out into the heat of the afternoon. I sat down next to Scrap on a bench and watched several students walking towards the next building.

“Scrap, you want something to drink?” I asked as I checked my pockets for some cash.

“Sure. Grab me a Mountain Dew, man.” Scrap replied.

I walked along a well-groomed stone path towards the cafeteria. Inside, I quickly located the self-serve drinks. I poured Scrap a Mountain Dew and helped myself to a bottle of Green Tea.

“It’s not coffee, but it’ll do.” I thought to myself as I handed the cashier a few bills.

A few feet away, a group of students sat huddled around a break table with laptops strewn about. One of the students had a beautiful 17-inch Apple Powerbook. I wandered over in their direction for a closer look and took a seat at the table next to them.

“Yeah, and check this out. Star Wars Episode I! I’ve got a couple of Jackie Chan movies and some serious cartoonage coming down the pipe, babe.” one student bragged to the girl sitting beside him.

I leaned over to get a better glimpse of the beautiful 17-inch widescreen display on the Apple Powerbook when a familiar logo caught my eye. A lime, ala Limewire.

“How the heck is he pulling down peer-to-peer files from in the cafeteria?” I wondered.

A quick glance at the menu bar near the top of the screen gave me my answer; an airport icon. The school had a wireless network connection present.

I grabbed my drinks and headed back towards Scrap. My mind was racing. I had an idea.

“Hey, thanks Chief. This will hit the spot.” Scrap said enthusiastically as he took a sip of his drink.

“Let’s roll back into the datacenter.” I said without a bit of hesitation.

Once back in the datacenter, Scrap and I were treated to a familiar pair of faces.

“Chief, I think you know Jared. Jared has an interesting story to share with us.” Thom said, nudging Jared forward a few steps.

The room was quiet with the hum of servers and Scrap sucking a few more gulps of Mountain Dew from his cup.

Jared looked like he was bursting at the seams.

Source: archived original

Part XII · Jun 15, 2005

“Tell me a story about a man with a secret, and I’ll listen long enough to determine if my name is mentioned or not.” – Anonymous

Jared was about as comfortable in his own skin as Scrap in a pair of thong underwear. His eyes shifted back and forth, he was sweating profusely and taking deep breaths.

“We’re all pretty confused about this entire situation, Jared.” I said as I sat on the table facing Jared.

“Tell them what you just told me.” Thom demanded as he made eye contact with Jared.

Scrap slurped up the rest of his Mountain Dew, breaking yet another painful silence in the room.

“Let’s start with the nameservers, since that is what brought me here. There is an entire rack of them right here, and there are two unmarked ones in that rack over there. Can you explain to me why the two ‘real’ nameservers are unmarked and in one rack, and the rest of these nameservers are marked and in another rack?” I asked Jared.

Jared took a deep breath and began speaking nervously.

“There are twelve nameservers total. Two of them are the primary resolvers for the two externally accessible domains of the school. The other ten are resolvers for internal domains only. Departments around the campus have their workstations pointing towards one or more of the ten servers depending on their needs.”

I scribbled some notes in my notebook, trying to ensure that all the holes in this strange situation were addressed.

Thom took a step forward towards Jared. “I’d like to know what was causing the load on these machines to spike and render the machines useless? We have lost a lot of man hours rebuilding servers and dealing with staff that can’t function without name resolution working.”

“The CPU load increases are a side effect of file serving software that is installed on all twelve servers. As the number of requests for files increases, the CPU load increases.” Jared replied.

“Might I ask WHY we have file sharing software on all twelve of our nameservers?” Thom asked.

“The machines had no load on them most of the time and the hard drives were barely used. Between the twelve machines we had over half of a terabyte of data storage available, an all of the machines sat on a 100MB ethernet network connection.” Jared said, taking some time to think about quantitative amounts before speaking.

“I’m not sure that you answered his question.” I interjected.

“Look man, I didn’t do any of this. I took over for Marcus last semester and all of this stuff was set up when I got here.” Jared snapped back.

“And you didn’t think to report any of this, man?” Scrap asked, clearly irritated that he was out of Mountain Dew and eyeing my half-full bottle of green tea.

“Of course I did.” Jared replied. “But I had other incentives.”

Now we were getting somewhere.

Source: archived original

Part XIII · Jun 20, 2005

Thom was chewing his cud, clearly agitated over the entire situation. I realized that with Jared spilling the beans and Thom’s patience running short, I had to step in and steer the conversations.

“Let’s talk about those incentives. Why the deadman switches on the boxes? The smoke bombs? The rigged logins?” I asked.

Jared slowly inhaled. “From what I understand, this is bigger than just Marcus and his friends stealing bandwidth and services on the school’s dime. Marcus gave me direct access to the servers to leech anything that I wanted off via secure ftp. The boxes were peering points and used to distribute a lot of ‘zero-day’ items.”

“Zero-day items? You mean like theatrical movies? Software releases? Music files?” I pondered out loud.

“Yeah. Marcus was making quite a name for himself in some circles. Before he turned everything over to me, he seemed a bit nervous when we talked. You know how those association groups are shutting down and prosecuting distro sites and their owners? Well I think Marcus thought he was going to get caught eventually.” Jared replied.

“I see. And?” I asked, leading Jared into answering the rest of my questions.

“Right, well I didn’t know the smoke charges were there until the first time I had to take a server out of service and re-image it. Ironically enough, all of the images for these servers Marcus made himself, so all of the peering stuff just magically appeared again. Anyway, the first time I popped the cover off of a server I got a face full of smoke and I was freaking out because the detectors up on the ceiling were beeping. I haven’t had the fire system on since.” Jared admitted.

“You turned off the da…” Thom said, flying off the table.

“Thom, wait. This isn’t helping the situation. Let’s get all of the facts.” I pleaded to Thom as he sat back down on the table with a scowl on his face.

“I got a question.” Scrap said, once again breaking the silence. “So, you kind of knew the whole time that re-imaging these machines was just wasted cycles? And then you’d hand off everything to the next guy that took over for you?”

“Well, I hadn’t really thought about it that far down the road.” Jared said, his eyes aimed squarely at his shoes.

“Ok, I’ve heard enough. Jared, let’s go have a conversation with the Dean.” Thom said, motioning Jared to exit through the door.

After the door was closed, I closed my eyes and exhaled. I felt like a train accident.

“Dude, this whole situation is freaking me out.” Scrap said, drinking the last of my green tea. “I mean, how much sense does any of this make?”

“I can relate.” I said as I rubbed my eyes. “But the sad truth is that this type of piracy isn’t exactly new or strange to school campuses. I’m really surprised that things were so lax here. Thom is a pretty sharp guy.”

“It’s like you always told me, Chief.” Scrap said. “People trust the people that work for them, and sometimes they trust them just a little too much.”

I brought my bottle of green tea to my lips and attempted to take a drink from a very empty bottle.

“You drank the last of my tea?” I asked in disbelief.

“Hey man, sounds like you’re trusting your people too much!” Scrap rattled back.

Sometimes the teacher learns from the student. And sometimes, you have a student like Scrap.

=*=

Epilogue:

Thom went on a blitzkrieg at the school. Law enforcement and several trade associations were notified about the activities of Marcus, Jared and numerous other students.

Scrap scored a major consulting gig with Thom rebuilding the school’s DNS architecture since all twelve servers (and numerous other items) were confiscated as evidence.

Lawsuits were brought against several of the students, and they were all kicked out of the university and enrolled into the school of hard knocks.

[NOTE: I apologize for the delay in publishing the conclusion of this case file, however I had to edit it several times for content.]

Source: archived original