Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.
Part I 路 Oct 15, 2003
Saturday mornings are one of the true pleasures in life. There’s nothing like waking up, making a fresh pot of coffee from ground organic Peruvian coffee beans, mixing in a creamy stream of Hazelnut flavoring and a dash of skim milk for texture, and then sitting in front of my beloved Powermac and catching up on e-mail, news and a little bit of work.
Minutes after loading up MailApp and connecting via imap/ssl to my mail server, one particular e-mail caught my eye - it was from the owner of a local ISP that I had befriended at a technology expo many years ago.
The e-mail was quite enthralling: _ Chief,
Sorry this isn’t a howdy-do e-mail. I have a problem that I think you may be able to help me with. I’ve lost your cell phone number, so sorry about the long e-mail.
As you know, we host a very large mail cluster that hosts approximately 3000 domains. On average, we receive about 12000 spam e-mails a day. Over the past two weeks, this number has jumped to 110,000. Our servers are so busy receiving all of this spam mail and pushing it through spam filters that we’re contemplating adding more machines to the cluster to help lessen the load. Unfortuantely, the number of spam e-mails is growing every day. This morning we received almost 150,000!!
Our security analysts have examined a lot of the e-mails, but they haven’t come up with much.
This is where you come in.
Help me obi-monkey, you’re my only hope!
Call me at XXX-XXX-XXXX asap.
Dick H. _
“Sure” I thought to myself. “Right after I have my second cup of coffee goodness.” I mused.
SecurityMonkey is on the case.
漏 2006 All Rights Reserved.
Source: archived original
Part II 路 Oct 17, 2003
Lucky for me, the drive to Cranial-Link (CL) was less than an hour in Saturday traffic. Their office was located close to a local university, so needless to say coffee shops were in abundance.
I entered the lobby of CL with a 16oz cup of joe in one hand and my laptop bag in the other.
No receptionist to be found.
“Oh yeah…” I thought. “It’s saturday - where the heck is Dick?” I pondered.
The lobby had three doors - one to the outside (that I just came in), one to what looked to be a restroom, and the last being a high-security door with a badge and palm reader attached to the wall.
BEEP!
A loud chirp caught my attention as Dick entered the lobby from the datacenter security door.
“Hey stranger!” he yelled as he helped me with my laptop bag. “This way…” he trailed off as he escorted me into the datacenter…
“Nice flypaper.” I joked as I walked through two feet of anti-static/dust matting on the floor. These guys were serious about a clean datacenter.
We were seated in a medium-sized conference room for a few minutes catching up on recent events when we were joined by Mark & Mike, the security analysts for CL.
“Mark, Mike… bring Chief up to speed on where we’re at.” Dick said as he attempted to drink out of my Starbucks cup.
“Get your own!” I screamed as I snatched the cup from his hand. Why is it that people want to drink MY coffee?
“Hi Chief. Mark wrote a shell script that drilled down into about half a million spam emails and did some data extraction and correlation of header/content components. Here’s the results.” Mike said as he turned our attention to the powerpoint presentation now showing on the conference room wall. The slide gave everyone a good glimpse at the frequency and source IP address ranges of the spam messages.
“Fascinating.” I accidentally said out loud.
“What?” Dick asked, sipping HIS coffee.
I popped open my linux laptop and took advantage of the ethernet cable that Mike offered me earlier.
“WHAT????” Dick asked.
“Hold on and I’ll tell ya.” I snapped back. People are so impatient these days.
“I hate it when you do that.” Dick remarked.
“Ok, who has the largest vehicle in the parking lot?” I asked.
“Uh, I would guess me.” Dick answered. “My suburban dwarfs anything out there. But what does that have to do with the price of tea in China?”
“We’re going for a ride.” I responded back. “But first, lunch.”
Tracking down spammers makes me hungry.
Source: archived original
Part III 路 Oct 20, 2003
Decaf.
Of all the restaurants near the CL building, these geniuses had to pick the one that was out of regular coffee and only had decaf.
Thank goodness for Coca-Cola. I had to make sure I was perky for the next leg of the investigation.
“Okay, now that I’ve fed you - what gives, Chief? Where are we going?” Dick asked as he lit up a menthol cigarette and made smoke rings the size of my head. We were walking back to his Suburban with the Mike & Mark show bringing up the rear.
“WE aren’t going anywhere. I have a hunch about something that I need to check out. Drop me off at my car and I’ll call you in a few.” I said.
“You’re a real piece of work, Chief.” Dick said under his breath…(Note)
While I was in the meeting with Dick & team, I had noticed something interesting about the statistics on the powerpoint presentation. A large amount of the spam email was originating from a particular subnet - a /26 net to be more accurate. This told me right away that either the spammer bought/rented a new ip block, or he was using someone else’s at will without their consent.
According to ARIN, that subnet actually belonged to a company and it wasn’t an internet service provider - it was a hotel chain!
(End Note)
After climbing into my car, I drove back over to the local Starbucks and used their wireless network connection to do a few more steps of research. I needed an I.T. contact at International Stars Hotels. Their website wasn’t much help, but I did find a number for interested investors.
“Surely they couldn’t refuse my $2.50 in my pocket!” I mused.
A quick call to the number listed on the website connected me to a very sultry-sounding woman named Christy.
“Um, hi - I was trying to reach your IT department.” I said in my best nerd voice.
“Who are you?” Christy asked.
“I’m a security consultant and I need to talk to your IT department about some unauthorized usage of your network.” I said.
“I’m not allowed to transfer anyone directly - you need to call the main number.” Christy snapped and hung up.
Some companies make helping them so hard. sigh Since Christy wasn’t giving up the goose on what that magical number was, I was going to have to work for it - and use some tricks from individuals that I track down and have prosecuted.
I ran into Starbucks for a caffeine fix, was back to my car in ten minutes and I picked up my cell phone again. This time, I was using my command voice.
“Hi Christy.” I began. “I think your our phone system is screwed up. I was being transferred to the help desk. Can you transfer me?”
“Um, I don’t have time to look up their extension - look it up in the directory.” Christy said as she hung up.
My relationship with Christy wasn’t over yet. I dialed her number again sixty seconds later.
“ISH - this is Christy.” was her familiar greeting.
“Hi Christy! I got transferred to you again! Grrrr! I think the phone system is screwed up.” I said.
“Jeezus, I swear I’m never going to get this TPS report done. HOLD-ON.” Christy snapped. A few touchtones and a click and I was transferred to… the help desk.
Works every time. Perhaps I should engage the CSO of ISH about some anti-social engineering training? Maybe get Christy an assistant?
Now… to locate their data security department and get some work done.
Clutching the clue banana in one hand and my cell phone in the other, I waited patiently for the next available help desk agent to help me… and themselves.
Source: archived original
Part IV 路 Oct 22, 2003
My ears were in pain - horrible, horrible pain. The hold music for the ISH help desk was a mixture of elevator jazz and Martha Stewart commercials. The Middle Ages knew not torture of this magnitude.
“ISH Help Desk, Jon speaking. May I have your employee ID number please?” came from my telephone.
“Hi Jon - I’m a security consultant and I’m trying to reach your data security department. Can you put me in contact with them?” I pleaded.
“Which building are you in right now?” Jon queried.
“None - I’m in the parking lot on my cell phone looking for an extra battery for my laptop.” I said. Oh, that was a good one.
“Gotcha - give me your name and number and I’ll have them call you when I track them down.” Jon said.
“Wow - a help desk guy with a clue, following procedures. This place isn’t hopeless after all!” I joshed to myself.
I gave Jon my name and cell number, and searched my pants pockets for some coins - I was out of coffee… RING
“Chief here.” I spoke into the little magic phone.
“Uh, hello. This is Mikel - I am with ISH data security. You wished to report something?” he said in a very rich Russian accent.
“Hi Mikel - thank you for calling me back. I apologize for the theatrics surrounding my call - but I knew of no other way to reach you. I have a large ISP client who is being hammered with spam email from one of your netblocks.” I said.
“Spam? Spam is bad. We do not allow spam at all. " Mikel responded.
“I’m sure you don’t - but this is spam that I believe is unintentionally being funneled through your netblock.” I explained.
“I see. And how do you propose that this spam is coming from our netblocks?” Mikel asked.
“Give me your e-mail address, and I’m going to send you an overview of what I think is happening. After you read it, please let me know what you think.” I said.
“Ok, here is my e-mail address…” Mikel said as he hammered away with a private address.
(Aside)
The document I sent Mikel more or less described the following:
-
There were approximately one hundred IP addresses in their netblock that had sent mail to CL.
-
The IP addresses were all in the same subnet, owned by ISH.
-
The hostname of the machine sending the spam was the same - localhost.localdomain and a very funny X-MAILER header - it said “Suck it down!”
My suspicion was that somehow a machine(s) was appearing on these IP addresses at random intervals of time and firing off thousands of spam e-mails that were apparently directed at the domains that CL was hosting.
With Mikel’s help, this was going to become one of my most fascinating cases yet…
Source: archived original
Part V 路 Oct 24, 2003
“This is not possible on our network.” Mikel said tersely.
He had read my e-mail regarding the spam transmissions from their network and how the e-mails were being directed to CL-hosted domains.
“Can you humor me and take a look anyway?” I pleaded. My watch was reading 1700 hours, and I had gotten out of bed way too early this morning.
“You say these e-mails are still coming?” Mikel inquired.
“After my discussions with CL staff, we have installed access control lists on their border routers and are currently dropping all traffic from the netblock in question.” I responded. “However, that’s a band-aid. When these e-mail storms kick in, utilization on their routers climbs dramatically.”
I managed to squeeze out some technical details from Mikel about their network setup. The troublesome netblock was routed over their primary internet connection which happened to be a 6MB fiber link. This would explain how the spam traffic was able to overpower CL’s two BGP4-enabled routers that sat on two T1 lines. ISH apparently had very few ACL’s on their routers, no firewall between the router and any machines living on that subnet, and Mikel seemed irate when I brought up the concept of intrusion detection.
“Is this your idea of selling us your security ‘services’?” Mikel asked in a seemingly harsh tone…“Mikel, I am a consultant working for another company - I am seeking your assistance in stopping this traffic from harming my client - nothing more.” I pleaded yet again.
After an hour or two of convincing Mikel that I wasn’t selling him anything, that I didn’t work for the U.S. Government, and that I wasn’t a KGB operative (joke! it’s a joke!) he seemed to lighten up quite a bit.
(Aside) One needs to be very careful when seeking the assistance of a corporate data security team in a situation like this. If you are wondering why I didn’t actively probe Mikel’s network to begin with, the reason should be quite obvious now. It could have been taken the wrong way, and if recent court cases can teach us anything about the ignorance of companies and the internet, odds are they might have sought to prosecute me as a ‘hacker’. Gain their trust first, then seek assistance. If you get the assistance, great. If not, you need to throw the case into the hands of the client’s general counsel as soon as possible. (/Aside)
Mikel continued to talk to me on his cell phone while he entered the ISH datacenter. He logged into a workstation in the datacenter and brought up their network inventory system.
“Ok Chief, according to records, there are twenty active IP addresses in subnet. Most are web servers, mail servers, dns servers, media servers… I will now verify addresses and call you back.” Mikel said.
“Great! One more thing Mikel… " I began. “I’m more concerned with the IP addresses that you think AREN’T in use. "
“Yes yes… I will setup span port on switch like you ask and sniff.” Mikel replied. As I ended the cell phone call, I was getting a little frustrated with Mikel. I had a feeling that he wasn’t very experienced in what I was asking him to do - what kind of results was he going to achieve? Would it help my investigation?
I decided to catch a movie and relax and the local theater. Nothing like a good SCI-FI flick to relax the grey matter.
After the movie, I headed back to my home office with every intention of calling Dick and bringing him up to speed on the situation until my cell phone rang again.
“Chief” I barked.
“Yes, this is Mikel. I am thinking that I have bigger problem.” Mikel said.
“Yes?” I asked.
“I found something plugged into switch when I was plugging laptop into switch.” Mikel explained in an exciteable voice.
“YES? YES?” I said, growing impatient.
“And it has cable running under floor…”
Source: archived original
Part VI 路 Oct 28, 2003
“I’m sorry - there’s a cable running under the floor ?” I interrupted.
“Yes! Yes! From switch to purple box.” Mikel replied dropping the phone on the floor. My ear did NOT need that.
“Ok, but what does it say on the ‘purple box’? What is it?” I inquired with increasingly less patience.
After about fifteen minutes, I finally established that it was a Linksys wireless access point!
Things were making much more sense… Now I had to ask Mikel a painful set of questions.
“Mikel, do you allow wireless networking on your network?” I asked.
“No, no wireless networking. Network people already asked, I said no. Not secure enough!” Mikel replied.
“Good answer.” I thought to myself.
“Mikel, I think you might want to interview all the network folks and find out if they installed that for their convenience - circumventing your network security policy.” I suggested. “But first, unplug that thing!”
“I already unplug box.” Mikel said.
“NOW he shows initiative. Sheesh.” I mused to myself.
Mikel agreed to start interviewing the network staff, and I quickly entered a few pages of notes into my laptop.
At this point, I could give Mikel some suggestions on what to do next. He could simply confiscate the WAP and create a security procedure to periodically scan his physical property for unauthorized WAP’s using common tools like NetStumbler or KISMET. Or (after confering with his administrative/legal/managerial team) he could plug the WAP back in and keep that network sniffer active - possibly find out the who/what/when/where/how of the situation.
Several days had gone by. Dick was happy that his spam traffic returned to ’normal’ and was eager to cut me a check for my consulting help. I of course had no problem with that - except that in my mind the case was far from over.
Sitting in my home office, my mind was racing with different scenerios. I wanted to be onsite and working with Mikel so bad on this one! Unfortunately not all of my wishes come true.
RING went my cell phone.
“Chief.” I answered.
“Hello Chief. I have proposition for you.” Mikel said. “I have talked with director of information systems. He would like to meet you and discuss our situation. We have complaints and legal papers from other companies regarding spam.”
I almost dropped the phone.
“Really?” I replied, with the thoughts of yet more consulting fees dancing in my head. “No problem Mikel - let’s set something up.” It was looking like IHS’s problem was much wider than just CL’s hosted domains.
Two hours later an e-ticket for America West Airlines showed up in my inbox.
Looks like I was about to get my wish.
Source: archived original
Part VII 路 Oct 30, 2003
What a flight. I actually prefer to fly at night because I get an opportunity to do something that I don’t do very often: sleep. ISH was kind enough to fly me to their HQ in the first class section, and I wasn’t complaining.
“Another glass of bubbly should put me right to sleep.” I thought as the flight attendant poured the champagne in my glass.
A few hours later I was awakened by the captain alerting us that we were preparing to land. I was still a bit groggy, but able to remember how to turn my cell phone back on and get off the plane.
Minutes later I was clearing the security checkpoint at the airport when I noticed a gentleman carrying a sign written in red marker that said “Chief.”
“Mikel?” I asked the stranger as our eyes met.
“Yes. Hello. I hope flight was good. I will drive you to hotel.” Mikel said as we headed to the parking garage.
Mikel proceeded to fill me in on several staff meetings that he had attended regarding our recent findings, some background on the ISH IT Director, and the address where I could send my invoice.
Things were looking up… for now.
The morning came way too early. I managed to sneak another few hours of sleep before my taxi ride to ISH HQ.
The ISH receptionist greeted me with coffee (good beans) and escorted me down a never-ending hallway to a very large conference room.
Introductions were made, and I found myself speaking in front of the IT Director, Mikel, General Counsel, the PR Manager, and the CFO.
The IT Director spoke first.
“Chief, we have received numerous complaints from companies and individuals that they have been the recipients of large amounts of unsolicited email that originated from our computer network. Two of the complaints are from very large ISP organizations - Canada Online and NSN (names changed). This is completely new to us. We want to know how to address these concerns.” said the ITD.
The General Counsel squinted at me and said “You and Mikel have done some impressive work tracking this down. What we’d like from you is a full investigation on where this all started, who is involved, and what remediation steps we’re going to perform so that the organizations complaining to us will not press us for remedies of any kind.”
“Understood.” I said.
The meeting ended, and the ITD caught me going out the door.
“Chief, we followed your advice and decided not to interview more staff members. You should know that you are here under the disguise of a SAS*70 auditor.” the ITD said.
“Great - now nobody will want to talk to me.” I joked.
Mikel and I grabbed some lunch at a nearby deli and returned to ISH a few hours later.
We walked to the datacenter. I noticed that Mikel had to scan his ID badge and enter a PIN code to enter the datacenter. A good use of two-factor authentication.
The datacenter was at the very rear of the ISH building. I asked Mikel if we could exit the datacenter through the loading dock door in the rear so that I could look around. As we exited the door, I was shocked to see that access to the rear of the building was wide open to the access road and the parking garage . The rear door had no external door handle or means of entry - it had to be opened from the inside.
Back in the datacenter, Mikel showed me the WAP that he discovered and the network switch that the WAP was plugged into. Mikel had configured a span port on the switch so that his laptop could monitor the traffic on the WAP port, only he had never gotten a chance to use it.
I examined the WAP thoroughly and decided that I wanted more information. I attached a USB cable to the WAP, and the other end to my laptop which was dual-booted into Windows 2000 Professional. I happened to have the Linksys management tools installed on my laptop (what’s the boy scout motto again?). I started the USB management tool and was browsing the configuration in no time.
There were no surprises - the WAP was configured as an ethernet bridge. All of the other settings were factory default. No clues.
Essentially anyone that connected to the WAP could appear as a machine on their external subnet as long as they knew the proper IP settings.
“Mikel, I’d like to setup a monitoring session using the WAP and your laptop. Did you clear this with the GC llike I requested?” I asked.
“Yes - he was okay with request.” Mikel responded.
“Good - here’s the application we’ll need.” I said as I handed Mikel a CD. The CD contained a Windows Sniffer application and a log monitor. I configured the log monitor to send an alert to my cell phone via the laptop modem if it detected any SMTP traffic on the span port.
We placed the WAP and the sniffer laptop under the raised floor and up against the wall where Mikel had previously found the WAP. Mikel managed to snake the POTS line for the modem from the wall jack down into the floor without cutting the line with the floor tiles.
After discussing our sniffer project and exchanging war stories about information security, we decided to break for dinner, but a quick stop at the restroom was required.
As I entered the restroom I was startled by a loud electronic noise echoing through the tiled walls.
My cell phone had received a text message.
Source: archived original
Part VIII 路 Nov 4, 2003
“That didn’t take long!” I yelled at Mikel as we ran back into the datacenter after patiently waiting to authenticate at the main door.
Mikel removed the floor tile and I lifted the laptop out of it’s hiding spot. A quick tap of the spacebar and the screen came to life.
“What are you seeing on screen?” Mikel asked while doing a little dance-in-place next to me. Evidently Mikel had to talk to a man about a horse as well. I guess we’re both out of luck.
“Fascinating.” I said, engaging my Spock eyebrow as my fingers flew accross the keyboard. “The traffic started at exactly 1800 hours.”
“I do not understand relevance. 1800 hours? What?” Mikel pondered.
“I adjusted the time on my laptop to match the time on your application server over there in the corner.” I said, pointing towards a Sun E450 about five feet away. The server had a 17” Sun monitor propped on top, the CDE desktop clearly visible.
“And?” Mikel asked.
“All your servers are synced to a master time server, correct?” I asked.
“Yes.” Mikel replied.
“You don’t think that’s a one-in-a-million coincidence?” I queried.
I smell CRON (or maybe AT?).“Ok, so what is plan?” Mikel demanded.
“First, we find out what machine is on the other end of this wireless connection.” I said.
The IP address that was showing up in our sniffer session was one of the same IP addresses that had been found in the SMTP headers of the spam emails at CL.
BEEP
My cell phone had received another text message.
“What the…” I said as I scrolled through the message.
I grabbed the laptop from Mikel and started analyzing the traffic sessions very carefully, starting with the raw traffic stream and stripping out protocols, then service types, and then IP addresses.
“What! What!” Mikel yelled.
“Look at the time - 1815 hours. Look at the originating IP address.” I said as I gave the laptop back to Mikel and pointed out my resulting data set.
“Different IP Address. So what?” Mikel asked in a confused tone.
“Remember the data that I shared with you from CL?” I asked.
“Yes.” Mikel responded.
“Look at the sniffer data again - the MAC address is the same. The IP address is different. He’s rotating the IP address every fifteen minutes!” I exclaimed. I had to laugh inside after I said this. What good is rotating the IP address when you’re in the same subnet? Any anti-spam sysadmin would catch on to this rather quickly if they were reading their logs… well… if they read their logs.
“We need to find guy now!” Mikel responded.
Right after a bathroom break.
Source: archived original
Part IX 路 Nov 6, 2003
It just stopped.
Two minutes after a most-needed bathroom break, and the traffic on the wireless link had gone bye-bye.
“Where is it? Where?” Mikel asked, taking the laptop out of my hands.
“This is just a guess…” I began, “but I’d say they figured out that their spam isn’t going anywhere but the bit bucket. Maybe they are paying attention to their mailer cough spammer cough logs.”
“I am wondering where to go? How to catch him now?” Mikel pondered as he looked up at the ceiling like it was going to give him an answer.
“Let’s leave the gear and call it a night.” I said.
I needed a slice of banana cream pie bad.A few hours later I was enjoying a hard hotel room bed and the satisfaction of a full stomach. The amber ale for desert was a wonderful idea - or so I thought at the time.
RING
RING
RING
My leash rang at 0742 hours the next morning.
“Chief.” I said in possibly my most lathargic tone ever.
“Mikel. I have person in my office that you need to speak with.” Mikel informed me.
“Who is it?” I asked while scraping sleep from my eyes.
“It is network nerd from next door - says ISH people hack his network.” Mikel exclaimed.
“What? What makes him think that your employees are hacking his network?” I asked.
“He says we hijack wireless link and eat up bandwidth with mail. Sounding familiar to you?” Mikel asked, finally getting my attention.
“I’m on my way - get the coffee brewing.” I demanded.
“Sigh.” said Mikel as he hung up.
Now this was interesting. The company next door to ISH is now having quite possibly the same intrusive problems that ISH had. The coincidences in this case were driving me mad.
(Aside)
Through this case, I had kept ISH management involved as the events unfolded. It’s important that you as the investigator keep the client close in these types of circumstances since network intrusions (whether wired or wireless) can have serious legal ramifications.
You’ll notice that I take an extensive amount of notes while I work with timestamps. I advise you to do the same and refer to them often. Often the clue you’re looking for is right under your… well… it’s on your paper.
(/Aside)
BEEP
A text message appeared on my phone. SMTP traffic was being detected again - the originating IP was the same - and guess what time it was - 0800 hours on the dot according to my watch (which was on ISH time now).
Another coincidence?
(Aside)
*** Bonus points to the reader who can pick out the significant clue found in some of the past 8 postings. ***
(/Aside)
Source: archived original
Part X 路 Nov 11, 2003
“You’re killing me, Mikel.” I said as the swill posing as coffee touched my lips for a second time. “What is this, instant Folgers?” I asked.
Apparently Mikel did not understand the importance of good java while working a case.
I had raced down to ISH to meet with Mikel and the network nerd from next door.
“Chief, this is Frank - Frank, this is Chief - a security consultant.” Mikel announced as he made introductions.
“What do we have here? You told Mikel that you have a wireless intrusion going on?” I asked.
“Yes, we installed a wireless network about two weeks ago for executive management to use in their conference rooms. I admit that I just learned about 802.11 networks, but the point is that somebody is sending out mail from inside our network - and the IP is from a DHCP pool assigned to our wireless network.” Frank explained.
“Ok, but where does ISH come into play?” I fired back. “That’s a pretty serious allegation.”
“Well, we’re the only two buildings in this industrial area for about half a block - and I’ve confiscated all of our laptops and it’s still happening. I had my network vendor work with me last night, and he installed a network analyzer on the port that our WAP is plugged into - and the traffic is definately coming in on that wireless WAP.” Frank said. Frank was clearly out of his area of expertise, judging from his hesitant answers.
“Did you capture any of the traffic?” I inquired.
“I think we did - let me call the vendor real quick.” Frank said as he dialed a number on his cellphone.
“Mikel, I have a working theory - but I’m going to need your help to provie it.” I said as we walked into an ISH conference room…Hours later, Mikel, Frank, the ISH IT Director, and I were seated in that very same conference room analyzing the traffic captures from Frank. We were comparing the samples from ISH as well.
“The picture becomes clearer…” I began as I scanned the traffic captures. “The MAC address is the same.”
“So going by what we talked about hours earlier, you suspect the same person is responsible for your intrusion as well as mine?” Frank asked.
“The evidence would suggest that.” I replied.
I tapped my fingers on the conference room table.
“What?” Mikel asked. The silence in the room was deafening.
“Did you get that list I asked you for?” I asked Mikel.
“Yes - I am sorry it has taken days to get that information. I am sure it is on way - I will check.” Mikel said as he left the room.
“What did you ask him for, Chief?” asked the ISH IT Director.
“The ISH computer equipment inventory report. Your hardware group does the job of recording equipment purchasing, issuing and retiring based on my earlier examination. I asked for their entire inventory report when I first arrived.” I said.
“I don’t get it…what are you looking for?” the ISH IT Director asked.
Mikel burst into the conference room at a brisk pace.
“Here you go Chief!” Mikel said as he flung a three-ring binder into my lap.
“Hey, watch the monkey jewels!” I screamed as I took five pounds in the nether-regions of my pants.
I flipped through the book with anticipation.
“Chief? What are you looking for?” the ISH IT Director asked again.
I slowed my search in the book until I found the entry I was looking for.
“CHIEF!!!!” the ISH IT Director asked again. One mystery was solved - I knew who drank all the bad coffee earlier.
“I’m looking for a Compaq Laptop, inventory tag # 0011067 - and your interview notes for it’s owner.” I said.
“Doesn’t ring a bell…” the ISH IT Director began to say.
“I’m not surprised - because you never interviewed the owner of that laptop. " I responded.
“Eh?” the ISH IT Director’s face said.
“Jeremy Chandra” I barked back. “There is no incident interview form for him anywhere in the documents you provided me.”
“But… we interviewed… the entire IT staff… I don’t…” he stuttered.
“You’re overlooking one key fact.” I said, turning the 3-ring binder around on the table and pushing it the director’s way.
“He’s an intern in Marketing.”
(Aside)
WHY is the fact that the owner of this particular notebook an intern in marketing significant at this point?
How did I know that it was a Compaq laptop?
I have left out some fact from the investigation to see if anyone can figure it out. :-)
Does this change any of our theories? Support some existing ones?
Discuss.
sips coffee
(/Aside)
Source: archived original
Part XI 路 Nov 18, 2003
It was a breakdown in policy that was more painful than a root canal without novacaine. Judging from the scowl on the ISH IT Director’s face, I could tell that Dr. Monkey hit a nerve.
The tension in the conference room was getting thick.
“I’m curious why you only interviewed the IT department staff regarding this issue, when the sales and marketing staff are equipped with wireless-capable laptops?” the CFO said as he sat down next to the IT Director. Looks like someone finally read my status e-mail to the executive staff.
“It was a matter of procedure - one that I’ll be sure to correct going forward. So what do we do now Chief?” asked the IT Director, swaying attention away from his policy blundering.
“I want that laptop, and an interview with Jeremy.” I said as I sipped a mouthful of cold coffee.
Shortly after lunch I entered the office of the CFO. Mr. CFO was seated behind his desk, and Jeremy was sitting in a guest chair looking spooked. Mr. CFO’s receptionist closed the door after me and I took a seat directly opposite Jeremy.
“Jeremy, you can call me Chief - I’m a security consultant. I’d like to ask you about your laptop.” I said as I locked eyeballs with Jeremy.
“Uh, what about it? I mean, it’s a laptop.” he replied. If his ass fidgeted anymore in that leather chair, I feared he was going to start a fire. “It wasn’t even at my desk when I came back from lunch. I’m sorry that I left it there, but I didn’t think someone would steal it.”
“I know where your laptop is Jeremy. It’s being taken apart by some associates of mine.” I said with my best poker face.
“Apart? Why? What’s this all about?” Jeremy asked.
I volleyed a bombshell. “Tell me about Mark Wargo.”
Jeremy sat silent.
“Answer him.” bellowed the CFO. “We’re not farting around here.”
“I don’t know any Mark Wargo.” Jeremy said quietly.
“According to several e-mail messages recovered from your ISH mail account, you have a relationship of sorts with Mark.” I stated. “Or should I say, a living arrangement? Isn’t Mark your roommate?” I asked.
“Yes.” Jeremy replied.
“And you are both co-owners of JMSSC, LLC., correct?” I asked - referring to my late-night research on the company that was behind several of the products advertised in the SPAM e-mails.
“Yes.” Jeremy replied again.
“How does JMSSC make money? Give me the 411.” I demanded.
“We do targeted e-mail marketing. It’s legal and an honest living. What’s your point?” Jeremy snapped.
“Do you recognize this?” I asked, tossing a pink invoice that I had recovered from Jeremy’s desk into Jeremy’s lap.
Jeremy scanned the paper, closed his eyes and exhaled loudly.
“Jeremy, according to that invoice, you owe Level-9 hosting about $20,000 in late bandwidth charges.”
Jeremy said nothing.
I proceeded to spend the next twenty minutes describing the research involved in isolating the MAC address of the wireless card and tracking it back to his laptop. Jeremy continued to be unresponsive to my queries, until I ran around third base and headed for home.
“You had an accomplice in planting that WAP, did you not?” I asked.
Jeremy sighed. “Yes. They wanted in on our business. They helped us get around a lot of the problems we had once our colo provider cut us off. "
“What’s his name Jeremy?” I asked, moving in for the kill. Time to see if Jeremy’s confession and the supporting evidence fingered the same person.
“It was …”
(Aside)
The case is wrapping up, and I’m revealing a lot of investigative details that I left out earlier.
Who is the accomplice? What was the evidence that made them suspect?
What would an appropriate punishment be for Jeremy?
BTW, I’m currently on assignment so the blog updates are going to be slow. Sorry y’all!
(/Aside)
Source: archived original
Part XII 路 Nov 25, 2003
Four hours had gone by in the CFO’s office. The game of mental chess that I was playing with Jeremy was getting old fast. Jeremy decided to spill the beans.
“It was Phil.” Jeremy said, staring at the ground.
The CFO winced. “Phil? You mean Phil Gyser?”
“Yes. " Jeremy replied, still staring at the ground.
I flipped through some papers I had placed on the CFO’s desk.
“Let me get this straight - you decide to use this company’s network resources to transmit targeted spam mail, and the damned janitor is your partner?” the CFO said in disbelief.
I placed several printouts in front of the CFO.
“Take a look at this…” I said.“The first printout is what is called a ‘syslog’ or ‘system log’ from the network switch that the unauthorized WAP was plugged into. The WAP was plugged into port 17. According to this syslog, that port went active exactly 10 hours before the first spam transmission took place that was recorded by the CL folks.” I said.
I drew the CFO’s attention to the second piece of paper.
“This is the security badge access log for the same day. If you correlate the times [Ed. All machines were NTP synced at this place - thank goodness.], you’ll notice that Phil’s badge was swiped 4 minutes before the switch port went online. No other access to that room is recorded for another two hours. Phil was nice enough to sign himself out of the datacenter as well - according to the activity log posted by the door.” I explained.
“Chief, I don’t understand - how did Phil know how to hook this stuff up? He’s not a computer guy at all.” the CFO asked.
“According to his personnel file, Phil’s last job was at DataTron Semiconducter - as a systems administrator. Apparently the downturn in the tech job market prompted Phil to seek another means of earning an income.” I said as I passed the CFO a copy of Phil’s personnel record.
“Ok, so Jeremy and Phil hooked up, got the WAP installed - who’s talking to the WAP? I don’t understand this.” the CFO said.
I was interrupted at this point by the IT Director and Mikel entering the office with a dishoveled looking guy in tow.
“I’d like you all to meet my new friend Mark.” the IT director said as he asked Mark to have a seat.
“He was sitting in car with laptop by door.” Mikel said, beaming from ear to ear.
I glanced down at my watch and started to laugh.
It was 1805 hours.
So much for the punctual drive-by spammers.
(Aside) It took twelve installments, but I think I covered most of the important points of this case. This is one of the longest investigations I’ve ever had to do. Most of my time was spent chasing after ISH employees to get me records, logs, interviews, etc. Organization is key to a successful investigation. The fact that all of ISH’s servers and equipment were logging to a central logging server and all the machines and network equipment were ntp-synced helped tremendously. Without the NTP syncing, correlating times would have been a lot more work.
Jeremy and Mark had no idea that they had accidentally used a neighbor’s network to start transmitting spam. When I asked them about this, they pleaded complete ignorance. While Jeremy and Mark had some decent tech skills (Phil wasn’t too bad either), they lacked depth and experience.
The custom scripts and software that they had written had numerous problems. They ended up spamming the same group of email addresses over and over again rather than using the new email lists that they had acquired through various means. If they hadn’t concentrated on the same addresses over and over, CL might have not called me - and I might have not caught them.
Jeremy was fired that very day. Phil was fired about an hour after Jeremy. ISH threatened them all with legal action, however under the advice of legal counsel the entire issue was dropped.
You win some, you lose some…
(/Aside)
Until next time.
~ SecMonk
Source: archived original