Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.

Part I · Aug 1, 2003

“Chief, you want anything from the Coffee Shop? We’re making a run!” Mike announced in the doorway of my office.

“Do you have to ask? Coffee of the day - hazelnut flavoring - and make sure you get a cup condom - I don’t want to burn my hands on their cups.” I replied, half laughing at myself for using the word ‘condom’ in regards to coffee.

The mood in the office had been quite somber in the past week. The date? September 18, 2001.

Fear. Anger. Outrage. These feelings raced through my system like a bad burrito with sour cream. Only a week had passed since the worst terror attack the United States had ever witnessed.

The glow of the Sun CRT on my desk lit up my spectacles as I browsed through CNN.COM.

“Bad news…. bad news… bad news… oh look, someone rescued a kitty from a well. How refreshing!” I said to myself, waiting for that cup of coffee that was taking way too long to get to my desk.

ring

Glancing at my office phone, I saw that one of the fraud investigators was calling. Being the ISO at a small telecom was an interesting experience to say the least.

“I hope this isn’t about my last expense report.” I mused as I picked up the phone.

“Chief? It’s John. Can I run something by you? " he asked.

“Sure, as long as you bring me coffee.” I replied….“Okay, I’ve been working fraud for five years, and I can’t make heads or tails of this!” John exclaimed as he slammed two large binders of dead tree on my desk.

“We have a user of our teleconferencing service who ran up $4000US in charges. His credit card was denied. I called the user, they pleaded ignorance and gave me another card number. That was denied as well, and now the user has disappeared! We can’t contact him at the numbers he provided us.” John said.

“You called the card issuers, right?” I said, cutting him off. “Stolen cards?”

“Good guess. So I had services flag the account for non-payment and it’s been deactivated.” John said as he took a sip out of MY coffee that he had brought ME.

“I’m not hearing anything new here John. What’s got your goat?” I asked, watching him drink more of MY coffee.

“Oh yeah… sorry. Well I had my two junior analysts start cross-referencing all the teleconference connections, and… well… we can’t find them.” John replied nervously. “It’s like they never existed, yet they ended up in billing!”

“What? The system doesn’t allow that to happen.” I spouted back. “Are you sure your monkeys aren’t eating fermented bananas over there?”

“Checked it myself. Help!” John pleaded as he went back to his desk.

“Hmmmm. Missing references in the calls database. Stolen credit cards. Disappearing users. Guess I’d better order in lunch today.” I remarked to myself as a I poured over the thousands of pages of information now occupying my desk.

I was intrigued. Our system was designed to record every call into our distribution system, and every call from an outside system that joined our teleconferencing system. The calls were referenced by a CID (Caller Identification Number) that is composed of the calling phone number, network id, etc. It’s essentially a big long string of characters like a VIN (Vehicle Identification Number) where every letter and number means something. Yet, these teleconferences had ‘ghost’ connections - no CID.

“Where are the ghostbusters when you need them?” I muttered angrily, wondering where the hell my Coffee Shop coffee was.

A few hours had gone by, no coffee, but lunch showed up in the form of a delicious sliced beef sandwich with au jous. Then I saw it.

“Holy crap!” I said to myself as my sandwich fell out of my mouth. I grabbed my phone and hit the ‘quick-dial’ button for telecom operations.

After two quick rings I was greeted with “TELOPS. Frank speaking.”

“Frank, Chief here. I need you to assemble the system developers, the database guy and anyone else that knows how the routing system works NOW. Conference room 2 in 30 minutes.” I barked into the phone.

“10-4” Frank replied.

Time for an exorcism.

Source: archived original

Part II · Aug 4, 2003

“Why is it so cold in here?” I muttered, looking around Conference Room #2 for a thermostat. I was half-tempted to start cutting a hole in the floor and start ice fishing.

One by one the critical members of the TELOPS development team scampered into the conference room. Nobody likes to be ushered away from real work to an impromptu meeting, but these guys expect last-minute meetings from me - the evil security guy.

“Uh-oh. Chief has a live one.” joked Pitre, noticing the stacks and stacks of dead tree surrounding my position at the head of the conference table. “And me without my PDA!”

“Quiet you!” I joked back while motioning for Mr. SmartySwede to close the door behind him.

I dimmed the lights in the conference room, unlocked the screensaver on my linux presentation laptop, and welcomed the team with slide #1:

“The system is haunted”…I quickly brought the team up to speed on my phantom records.

“The real kicker isn’t what you’re seeing on the paper, but what’s NOT on the paper. If you’ll examine the teleconference logs, you’ll notice that not only is the CID missing, but the OCN (Originating Connection Network) is zero.” I said.

“Chief, we don’t have an OCN with an ID of zero - that must be a table-level error in the database.” Pitre speculated.

“Wrong - I’m querying the database now, and the rows for these records do indeed have zeros in them.” announced Andre, our resident DBA lord as he glanced over the scrolling text on his laptop.

“Then there’s an error in the population application - we’re not reading the OCN correctly at the point of hand-shake.” spouted Phil, one of the application developers.

“Can you verify your code?” I asked Phil while generously offering Phil the laptop of Andre.

“I’m only one SSH session away from answering that question.” joked Phil as he typed fevorishly on the laptop.

A few minutes of pondering and questions had gone by when Phil announced “Well, there goes that theory - CVS shows no changes to the production code in 3 weeks, and I just walked a connection through the entire application - flawless.”

“Let’s look at where we’re at here. The record of the teleconference IS hitting billing, so we know that process is working. Billing is reporting what it’s handed - so something is amiss in the telops infrastructure.” said Andre as he clawed at Phil to return his prized notebook.

I sat for what seemed an eternity listening to the team volley ideas, speculations and accusations around the table. In between two sips of cold coffee, I turned to Phil and injected my thoughts into his conversation. “Phil - in the code snippet that you walked me through, you’re more or less doing a case-by-case ID assignment based on OCN, correct?”

“Yep.” Phil replied. “Why?”

“And those OCN’s are fed by the provider..” I trailed off.

“Yes, yes…” Phil nodded his head.

“But we’re counting on the provider to do some homework while making the connections - and if they fell down while doing their accounting…” I trailed off yet again, lost in thought.

“We’d still see the handoff phone number!” Phil said, glancing around the table at befuddled team members.

“Not if the connection is another… teleconference!” I blurted out in mid-thought.

“Huh?” Phil asked. “You’re saying…”

I stood up in front of the table and rolled a fifty-page diagnostic from one of the calls out onto the conference room table.

“The bastards are conference jumping - and they’ve just found a hole in our system that didn’t show up because nobody is looking for this sort of thing - until the next billing cycle comes up - and then it takes 30 days for fraud to be notified. Sixty days of anonymous connections with the system!” I exclaimed while shuffling around more dead trees, trying to find the phone number of the ISO at one of our largest providers.

As I hit the speakerphone button on the phone on the table and dialed his number, my heart was racing because I already knew what he was going to tell me.

He had ghosts on his network too.

Source: archived original

Part III · Aug 5, 2003

Spending all day on the phone can have some strange psychological effects on a person. For instance, you sit down at a restaurant afterwards and give your order to the little shiny box in the center of the table. Then you realize “Hey, that’s a napkin holder! Not a speakerphone!” The glaring eyes of the people sitting around you is rather amusing though. Like THEY have never talked to a napkin holder before.

Our conversations with our providers’ ISOs (Information Security Officers) were very productive. It would seem that all of the providing networks which feed teleconference callers into our system had the same ‘ghost’ records in their billing systems. Most weren’t aware of the problem until our phone call.

One particular provider was especially helpful, and I had worked with their ISO on a number of occasions relating to inter-network security. AJ was an all-around cool guy, former black hat-turned-white hat and pretty darn funny. We decided that we’d work together as a team on this investigation.

We started off backtracking connections to the teleconference systems call by call. It was a very slow, painful process that took several hours. The clock was flirting with midnight when AJ called my office.

“Chief - you’ll never guess what I did.” AJ asked.

“Fell asleep at your desk and were woken up by that annoying beep that the spacebar makes when your head slams into it?” I replied in my most groggy kermit-the-frog sounding voice.

“Nice try, but no. I found an originating phone number into a teleconference that started on my network and jumped to yours!”

I almost dropped the phone.AJ and I were both exhausted, so we agreed to call it a night and start again in the morning.

One pot of coffee and 6 hours later I was back in the office and my mind was racing. Almost on cue my phone rang - it was AJ.

“We ran a records trace on this number last night.” AJ said. “You asked me for the last three months of call data, and boy do I have a present for you!”

“How much data are we talking?” I asked apprehensively.

“I just dropped you a PGP-encrypted email with a document attached. This guy was BUSY!” AJ laughed.

“What do you mean?” I queried as I popped the passphrase into my mail client. “Busy as in ordering undies from Macy’s or what?”

“Busy as in calling all over the damn world!” AJ said in a now-increasing southern accent.

I examined the document AJ sent, and made notations on countries, times, dates… trying to wrap my mind around what this guy was doing with our systems. The caller was dialing locations in England, India, Pakistan, Afghanistan, Iraq, Kuwait, Germany… but the calls were short - under a minute. And then the same teleconference network received inbound calls from those same countries within minutes.

“And Chief, it gets even more interesting. Flip to the last page of the report.” AJ said.

I quickly flipped to the back page marked “Reverse Number Lookup”.

“Ah, cool. You found the location that this number belongs to!” I exclaimed.

As I read through the detail dump, the report almost fell out of my hands.

“AJ, this address is 2 blocks from where the world trade center stood! " I blurted out.

The coincidence was astounding.

Time to call the FBI.

Source: archived original

Part IV · Aug 6, 2003

“Um, I’m new to this region - how do I get to your office again?” asked Special Agent Jones. He had called me on my office number from a pay phone.

After giving SA Jones driving directions in excruciating detail, I hung up the phone and alerted my secretary that we’d be having a visitor shortly.

“No wonder they never found Jimmy Hoffa” I cackled as I placed the finishing touches on the three-ring-binder holding all of the information that SA Jones requested. I burned a copy of all documents, reports, and files to a cd-rom for their convenience. I imagined that the FBI would have an easier time grepping through a million lines of information in digital form than paper.

I carried the binder and CD into the office of the company Attorney, alerting him that the FBI was going to present us with a warrant for the information. The look of relief on his face was a sight to behold - if you were into bulldogs I guess.

buzz

The intercom in my office crackled loudly.

“Chief, you have a visitor from the FBI waiting in the lobby.” announced the receptionist.

“On my way.” I said to myself as I broke into a quick trot down the hallway.SA Jones looked a little frazzled sitting in the lobby. He was outfitted in the standard “look at me I’m a federal agent” outfit. Short-sleeved dress shirt, ‘outback’ vest, and tan SWAT pants. His badge and gun clung to his waistline like a couple of Krisy Kreme doughnuts.

We settled down in a conference room close to the entrance, and I presented SA Jones with a fresh cup of coffee.

“Any flavored creamer?” he asked.

“I like this guy already!” I mused.

SA Jones presented me with a warrant for the information, which I glanced over and gave to the receptionist. She walked the warrant to the Company Attorney’s office.

“I have to say Chief, this is impressive” SA Jones said as he flipped through the book. “You wouldn’t believe the type of evidence that people give us - we usually need a team of SA’s just to organize it into something understandable.”

“We aim to please.” I joked.

“We’ll contact you if we need more information or clarification of what you’ve given us. You might want to give me your SSN just in case you move on to another company. We have ways of finding you.” SA Jones said with a half-smile on his face.

I escorted SA Jones out of the building, and returned to my desk. I knew that I should be feeling happy that I was able to assist the FBI in an investigation, but I was frustrated… frustrated that my contribution ended with me passing SA Jones the book.

Two weeks later, I was contacted by SA Jones. They had some ‘hits’ on some of the phone numbers that were in our notes, but that was all that he could tell me.

(fast forward to today)

Sitting in my living room at home and watching CNN I see that more suspected terrorists were found around the world.

I always sit and wonder… did the information that I helped discover have anything to do with their capture? Did a phone number lead them to a person who knew something about somebody’s half-sister that was involved with a brother of someone who likes to blow up innocent people?

The great thing about information security is that you never know what you’ll find while looking for the truth… like finding out an employee is selling confidential information to the black market for extra spending cash.

Special Agent Monkey is on the case.

Source: archived original