Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.

Part I · Jul 19, 2004

New Format Announcement From The Chief:

I have decided to do smaller, more frequent updates to the case files. You’ve all been very vocal about this, so I’ll do my best to comply. Enjoy! ** ~ Chief**

I hate planes. I hate flying. Unfortunately, it’s required to get around in today’s world. The worst part has to be the coffee.

“Where do they find this stuff, the back shelf at Wal Mart?” I joked to myself.

It was a long flight back home after a well deserved vacation, and I was looking forward to sleeping in my own bed and the familiar aroma of quality java beans being ground at exactly 05:00 hours. Yum.

I was off the plane a mere three minutes when I turned my cell phone on to check my messages.

“Twenty Nine new voice mails? UGH.” I moaned as I began the daunting process of listen-and-delete. The last voice mail was left a few hours before, and caught my interest.

It was a referral call from a large financial clearinghouse in town. They had a confidential matter to discuss with me.

“God help me if these people just want to refinance my house.” I said as I closed my phone.

I would return their call first thing in the morning. Five o’clock came way too fast the next day. I had to spend the next few hours getting caught up on e-mails and snail mail. The coffee went down smooth and I could feel it activating every neuron in my monkey-like brain.

“Time to give BigHouse Financial a call back.” I thought. I pounded their number into my cell phone.

“BigHouse Financial, how may I direct your call?” said the perky voice on the other end of the phone.

“Michael Cash, please.” I said with a straight face. This receptionist was straight out of the movie “Valley Girl”.

“This is Michael.” a familiar-sounding voice said.

“Michael, it’s Chief - returning your call. What’s up?” I asked.

“Thanks for calling back Chief - I have a sensitive issue brewing here at BigHouse and I’m not quite sure how to handle it.” Michael said slowly. He was thinking about each word he said so loudly that my ears were bleeding.

“What kind of ‘issue’?” I asked.

“It’s best that we don’t discuss this on the… phone. When can you come in?” Michael asked.

“I’m jumping into the car as we speak.” I replied back.

Somebody sounded very paranoid.

Source: archived original

Part II · Jul 21, 2004

BigHouse Financial occupied a large glass-faced building in the ritzier part of town. The building was less than a year old, and hosted one of the most technically innovative physical security systems around. I had read about this building while it was in the planning, building, and completion phases in a lot of trade magazines.

“Now’s my chance to see the goods.” I thought as I checked in with the guard at the parking garage. He took my driver’s license, scanned it into a machine, printed and laminated an id badge for me and my vehicle and assigned me a parking space - next to the elevators even!

I exited my vehicle and looked around the parking garage - I was being watched by numerous cameras mounted around the ceiling. As I scanned my new badge to enter the elevator, I was surprised to see tah dah… yet another camera in the elevator.

“This is right out of movie.” I said outloud. I hadn’t seen security this tight since touring a few casinos in Las Vegas.

I walked accross a bridge that extended from the second floor of the parking garage to the BigHouse Financial office building. I scanned my badge again at the door to the building and walked into a large reception area, complete with perky receptionist…

“Hello! How may I help you?” the receptionist said without blinking an eye.

“I have an appointment with Mike Cash.” I said looking around the reception area. A camera was overhead, and one on the ceiling overtop each of the intersecting hallways nearby.

“Please have a seat, Mike will be right with you.” she said as she motioned me to sit on a very comfy brown leather sofa.

I was reading a fascinating article about Michael Jackson and his fascination with monkeys when Mike approached me.

“Chief! Good to see you… let’s take a walk to a conference room.” Mike said as badged us into one of the nearby rooms.

Mike closed the door behind us, asked me to sit down and proceeded to pour me a cup of coffee from a nearby coffee pot.

“What’s up Mike? I talked to the person that referred you to me, and he didn’t have a clue about your ‘issue’.” I said as I sipped the hot coffee slowly.

“Chief, we spent a lot of money making this one of the most secure buildings around. We process thousands upon thousands of sensitive financial transactions a day. We keep several terabytes of financial documents online and accessible to our service staff. We have a large security staff, and we pass every security audit that we go through. That’s what makes this especially difficult.” Mike lamented.

“I don’t understand, makes what difficult?” I asked, sipping more coffee.

“We have a mole.” Mike said, folding his arms in front of him.

“This mole has somehow figured out how to…”

To Be Continued…

Source: archived original

Part III · Jul 27, 2004

“To what?” I asked, annoyed that Mike wasn’t completing his sentences properly.

Mike smiled. “Well, someone has figured out how to get a large amount of our customers’ data out of this building.”

“Fill me in.” I said, intrigued.

“Well, this has been going on for about two months now. You need to understand that there is no external access to this network, aside from private network connections to clearinghouses. No internet access, no analog phone lines laying about… none of the obvious electronic stuff. Everyone is searched coming and going from this building. We’re stumped.” Mike said, offering me a warmup on my coffee.

“I have to say I particularly enjoyed the brief search that the guard gave me.” I joked. I was wanded for metal objects - that was about it. Of course, I brought nothing into the office aside from my trusty pen and notepad…

“You have this place under pretty heavy CCT surveillance man - cameras are everywhere.” I said. “And I’m guessing that you have an adequate internal audit department that is analyzing employee access to systems? Nothing out of the ordinary?”

“You are correct - and no, we’ve found nothing. We first caught on to the information leakage when ten people over a four week period were the victims of identity theft. The only thing these people really had in common was that their identities were used around the same time, and they all had a major financial transaction go through our shop at some point.” Mike pointed out as he flipped through some drink coasters on the conference room table.

I raised a finger to my bottom lip, thought for a moment and said “So your assumption is that you have someone lifting documents in some form or fashion from your company and using the information themselves…”

“Or selling the information to someone else perhaps.” Mike responded, a smile crossing his face.

“That’s quite common nowadays.” I commented, taking a few moments to collect my thoughts. “I think I need to sign those NDA’s you talked about, and then pour over these ten individuals’ transactions.”

“Let’s go to my office - I have everything ready.” Mike said as he ushered me out of the conference room and into an elevator.

A few minutes later, I was signing my John Hancock to some legalese and glancing over what appeared to be papers associated with home mortgages.

“Woah.” I said out loud as I dropped my pen.

“Whoa what?” Mike asked.

“Mike, all of these people have yearly incomes that exceed half a million dollars.” I said.

“Yes, that too is quite common - we see them all the time.” Mike pointed out as he flipped through the papers.

“Yeah, but are all your customers self-employed?” I asked.

“What?” Mike spouted as he took the pile of papers from my hands.

“Someone is profiling your customers… looking for specific characteristics… something that works to their advantage.” I said as I sat up in my chair.

“Well, we somehow missed that… but how does this help us? Where are you going with this?” Mike inquired as he stared at me in disbelief.

“Honeynuts.” I responded.

Time to bait the line and see what bites.

Source: archived original

Part IV · Jul 29, 2004

“Breakfast cereal?” Mike asked as he stared at me with disbelief. Mike looked more like a ‘Wheaties’ kind of guy, so I’m sure that I had just offended him.

“No - think of it like this. What happens when you go out to the park and leave a trail of breadcrumbs from the bottom of a tree to the bench you’re sitting on?” I asked Mike with a grin.

“You’re arrested for littering?” Mike said sarcastically.

“The pigeons flock to the breadcrumbs, and eat every one… working their way up to the last breadcrumb which is right in front of your feet.” I said, downing the last of my cold coffee…“So you propose a trap - ok, what are the breadcrumbs? How do we pull this off?” Mike said with a raised eyebrow. Spock would have been proud.

“I’m going with the assumption that your thief is doing customized searches against certain fields in these records. We’re going to plant some fake records out there that nobody would ever access in their normal course of work.” I spouted as I tapped Mike’s desk with my pen.

“Ah, and since we don’t trap searches - but we DO trap record access - the pigeon would be walking right to our feet?” Mike asked, blinding me with the lightbulb that suddenly appeared over his head.

“Exactly. I need you to hand pick a small team of people with skills in these areas.” I said as I scribbled some monkey jibberish on a piece of paper. “We’re going to have a briefing in the morning at 0600 hours before most of your staff arrives. Stealth is the key here - and you want to make sure that these are people that you would trust with your life. We’re not going to get too many shots at this.” I said as I stood up to adjust the clue banana hanging proudly from my belt.

“You got it Chief - so when do you think we’ll see the pigeon - or possibly pigeons?” Mike asked as he shook my outstretched hand.

“Soon.” I said.

Very soon.

Source: archived original

Part V · Aug 3, 2004

The clock on the wall read 06:05 hours. The conference room was a bit stuffy from the air conditioning being turned off over night for maintenance. I was sipping the last of a beautiful french roast, waiting for Mike to assemble the crew I asked for.

“So much for starting on time.” I mumbled as I released a rather loud yawn.

Monkey business is tiring business, no two ways about it.

A few minutes later Mike entered the conference room with three individuals. Database guy was the architect of the system suspected of being used to steal the information. Data security guy was third in charge of security and a former network monkey - very clued individual. Systems specialist girl was an expert in the applications used in house to access the database in question. But why these three people? Some explanation is necessary. First, I wanted someone who was a key individual and very low-level to the database. We’d need knowledge of how the database was composed and what low-level solutions were available in a pinch.

Data security guy was a grunt that had access to the security infrastructure, but also had a networking background to help us with our search endeavors.

System specialist girl was going to help us in figuring out each nook and cranny of the document-retrieval system that the users had access to.

The plan was simple. Plant the nuts, post a sentry and set some alarms. When the alarms were triggered, we wanted to do a capture-and-trace of the network traffic between the workstation and the application server, and perform as much physical surveillance as possible in the shortest amount of time. This is where data security guy was going to come in real handy - he had access to the entire security infrastructure - net-powered cameras and all.

I discussed the plan in great lengths and received some feedback from the team. As usual, there were surprises involved.

“Capture record access? Hell yes we can do that. Who told you we can’t do that?” database guy asked Mike. Database guy was also an old Unix hack, and had some great tricks up his sleeve.

“I recommend we modify my debug scripts that I use for testing to look for database requests containing the bait names.” database guy said. “One script in particular would be perfect - and it runs in real time. We can setup an email alert too.”

“That’s great, but all the transaction requests come from the application server - they all appear to be from the application server. We need a way to correlate when the search happens against the database with the user logged into the application and executing the search. Unfortunately, there is no such facility in the software.” systems specialist said with a frown. “The best we could do would be to catch the alarm from the database, and then immediately perform a manual search on the application server for the request - which could take a while.”

“And what about the network traffic capture? This is all web-based, and it’s not SSL-encrypted for whatever reason.” data security dude pointed out with a grin. “If we want hard proof of the request, we’ll need to setup an http sniffer box to monitor the switch port that the application server is plugged into.”

I was grinning from ear to ear - I was making them think!

“Ok, let’s talk about surveillance. Once we identify the ip of the workstation, we’ll need to know where the workstation is located. We have a network map showing each workstation, right?” I asked.

“Well, we’d need to query the IP for the hostname since we’re running DHCP here - but once we have the hostname it’s a snap!” data security guy proudly announced.

“Database guy, let’s get those records in the system and that script running.” I suggested as I stood up from the table.

The game was on.

Source: archived original

Part VI · Aug 10, 2004

Fish are a real pain in the arse. You’re sitting there in a boat with your fishing pole extended, bait in the water… waiting… waiting… and those little suckers ignore you - almost as if they can smell you.

“Well, they wouldn’t get near me right now - I smell like I want to be alone.” I joked to myself.

Three days had gone by since our tiger team had jumped into action, and not even a nibble.

This isn’t entirely unusual. In data collection cases such as this, the offender typically works in two fashions:

Slow and easy - records are accessed and copied one at a time over a very long period of time. Fast and quick - records are accessed and copied in bulk using the fastest method available, hoping to ride in between security checks and not get caught.

My stomach was grumbling - it was getting very close to lunch time - I couldn’t fathom another day in the company cafeteria…. One half an hour later, I was enjoying an amazing bowl of japanese food with green tea.

Note: I should tell you all now that the Chief’s doctor had advised him to cut back on coffee consumption and switch to tea. Yes, it was a sad day.

ring

“Chief.” I said holding my cell phone with one hand and my chopsticks with the other.

“Chief, it’s database guy. I have a hit! Actually, I have three hits! We’re rolling up the incident now… you should probably get back here asap.”

I almost choked on my rice bowl. What’s that saying, “a watched pot never boils?”

I drove back to the building and breezed through the security checkpoints. I entered an office with a swipe of my badge - an office that we converted into a makeshift “watch room”.

“Ok, I got the queries right here… see.. right here!” database guy was almost screaming with glee. I really just wanted to slap him at this point.

“Have we run through the application logs yet, anything?” I asked systems specialist girl.

“I’m rolling through them now… I have several hundreds of thousands of lines of logs to search through. Shouldn’t be much more than a few minutes longer.” she said, her eyes not leaving her laptop screen.

The tension in the room was mounting, and data security guy looked like he was about to explode - foot tapping, nervously sipping a Coke. It wasn’t warm in the room, yet he was sweating profusely. I hope he remembered to be “sure” this morning.

“Got it! I got it!” systems specialist girl said as she swung around in her chair. “Data security dude, bring up a camera on workstation 5B003.”

“5B003? Where’s that?” I asked.

“You wouldn’t believe me if I told you.” data security dude said as he manipulated the security cameras on the workstation in front of him.

“Try me.” I said.

“That’s on the fifth floor, section Bravo (north wing) in cube #3. That cube belongs to my partner Maria.” data security dude said, more flustered than five minutes ago.

Data security dude brought up the camera, engaged the zoom mode and found the workstation.

“There’s only one problem here, Chief.” data security dude said.

“Eh?” I asked.

“That’s not Maria at the computer.” data security dude replied.

The room was dead silent.

Source: archived original

Part VII · Aug 21, 2004

“Somebody want to tell me who this person is?” I asked as the camera zoomed in closer to the woman sitting at Maria’s workstation.

“Darla Dominique.” Data Security Dude replied, lost in a blank stare as he moved the camera around.

“Darla used to work in Data Security, but was moved to an analyst position within the IT department because of a political issue with the Director of our department.” Data Security Dude said.

“What issue would that be?” I asked as I stepped closer to the LCD screen, watching her every movement.

“He actually expected Darla to be at work and do her job. Darla is in the middle of a nasty love triangle from the last information I heard, and wasn’t making her job a priority. The Director didn’t want her in a sensitive position, so he gave her a lateral promotion to somewhere else in IT.” Data Security Dude elaborated as he began to stand up.

“What’s the plan, Chief?” Systems Specialist Girl asked as she closed the lid on a nearby laptop.

“We watch and learn.” I said as I sat down in Data Security Dude’s chair. The next ten minutes went by very slowly as we tracked and documented Darla’s moves. I was most curious about how she was getting the information in such large quantities out of the building.

“She doing her criteria queries, getting the output through the document application, then what is she doing after that? It’s hard to see.” Database Guy said as he nudged closer to the LCD screen.

“The document images that she’s seeing - what format do they appear as in the browser?” I asked Systems Specialist Girl.

“They are converted on the fly from PDF to JPG - they show up in the browser as a JPG.” she said, holding her chin in one hand and scratching her head with the other.

“She’s dragging and dropping something… look at that… one window into another window… see that? And the image moved up, then down…” I pointed out. My eyes were really beginning to hurt.

“She’s getting up!” Data Security Dude said as he tried to find the ID of the camera she was approaching.

By the time the other camera was located and adjusted, Darla was already walking towards the elevator.

“Where does she normally sit?” I asked.

“She’s on… the third floor… in the corner.. cube 12 it looks like. Let me find the camera…” Data Security Dude said as flipped through the camera software.

“Got it!” he said as he zoomed in. Darla had made it back to her desk and appeared busy with some menial task.

“What the hell is that - a three hole punch? What the hell is she doing?” Database Guy asked.

I started laughing - hard. I had to sit down.

“What’s so funny Chief?” Data Security Dude asked.

“Get your facilities security team on the phone - and get Mike up here now. We need to grab her before she heads home.”

“But why are you laughing?” Systems Specialist Girl asked.

“Because, folks, the joke is on us.” I said.

I needed coffee.

Source: archived original

Part VIII · Sep 1, 2004

“Medium-roasted beans, oh the pleasure….” I thought as I leisurely sipped 24 ounces of the creamy concoction in Mike’s office. “I’m craving Oatmeal Cookies. That’s weird.” I thought to myself.

“Uh, Chief?” Mike asked, garnering for my attention.

“Huh?” I replied, snapping back to my senses.

“Sorry, you were lost there - tired?” Mike asked.

“You could say that - what time is Darla’s shift over?” I inquired, glancing at my monkey watch.

“About 30 minutes to go. We should probably head down to the security desk.” Mike suggested as he stood up behind his desk.

“Let’s do it.” I said, slamming the rest of my coffee in a gulp. It’s a talent, I tell you. A quick ride down the “secure” elevator and we were at the security desk,watching the clock tick.

A few minutes later, emloyees started to pour down the elevators and stairs, heading past the security desk and badging out the exit doors.

“There she is.” Mike said, walking towards Darla - security guards in tow.

“Darla, could we have a few minutes?” Mike asked as he motioned towards the door of the security office.

“Uh, ok I guess. Is something wrong?” Darla asked, looking around nervously.

“We just need to ask you a few questions.” Mike said, spouting out a line that has been in way too many detective movies.

Mike, Darla, and a security guard joined me in the security office. It was a bit crampt, but adequate.

“Darla, as you know we have a company policy on inspecting items leaving the company for security purposes.’ Mike said.

“Uh, yes. I know that.” Darla said, staring at her feet.

“Mind if I take a look at your binder?” I asked, reaching for the 1-inch thick blue plastic book.

“Uh, ok.” Darla reluctantly agreed.

Mike peered over my shoulder as I opened the binder.

We weren’t looking at meeting notes, recipes, or software instructions.

On each formatted page was a neatly printed graphic image of a document - the first page of what appeared to be some type of financial application. I could clearly see fields for name, address, phone number, date of birth, social security number, etc.

I flipped through the binder and realized there were approximately 30 different prints.

Darla was staring at the floor.

Under questioning, the following information was established:

• Darla had become angry at her Director for ‘banishing’ her from the data security department.
• Shortly before she left data security, she had installed a key-logging application on a few workstations.
• Darla knew the value of the information that was in the binder.
• Darla was selling the information to a ‘buyer’ that she met online. $100 per usable identity. Darla was making an extra $3000-5000 a month in cash!

Based on this information, I made several recommendations to Mike & crew:

1. They spent a lot of money on facility and data security - however they had their ‘clocks cleaned’ by that one random element: the user. Policies were examined and changed.
2. They needed to tighten up their user access to systems and applications - there is no reason that data security personnel needed administrator access to all systems. Minimum necessary access is the key.
3. They needed to monitor for keylogging software better, especially on ‘secure’ workstations. We also located about 280 workstations that had some nasty malware/adware/spyware infections. Windows, windows, windows… sigh
4. They needed to monitor print server logs. I found that they only kept about 24 hours of print logs before the log files were erased. Not good.
5. They needed better due diligence when it came to employee hires/transfers/terminations. After auditing about three months of HR activity, I found terminated/transfered users that still had access to sensitive systems.
6. Password policies were not consistent accross systems and applications. Some sensitive applications would accept a one character password! Eek!

Mike followed up with me several weeks later regarding Darla. Apparently Darla’s case was referred to a federal agency who sought her cooperation in nailing her buyer.

Let’s hope they got him.

Source: archived original