Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.
Part I · Sep 10, 2004
Two O’clock in the morning, and I wasn’t even close to falling asleep. My eyes were peeled, aimed towards the LCD screen of my trusty Apple Powerbook laptop…
“Ten seconds… 7 seconds… CLICK!” I screamed as I mashed the mouse button down in a fit of pent-up jealousy.
“High Bidder: MadClown007? NOOOOOOOOOOOOOOOOOO” I howled. I had been outbid on the mother of all coffee bean grinders by a frick’in clown.
“It just wasn’t meant to be.” I said as I closed the lid of the powerbook, and watched it instantly go to ‘sleep’ mode.
All of the eBay excitement prompted me to brew up a fresh pot of Starbucks house roast and pretty much write off sleep for the night. I sat down in the living room and stared at the TV screen, longing for sleep to take it’s toll on my over-caffeinated brain.
I gave up on the Discovery channel and cracked open the Powerbook, this time wandering around through email.
“Now that’s interesting…” I said to myself. “Why would this guy be emailing ME?….“It was an e-mail from a sysadmin-type that I had worked with at a company a few years ago. He apparently had escaped from that company unscathed and was pimping himself out as a Unix systems administrator for a local company that was known for selling ad circulation services to various websites - most of them “adult entertainment” sites.
The interesting parts of the e-mail consisted of:
_ …I just don’t get it Chief, somehow someone is changing the destinations of our ad-clicks. There’s only about 90 of us here at the office, and very few of us have access to the database. I’ve checked every log file I could find, and I just don’t see it…
…This is costing us several thousands of dollars a week in lost revenue. Whoever is doing this is sending all our would-be clickers to other sites…
… essentially they are stealing our potentional customers - the eyeballs and mouse clicks that help us pay our bills…
_
“Fascinating” I said to myself. Glancing through iCal, I noticed that I didn’t have a gig lined up for another week.
The Chief is exhausted, but he’s on the case.
Source: archived original
Part II · Sep 11, 2004
(Aside) Helping companies whose sole purpose in existing is to throw advertising in our faces is a moral dilemma. I thought long and hard before agreeing to accept this case, even though in ‘Part I’ I seemed to have no reservations.
Staying impartial is hard sometimes, especially if you want to keep a steady caseload. Imagine if you were asked by a SPAM-spewing company to come assist them with a security issue. Would you accept the case? For your usual fee? For an increased fee to ease your moral pains?
These type of philosophical arguments aren’t thrown around by many security professionals these days. It would appear that the almighty dollar cures most people of their hesitations.
So why did I accept this case? As a favor to an individual that you’re about to meet. (/Aside)
Read on…I was wiping sleep from my eyes as I drove down the freeway, heading towards my new gig. The sun had been up for a while, but not as long as me. My car was filled with the aroma of freshly brewed Peruvian beans, and my head was full of curiosity - just who decides to redirect ad clicks on the fly? What’s the motive? What’s the method? I had lots of questions (as usual) and little answers.
AddaBill Systems didn’t look like much from the parking lot, but the face of the building was quite deceiving. As I opened the wooden double doors and stepped inside, I was immediately struck by the innovative architecture and technical prowess of the reception area. Large LCD panels occupied the walls. Their screens displayed various web pages, all marred by banner advertising - even flash animations that begged the viewer to “click and win!”
I inhaled deeply as I approached the receptionist, and was immediately struck by an all too familiar aroma. Suits. The placed reeked of corporate Americana.
“Hello! How can I help you?” asked the receptionist as she put three callers on hold with the familiar “Can you hold? Thank you. Can you hold? Thank you…”
“I’m here to see Scrappy.” I said.
“Excuse me? Scrappy?” she said with a blink of heavily-mascara-applied eyes.
“Oh, sorry - Fred. Fred Stoneflint.” I replied and feeling rather embarrassed. “Scrappy” was Fred’s nickname and online moniker.
“One moment please and I’ll page him. Have a seat.” She said as she motioned me to sit down on a very swank purple couch.
I whipped open my PDA and made some notes while waiting for Fred. I briefly brought up Ministumbler to see if any wireless networking goodness was available. One network appeared, with the token 128-bit WEP encryption.
“Bedrock” I mumbled, noticing the ESSID. “How quaint.”
“Yo, Chief!” yelled a familiar voice in my direction.
“Hey Scrappy, I hear you have a small security issue.” I joked as I shook his hand.
“Yeah, you could say that. Let’s go back to my cave and I’ll fill you in.” Scrappy said as he escorted me through a maze of hallways that were all perfectly curved like the letter “S”.
Scrappy’s “cave” was a small office about ten feet from the datacenter door. The datacenter appeared to be badge-access only, along with a palm scanner. Scrappy was up to his usual tricks it would seem.
“This whole situation is just complete dino-poop man.” Scrappy said as he sat down and lit up a cigarette. Apparently AddaBill bucked a lot of trends, including the local building codes.
“As you know, we are an ad-injection service. We keep an inventory of ad products like banners, flash clickers, even those little deceptive ads that look like window popups. All of these ads are stored on ad servers which are basically just web servers. We also have ‘destination databases’ that contain URLs of where certain ads should take the viewers. We can, in some cases, sell the same ad to several different companies or page owners. " Scrappy disclosed as he took another hit off of his cigarette.
“And judging through the webpage source code I looked through before I came over here, you’re simply pointing the click to an ad server, feeding it an ID number of some sort, and the webserver sends the viewer on their way to advertisement for Viagra or adult escort services.” I joked, longing for a cup of joe.
“Exactly. Our business viability relies on the accuracy of our ad redirects, and lately we’ve had some ‘problems’.” Scrappy said, making little quotation marks with his fingers.
“And what exactly are these ‘problems’?” I replied, also making little quotation marks with my fingers.
“Somehow, somebody is modifying the destination databases here at random and sending our viewers to websites in China, Russia, Taiwan… the list is pretty long.” Scrappy moaned as he rubbed his eyes.
Scrappy put out his cigarette, sat up in his chair and looked me directly in the eyes.
“Chief, I got special permission to let you poke around here and try to figure this out. I’ll go over everything that I’ve already tried, and you can take it from there.” Scrappy said as he removed an envelope from his desk drawer and slid it across the table towards me.
“What’s this?” I asked as I picked up the envelope.
“A retainer.” Scrappy spouted with a grin on his face.
The number of zeroes on the check brought a smile to my face.
Time to get busy.
Source: archived original
Part III · Sep 18, 2004
One thing about Scrappy that I always admired was his attention to detail. I spent the past few hours listening to him describe the infrastructure of AddaBill’s network resources, the steps he had followed after the first instance of database modification had occurred, and how he wished that AddaBill would use whole beans in coffee preparation. The mere thought of Scrappy having to drink a strange Kona ground product early in the morning sent shivers down my spine.To recap, AddaBill had a pretty decent internet presence. A master set of servers resided at headquarters that replicated changes out to a farm of web/ad/database servers on both coasts of the United States. Scrappy had planned for disaster recovery and availability from the get-go.
The headquarters network was fed via four T1 connections, each connected to a Cisco router, and then barricaded with a simple linux firewall that Scrappy had hand-crafted himself. What a perfectionist. The ad, web, and database servers were all isolated into their own DMZ segments - separated by another series of linux firewalls.
The remote networks had a very similar configuration - Cisco router and linux firewall between the internet and the segmented DMZ networks.
I wanted to start my investigation at the master database server, and then work my way to the two remote database servers - very similar to what Scrappy had done with his investigation.
AddaBill had a nice out-of-band management network that talked to the servers via a serial terminal server. After authenticating to the terminal server via a SSH session using a key and a password, I was then authorized to initiate a terminal session with the database server.
The database server was running a slightly modified MySQL server daemon - which shocked me, as I remembered Scrappy being a PostgrSQL guy. Hey, it’s open source - what the hell.
I stepped through Scrappy’s list of steps, and verified his findings.
The integrity of the server itself was intact, according to Tripwire. Key binaries matched MD5 signatures, logs were intact, no unauthorized user accounts, etc.
Scrappy took over, and initiated an interactive session with the MySQL daemon. He gave me an idea of how the database was structured, a look at the tables, etc. What I was most interested in was the MySQL configuration files and the user table. A quick glance through the config files revealed nothing interesting. Hmmm. Back to the interactive MySQL session - and stepping through the table row by row.
“Dude, we change passwords every thirty days on the user accounts that the system uses to do queries, inserts, etc. You can see the hashes there in the table… I just don’t get it man.” Scrappy said, his neck twitching a bit - maybe he needed a smoke.
“Um, Scrappy - not to bust your marbles - but is there a reason that your maintenance user “dbmaint” can connect directly from the inside interface of the DMZ firewall?
“Huh?” Scrappy said, pushing me out of the way.
“WHAT THE HELL MAN!!!!” Scrappy yelled, his fists clenched.
Looks like Scrappy missed something.
(Aside)
Scrappy has egg on his face. What did Scrappy do wrong here? How did he miss that? And what is the signifigance of what I found in the MySQL user table?
Drink Coffee.
Discuss.
(/Aside)
Source: archived original
Part IV · Sep 20, 2004
Talk about a long silence.
Scrappy stared at the MySQL prompt like he was waiting for it to do a trick.
“Chief, I swear on a fresh pack of Newports that I looked this table over row by row. No way I could have missed that!” Scrappy whined, placing his forehead on the keyboard.
BEEP! BEEP! BEEP!
“Scrap, get your head off the keyboard man. Look, whether you missed it or not is irrelevant at this point in time. We have a lead, so let’s follow it. Let’s log into that firewall and have a look around.” I said, giving Scrappy a gentle elbow in the side.“I’m guessing you want me to copy this table out somewhere so we can look at it again?” Scrappy asked.
“You read my mind, Scrap.” I said with a smile. A few keystrokes later, Scrappy had the table copied. He spent the next few minutes engulfed in self-pity as he changed the password of the dbmaint user, changed the permissions, and exited the session. Apparently that user was supposed to be used by administrators for maintenance.
A few moments later we were logged into the linux firewall, having a look around. Nothing out of the ordinary was noted. No strange user accounts, nothing of consequence in the logs, shell history files were intact. Strange.
“This makes no sense man - no sense at all.” Scrappy said angrily. “There’s nothing here Chief!”
“How many other admins have access to these boxes again?” I asked, flipping through my notepad.
“Just the other two admins, and I’d trust them with my life man.” Scrappy replied.
“Well, if you’re sure of their integrity, give them a call real quick and make sure they didn’t change that table.” I suggested as I handed Scrappy the phone off of his desk.
While Scrappy was making his phone calls, I continued to poke around on the firewall. Something didn’t feel right - similar to how one feels after a bad meal of Chinese take-out.
“Scrap, I’m going to do further analysis on this firewall - why don’t you log into your web farms and make sure the database servers there haven’t been modified in the same way?” I asked Scrap as he cupped the mouthpiece on his phone.
“Cool - I’m on it.” Scrap said with a renewed sense of purpose. He quickly cleared some papers and empty Coke cans off of a nearby laptop and went to work.
Time passed. I was still poking around in the firewall when I got a confirmation from Scrappy that the web farm didn’t have the same type of modification we had seen here on the HQ network.
“Hmmm, either our gremlin knew he only had to modify this server to get his changes replicated out, or he was completely unaware that the web farm existed.” I thought to myself. Well, it was more of an assumption actually.
The firewall was very basic, and as stated before, could only be managed through a serial connection directly to the box. This would suggest someone with physical access to the box was using it as a jumping off point perhaps?
I was busy talking myself into several different scenarios when a lightbulb suddenly appeared over my monkey-like head.
“I need to plug in to that DMZ switch.” I yelled over to Scrappy.
“You do? Uh, ok - I’ll badge you in.” Scrappy said as we walked to the datacenter door. A quick swipe of his badge and a scan of his hand and Scrappy pushed me through the datacenter door.
“Port 12” Scrappy said as he pointed to the switch.
I quickly plugged my trusty Powerbook into the switch, configured my Ethernet port with a floater management IP provided by Scrappy, and brought up a terminal.app. A few key clicks later and I was performing an nmap scan of the outside interface of the firewall.
Scrappy was on the phone with his girlfriend when I rushed into his cave.
“Dude, what service is listening on port 2929 on that firewall?” I asked.
“Huh? Nothing! That thing firewalls connections, and that’s it. No services are running on that box.” Scrappy responded as he hung up on his girlfriend.
“Any reason that when I telnet to that port I’m greeted with a shell prompt?” I asked.
Scrappy sat motionless.
(Aside)
Uh-oh. Scrappy gets caught with his pants down again.
How am I getting a shell prompt on that port?
What am I really talking to?
Drink more coffee, discuss.
(/Aside)
Source: archived original
Part V · Sep 29, 2004
“What in the wide world of sports is going on here?” blared the TV in Scrappy’s office. He had a DVD of “Blazing Saddles” in the DVD player, and it was quite ironic that I was thinking the exact same thing as I shifted my eyes back to the Apple Powerbook in my lap.
I felt bad for Scrappy - he was a top-notch Unix admin, so this was a big hit to his ego. If I had to hamper a guess, I’d say he was spending too much time with his newly-acquired girlfriend that he met at DefCon a while back. She was very cute, and very smart, but very demanding of Scrap’s time. Can you say “Lack of focus?”
Scrappy was busy in the datacenter running nmap scans of every network device, every server, and every workstation. He was a man on a mission.Back to my investigation, and a wonderful venti white chocolate mocha from Starbucks.
Scrap had done some quick network-jack patching in the datacenter and the phone room so that I could work in the DMZ by plugging my Apple Powerbook into the spare network jack in his office. How convenient.
The mysterious firewall service was bizarre to say the least. No running daemons on the box (verified by running MD5-verified copies of ‘ps’, ’netstat’ to name a few), no logged in users but myself… nothing. But here I was, having a conversation on TCP port 2929.
MonkeyBook:~ chief$ telnet firewall.internal 2929 _Trying 10.1.1.1… Connected to firewall.internal. Escape character is ‘^]’. _ My keystrokes weren’t being echoed back to me.
No output.
What could this be? It almost looks like a … a… webserver?
GET filename HTTP/1.0 _ HTTP/1.0 200 OK Date: Thu, 13 Jun 2002 17:37:53 GMT Connection: r00t3d Content-Type: text/html _ Fascinating! There’s a webserver running on port 2929 on this firewall?
Not quite. Examining the banner, I got quite a chuckle out of the “Connection” screen. Someone was going to a lot of trouble to make this LOOK like a webserver.
“Scrap?” I yelled out the door, hoping he could hear me in the datacenter.
“Yo - almost done man. Looks like we’re clean!” Scrappy replied back.
Scrap walked in the room and planted himself in his chair and reached for a smoke. “Did you find anything?”
“Dude, someone has…”
(Aside)
What has someone done? Explain the output I saw above.
(/Aside)
Source: archived original
Part VI · Oct 1, 2004
(Aside)
First of all, hats off to reader “ut-admin” for nailing the breach. It is indeed, adore - well, adore is half the problem. Second, hats off to UcantBeSerious for finding a very serious clue in the last posting. You all amaze me! Great work.
(/Aside)
And our story continues…
“Dude, someone has left you a present on your firewall.” I said as I kept punching keystrokes into my GLterm window.“What do you mean present?” Scrap said, almost dropping his lit cigarette on his desk.
“This is a smooth backdoor. Whatever application I’m talking to, it’s configured to spew this HTTP response header no matter what. I wonder what happens when I issue a shell command.” I wondered.
My monkey paws quickly hammered in “pwd” , the UNIX command that displays the working directory name.
/root
“Holy moldy French roasted beans!” I thought to myself. I then entered a bunch of garbage commands and received… not to my surprise… that same HTTP banner.
“Let’s try something obvious.” I thought to myself, a slight grin spreading over my face.
#mysql -u dbmaint -p -h 10.1.1.100 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2933 to server version: x.0.20
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.
mysql>
“Chief, what are you thinking man? WTF? Where is this thing?” Scrappy pleaded with me, putting his cigarette out on the floor. [ed: floor was carpeted. The next two minutes were spent finding glasses of water to put the fire out.].
“LKM Backdoor.” I said as I massaged my temples. “I’d bet a Starbuck’s giftcard on it - and I’m thinking netcat on port 2929 is our doorway.”
“Linux Kernel Module? Netcat? You have to be kidding me. Well, let’s kick it’s ass, get my firewall rebuilt and we can still make it to dinner before 9pm! You can meet my girlfriend!!” Scrap said as he reached for his laptop.
“Wait.” I said.
“What? Let’s do this! Like old times!” Scrap said, his eyes blazing.
“You want to find out who did this, or just clean up their mess and hope it doesn’t happen again?” I asked as I tapped the top of my Powerbook screen.
“Well, the owners just want the ad redirection to stop - that’s why you’re here. But I bet we could set a trap for ’em right before they go to manipulate some bits!” Scrappy said as he stood at attention. This was getting way too dramatic for my tastes.
“I thought you’d say that.” I replied with a grin.
It was on.
Source: archived original
Part VII · Oct 9, 2004
Scrappy looked like a young boy on Christmas morning. He was so excited that we were going to possibly find out who was penetrating his network that he was almost doing a Steve Ballmer-ish monkey dance in the datacenter.
“Let’s go over the plan.” I said as I waved my hands in the air, trying to get Scrappy to calm down.
Scrappy sensed my displeasure at his monkey dance and quickly walked over to the rack of equipment I was standing in front of. “First things first, let’s verify that my monitoring box is working.” I announced as I brought up a serial console to “seeing-eye”, a Sun Ultra 1E box that I had neatly placed in the rack next to the firewall. Seeing-eye was a custom-crafted box that did one thing and did it well: snarfed packets off the wire, analyzed them, stored them, performed some type of action based on the traffic (like send the Chief e-mail alerts on his phone) and be as undetectable as possible.
Scrappy had plugged his laptop in outside of the firewall and proceeded to make a telnet connection to the mysterious service running on tcp/2929.
“Beep!” went the console on Seeing-eye. It logged the connection and Scrap’s request for a mysql session. Around a minute later, my cellphone beeped with a message containing a brief dump of Scrap’s commands.
“Excellent. Now, let’s go get that dinner you promised me.” I told Scrappy as I listened to my stomach perform a musical rendition of “Feed me dammit!”.
Scrap left the office a bit earlier than I did in order to pickup “The Girlfriend[tm]”. He really wanted me to meet this girl. “True love.” I said as I laughed to myself. I pointed the car in the general direction of the restaurant and prepared for massive quantities of ingestible calories.
I arrived at the restaurant in record time thanks to my brain communicating properly with my feet. Apparently there was a cerebral coffee shortage and my brain wasn’t too happy about it.
The restaurant was rather nice - a traditional steakhouse with all the usual trimmings. The host sat me down in a nice booth, decent lighting, and brought my coffee within a few minutes. I liked this place already.
Scrap was taking his time, so I whipped out my Treo and attempted to catch up on my e-mail and web-surfing habits.
I was right in the middle of winning an auction on eBay when a familiar voice reached my ear.
“Hey Chief! Sorry I.. err… we’re late. We had some issues at the gas station.” Scrappy said as he sat down in the booth next to a rather attractive girl.
“Chief, this is Abby. Abby… this is the Chief.” Scrap said as I extended my monkey paw towards her. We were in mid-shake when the caffeine from my coffee stirred up some neurons and I suddenly felt a bit sick in the stomach department.
“Abby… Abby…” I said to myself over and over again, staring at her face and waiting for the neurons in my head to settle down.
The waitress brought us an appetizer, and I politely listened to Scrap and Abby talk about something that I had absolutely no interest in. I started to choke on my crab cake surprise.
“Chief! You OK?” Scrap said as he broke off in mid-sentence from Abby.
I had to do a bit of “Photoshop-ing” in my head, but I knew that Abby was a familiar face. I had met her in Vegas a few years back at Black Hat.
Although when I met her, she was going by the name “DarkAng3l”.
Scrappy, what have you gotten yourself into.
Source: archived original
Part VIII · Oct 16, 2004
“DarkAng3l” was no ordinary chix0r. She made a permanent impression in my memory with a demonstration of her skills at network penetration.
(fade to black, a memory from the Chief’s way back machine…)
I had joined DarkAng3l and a group of friends for some wardriving during BlackHat. The goal was to locate a seemingly valuable (and protected) wireless network, penetrate, record a quick nmap scan and leave. This was ordinarily something I wouldn’t have participated in, but it went back to the old mantra “hero, know thy enemy”. [Note: this was not a sponsored event of BlackHat, but the result of a good beer, dinner, and boredom.]We were driving for a little over twenty minutes when another person in the vehicle got a good reading on a network called “hidden”. Obviously this name was a misnomer.
The network was fairly well protected aside from the SSID being broadcast - a practice that I discourage to all my clients.
A few minutes later we had identified the access point as a Cisco product. DarkAng3l wasn’t saying a thing, simply typing away on her linux-powered HP notebook.
I was peering over the shoulder of the individual who found the network, and saw that a traffic capture was in progress. Apparently this person had come to the conclusion that the WAP was configured with a 64-bit WEP key. Shouldn’t take very long to crack.
We were both startled at that point by a loud click and a sigh. I turned to my left to see DarkAng3l putting her laptop in her bag and handing me a floppy disk.
“What’s this?” I asked as I stared at the label-less media.
“It’s the scan. I win.” She said with a creepy half-smile.
I verified the validity of the key, and the results of the scan by duplicating her efforts on my own linux laptop.
I was impressed with her skills and made it a point to get to know her over the next few days before I was scheduled to leave.
She belonged to (and created) a group named “Dark0n3s” that was going to be a hacker-for-hire company. She, along with two other members, were all very well-trained in the “arts”.
(End of flashback)
“Chief! Dude, wake up! You’re zoning out on us man!” Scrappy said as he waved his chimp paw in front of my face.
“Sorry.” I said as I finished off my coffee. “It’s been a long day.”
(buzzing sound)
My cellphone was vibrating in my pocket. I had received a text message.
“Late for a date Chief?” Scrappy asked.
“You two have fun, I need to go.” I replied as I slapped down a few bills to cover my drinks.
“But Chief you…” Scrappy moaned as he watched me get up from the table.
I wanted badly to tell Scrap why I was leaving, but I was wary of Abby’s presence.
Somebody was connecting to Scrappy’s firewall.
Source: archived original
Part IX · Oct 25, 2004
This case was developing more curves than Anna Nicole before she discovered TrimSpa. I let the details of the case swirl around in my coffee mug of a brain while I was stopped at a redlight. The drive back to my home had given me a good idea or two from time to time.Scrappy had a linux firewall with what appeared to be a linux kernel module backdoor installed. Scrappy was dating a known hax0r (and a rather cute one too, I might add). We have a method and a suspect, but what of a motive? Was Abby the only suspect? Was she using Scrappy for personal gain? What type of gain? What type of connection was there (if any)?
Once home, I cracked open my trusty Powerbook and performed an nslookup on the IP address that had connected to Scrappy’s firewall. Having the entire session sent to my Treo in a few email messages was quite convenient for this type of monitoring.
chimptop:~ chief$ nslookup 210.x.x.x
Server: 192.168.1.254 Address: 192.168.1.254#53
Non-authoritative answer: x.x.x.210.in-addr.arpa name = x.x.x.210.somecompany.co.th
Wonderful - Thailand. This was going nowhere fast. Gee, I wonder if this server is compromised and simply a backdoor for some mischievous folk?
Chimptop:~ chief$ nmap -sS -O -v 210.x.x.x
Host (210.x.x.x) appears to be up … good. Initiating SYN Stealth Scan against (210.x.x.x) Adding open port 8080/tcp Adding open port 21/tcp Adding open port 23/tcp Adding open port 80/tcp The SYN Stealth Scan took 13 seconds to scan 1601 ports. Interesting ports on (210.x.x.x): (The 1588 ports scanned but not shown below are in state: closed) Port State Service 21/tcp open ftp 23/tcp open telnet 25/tcp filtered smtp 80/tcp open http 135/tcp filtered loc-srv 136/tcp filtered profile 137/tcp filtered netbios-ns 138/tcp filtered netbios-dgm 139/tcp filtered netbios-ssn 445/tcp filtered microsoft-ds 1720/tcp filtered H.323/Q.931 8080/tcp open http-proxy 27374/tcp open subseven
Remote operating system guess: Windows Millennium Edition (Me), Win 2000, or WinXP
I sighed in boredom. Another Windows machine with a subseven backdoor in use. What a surprise.
The connection was a dead-end, however the command capture was quite interesting:
**mysql -u dbmaint -p -h 10.1.1.100 seckritpasswerd UPDATE sites.destination SET finaldest = “massivechix^H^H^H^Hpr0nsite.com” WHERE origdest = “legitsite.com”; **
Let’s think about this for a second.
- Firewall is compromised.
- Attacker installs backdoor through an LKM
- Attacker places workable mysql client on firewall (and makes no attempt to hide it)
- Attacker knows the NEW password to the ‘dbmaint’ account.
- Attacker had a type-o in their UPDATE SQL statement…
But what is the significance of the TYPE-O?
Drink Coffee. Discuss. Bonus points to who figures it out first.
Source: archived original
Part X · Oct 30, 2004
(Aside)
My readers constantly amaze me with their abilities to pick out the key parts of a case file that are important and draw some pretty impressive conclusions. As you’re about to read, several of you were right on the money…
(/Aside)
It’s a good thing that decaf coffee doesn’t really mean “without caffeine coffee”. I was determined to stay awake while scribbling in my case notebook. I was using a coffee brewing trick I learned in the military when “real” coffee was scarce. Double-brew the decaf, and say “hello” to wake-up juice. I went back to the basics. I drew a timeline in my notebook of the events so far, indicating the systems involved, the players as I knew it, and drew in “theory” bubbles around certain events. (These are helpful to attach motives to people based on events.)
After staring at the diagram for a while and working my way towards the bottom of the super-decaf pot, I decided that I needed to either eliminate Abby as a suspect, or build the case around her. To do this, I would need to play a bit with Scrappy. (What are friends for?) A plan was formed in my head - and a good plan at that.
I dialed Scrappy’s cell phone.
ring
“Yo! Chief, it’s late man - what’s up? Why did you take off?” Scrap asked in between loud drags off a cigarette.
“We had another connection to the firewall this evening - I think you need to go into the office, change the ‘dbmaint’ password again, and consider changing out the firewall. I don’t think your employer wants to play this game very much longer.” I mumbled into the phone.
“You kidding me? How did they get the new password? We just changed it man! What the f…” Scrap said as I interrupted him.
“Obviously we missed something. Grab your laptop and get over to the office.” I said with an impatient yawn.
“No can do Chief, Abby has my laptop. She dropped hers at her apartment and the screen is crack-o-la. I lent her mine.” Scrap replied with a dramatic cough. “But… wait… I got that Toshi sitting on the bookcase. I guess I could use that one.”
“Hey out of curiosity Scrap - which laptop did you loan her?” I asked - knowing that Scrap has quite a few laptops laying around at Casa De Scrap.
“My SparcBook - you know… the tadpole.” he said matter-of-factly. Interesting.
“Scrap, I want you to grab the old Toshiba Satellite laptop in your office and use it to change all the passwords. Then, I want you to swap out the firewall with the freshly built one under your desk. Ok?” I demanded.
Sometimes Scrap needed a push to do the most obvious things. It drove me nuts.
“Yeah, ok, ok… calm down man. I’m on it.” Scrap said. “So have you found anything yet?”
“Nothing conclusive.” I lied. “And before I forget - why were you guys so late to dinner?”
“Oh, yeah… " Scrap replied with a snort. “Dude, it was soooo wrong. Abby was wanting to check her mail on the tadpole right - so she brings up Netscape and she’s like all happy and stuff because we were down by the University, you know - hotspot central - and she gets right on. She’s totally digging the tadpole, and kind of impressed by how snappy Solaris is on that bad boy. Then she got all pissed because I had some pr0n sites in my browsing history. It got ugly man.”
“Uh, Scrap - you let her use your tadpole logged in as YOU?” I asked as I slapped myself on the forehead.
“Yeah man, it’s not like my girlfriend needs a permanent account on my tadpole.” Scrap responded, reeking of ignorance.
“Scrap, I’m going to meet you at the office. We need to talk.” I said as I looked around for my shoes.
“Uh, ok. About what?” Scrap asked.
“Abby.” I said.
The chess game was in motion once again.
(Discussion)
What have we learned that proves a few readers’ suspicions correct?
Have any of your theories changed? Is Abby still the main suspect?
(/Discussion)
Source: archived original
Part XI · Nov 3, 2004
This case had more twists and turns than my last tax return.
A slight drizzle of rain had started to pour from the clouds overhead as I made my way towards AddaBill. I had diverted myself for a few moments to locate a late-night latte, and was becoming mesmerized by it’s sweet aroma as it filled my car.
I took the last few minutes of my drive to collect my thoughts, and to prepare for my discussion with Scrappy. This wasn’t going to be the most enjoyable moment of my career.I walked to front doors of AddaBill and saw Scrappy standing outside, huddled in a corner and nursing a cigarette.
“Hey Chief - mission accomplished. Password changed, and the new firewall is in place.” Scrap said as he tossed his smoke into a pile of rain water.
“Good work - let’s get out of this rain.” I said as I motioned Scrappy inside.
I took up residence in a chair next to Scrap’s desk and stared at the linux firewall that Scrap had pulled out of the networking rack in the datacenter.
“It’s all yours, Chief. You want to take it with you - or you’re more than welcome to do your magic on it here.” Scrap said with a big toothy grin.
“I’ll take it with me.” I responded, looking over Scrap’s shoulder. “Scrap, I’m a little concerned about your relationship with Abby.”
“Why? She’s really cool. I don’t understand.” Scrap said with a look of concern.
“You’re breaking one of the cardinal rules of security, bud. You’re giving someone else privileged access to your main laptops, which we both know could give her enough information to do a lot of things.” I said without missing a beat.
“I’m not sure I understand where you’re going here, man.” Scrappy said, staring at the ceiling tiles in his office.
“Scrap, you know her background probably better than I do. I’ve met her before - and I’ve seen her in action. She is the definitive ch1x0r.” I pleaded. “Did it ever occur to you that Abby might have something to do with all the weirdness going on with your little database problem?”
“Oh come on man. Are you serious? Abby isn’t like that - you don’t know her like I do. She is legit - she’s a security consultant now, ya know.” Scrap said with a hint of pride. “She’s not involved.”
“Then you won’t mind if I ask to look over your Tadpole?” I asked, looking Scrap straight in the eyes. “I need that laptop.”
“Hey, that shouldn’t be a problem Chief. She should be here any minute. I wasn’t about to let her keep my ‘pole over night - how am I supposed to get any work done?” Scrappy said with a laugh.
One half-hour later, Abby arrived with the Tadpole under her arm. Abby gave me a rather strange look, handed the laptop to Scrap, gave him a kiss on the cheek and hurried out the door.
“What’s her deal? I didn’t even get a ‘hi’!” I cracked.
“She’s flying to Atlanta tomorrow, or maybe it’s Miami… I forgot. Anyway, she probably wants a good night sleep or something.” Scrap said as he handed me the Tadpole.
“I need to take the Tadpole with me too - you’re going to have to make due with the Toshiba until tomorrow evening.” I said as I placed the linux firewall under my other arm and made my way to the door.
“Chief! Come on man, can’t we do that tomorrow?” Scrap whined.
“Dude, you live in a world of console prompts. Use the Toshiba. Pine works just as well on that box.” I joked as I walked down the hallway.
‘You’re killing me, Chief!” Scrap yelled from his office.
The feeling was mutual.
Source: archived original
Part XII · Nov 11, 2004
The rain was pouring from the heavens faster than Oprah eating at a dessert bar. There I was, sitting in my car outside of my house, watching millions of gallons of water coming from the skies. I was debating whether to leave the linux firewall and tadpole laptop in the trunk of my car and make a run for the house, or brave the storm and move the equipment into the house. Decisions… decisions…I made a compromise with myself - I made a run for the house, and came back to the car with my trusty JumboBrella. I remotely opened the trunk, grabbed the gear with one hand and ran for the house. As luck would have it, I made it through the front door - but the tadpole didn’t. I was four paces from the front door when I felt the tadpole slipping out of my grip. Before I could compensate for it’s gradual descent towards the ground with my knee, the tadpole had made a beeline for the nearest half-foot deep puddle of water with a suicidal flip from the back of my arm.
Dammit.
What an amateur mistake. I ran outside, plucked the laptop from it’s watery grave and ran back to the house. My clothes were soaking wet, and I hadn’t even started a fresh pot of coffee to warm myself up. It just wasn’t right!
A quick change of clothes later, I inspected the tadpole - now hanging from a clothesline in my office like a pair of worn-out boxer shorts. Water was dripping at a regular pace from every nook and cranny. Not good. My only saving grace was that I had removed the battery from the laptop in the car before I had left Addabill. Well, I really didn’t have a choice because Scrap had broken the case holding the battery in. Nice.
I set up a blower fan, pointed it at the tadpole, crossed myself and wished the tadpole the best. I hope the hard drive had survived. Thank goodness they are self-contained.
I turned my attention to the linux firewall.
“What secrets do you have in store for me, my little friend?” I asked it, almost expecting a verbal response. I popped the cover off of the firewall (Scrap wasn’t big on using screws on anything - what gives?) and located the IDE hard drive. I snapped a few reference pictures with my trusty Olympus digital camera and proceeded to disconnect the IDE drive and attach it to my good friend Roscoe.
(For those of you that haven’t read all my case files - Roscoe is my portable forensic workstation.)
Double-checking that my write-blocker was installed on the IDE chain, I began the imaging process and wandered off to the kitchen to brew some go-go juice.
A few hours later, the laptop was still dripping and my imaging process had nearly finished.
BEEP
A friendly console prompt was staring me in the face - I had a nice image of the root ( / ) filesystem of the drive and an image of the SWAP partition. Sweet.
I quickly mounted the root partition as a read-only filesystem using the nifty loopback device, and attempted to have a look around the filesystem.
roscoe$ cd /mnt/firewall roscoe$ ls -la total 0 drwxr-xr-x 3 chief chief 102 11 Oct 12:02 . drwx—— 35 chief chief 1190 11 Oct 10:38 ..
roscoe$
What the…
Did I goof? Did I not image the drive properly? I quickly checked the cables, the write blocker… my dd command… everything was perfect. There was nothing on that filesystem.
And if there was, I suspected it had been deleted or formated.
My patience had worn thin.
I picked up my cellphone and dialed Scrap’s number.
ring ring ring ring ring
No answer.
He had some explaining to do.
Source: archived original
Part XIII · Nov 13, 2004
I was losing my mind. I was trying very hard not to let my friendship with Scrap interfere with my work, but it was beginning to take it’s toll. I was frustrated at the challenges that were being thrown at me left and right (and even more so - the ones I was creating, like the wet tadpole laptop).
I picked up my land line and called Scrap’s apartment. If he was screening his calls, he wouldn’t know this was me because my land line was blocked - no caller ID.ring ring
“Hello?” said Scrap, sounding mildly distracted.
“Scrap, what’s going on?” I asked in a most annoyed tone.
“Heh?” Scrap responded.
“I just imaged this firewall, and the drive partitions are blank - empty. How might this have happened?” I asked.
“What do you mean they are blank? I pulled that out of the rack, sat it on the floor and swapped the other firewall in. I didn’t touch that thing!” Scrap fired back.
“I want you to come over here right now.” I demanded. “I want you to eyeball this thing.”
“Alright - give me a sec.” Scrap said as he hung up the phone.
I decided to rest for a few minutes and let my mind clear. I had to get back into an investigator mindset and calm down. I was really irritated.
Two cups of finely brewed java later, my doorbell rang.
dong!
I opened the door to find a quite drenched Scrappy huddling under the dry safety of my porch.
I motioned for Scrap to join me in my office and I pointed to the linux firewall setting neatly on my lab table.
“Care to have a look there, bud?” I asked.
Scrappy approached the firewall and peered inside. The hard drive was removed and setting next to Roscoe, IDE cable still attached.
“What are you expecting me to see here, Chief? This is my firewall. There’s my ID sticker on it - FW-1.” Scrappy said, running his index finger along the ID tag on the front of the firewall case.
“You want to explain how that hard drive is wiped clean?” I asked, glancing over at the hard drive sharing desk space with Roscoe.
Scrap approached the tabled and looked down.
“What hard drive? This drive?” Scrap said as he turned the hard drive 180 degrees. “This isn’t my hard drive. This is a Maxtor - I only use Western Digital drives in these babies. "
I stared at Scrap like he had just called my mother a derogatory name.
“What do you mean that’s not your hard drive? I took it out of the box!” I exclaimed.
“I’m telling you Chief, that is NOT a hard drive that I would use. That model of Maxtor has a ridiculous fail rate.” Scrap said as he shook his head.
I disconnected the hard drive and placed it back in the linux firewall, screwing the two screws holding it into the drive bay.
“Here.” I said as I handed the case cover to Scrap. “Finish putting this back together.” I sat down in my office chair and crossed my arms in my lap.
Scrap popped the case cover back on and slid it neatly into place.
“Got a screwdriver?” Scrap asked, stretching out one arms towards me.
“Why? There weren’t any screws holding the thing on when I got it.” I replied.
Scrappy put his arm down and stared at the box. He stood motionless for what seemed an eternity.
“You ok?” I asked.
“Chief, you know me pretty well. I always screw case covers back on. This case takes four case screws - and one of the holes is stripped - this case cover had three case screws holding it on.” Scrap moaned as he sat down on the lab table.
I left the comfort of my office chair and examined the case cover. I opened a filing cabinet drawer and pulled out a baby food jar full of case screws. Two screws into securing the cover and I found that one wouldn’t tighten.
“Son of a gun… " I said.
Scrappy continued to sit on the table, head down… thinking intensely.
I glanced over at the tadpole, still hanging from the hastily-fashioned clothesline.
“Hey Scrap, how are your surgical skills?” I asked with a grin.
“Heh?” Scrap said, not lifting his head.
“I need your help with a little surgery on your laptop.”
I had all the questions, and Scrap was about to give me all the answers.
Source: archived original
Part XIV · Nov 19, 2004
“I need a smoke first.” Scrap said as pulled a pack of Newports from his pocket and headed toward my back porch.
My heart was starting to beat a little faster, and I suspected it was from the excitement surrounding the tadpole. Anyone else would blame the coffee, but we all know that the Chief has a high tolerance. I removed the tadpole from the clothesline and placed it on the empty half of my lab table, still warm from Scrap’s strategic placement of his arse.
I had just begun to slide the drive door down when Scrap re-entered the office, a blank look occupying his face.
“Ready to do this?” I asked as I slid the drive sled from the tadpole.
“Yeah.” Scrap replied as he volunteered to hold the tadpole steady.
Scrap had apparently gotten sick of the small SCSI drive that came in the tadpole and replaced it with an IDE drive equipped with some sly IDE-to-SCSI adapter. Very cool.
I scrounged around in my forensics kit for the proper adapter for the drive, and before I knew it we were in the process of imaging the drive.
The room was getting uncomfortably quiet, so I recommended to scrap that we step out the living room and watch some TV.
A short time later the familiar chime of a console beep alerted me that the drive image was completed. Repeating the same process as earlier, I mounted the partitions and prepared for some serious investigation.
With scrap looking over my shoulder, we went for the obvious - shell history files. Scrap was a bash man, so my attention was drawn to /.bash_history and ~/.bash_history.
We breezed through root’s history file (/.bash_history) and saw nothing entertaining. My faith was partially restored in Scrap - he wasn’t doing too much work as the root user.
The history file for his user account was a little more… fascinating. Scrap and I stepped through his commands, and analyzed the sequences very carefully… until they abruptly ended.
“I don’t get it.” Scrap said. “I don’t see any commands past where I checked my mail before leaving to get Abby - see, that’s where I ssh’d to the mail server… there’s where I ran pine… and there’s where I handed the ‘pole to Abby. "
“How does she usually get her mail?” I asked.
“She’s a console gal, man. Otherwise I wouldn’t be dating her.” Scrap said all too seriously.
I went with my gut and dived into the Netscape history and cache files. Scrap verified all the sites, and even pointed out the pr0n links that Abby was so upset about. We weren’t getting anywhere. I then decided to do a comprehensive search of the root partition for strings of interest. A little magic with ‘find’, ‘grep’ and ‘strings’ and we were sailing through thousands of files - and of course, nothing of interest was revealed.
I noticed the number of partitions that we imaged… a root partition, a swap partition, and two others that seemed rather strange. One was relatively small, and the other was the exact size of the root partition.
“Hey scrap, what are these other two partitions?” I asked, pointing to the images files - not yet daring to delve into the mounted versions.
“Uh, that small one is used for suspend/restore - and the other one is my backup partition.” Scrap responded.
“Backup partition? How are you backing up your root partition?” I queried. I was almost giddy that Scrap was proving to as responsible as I remembered.
“I have a little script that fires off when I login, and it performs an rsync between the two partitions.” Scrap said.
“Wait, you mean when you logged in to check your mail BEFORE you picked up Abby, it made a backup… and then it made another backup when you logged in to give the tadpole to Abby to check her mail?” I asked.
“Right - well, there’s a delay of a few minutes at first because it has to figure out what files have changed, but it’s pretty darn faster after that.” Scrap said.
I wasn’t even paying attention to Scrap’s words at this point. I had already mounted the backup partition and went straight to ~/.bash_history. A quick SHIFT-G in vi and I was at the bottom of the file.
“Holy Jeezus on a pogo stick.” I whispered.
Scrap didn’t make a sound.
Source: archived original
Conclusion · Nov 29, 2004
I’m not good at awkward silences. I start to say something, but then I stop because I think someone else is going to say something. I decided that I’d pass the proverbial speaking torch to Scrappy. I was waiting a long time - then I noticed that Scrappy was standing there with his eyes closed.I looked back at the console screen - there in the .bash_history file was the straight dope:
… telnet 210.x.x.x 27374 ssh mail.chix0rsrus.com …
“Scrap.” I said as I tapped him on the shoulder.
“Huh?”
“So you’re telling me that this telnet command is Abby?” I asked, pointing at the screen.
Scrappy sighed loudly.
“Yeah. I guess it is.” he replied, closing his eyes again.
“Scrap, the machine at that IP address appears to match what we caught using the backdoor on the firewall. Any ideas?” I asked, smacking Scrap in the arm a few times.
“Yeah, I suppose I have a few.” Scrap said. “Let’s go outside - I need a smoke.”
I followed Scrap out to my Patio, grabbing a fresh cup of Joe from the kitchen.
“I’m sorry.” Scrap said as he exhaled a plume of Newport smoke into the atmosphere.
“Sorry for what?” I asked, tapping my finger on my coffee cup.
“Sorry for getting you involved in this.” he responded.
“Can I make a guess on what you’re about to tell me?” I asked Scrap after a mighty swig of go-go joe.
Scrap nodded as he sat back into the patio chair.
“Abby is every geek’s dream girl. She’s hot, she’s smart, and she has a personality. You got involved with Abby and found geek nirvana. After a few weeks, you started having differing opinions on all things geek. Your relationship turned into a festival of one-upsmanship. How am I doing?” I asked Scrap.
“You forgot to mention her incredible taste in geek boys.” Scrap cracked as he flicked a burnt cherry off the end of his cigarette. “But please, please continue.”
I elaborated on my theory.
“A series of ‘geek olympics’ has been going on. She hacks your hotmail account, you socially-engineer a credit-card in her name, she has your electric turned off, you deface her weblog…. am I still doing ok?” I said as I put down my coffee cup and scratched my arm.
“It’s like you’re reading my mind, man.” Scrap joked as he lit another Newport. He was taking my intuition a little light.
“Eventually, you two ran out of ways to best each other. One night, Abby turns her attention to your little server farm at AdaBill. She’s so good at not being noticed once she’s penetrated your firewall that she decides to liven things up. She wants you to know what she’s done. She starts manipulating your link traffic. You see what’s going on, but you’re unsure of who is doing it. You suspect Abby, but you don’t want to ask her - because it might not be. You don’t want to mention it to me because, hey - you appear to be getting your geek butt kicked by a chiX0r. So you remain silent.” I said, looking back at Scrap for a reaction.
“That’s mostly correct.” Scrap responded. “I honestly didn’t think it was Abby until you showed me the hard drive you took out of the linux firewall. There’s no way in hell that I’d use a cheap ass drive like that one. And I knew that the only person who would cover their tracks that well AND that had access to my office would be Abby. She wanted to preserve her hack, and the only guarantee she had was to replace the drive.”
I raised an eyebrow.
“Scrap, I need to know where to go from here man. So far I’ve figured out what has happened, who was responsible, and I gave your rather expensive sparc laptop an unneeded bath. I still owe your employer a report. I can write up my findings and recommendations, or you can go talk to them yourself. It’s your call.” I said as I sat forward, placing my head in my hands. “What’s it going to be?”
Scrap put out his cigarette and leaned forward in the patio chair.
“I’ll talk to management. You’ll get your check either way. Let’s just hope I have a job after tomorrow.” Scrap said as he stood up and stretched.
CONCLUSION
Scrap met with the management of Adabill the next afternoon and explained to the three most senior officers of the company what had happened. Adabill hadn’t finished putting together a loss estimation from the misguided traffic, but from my exit conference with management I learned that the CFO had guessed around $120k. Adabill made a move that surprised me. They didn’t fire Scrap. Scrap paid for the $120k loss in good ol’fashioned humiliation. Adabill had Scrap explain the loss himself at the BoD meeting. Needless to say, the BoD tore Scrap a new orifice and then laughed through his entire presentation. Comments like “she sure schooled you!” peppered the conference room.
Abby was confronted by Scrappy a week later after she returned from a security gig out of town. Abby confirmed my suspicions, however Scrappy left me out of his conversation. I’m not sure if that was a blessing or not. Scrappy and Abby broke up within two weeks citing geek incompatibilities. (Scrappy won’t admit it, but her skills were far more superior than his) Scrappy is currently employed as a systems administrator for a hosting company. Abby is still a security hired gun. The Chief is still tired and in need of coffee.
(Aside)
This goes to show you that no matter how much work you do in this industry, you haven’t seen everything.
What would you have done if you were Scrappy?
What would you have done if you were on the Adabill Board of Directors? Would you have been easier? Harsher?
(/Aside)
Until next time - beware of good-looking ch1X0rs in leather pants - and remember that computer investigations aren’t all about computers.
Source: archived original