Originally published by Chief Monkey on the A Day in the Life of an Information Security Investigator blog (the “Security Monkey” blog) at it.toolbox.com. Mirrored here from the Internet Archive. All rights remain with the original author. See the section intro for full attribution and takedown contact.
Part I 路 Nov 30, 2007
“How do I keep getting talked into doing these things?” I asked myself as I shoved my powerbook into my laptop bag.
I had been in Las Vegas for two days, dished out three presentations on information security best practices, and was eager to hop on a plane back to the monkey crib to enjoy some rest and relaxation.
I scanned the emptying room and noticed that something was missing.
Where was Scrap?
I was twenty paces out of the room when a familiar voice called out.
“Chief! Wait up!”
I turned momentarily to see Scrap and another gentleman I didn’t recognize trying to catch up to me while balancing drinks in both paws.
“Where’s the fire, boss? Hey, I want you to meet Kip. He just bailed me out of a shortage problem over at the bar.” Scrap said in between gulps of air.
“Shortage problem?” I asked.
“Yeah, of the cashola kind.” Scrap replied.
Why wasn’t I surprised?
“Chief, it’s a pleasure to meet you. Kip Kerpatrick - I’m the security officer at PPP productions. Scrap and I were talking at the bar and I happened to mention a problem that my staff is working on right now. He seemed to think you could help me?” Kip asked.
I could only imagine what Scrap had filled this poor chap’s eardrums full of after six or seven long island iced teas.
“Whatcha got?” I asked.
“Well, I guess you could say that it’s not a matter of what I have, but what I don’t have.” Kip replied.
“Beg your pardon?” I replied while raising an eyebrow.
“Control of my network.” Kip said while glancing at the screen of his vibrating phone. “Christ, it’s happening AGAIN. Excuse me for a moment.”
Kip scampered off and started what appeared to be a serious conversation on his phone.
“Scrap, how much have you had to drink?” I asked, not really wanting to know the answer.
“Um. Lemme see. One… two… three…” Scrap yammered.
“Never mind. What’s the deal with Kip?” I asked.
“Check this out. So he’s the security monkey at PPP. They do a lot of indie film work and commercials. They have this new office in California. Well, it’s not really ’new’ exactly… they bought this other production company… inherited a lot of staff and equipment… yadda, yadda. Kip says that every once in a while they have these killer network outages that they haven’t been able to trace back to anything. They are on a shared pipe with a bunch of other tenants in this building… it’s a fat pipe man. 10MB to the net. Okay, so they have all these disruptions and the tenants in the building are cat fighting over who is at fault. It sounds pretty ugly.” Scrap blurted out in broken english.
“Sounds like it’s right up our alley.”
脗漏 2007 All Rights Reserved.
Source: archived original
Part II 路 Dec 7, 2007
“I feel so violated.” Scrap moaned as left the airport security checkpoint station.
I laughed. “Scrap, that’s like Rosie O’Donnell announcing that possibly gained a pound from that last doughnut.”
“I swear that guy had three hands.” Scrap replied as we took seats in the boarding area.
ring
“Chief” I announced as I answered my phone.
“Hey, it’s Kip. I really appreciate you guys jumping on this and missing the last day of the conference. I should be in town about an hour before you land. I’ll try to meet you at the office like we talked about.” Kip said with a smile that was radiating over the phone.
“Not a problem. Did you get a chance to e-mail the materials that I asked for?” I queried as I removed my powerbook from its protective sleeve.
“Affirmative. If you get a chance to look them over before you arrive I’d appreciate it. My flight is boarding so I have to go. See you at the ranch!” Kip said as he hung up.
I was pleased to find free Wi-Fi moments after opening my powerbook and did a quick check of my e-mail. As promised, the requested documents appeared in my inbox.
The first document was a network diagram that resembled any of the hundreds of others I had encountered in my case work. PPP had one large flat network sitting behind a single firewall. PPP’s Internet-facing servers were located at a colocation facility.
The second document was a bit alarming. It was a configuration dump from their Cisco Pix Firewall.
“Scrap, take a look at this. Am I seeing what I think I’m seeing?” I said as I tilted the powerbook towards Scrap.
Scrap spent a few minutes perusing the configuration file and then furrowed his brow.
“Uh, they’re kidding, right?” Scrap said as he scratched his head.
access-list outbound permit tcp [Inside Network] [Inside-Subnet] any any
“Apparently not. The inbound is nicely limited, but there’s no outbound controls? What year is this again?” I said in disbelief.
The third document was a recent traffic graph gathered from traffic on the PIX firewall. It was equally disturbing.
“Holy crap!” Scrap stammered. “That graph has more peaks and valleys than Pamela Anderson’s cleavage!”
I stared at Scrap for a brief moment and continued.
“It’s obvious that there is something on the network generating massive outbound traffic on a periodic basis. The questions are plenty. Is it authorized traffic? Unauthorized traffic? Are we looking at some massive P2P client usage on their network?” I asked.
“It’s always the users, man. Always.” Scrap lamented.
I opened the fourth document and raised an eyebrow.
“Or maybe not.”
脗漏 2007 All Rights Reserved.
Source: archived original
Part III 路 Dec 15, 2007
“You could smoke a ham in this joint.” Scrap whined as we waited patiently in the reception area of PPP Productions.
The summer weather was brutal and a bit amplified by the lack of air conditioning in the large, warehouse-style building. PPP occupied the entire first floor of the three-story building. The other two stories were occupied by other tenants. I found it odd that most of the parking lot was empty, and that there was very little foot traffic in and out of the building.
“Agreed.” I replied as jogged through some E-mail messages on my treo and wiped a bit of sweat from my monkey brow.
Moments later a familiar face came cruising through a pair of doors to our right.
“Guys! Great to see you. Thanks for making the trip out. I see that the receptionist got you your guest badges, so we’re all set. Follow me.” Kip said as he motioned for us to follow him through the double doors.
“Nice place, but what’s up with the air conditioning?” I asked as I fanned myself with my notepad.
“Ah, yes. Sorry about that. We’ve been having some issues with the HVAC in this place. I really wish that we had stayed in our previous office downtown, but you know how acquisitions are. Consolidation is always lots of fun.” Kip explained as he badged us into a room near the end of the main hall. The wall placard near the door was rather entertaining. At one time it had said “Storage”, however it now bore a piece of paper with the inscription “IT”.
Scrap and I took a pair of seats located inside a large cubicle belonging to someone named “Cookie”. From the cubicle’s adornments it was rather easy to guess that “Cookie” was a female and had a strange fascination with cats dressed in costumes.
“Creepy.” Scrap whispered as he tried to contain his laughter.
Kip took off to round up his troops.
My eyes gazed around the cubicle to find a couple of standard PC machines running a version of Windows and three different telephones. Network activity graphs were neatly thumb-tacked to portions of the cubicle’s walls along with a lone white board bearing a number of diagrams and scribbling.
“Hey Scrap, looks like Cookie is a big MRTG fan. You too should get along fabulously.” I said with a smirk.
Scrap was seconds from a witty reply when a girl with pink hair whipped around the corner of the cubicle. She appeared out of breath.
“Great timing. Come with me.” she said as she took off towards another cubicle in the IT area.
After a simultaneous shrugging of shoulders we grabbed our laptop bags and tried to catch up with her.
We landed in a cubicle belonging to “Cedric” and were quickly introduced to our new hosts.
“Oh, sorry. I’m Cookie. This is Cedric. I’m the IT manager, and Cedric is my network guy. Kip told me that you two could help us with a little weirdness we’re having with our network. It’s happening right now. Show ’em Cedric.” Cookie said.
Cedric turned his monitor a bit so that we could examine the graphs populating his screen. We were looking at MRTG graphs of inbound and outbound traffic that bore a striking resemblance to the graphs in the documents that Kip provided us. Outbound traffic was peaking and inbound traffic appeared to be normal.
“When these events happen our entire access to the internet slows to a crawl. It’s very frustrating.” Cookie said with a frown.
“Have you isolated which inside addresses are generating this traffic?” I queried.
“All of them!”
脗漏 2007 All Rights Reserved.
Source: archived original
Part IV 路 Jan 10, 2008
“Holy packet storm from hell, that is nuts!” Scrap said as he watched Cedric pull up MRTG graph after another from the various network gear on the network. It appeared that every switch port was pumping data at capacity and the lone PIX firewall was taking a beating - although it was a bit of a sucker bunch. Lots of outbound data packets and nothing stopping them.
“Scrap, let’s break out some gear and sift some packet goodness of the wires. I want you to…” I trailed off as I was interrupted by Cedric.
“It’s gone!” Cedric exclaimed. “What the holy hell is going on?”
I glanced over Cedric’s shoulder and confirmed that the traffic had disappeared in an instant.
“As I was saying, Scrap, let’s get some gear set up. Cedric, would you mind assisting Scrap with some taps?” I asked.
“Not a problem.” Cedric replied as he sauntered out of the cubicle with Scrap in tow.
“Cookie, I have to ask this out of curiosity. These traffic anomalies have been occurring quite frequently from what Kip told us. You haven’t captured any of the data?” I queried.
“That’s correct, and I know it’s the wrong answer. To be honest, we know enough to keep this network up and functioning. Investigations really aren’t our thing. Don’t tell Kip I admitted to that!” Cookie said with a chuckle.
“No worries.” I replied. “Out of curiosity, you appear to be running one large flat network in here. How do the other tenants in the building connect to the internet? Do they have their own point of presence or are you acting as their ISP?”
“That’s a long story.” Cookie said. “We have a ten meg pipe to the net and before all of this started to happen, never worried about filling it up. We offered to play ISP for the other tenants. They are all behind NAT appliances.”
“So their traffic appears to originate from single IP addresses - the addresses of the appliances?” I asked.
“Exactly.” Cookie replied.
I filled a page in my notebook with notes and was interrupted by a text message from Scrap: “We’re a go.”
I barely had a chance to share the good news with Cookie. Her eyes were already nervously spying on her MRTG graphs.
“Here we go again! They don’t normally come so close together.” Cookie commented as she clicked around her management interface.
“It’s like we’re monitoring labor contractions, and the woman is not in a good mood.” I joked.
I picked up my phone and called Scrap.
“Are you getting this? We’re spiking again.” I asked.
“Affirmative boss. Wireshark is rolling packets by on the screen… and… wait one.” Scrap said as he put his phone down.
“What? WHAT?” I asked into my phone, expecting the table supporting Scrap’s phone to answer.
I heard a rustling noise on the phone.
“I knew that traffic looked awfully familiar!” Scrap yelled into the phone.
“Do tell.” I said.
“Its like 50 million geeks cried out all at once.” Scrap joked, attempting some Star Wars humor.
“And?” I asked impatiently.
“Well, let’s just say they are all crying out… in IRC.”
脗漏 2008 All Rights Reserved.
Source: archived original
Part V 路 Feb 12, 2008
“Wow. Just… wow.” Scrap said as he scrolled back through his packet captures. “This is just mind blowing. It’s like IRC soup in here: channel commands, DCC requests, a zillion conversations… it’s like a snapshot of every conversation on the planet all at once. Chief, this is going to take forever to parse.”
I chewed on the end of my pen as I looked over Scrap’s shoulder. “Well we’d better get started then.”
The hours crawled by as Scrap and I wrote shell script after shell script to parse the packet data. I needed to answer a few questions immediately:
From how many points on the network was the activity coming from?
How many different points outside of the network was the traffic going to?
What type of data was going to/from which addresses?
Was the time/duration of each burst random, or part of a pattern that would provide a clue?
What was generating the traffic in the first place, why was it generating that traffic, and how did it get inside the network(s) to begin with?
“Coffee time!” Cookie said as she sat a familiar carry-out tray in front of us. Two large venti cups stood before me giving off a familiar vapor trail.
“I love you.” I said as I took several quick sips of the caffeinated goodness. I could actually feel my brain recharging.
“Any news?” Cookie asked as she sat down in a nearby chair and started twirling her pink locks with her right hand.
“A bit.” I began. “We’ve isolated the traffic origination points within the network. They are the IP addresses of all your neighbors’ NAT appliances and nearly one hundred machines on your local LAN.
“Jeezus on a pogostick!” Cookie whispered.
“Oh, but that’s not the interesting part.” I continued. “Every fire starts with a spark. This fire starts with query packet sent from every single machine here to an assortment of machines out on the internet. It’s a HTTP POST that looks like this:
POST http://some.unknown.IP.address/check.pl?ping
Each box then receives a strange encrypted response string that is 128 characters in length. They are always 128 characters, absolutely no deviations.
“Am I hearing you right?” Cookie asked. “All these machines check in with some random machine or machines out on the internet, get an encrypted response, and then the traffic goes nuts? I don’t get it.”
“Allow me to finish… so after the encrypted response is received… and we are talking milliseconds… every machine then connects to a series of IRC chat servers. I’m guessing here based on previous experience, but I’m pretty sure that the encrypted string is an updated list of IRC servers, channels and commands.” I said with a scratch of my head and motioned for Scrap to continue my thought.
“We’re also seeing a lot of file transfers between these IRC users. The machines here are transferring unknown, encrypted files to other users in the IRC channels. That is never a good sign!” Scrap lamented.
Cookie reached into her pocket and pulled out a small bottle of aspirin. “Well, per Kip you two have the run of the place. What’s the next step, Sherlock?”
“We lock this place down.”
Source: archived original
Part VI 路 May 6, 2008
“Talk about stopping the bleeding with a bullet.” Scrap joked on the phone as he removed the network cables from the Cisco PIX firewall. “Nobody is getting in or out now!”
I gave Scrap a few other things to check out and went back to enjoying my coffee.
Cookie was looking around nervously, like a lamb waiting to be eaten by a pack of lions.
I sipped coffee goodness from my cup and motioned to Cookie that I needed her undivided attention.
“Cookie, I need you to think back and remember when was the first time that you noticed this strange behavior. Did it involve a particular network or machine? This is important.” I asked as I dove back into my coffee cup.
Cookie put her head in her hands and swayed back and forth like a drunken ballet dancer.
“Cookie.” a disembodied voice called from a few rows of cubicles over.
“Cookie!”
Cookie sprung to attention.
“What’s up Cedric?” she asked loudly.
“Just got a call that internet e-mail isn’t working. You doing something?” Cedric asked.
“Yeah, we just disconnected from the internet. Send out an e-mail, would you?” Cookie joked.
“Very funny, boss.” Cedric replied as he appeared in front of us.
“Cedric, I was just asking Cookie if she remembered…” I said but was quickly interrupted by Cookie.
“Mail! Mail! Holy killer cat box, it was MAIL!” Cookie said as she slapped herself on the top of the head.
I was beginning to become concerned for her sanity.
“Chief, the mail server! That’s the server that started giving us issues first. It was just a day or two after those problems that all of this network nonsense started! I’m sure of it!” Cookie exclaimed.
I picked up my phone and dialed Scrap.
“Yo.” Scrap eloquently answered.
“Scrap, Cedric is on his way over to your location. I want you to pull the logs off of their mail server. It’s an exchange server.” I barked.
“Roger that.” Scrap said as he disconnected.
Cookie and I spent some more time together going over recent events and trying to map out an intelligible timeline of events, starting with the mail server issues that she described. The timeline was incredibly helpful.
According to Cookie their mail server stopped responding to incoming and outgoing mail one day. After performing a number of basic troubleshooting steps they resorted to rebooting the server and the problem went away. The next day at approximately the exact same time the issue happened again. After a reboot, the problem went away again. This cycle repeated itself once more and then stopped occurring. However, shortly after the problem disappeared on the exchange server they started to notice misbehaving network traffic, traffic spikes and self-induced denial of service attacks. The mail server was the obvious place to take a first look.
ring
Scrap was calling me with what I hoped to be good news.
“Chief, I’m on my way back over there with the logs on a flash drive. I hope you’re in the mood to do some event correlation, man!” Scrap said.
I was more excited than waiting for a liver biopsy.
Scrap arrived shortly thereafter and we began the long tedious process of looking through the server event logs, messaging logs, and application logs. Cookie and Cedric didn’t believe in log rotation, so we had some rather large log files to look through. While a bad system administration practice, this may have worked to our advantage by providing us an adequate event audit trail.
“Oh, what do we have here?” I said as I rolled up and down through the system logs.
“What ya got, boss?” Scrap queried.
“Interesting. Just before each outage on the mail server, the anti-virus service on this server stops and reloads itself. When we correlate that to the mail log, it appears that a certain e-mail is being scanned for attachments at each one of those times.” I mumbled.
“Are you thinking what I’m thinking?” Scrap asked as his eyes rolled across the logs on my laptop’s screen. “Paydirt?”
“Payload.” I responded.
漏 2008 All Rights Reserved.
Source: archived original
Part VII 路 Jun 17, 2008
“Why me?” Cookie moaned as she stared into the ceiling. “Why can’t this stuff happen to someone who doesn’t have to go home and knit small colorful costumes for her cat’s talent show?”
Scrap and I stared blankly at each other. Neither one of us could blink for the fear that we’d bust out in inappropriate laughter.
“Cookie, we’ve been through your mail server logs and we’re now 100% positive that we’re looking at either a mailer exploit or some kind of anti-virus software exploit. We’re not going to know more until we start tearing apart the e-mail (or e-mails?) that keep triggering this behavior. I need you and Cedric to do something important for me. Can you do that?” I asked.
Cookie and Cedric both nodded affirmatively.
“Good. I want you to visit your neighbors upstairs. According to your firewall configurations you’re forwarding smtp traffic up to a single box on each of their networks that I’m assuming is a mail transfer agent of some sort. Bring me log files. You now know what to look for.” I barked.
“Roger that!” Cedric hollered as he headed towards the stairs.
Scrap was rubbing his eyes with his paws while scanning through the raw traffic dumps that he’d collected earlier using Ethereal [Edit: Ethereal is now called ‘Wireshark’].
“Chief, we’ve seen a lot of bizarre stuff in our travels but this is just monkey madness. I’ve pieced together what I think is a rough example of what is going on when these network hemorrhoids flare up.” Scrap said with a grin.
Thanks for that image, Scrap.
I stood behind Scrap and reviewed his notes.
By George, he was on to something.
“Let’s run it down. Go.” I said with a snap of my fingers.
“Bogon e-mail is received by the mail server. Either the MTA or some other process on the server barfs on the e-mail and some type of code execution takes place. The mystery code that I’ll call ‘Scooby’ then tries to make an immediate unencrypted IRC connection to irc.badguy.foo. It’s logging in to a channel called ‘wtfover’ and receiving a series of instructions from what appears to be a bot named ‘Fred’. Scooby disconnects from the channel and then…” Scrap trailed off as he brought up a series of terminal windows.
“And then, it starts scanning the local subnet for hosts. When it finds an active host, look at what it does… do you recognize that pattern?” Scrap asked with both eyebrows in a Spock formation.
I chuckled. “You have to be kidding me. That’s a buffer overflow attack… it’s old… wait… if they keep their workstations as updated as they do their servers… oh dear God!”
“Exactly. This thing infects everything on the local subnet. All of those boxes then repeat the process and check in with the channel over and over again.” Scrap reported.
“So let me guess… eventually ‘Fred’ tells them to go to sleep?” I postulated.
“Bingo! But sometimes Fred doesn’t want the litter of Scoobys to sleep. He wants them to be bad puppies.” Scrap said dramatically.
“How bad?” I asked.
“How about search-the-local-file-system-for-keywords bad?” Scrap said as he pointed to a command sequence intercepted from Fred.
“Wait, you mean this thing back-hacks itself into a network, and then periodically looks around? It’s a freaking spider bot?” I asked. “What the hell is it looking for?”
Scrap scrolled down the command sequence and tapped on the screen.
s/ISO/AVI/MPG/MP3/MP4/JPG/GIF/TIF/TIFF/RAW/DOC/CSV/XLS/PPT/TXT/RTF/
“Looks like we have a warez bot infestation. Wouldn’t be the first one we’ve ever seen.” Scrap said.
I pondered the commands on the screen and rubbed my chin with a paw. I removed my phone from my pocket in a robotic fashion and quickly dialed a number.
“Kip? Chief. Just exactly what kind of production work do you and your neighbors do here?” I asked.
“Why?” Kip asked.
“Because you have more fans than you’re probably aware of. And not the kind that you want.”
Source: archived original
Part VIII 路 Jul 29, 2008
Click. Click.
“That e-mail has to be here somewhere!” I murmured to myself as I looked through the E-mail quarantine directories on the exchange server. I was determined to find one of the E-mails that kept causing the anti-virus process to reload itself.
Unfortunately for me, the anti-virus logs were lacking in detail. I manually scanned through each E-mail using the date and time as my approximate guide. It was a slow process and the chair in the server room was extremely cruel to my lower back as I worked on the exchange server.
“Chief?” a voice said cautiously behind me.
“Hey Cedric. What did you two find on your mail server log investigation?” I asked as I rubbed my tired eyes with my forefingers.
“We think we hit the jackpot. We found similar entries in the mail and anti-virus logs. The real interesting thing is that each of the exchange servers handles mail for completely different domains.” Cedric responded.
“Different domains.” I said while quietly postulating and scribbling in my notebook. “Interesting. I think it’s safe to assume that if all of the mail servers on each of the segments fell victim to this same attack, then an unknown number of machines on your neighbors’ networks were infected.”
Cedric nodded in agreement.
“Now we need the last piece of the puzzle.” I said as I continued clicking through the E-mail quarantine directory.
“And there… it… is!”
Cedric walked up behind me and glanced over my shoulder. “There’s what?”
“The mysterious E-mail that pains your mail servers.” I responded in an increasingly louder voice. “And look at that! It has meaningless text in the e-mail that appears to have been copied out of a bad Nancy Drew book and a very unfriendly-looking attachment.”
I picked up my phone and dialed Scrap.
“Yo.” Scrap answered.
“I’m ready to test your theory.” I said. “Are you ready?”
“Affirmative. The taps are still in place. Call it.” Scrap replied amid a lot of typing noises in the background.
“Plug us back in.”
Source: archived original
Part IX 路 Jul 29, 2008
“3… 2… connection restored.” Scrap said over the phone as he jacked the PIX firewall back onto the Internet.
“Hold on to your ponies!” I exclaimed as I right-clicked on the suspect E-mail and selected “scan attachment”.
Within seconds event viewer was showing a service failure by the anti-virus scanning service and a quick restart.
“Scrap - confirmed. The anti-virus service just reloaded itself after the scan. You should see a…” I said but was cut off by Scrap.
“BAM! There it went. Just saw that ping post go sailing by. Here comes the reply. Traffic levels are increasing. It’s the same as before. I think we can safely say that we isolated the issue.” Scrap announced on the phone.
“Pull the plug. Strategy meeting in the IT conference room in five minutes. Let’s round everyone up.” I barked as I disconnected the call and looked around for my notebook. Things were about to get even more interesting.
Cedric and I wandered down to the conference room and met Scrap, Cookie and Kip. I pulled out a chair from the end of the conference room table and jumped right in.
“We are at the moment of decision, folks. We have established what is causing the network outages and we’re nearly complete in our assessment of how much damage has been done to the machines on both this network and your neighbors’ networks. Obviously these companies can’t stay removed from the internet forever. We need to.. well, you need to make a decision on how you want this situation dealt with.” I said.
Kip maintained eye contact with me and nodded in understanding.
“Scrap and I are going to suggest two approaches to your situation. One is cheap and quick, the other is expensive and will take a good deal of time. Here are your options. Scrap, please do the honors.”
Scrap shifted his weight in his chair and cleared his throat. “The first option is to have us take a tactical approach to the problem. We’re going to need to fix the issue on the mail servers and then clean them of any malware or cruft. If we determine that they are not salvageable, then we’ll need to reload and rebuild them. At that point we’ll need to take a similar approach to all the workstations. Only after a thorough review and certification process should any machine go live on your network.”
I scanned the room and noted that everyone was aware of that requirement.
“The second option is that we treat this like a crime scene. The network would stay locked down and forensic copies of all affected machines will be made. Research would be done to determine if this was an isolated case of an errant bogon E-mail, or if this was a targeted attack. During our research we would work with your staff to implement the salvage operations in the previous option after we have determined that no additional evidence can be gathered at the scene.” Scrap concluded.
Kip’s left leg was bouncing up and down and he was pinching his lips with his right forefinger and thumb. After a few seconds of contemplation he sat straight up in his chair and looked directly at me.
“Option two. I need to know!”
Source: archived original
Part X 路 Oct 22, 2008
Scrap and I had clocked nearly an entire week of billable hours since Kip’s request for help. We rigged a temporary network of sanitized machines behind a known good firewall for temporary internet access and were nearly done with our forensic examinations and rebuilds of all the servers.
We were beat. Clues were few and far between. Scrap forgot to wear deodorant two days in a row.
Our luck was about to change.
“Chief, do you have a minute?”
Cedric’s voice woke me from a numbing gaze at my laptop’s screen.
“Sure. What’s up?” I replied.
“I was updating our network documentation for Cookie and I spotted a machine that I completely forgot about. It’s a machine that I think might be of some help.” Cedric said.
“Cedric, we’ve torn down every server in this place. What are you talking about?” I queried.
“This.” Cedric said as he pointed to a box on a visio diagram labeled “Piggy”. I noticed that the box was jacked into the same switch as the router. My mind performed a quick network nerd calculation.
“Span port configuration? Cedric, is that a SNORT box?” I asked with a look of astonishment.
“Yes it is, Sir.” Cedric replied. “That box was my personal project while working towards a security certification.”
I took a sip of my cold coffee and sat up straight in my chair. I was completely taken aback by this event and felt guilty that I had sold Cookie & Cedric’s technical skills short on our arrival.
“Hell yes that’ll help! Let’s get Scrap in here.” I announced.
“Already here, boss. What’s up?” Scrap said as he bounced into the room, nearly knocking me off my chair.
Cedric filled Scrap in on his project and proudly displayed a web-based console to the box based on BASE. BASE is the Basic Analysis and Security Engine which was developed from the code of ACID, or the Analysis Console for Intrusion Databases. The interface was very useful and allowed Cedric to perform quick queries of the SNORT IDS system.
“Hell ya.” Scrap nearly yelled as he buzzed through the SNORT logs at a lightening pace.
“While you’re having fun there Scrap how about looking for some events occurring before and after our little network outages?” I asked.
Scrap clicked around the screen like a highly caffeinated security Mozart. Within a matter of minutes we were scrolling through some interesting events.
“This is beautiful, man. It looks like shortly before those outages occur, a sneaky little SMTP transaction is executed to check the banner on the mail server. The banner reveals both the mail server type and version… and the version of the anti-virus software.” Scrap proudly announced.
I popped up an eyebrow on this news. “So if this happens every time, we should be able to go back through the SNORT logs and find the first time that query happens. What are the odds that it’s coming from the same IP address?”
Scrap wore a large smile on his face and cupped his hands behind his head as he leaned back in his chair.
“I’ll take that bet, boss… because they are all coming from the same IP address.”
I looked at the IP address highlighted on the screen. Before I could open my mouth, Cedric made a small squeaking sound and proceeded to drop a bombshell on us.
“I know that address!”
Source: archived original
Part XI 路 Nov 14, 2008
“I beg your pardon?” I asked after fully digesting Cedric’s admission. “You KNOW that address?”
“I sure do. That box is at Cookie’s house. It’s a company-owned machine that she uses to monitor the network. It’s a linux box. I built it myself about a year ago while I was in night school.” Cedric said. “But how are these attacks coming from that box? That makes no sense.”
Scrap glanced at me and rolled his eyes in frustration. I knew what he was thinking. We had seen this situation far too many times and had become nearly numb to discoveries like this one. For whatever reason IT folks tended to assume that boxes on their home networks were safe and immune from the same security threats as their work networks.
I rubbed my eyes for several minutes while Scrap wandered over to Cookie’s cube and interviewed her again. I thought really hard about all of the facts of this case. We now knew that a single box at Cookie’s house was responsible for probing the network, crashing the mail server, executing some arbitrary code, scanning the local network for vulnerable machines, installing malware, scanning all of the local file systems for specific file types and then reporting back to a master of some sort through a http ping.
But what was the motivation? Why hadn’t we witnessed any files actually copied off the network? Why would someone simply inventory a location, ping a web server, relay some IRC traffic and then disappear for a while? And, why would they do so in such a loud and noticeable manner?
“Boss - check this out. Cookie actually has an ssh connection open to that very box right now. She wants to know if we want to look around?” Scrap queried.
“Absolutely.” I replied as I followed Scrap over to Cookie’s cubicle and took a seat in front of Cookie’s computer.
Typically I would have taken a snapshot of the running processes on the box, the contents of RAM and then imaged the box. For this particular case, I followed my gut and dove in head first.
A quick ‘ps -aux’ showed nothing of interest. There was nothing in any of the crontabs, nor was there anything interesting in the logs.
“Scrap, plug us back in.” I said without turning my head.
“You want me to plug the network back in to the router? Say what?” Scrap asked.
“Just do it.” I mumbled. Scrap turned and left the room.
A few minutes later Scrap reported back that the network was up.
“Keep your eye on piggy. If I see anything weird on this box I want to be able to correlate it to…” I trailed off as my keystrokes froze.
“What the?” I asked as I pounded the keyboard with my paws. My ssh session was dead. We weren’t using the local LAN - we were using the secondary network that Scrap and I had set up while doing our examinations.
“That happens all the time.” Cookie said. “That box has a bad power supply. It reboots when it wants to.”
I pinched the crown of my nose with my fingers in an effort to not shake my head.
“Is it going to come back up?” I asked quietly as I pinched tighter.
“Sure, it’ll come back in a couple of minutes.” Cookie said with a laugh.
“I’ll be in the little monkey’s room. Be right back.” I growled as I lifted myself out of the chair and wandered toward the restrooms.
“Chief! Hey, where is he?” I heard Scrap asking outside the restroom.
I was in the middle of washing my paws when the restroom door swung wide open and Scrap slid through the door.
“You were right!” Scrap exclaimed.
“Right? What was I right about? I didn’t say anything…”
“About plugging the network back in. I just got a ping from Cookie’s box!” Scrap said while rubbing his paws together. “What did you do?”
I dried my paws with a paper towel and tossed the crinkled mass a few feet across the bathroom into the wastebasket.
“I think I have this figured out.”
Source: archived original
Conclusion 路 Apr 3, 2009
“What’s he doing?”
Cedric was whispering questions into Scrap’s ear as I busily typed away into a shell window.
“He’s doing what he does. He’s looking for a contact lens in shag carpet.” Scrap joked as he watched me searching through files on Cookie’s remote machine.
My paws stopped moving melodically over the keyboard and I exhaled loudly. Tension filled the room.
“Very sneaky.” I blurted out as I opened a file that was discovered by my carefully crafted grep command.
Scrap scratched his head and leaned in a bit closer. “Aw, that is just EVIL.”
A glance inside /etc/inittab, the general process “dispatcher” of the linux system, revealed a line that looked like this:
pf::powerwait:/etc/init.d/powerfail
which looked a lot like the original line that shipped with the system:
pf::powerwait:/etc/init.d/powerfail start
I nearly missed it.
This was an instruction to the system to execute /etc/init.d/powerfail after power was restored. Execute the file, wait for completion, then move on.
An inspection of /etc/init.d/powerfail revealed a single entry:
"/root/.kde/sentinel”
This was a dead giveaway. Not only was the KDE desktop environment never installed on this system (as verified by Cookie), but the directory permissions on the directory were all wrong:
drwxrwxrwx 2 root root 4096 2001-07-14 12:31 .kde
The KDE preferences directory permissions for a user would never be 777!
This rabbit hole was pretty deep. I had a hard time keeping up with my fingers as they checked the file type of “sentinel”:
$file sentinel
sentinel: ELF 32-bit MSB executable, ARCH, VX+ Required, version 1, for GNU/Linux 2.2.0, dynamically linked (uses shared libs), for GNU/Linux 2.2.0, stripped
_It’s a binary. Nice. Someone dropped a kit on this machine, but did a reasonably good job at covering their tracks. But what about the content? Was it compiled on this machine or elsewhere? 聽聽_
$strings sentinel
Several key strings appeared including the callback URL that was found previously, various file type extensions, and… the IP of the mail server? I thought back to Scrap’s comment days ago when we presented Kip with his two options and my currently tingling spidey-sense. Maybe this was a targeted attack?
Scrap, like all great investigative partners, was reading my mind and decided to share our assumption with those standing close by.
“We need a reload on coffee, and this is looking like a very targeted attack now. Typically one would find code that computes an array of IP addresses for recon and exploitation. This one looks like it was made to attack your mail server.” Scrap said calmly.
By the time Scrap had re-focused his concentration on my mad typing, I had already started to dig for the source of this mysterious sentinel application. A quick find command verified that there was no development environment on this system - it was a default install of the distribution. Interesting.
Netstat verified that there was only one listening service on the entire box: sshd. Searches through /var/log showed no evidence of previous services being installed or used to transfer files. No evidence of log erasing or activity hiding were evident until I did one quick scan for a well known trick.
$ sudo find / -name .bash_history -type l
/home/admin/.bash_history
$ ls -la /home/admin/.bash_history
lrwxrwxrwx 1 admin admin 11 2001-07-14 12:28 .bash_history -> /dev/nul
“Holy linkage to nowhere. That shell history log is symbolically linked to /dev/nul. Nice way to cover your tracks!” Scrap exclaimed.
“It gets better.” I touted as I flipped back through my notebook. According to Cookie’s timeline and the evidence gathered so far, this symlink was created just minutes before the ‘sentinel’ binary was created, and THAT was just a few minutes before the very first payload e-mail was processed by the mail server.
Cookie scowled. “I have a question. I just watched you look through the sshd log file, and there were no logins by that user. How is that?”
“Easy peasy.” I said. “Your logs rotate weekly, and you only keep 4 weeks worth of logs according to your logrotate.conf file. Any evidence of those logins was erased months ago. But I have a feeling that you know who used that admin account, don’t you?”
Cookie’s expression was growing uglier by the second. Scrap was on to me and played dumb. Cedric appeared completely lost.
“Unreal.” Cookie lamented. “Son of a …”
“Can I see how close I am to what happened here?” I asked Cookie while raising both of my eyebrows.
“Sure.” Cookie replied, still steaming about her revelation.
“PPP Productions has been buying up a lot of other production houses. What PPP really wants is the intellectual property of these companies, so they end up letting a lot of staff go and absorb the IP into their existing portfolio. During the last purchase, which coincidentally included every production house in this building, PPP management starts the latest round of layoffs. That’s why this building and the parking lots are nearly empty. Our suspect starts to fear for his own job and devises a plan to keep himself employed out of the necessity for his skill set. How am I doing so far?” I asked.
“Fine.” Cookie blurted out as she hung on my every word.
“The suspect uses his security background to hatch a pretty interesting plan. He has to create a situation where those in power would not question the need for his existence. He has to do it in a way that won’t point back to him, and will look so elaborate that everyone will assume it’s an organized crime group or a bunch of kiddy haxors. The plan unfolds with a misdirection; placing the source of the disruption on a box that is owned by someone else with the technical knowhow to accomplish the task, however assuming that nobody would ever put all the pieces together and look at this linux box.” I said with a glance towards Scrap, sending him a non-verbal hint.
Scrap cleared his throat. “The second stage was to figure out a creative attack vector. The suspect knew all of the technical specifications needed to find some attack code on the net, craft it to exploit a vulnerability on the mail server’s anti-virus software, and then unleash a trojan/worm to cause mayhem. The objective was to create havoc, make a lot of noise, but not actually do anything that a typical attacker would do like set up a bot master platform, a huge warez site or steal files. The fact that it kept doing it over and over was a complete accident, thanks to your failing power supply. It was only meant to execute every time you happened to reboot, which on a linux server is typically quite infrequent. The sequence went something like
SMTP transaction to check mail banner
Send payload e-mail
E-mail parsed, explodes, launches malware
Sends HTTP POST to command server
Gets REPLY with list of targets
Scan local subnets
Propogate with new command set
Scan local file systems
Report back/forth with IRC command centers
and then die.”
“And the ultimate subterfuge, which we’re well acquainted with, is to bring in the ’experts’ to increase the severity profile of the incident, all the while confident that the ruse will never be figured out. Unfortunately for the suspect, he was wrong.” I explained.
Cedric gave the international “time out” signal with his hands. “Hold up, Chief. Everyone seems to know what’s going on here. Who are we talking about?”
“The one person who should not have been, but was, absent throughout the majority of this investigation.” I replied.
Cedric thought for a moment and then repaid me with a wild-eyed stare.
“KIP!”
=*=
Four days later, Scrap & I were on a plane to our next gig. Our case work at PPP had been exhausting, frustrating, and just plain strange.
Cookie had arranged for us to meet with Kip’s manager, who happened to be a Vice President within PPP. Scrap & I presented our findings in a clear and easily understood manner. Cookie & Cedric took turns providing additional details and color. The VP gave us permission to search Kip’s office and image Kip’s laptop’s hard drive. It took us less than a day to find the incriminating artifacts that included source code to bots, trojans and worms. One particular source tree actually compiled into the binary that we found on Cookie’s linux machine. The bow on the proverbial present was finding ssh commands to Cookie’s linux machine in Kip’s shell history.
Kip was suspended with pay until PPP could have another investigator review our report.
One week later, Kip was unemployed, and the new security monkey of PPP moved into her office - complete with pictures of cats in costumes.
Source: archived original