Musings on SOC and CSIRT and “Upward Trajectories”

I’ve been working with a company recently to help them hire a new Senior CSIRT Incident Responder. There have been some changes to their team recently which means they can’t afford to take someone more junior and train them up into the position (which is this company’s preferred method of building seniors, and I love), they need someone who can hit the ground running and lead things today.

We’ve had a lot of people who are SOC analysts, and have been SOC analysts for at least four years, applying for the position. I love this, SOC staff are at the coal-face of incident response, dealing with everything from imaginary mouse movements to the very first detection finding an APT in your environment. It’s a very hard job, and honestly one that I think is a security specialism in its own right, but I digress.

A lot of people seem to think that SOC is an entry level job, and that if you put in enough time and get enough experience, you’ll naturally learn the skills you need to become a solid incident responder. And in certain cases this is true, but I think that the way SOCs are made up today, most of the time, you’re not going to get the well rounded diet of activities growing incident responder’s need.

Incident responders (when they’re doing incident response and aren’t stuck in the prepare loop from hell) are specialists at system forensics as well as tracking an attacker as. they move through an organisation. SOC work can certainly prepare you for the former here, as triaging alerts will naturally give you a really good understanding of system artifacts and where to look to prove hypothesis (persistence? Better check those run keys!).

I don’t like “tell me about a time when” based interviews. The problem with these interviews, is they have the benefit of hindsight, and the answer I tend to get from candidates is them telling me an idealised version of what went down, where the mistakes they made are covered up by what they learned at the time, making the candidate out to be better than they might be in the heat of the moment. Incident response is all about how you handle the unknown, the scary, and relying on your whit, which is why I will always incorporate scenarios at some point in the interview chain.

The challenge I’ve seen in interviewees recently is that they struggle to pivot their thinking from single host analysis to assessing the business as a whole. In an interview scenario, I might give them an IP address that the malware talks to, or a username that falls out of a command the attacker executed, and I’m ready and waiting for “neat! What other systems have connected to that IP address?!” or “bloody choice, what other systems has that user logged in to? All of them? By design? Fantastic, let me tell you how I’d separate the good from the bad!”. These conversations are awesome give the candidate a chance to really show off their stuff, and, most importantly, keep the interview interesting for me!

What tends to happen instead is that this critical information is glossed over, and the candidate instead tries to chase the things they know how to investigate. Part of this is absolutely to show off their knowledge, this is an interview, after all, and rattling off the various Powershell logs and their locations on disk is totally impressive. But I think another part, a bigger and scarier part, is that pivoting based on IOCs outside of the host in question just isn’t considered as part of their investigation.

I’ve looked around, and there’s not a lot of training on this. SANS will give you weeks and weeks of content on the difference between a prefetch and MFT file, how you can parse them, how you can feed them in to the massive list of findings you’ll generate for that host. SO will youtube, and home labbing, a zillion blog posts out there, etc.

What I struggle to find is training on how to lead an organisation-level compromise, filled with lateral movement, multiple compromised hosts, sleeper compromise that will only activate a month after you kick the attacker out and lead to recompromise of your environment, etc. Honestly, this baffles me.

But that’s for people who want to become incident responders. That’s what I would consider a critical skillset to develop. Is that was every SOC analyst wants? Is CSIRT the natural progression of SOC? I personally don’t think so. I think SOC is a specialty that stands on its own. SOC don’t get to take their time on a single host - SOC need to develop specialised skills to triage systems into “needs more work” or “this is a nothingburger” absurdly quickly so they can get through the deluge of alerts that make up their day. And I take my hats off to these people - I can’t do that. I’m not fast enough, and I dive down rabbit holes way too easy.

So for you SOC analysts who want to become incident responders, do it! You’ve got an amazing foundation of knowledge in your current role, and with effort, you can build upon that to learn how to incident respond well. Please, though, don’t think your 5 years in SOC with little extra effort has prepared you to step straight in to an incident responder role, because it probably hasn’t. You’re going to need to put in the effort.

And to those of you in the SOC who have made the coal-face your specialisation, who own it and truly excel at what you do, I salute you, and I look forward to begging you for help in my next incident response engagement, because you know I’m going to need you.