Five Basic Forgotten Security Alert Truths
Here is a fun one: everybody whines that organizations have too many alerts, even the makers of the tools that produce alerts. Everybody! Everybody!! Everybody!!!
When people whine, their lamentations often obscure a few basic truths about alerts. These are:
- Some organizations don't have too many alerts, they just have too few people - their alerts are all legitimate alerts that need human triage; automation already did its job, now people must.
- MSSP is often seen as "solution for 'alert problem'" - but guess what MSSP would send you?! Yes, ALERTS! Thus, it is not The Answer. Will there be cases where MSSP sends you 'bad' alerts that waste your time? You bet!
- If you have a magic Wand of Alert Handling, and you wave it - and magically achieve perfect [however defined!] alerts handling, is that a WIN? Yes, a WIN - of a 1998 battle ... in 2015.
- Specifically, perfect alert handling does not give you ANY recourse against things do not produce an alert, even an obscure low severity one. This is definitely the case for the "unknown unknowns" and likely also for "known unknowns" (like things you know may come your way, but that you have no way of detecting).
- Still, we need alerts with better context, we need more triage automation, we need deeper data on endpoints/traffic, etc – however, there will always be alerts, and some will be false. If your “false positive” rate is zero, I can bet anything that you are missing the important, but weak signals... and they do matter!
There you have it -- when you think of channeling all your energy towards better alert handling, keep this in mind!
Possibly related blog posts:
Second the comments from Jeff Multz, CFS on generalization. However, I don't think this should be considered a negative thing. My read of this post is simply - "Stop whining about alerts and think about how to really handle them - not just by passing the buck on to someone else". Classic case of where an MSSP can never replace the customer is on how important an alert is to the business - no MSSP can wave a magic wand and cover 100% of the use-cases. A large number will be unique to every single organization. And that's where the point of "bad" alerts comes - it's not bad. It's just, customers just assumed MSSPs had magic wands.
6. Sending certain alerts to a group of people increases accountability within that group. 7. Tuning out alerts means you might miss something important. Accept this fact so that you can receive fewer alerts that are more meaningful. If you don't, your brain will tune them for you. 8. Re-baseline your alerts at least once per year. Are you still ignoring that one server that spewed thousands of alerts that one time but was fixed two days later? 9. You can't get the alert if the SIEM isn't getting the data. 10. Don't send alerts to your phone for things that can wait until the next business day. You'll be annoyed at being woken up and will likely shut off or silence your phone.
A good case for automating those repetitive alert triage tasks (ie, bouncing IPs/domains off threat intel, pulling in contextual information about affected internal systems/accounts, etc). Create a situation where analysts are looking at data, using their brains, and making decisions, rather than looking at raw alerts and manually working through admin tasks to pull together the relevant contextual data.
Anton, what metrics or indicators would you suggest focusing on as opposed to numbers of alerts, alert response time, false positive rate, etc.? How can an organization measure and improve its ability to pick up on the weak signals you mention?