Big Data in Security: Who should be doing the analysis?
One of my current projects includes the ability to replicate traffic into and "offline analysis" stream for those tools that are too slow or too complex to process traffic inline. Part of this is the "big data" component, which means things like Hadoop, etc. As a forensic activity, where forensic is meaning "after the fact" more than anything else, where is the true value? Anything that is found to be an attack, or exfiltration, or whatever is great, but it is "after the fact." If the attack succeeded or the data has been exfiltrated, you can stop the same technique, but you've already lost, as it were. So what then, is the value if it's only really preventing "next time"?
The part that is lost in many cyber security organizations is that there are two parts to good cyber security: offense and defense. The two halves should not function separately; two halves make a whole. In the big data of cyber security, it is equally, and I would argue vitally important that your offense, be it internal red team or a contracted penetration testing organization, have access to the data. In order to form the complete picture and gain the most value from the non-trivial investments required to make use of big data analytics, all interested parties should have access, and they should be collaborating and collating and coalescing findings together. Big data is about shared data, and in the cyber security world, you must have as broad and as clear a picture of what is flowing through your network as possible, or you're only going to be getting a fragment of your potential, and a fragment of your potential return on investment.
Posted in Newspaper Post Date 05/22/2016