Analytics -Yes Please! (Help?)
CIOREVIEW >> Cognitive >>

Analytics -Yes Please! (Help?)

Nathan Romine, VP Artificial Intelligence & Security Analytics (AISA), Optiv

As a CIO or CISO, you know that the business has to make money, the budget needs to align with the next 1-3 years of business strategy, and you have to show progress towards a maturity score, a set of compliance or regulatory or legal requirements, or a corporate monetary figure.

Building this strategy is easy at first blush, you put a high value on eternal “hot” topics such as automation and “next gen” toolsets, you place emphasis on defining and prioritizing your highest risks and vulnerabilities, and there is always a need to drive compliance or regulatory or legal requirements.

If, however, we want to meet the ever increasing demands of security and technology with a shrinking budget, hard to find skillsets, and highly dynamic tech environment, we need to have a much cleaner way of knowing ourselves.

  ‚ÄčThe entire Analytics process is designed to drive you towards maturity and insight, without spending too much time and effort on the wrong things, usually in hindsight   

How do we do that? Which tools fit these needs? How can analytics (with a focus on security specifics) help us meet or define this strategy? Let’s dive in and review tools versus process, and how we can best utilize them for our needs.

Tool and Tech Section

Alright! We start with a budget that has Capital Expenditures (CapEx) and Operational Expenditures (OpEx) dollars. Most of the OpEx is spent on internal resources, with some reserves for things like compliance, IR, and professional services. There is usually a good chunk of CapEx each budget year, and the typical 4th quarter scramble to figure out what the next-gen thing for security is, and see how much it costs.

Success! The major conferences have huge booths with cool words like “Analytics” and “Machine Learning”, and they seem to have a 99.999 percent success rate for doing the thing they do, according to them. Let’s get that tool!

9 months later, you have the tool implemented, and you are sitting at the web page or command prompt. You have seen the initial bar graphs and pie charts that is their standard dashboard…pretty cool. In your companies colors. Now what?

If you have the training and the understanding of how the tool works, you can probably pull together some interesting in-the-box data points about various log sources you have. Bar graph of today’s attack alerts? Check. Pie chart showing the top 10 IP addresses from your firewall? Check.

In some scenario’s, this is enough. This ticks the box for a compliance need, or a reporting need, or a senior leadership need. This is great if it is the case. You bought, built, and use the tool for your environment, and no further action is needed.

There is in most environments, a much deeper need. If you get the top 10 addresses from your firewall, you may find out that they are Facebook, Google, YouTube, Twitter, and a few streaming sites (especially in late March, early April!). Most analytical tools will show you this. The questions that we really want to answer though are: What does this mean for your risk profile? What does this mean for your security profile? What do I DO with this info?

[If it isn’t explicit or obvious from the Top 10 IP’s above: most attackers know that the top 10-20 IP addresses organizations see are things with Facebook, Google, Twitter, etc. that is why those are such easy and appealing attack vectors–it can hide in plain sight. It is very difficult to distinguish a good tweet from a bad tweet, and even if you could, what happens when it switches to Spanish, German, or French? What happens when it uses port 9001? Should you block that URL only to have the same malware show up on 600 others? How much time did you spend putting in change controls on your proxy to block the bad URL’s? We could go on…]

In order to get the most impact out of our tools AND our internal resources (people, skills, and data), we need to gain insight, not just more data points. That nine color stacked 3D bar chart that uses flash animations is pretty cool, but if the security team ignores it 100 percent of the time, it may be time to look at doing it another way.

The Process Section

Let’s build an analytics process, using the same budget.

An analytics process involves taking the time to build a profile for your needs. This is the first insight you will gain, and no log data or tool is required, just some time and experience. If you or your team doesn’t have the experience, gather that information from your org. If you can’t get any data from your org, look outside your org. You are unique, but not 100 percent unique. At least 80 percent of the strategy you will struggle to implement is shared by every other organization out there.

Once you have this profile, take some time to define how you would measure its effectiveness. You will need to define what “effectiveness” means for your organization; generically it means having data for past states, current states, and a definition of what the goal is (future state). You then strategically apply resources (money, people, and services) to get to that future state.

Few tools can do this. If you compare how much time and skill it takes to pull some data and put it in a spreadsheet, you could compare the same amount (or more) of time and skill is needed to understand the nuances of a new tool. Training is usually expensive, the time for your resources to take and use the training more expensive, and when you factor in the work it takes to coordinate all of the logistics for implementation of a tool (rack space, power, shipping, installing, configuring, etc.), it makes sense to start with a profile, a process, and some ideas rather than an expensive tool.

Does the spreadsheet have the nine color stacked 3D bar chart that uses flash animations? Probably not. Does it answer the question you really have, the one we defined generically as “Where do I spend my resources”? Yes, or at least it got us to the point where we know what we need to gather to get that answer.

How Do I Start?

It’s easy: define a multi-variate non-linear weighted statistical algorithm that consistently updates itself based on sophisticated parsing engines and probabilistic machine learning. Oh, and we want it to be beautifully visualized with lots of animation. Right?

Let’s start simple–You need to analyze your current state with the data you have and the tools you have implemented. This is usually where your current stable of tools and skills will do. Pull vulnerability information. Gather metadata on your IPS, Firewall, and proxies. You can usually pull around 9-18 months of data from these tools (retention rates and data storage vary). Throw the information in a spreadsheet and graph it. Just a simple graph! You now have trends.

Next simple step–how much money have you spent on each tool? See if you can tie something to measure together from this data (think “Miles per Gallon” or “Dollars per Ounce”), and try to show the same measurement across tools. You can generically think in terms of percent of FTE’s per tool to build, utilize, and maintain; or Dollars per endpoint per geography; or even simply Cost per Vulnerability to Detect (or patch, or test…)

Which trends jump out? Now you can do Analytics, diving deeper into some things and pushing others out of the way; you can start to focus on gathering better or different data to define points more clearly, as well as set strategic 1-3 year plans in motion to reach that next maturity level. You are no longer a slave to a Top 10 list, and can utilize this method for driving insight from your data.


The Analytics process outlined above may drive you to buy an “Analytics” tool. In most cases it usually involves the death of one set of tools and the birth of another set. The entire Analytics process is designed to drive you towards maturity and insight, without spending too much time and effort on the wrong things, usually in hindsight.

The most important take away from the process is:

• You know what questions you need to have answered
• Which data to gather that gets you closest to those answers
• An ability to analyze past and current states to drive future state / goals

Best of all, you have most of this information already in your environment and probably have the spreadsheet software to use it…Now go analyze!

Read Also

International Expansion In Apost-Pandemic World

International Expansion In Apost-Pandemic World

Pearl Mathew, Director, Fintech Corporate Banking at Standard Chartered Bank
Intelligence-Led Investigations And Better Information Sharing Hold The Promise Of Improving The Disruption Of Financial Crime

Intelligence-Led Investigations And Better Information Sharing Hold...

Lora von Ploetz, Head of Global Financial Crime Unit, Commerzbank AG
How Dutch Bros Coffee embedded culture into its mobile app

How Dutch Bros Coffee embedded culture into its mobile app

John Graham, Chief Marketing Officer, Dutch Bros
The Calculus of Cyber in the Context of Enterprise Risk

The Calculus of Cyber in the Context of Enterprise Risk

Seema Sewell, Director of Cyber Assurance and Architecture, Maricopa County Lester Godsey, CISO, Maricopa County