Continuous Monitoring and an 85% drop in Risk

This Article reminds me of the presentation I saw by Allen Paller at Infosec last year. Alan Paller is the director of Research at SANS. He presented a testimonial of his work within the State Department to congress. He highlighted the development and use of a continuous monitoring methodology, which has led to an 85% drop in measured risk. In part this was achieved by using a continuous monitoring approach which was an IT-driven system and replaced the existing paper-based reporting system. What made this approach effective was the usage of the metrics that were Comparative, Numeric, Reliable and Authoritative. At infosec Paller went into a little more detail about these aspects.

Comparative: Comparative monitoring needs to be something that can be used to show the relative quality between the effort undertaken by different teams. This creates a healthy climate of competition and motivation

Numeric: Existing FISMA standards dictate that the reporting systems should produce a risk report every few months or quarter. This is a terrific delay in terms of response time, and ability to gain instant situational awareness. One important factor in ensuring success was to automate the measurement of these controls. Without it, the overhead would have been a barrier to it being effective. The monitoring period was reduced to 72 hours thus having the effect of allowing a better response time, as well as illustrating gains being made.

Reliability: Based on repeatable tests, two or more evaluators would get the same results.

Authoritative: By getting a consensus from an acknowledged group of experts allows you to get buy-in from the very individuals who will be assessed by the measurements.

It is an interesting approach as it advocates a far more scientific methodology to monitoring . However, Paller also highlighted the fact that the human element needed to be considered as equally as important. Therefore, be fair when measuring metrics. If a team cannot change or effect something, it is not fair to measure it. Finally celebrate success and make progress visible.