In early September, Forrester published its “ The Forrester Wave™: Network Access Control, Q3 2008 .” Forrester’s findings revealed that Microsoft, Cisco Systems, Bradford Networks, and Juniper Networks lead the pack because of their strong enforcement and policy, but that Microsoft’s NAP technology, despite being a newcomer, has become the de facto standard.
Any time you try and put some order to vendor solutions, you are bound to find people in agreement – and to raise ire in others. However, reaction in the blogosphere to a recent Network World article on the research has raised some questions about Forrester’s Wave methodology which I’ll aim to address:
- Security Incite writes, “The Forresters checked out a bunch of data sheets and decided Microsoft was “top of the NAC heap.”” If only it were that easy! More than 150 hours of analysis went into the Network Access Control (NAC) Wave, in which we analyzed more than 70 criteria (encompassing more than 200 attributes) for the 10 vendors that were included in the study. The criteria was based on more than 200 client inquiries I’ve fielded in my five-plus years covering the space.
- Our study was not based on the number of units sold or performance tests – it was based on real-world challenges faced by very large enterprises and not an academic exercise (for the record, the sky is blue in our real-world ). Those common metrics are, for lack of a better word, useless. I know for a fact that several of the vendors in the study are giving their product away to gain market share, so how can unit volume equate to the quality of the product? Where does that fall on the BS-O-meter?
Performance tests are even less dependable because there are too many variables to consider and all too frequently are vendor-sponsored. Security Incite writes:
“…People that really buy products understand that a good RFP response gets you in the bake-off. That’s when things like “performance tests” start to matter.”
Since when is performance a critical factor in security? When’s the last time you heard a security pro say, “It doesn’t protect us, but boy does it scream with speed when it lets harmful users get by!”
Bottom line, I believe the NAC Wave was a fair, balanced and comprehensive study that looked at the products of 10 leading vendors in the space and concluded that Microsoft was the leader based on vendor strategy moving forward – even Security Insight and Security For All acknowledged it would be a standard in the future – and their presence in the market. While we may have to agree to disagree on the results, the methodology that helped us reach those conclusions should not be called into question.
Check out Rob's research