I’ve been approached a couple of times in the past few weeks regarding the “scanner” and “vulnerability management” vendors that are touting their all-in-one approach to security vulnerability assessments and compliance scans. The interest has been around PCI DSS and specifically Rapid7’s solutions (apparently their marketing folks are doing a good job). There are other vendors coming into the space as well including a big one being announced at RSA next week.
Here’s the deal. Let me be clear. Regardless of what you read and regardless of what you’re told, you have to understand that these scanners are *not* going to find all of the vulnerabilities in your hosts and Web applications. I found this out the hard way a long time ago and started writing about this years ago here and here. In my experience scanning tools are great for a baseline assessment but you can’t depend on them to find everything. Nor can you expect that everything they find to actually be of any consequence in your environment like I talk about here. It’s all about context and common sense…security scanners in and of themselves don’t have much of either.
In every single security assessment project I’ve completed over the years not one automated tool found all the things that mattered. In every situation I uncovered big, big stuff (like files containing PII sitting on public servers, login weaknesses, application logic flaws, and more) only by using manual analysis. That is, picking up where the automated scanners left off and poking around on the system using manual hacking techniques and trickery.
These automated scanner solutions are also have a heck of a time figuring out how to login to certain Web applications to check for weaknesses as an authenticated user like I discuss here. With certain form-based logins and multi-factor authentication systems it can take hours of manual analysis – by a human – to figure out how the login mechanism works just so you can record a login macro or startup script that allows the tool to gain access and perform an authenticated scan. Unfortunately some people (and scanners) blow through this, assume that authentication was successful and not uncover any vulnerabilities – even though they still exist….they just haven’t been found because the authenticated scan wasn’t performed properly.
Don’t get me wrong. I use these tools. I tout them all the time here on my blog, my articles, my books, and my presentations. I couldn’t do a lot of my work without them! I’m even looking into expanding my service offerings to include such scanning services.
All of this said, before doing any sort of testing, you have some homework to do. So prepare internally and know all the facts before you get into this…and never ever assume that automated scanners are going to find every security flaw that matters to your business, or your auditors, or your regulators. They won’t.