poniedziałek, 10 sierpnia 2009

Static code analysis - how much is it worth?

Recently, I've been using some tools to statically analyze my code. These tools are:
StyleCop
FxCop
CloneDetective

The sooner you start using these tools, the better. If you start using them in early development of your project, they may give you some hints about structure of your code and lead you to some good patterns. However, if you try analyze a project that consists of thousands of lines... well, what can I say - you'll just get the message that you did something wrong.
For instance, if you always use ToString() without specifying CultureInfo you'll get so many warnings that it would be almost impossible to correct them. To get meaningful notices that are important to your project you'll have to uncheck lots of rules in FxCop and StyleCop.
Clone Detective is a bit different from the previous tools. It analyzes source code to find code duplication. It's nice to have such a tool, especially when working with rather big projects. It may show that you're writing something that has been already written and also it encourages people not to use copy and paste. In one of my previous posts I wrote that code generation is just another way of doing copy and paste, my buddy Clone Detective says same thing ;) Yet again, if you start working with Clone Detective early, you'll be glad to see code duplication statistics showing maybe 5% of duplicated code. However, if you run it against a big project and see statistics at 40% or higher, it will just show you how miserable your life is.

There are lots of resources about all of these tools (guides, podcasts, etc.) so I don't want to talk about it in details. Yet, I wanted to state my current opinion on all of these tools.
If you're writing code that cannot pass static code analysis tests, I don't think these tools will "show you the way". These tools are for people who want to write good, bug free code. Good code is far more than sequence of brackets and letters. Good code has meaning, has design and all these little things that static analysis just cannot check. If you have some time and want to write and learn how to write code according to standards, then, hell yeah, give it a shot. However, if you are going to spend one hour writing unit tests or one hour adding all those CultureInfos - write some unit tests.
I was using these tools on my project that has lots of unit tests and I think that it was the main reason I was able to easily resolve all warnings. To be honest, I don't really think that the code is better after the analysis, but it's nice to see that I don't get any warnings. Also, I agree that these tools showed me some nice patterns that I didn't know about. But the bottom line is that I wouldn't be able to correct my warnings without unit tests. What is more, maybe my application is more portable and code is sticking to the rules, but it passed the tests before, and these tests are the real mark that everything is fine and the project is healthy.
Recently I ran a static analysis against some big project at work and they just showed me that the project sucks (as if I didn't know that before). I also think even if it would pass fxcop and stylecop tests it would still suck (clone detective is a different story). As Homer Simpson said: "This book has no answers". These tools also don't have real answers. Maybe some day these kind of tests will be run by compiler but we all know that code that compiles is not an indication of a bug free application.
I'm still under the spell of Bruce Eckel's article Strong Typing vs. Strong Testing.
Static analysis, in my opinion, is somewhat like strong typing.

P.S. My friend wrote an interesting post about static code analysis - Static analysis tools – one step closer to writing bugfree software.

2 komentarze:

  1. Hi there,

    Your post appeared courtesy of a Google Alert and I thought it might be worth commenting. Whilst what you say in your post is very true of a class of analyzers (notably those you tried), don't confuse style checkers with static analysis as a whole. Commercial/research SA tools tend to focus on defect detection, or architectural coherency, or security vulnerabilities, leaving style checking as an "also ran" part of the process.

    But talking of the types of tool you tried, Microsoft themselves published a study of how people used FxCop, for example, and found that the most useful checker they had in the box was one that verified that the various components of CamelCase tokens were correctly spelled English words. Hardly a great achievement in terms of program analysis...

    Try a real tool before writing off a research domain that holds considerable promise. All of the commercial vendors have free trials available, and there are some open source products (notably FindBugs in the Java space) which do a very commendable "starter" job in terms of introducing real static analysis.

    Hope you continue to explore and enjoy your profession.

    Regards,
    Gwyn Fisher
    CTO Klocwork, Inc.

    OdpowiedzUsuń
  2. Hi,

    Thanks for your comment. I'm going to check other tools. Could you recommend some products for C#/.NET Framework platform?

    OdpowiedzUsuń