Ways to measure test coverage
By DeviQA on Fri May 26 2017 00:00:00 GMT+0000 (Coordinated Universal Time)
Anyone who relate to QA will tell you that there is no way to measure Test Coverage with 100% accuracy. No matter what tools or metrics do you use, if test performed manually or by automation. It’s like you want to measure results which are currently shown as object in three-dimensional coordinate system with using of the two-dimensional coordinate system diagram. Don’t matter how much metrics, tools or methods you will try to use you will always have gaps in results - areas which were not measured. And even if you will succeed with gathering of such massive of data it will require inconceivable amount of resources and time.
1# Advantages of test coverage measuring
Some metrics are better than non of them at all. By measuring your test coverage you will get next important advantages:
- Let it not be 100% accurate result, but it still will show how good your current test coverage is which will help to understand how much time it will require to finish with project tests.
- According to gaps which will be noticed in test coverage you will be able to add new required tests which will improve quality of project.
- According to test coverage statistic you may notice which test do same job and reduce amount of overdone tests on your project which will save some additional time and resources.
2# Basic methods to measure your project test coverage
There are lot’s of methods which are using different metrics and tools to gather information which will be later used to indicate the actual test coverage. Many of them are used only for specific types of projects which provide specialized data and can’t be used out of scope of such kind of projects. So let’s take a look at most popular from methods which may be applied to almost any type of project and way of testing (manual or automation):
- Test coverage by feature - Is most simple one. First you need to prepare a list of all features which are available at your project and then check that all of them are covered with tests. This way of test coverage is not really informative regarding how good those features are tested but it will quickly show if you don’t forgot to cover with test any of features. It’s like first step in proper test coverage - first you need to make sure that all features are covered and then you need to check how good those features are covered.
- Test coverage by instrumentation (test code coverage) - Not so simple method as first one. It use a special tools which will check what part of code was executed while you perform some action in system. Results which will provide such tools will indicate the percentage of code which was executed during your set of tests (usually automation tests). But pay attention that if code was executed it don’t necessary mean that expected result was received during this action. So same as in first case it also don’t indicate how good the project was really tested. But knowing that all of the system code is affected by your set of test is also one of goals which you need to achieve to have most top test coverage of your project.
- Test coverage by GUI - Any project like mobile, desktop, web etc. has some user interface which may have lots of screens, pop-ups, drop-downs, tooltips, buttons, links, scroll-bars and other stuff which also should be covered with your tests as all those elements will be seen and used by user in first place. You need to make sure that all buttons which user may want to click will work exactly as required.
- Test coverage by transition - This method count all number of “ways” or “path” which may user try to use to receive the required results. It’s like you may get to job by car, train or jet and return home from job also by all this ways. So all those ways should be tested well. It’s really important to cover this for web applications where user may also navigate through URL or links and not only by buttons and options in drop-down menus.
- Test coverage by scenario (use cases) - One of most important measures of test coverage but in same time it really hard to measure it. At least without using of relevant tools. Now there are many products which allow to trace your users activities and record all actions and ways which they used on your site for example. This will allow you to have list of use cases which are performed by users while they use your project functionalities. And if you covered all of them on high level - it’s a really low chance to miss some critical bug during the testing. But what about case when you have a new functionality? Case when there is no statistic of how users will decide to use this new feature? Yeah... In such case this method of test coverage measuring sometimes becoming even less accurate than first three methods. But still qualified QA team is able to prepare such list of use cases which will be most close to what will real users do so still it good to have this test coverage measure results as well.
You can’t get 100% accurate way to measure your product test coverage but by combining of different metrics and tools you may get number which will be most close to actual one. The methods of test coverage measuring which you should choose are totally depend on your project functionality and available resources, time. But by having such data you will be able to have more or less stable and high test coverage level of all project aspects during all time of development and supporting which will significantly improve quality of your product!
Hope that this article didn’t leave any open questions for you and it was interesting to read it. Have a nice day and make good effort in improving the quality of software in the world!