I really like this quote:
> a lot of the value of code coverage data is to highlight not what’s covered, but what’s not covered.
Be liberal about marking code not worth writing tests for skipped for the coverage report, and set your coverage threshold to 100%. The report can then remind you about parts of your code you meant to but forgot to write a test for.
This is also a good way to introduce tests to an untested code base: ignore all existing files for the coverage report, then set the threshold to 100%.
Arbitrary thresholds are a great way to invoke Goodhart's law. Code coverage should be an assistant, not a stick to hit you with.
Shouldn't we try to change how people see code coverage values, instead? When I see a code coverage of 100%, I'd rather it meant that the whole code is covered (even though that does not mean there are no bugs) rather than having to check the coverage settings to see how many files where ignored to reach 100% ^^"
@tfardet Not IMHO - whether the code is covered isn't that interesting to me. The only thing I care about is whether a human has looked at the code and decided that the necessary tests have been written. If it's been ignored, with the above strategy that should mean that that was the case, so no need to look at how many files were ignored.
@tfardet Note that the transition strategy is to ensure that new code is tested, leading to a higher-quality codebase as larger and larger parts of your codebase are "new".
But yes, if you also plan to spend the effort to write tests for existing code, you will have to manually unignore them. But you can now do that one by one, rather than having to cover them all in one big bang.
@vinnl what I meant is that, rather than "hiding" the fact that some code is not covered, it should be OK for a project to start with a low code coverage, and increase it bit by bit.
That's what I'd like to see as an answer to
> Code coverage should be an assistant, not a stick to hit you with
I feel that checking all files also forces contributors to pay attention to tests and coverage when they make PRs, otherwise it's easy to miss the fact that some parts of the new code are not covered.
@tfardet With that strategy, at what point do people write tests for the old code?
@vinnl when they address an issue concerning old code, when a discussion occurs that raises concern about said part of the code, when they want to increase code coverage... plenty of occasions ^^
I'd return the question: in the strategy where the files are ignored, when do they write tests for old code that is not "dragging the coverage down"?
In my view there are now fewer incentives ;)
@tfardet So... The tests for the old code are not written because of the coverage threshold, but because the engineer is a good engineer and adds those tests on their own accord.
So the answer to your return question is the same: on their own accord. And meanwhile, the coverage report helps them not forget to write tests for new code.
@tfardet Btw, I think your last sentence is interesting: for me, it's not an incentive, but a tool that can help engineers. Hence an assistant, not a stick :)
@vinnl I see, maybe we have different conceptions of incentives, to me an incentive is no stick, it can be a carrot too.
I think the main issue here is people considering that it is bad to have low code coverage. While it is usually better to have high rather than low code coverage, I think the main issue is to make people understand that a low initial code coverage is perfectly understandable and should not be used for shaming/as a stick.
@vinnl however, people who like their project will usually want to see code coverage increase (because of various psychological effects) and may therefore just write a PR increasing code coverage when they have time.
This is I think a good thing.
@vinnl having said that, I myself do ignore some of my folders in one of my main project, simply because I currently cannot test these files on Travis, yet I kept all others and I had a rather low coverage of 30 for quite some time.
Whenever I make a PR, though, I do try harder to cover everything now that I have this additional "increase the coverage incentive".
I'm not saying it's especially smart, it's not, but it works...
@tfardet Fully agreed that people need to want to increase the quality of their code base (and need to know what tests to write to do so). And I see your point about a rising number serving as motivation - for me, a shrinking list of ignored files would have the same effect, but that will be different from person to person.
If a team wants to improve their coverage and think a rising number motivates them best to achieve that, I'd be happy to be on that team :)
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!