#3. Beware of Cobra Metrics
‘Management by Metrics’ does more harm than good. I will talk about three of the many such metrics used in software development — calling them ‘Cobra Metrics’ — that actually are detrimental to the purpose they are used for.
Before that, let me take you back to the British Rule in India. To control the snakes population in India, the British authorities at that time initiated a program that rewarded any person who brings in a dead Cobra. This program, after initial success, led to a reverse effect — as people started breeding Cobras to get more rewards — thereby leading to an increase in snake population — much higher than it was before the program. This reverse effect of a policy is called the Cobra effect.
The same Cobra Effect can be observed in software development when we get too fixated about metrics numbers and start incentivizing teams based how they perform against those numbers. 3 of the many such metrics (Cobra Metrics) are:
1) Unit Test Code Coverage
One of the quality gates set by (for) engineering teams is >X% (80,90, 95…) unit test code coverage. Setting such targets leads to two common issues: (a) High Unit Test Code Coverage gives a false sense of good product quality and hence a tendency to ignore other tests (like integration tests etc.) thereby leading higher bug leakages. (b) In order to achieve high coverage scores, developers begin to write junk unit tests that target ‘petty functions’ with no business logic/algorithm. These unit tests in many cases become mere replica of the logic written in the functions being tested and require to be modified whenever implementation changes. Thereby, increasing the code surface area where bugs could occur and leading to additional maintenance.
So, instead of improving the product quality — too much fixation on Unit Test Code Coverage could actually lead to deterioration of the quality and making it a Cobra Metrics.
2) Functional Test Automation Percentage
Many teams’ efficiency is measured by the percentage of Functional tests that are automated. So much so that they establish a separate ‘Test Automation Team’ with the sole objective to automate functional tests. In my experience, such a strategy instead reduces the overall efficiency of the team.
As the tests are automated by a separate ‘Test Automation Team’— this team also becomes responsible for executing and maintaining them. Such tests suites does not see the light of the day as they never get integrated in the Dev workflow — Devs tend to bypass the tests as they do not understand them or feel responsible to maintain them. The test suites are then always executed later in the cycle and at that time it is realized the use cases have been modified which requires updation of the tests.
Despite Test Automation percentage getting increased, the overall team’s efficiency actually decreases— making it one of the many Cobra Metrics.
3) Velocity Variance
I have been a part of many such management reviews where questions were raised on the variance in Velocity with each sprint. This variation is a normal phenomenon — and ideally velocity is an average concept and to be used internally by the Dev teams. When velocity variance becomes a metrics with a target range, teams rather than inspecting and adapting from the velocity trends start either burning out to ensure the committed velocity is met or they start planning for lower velocity.
Velocity is a measure that helps team to plan better but when its variance starts being used as a measure of team’s commitment and estimation accuracy it becomes a Cobra metrics.
Conclusion
‘Management by Metrics’ is not bad — as the expectations are objectively defined — it helps in clear communication and measurement. Just defining the targets numbers without a clear understanding of the intent of using a metrics is a problem which could lead to more harm than good.
Beware of such Cobra Metrics!
Have you encountered any Cobra Metrics in Software development? Please share your thoughts.
Title Photo Credit: My 9 YO.
Further Readings:
https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
Good article