Often just know that something is working is not enough. Knowing how and why something is working can be important to its future success. Conversely it is just as important, if not more so, to understand why it’s not working. Tracking metrics is one way to point out an open source community’s strengths and weaknesses, but which metrics should be tracked?
Activity – The evolution of your community is a particularly helpful metric to track as it will provide an overview of the activity of your open source community. You can measure the number of commits to get an idea of the development effort volume. The number of opened tickets will give an indication of the feedback you are receiving, whether it is bugs being reported or proposed improvements being submitted. Comments, posts and messages will give you an indication of how much discussion is going on.
Demographics – A community can change over time as past contributors leave and new contributors come on board. The age of the community will vary as a result, based on the time since members first joined. To get a good measure of the community’s retention rate you will need to measure the number of new members within a pre-determined time period and measure how many members still remain at the end of that pre-determined period. It is not uncommon for the retention rate to be high in the early stages of a community and reduce over time.
Diversity – Diversity can significantly impact the resiliency of a community. In general the more diverse the community is the better able that community is to recover from the loss of a contributing member or company. For example, if there are 20 contributing members but only 2 members are contributing 90 percent between them and one of those 2 contributing members leaves it will be far more difficult for the community to withstand the loss than it would be if one of the other 18 members who are contributing the remaining 10% collectively leaves.
Performance – This is a measure of the likes of how quickly processes are being finished. Another indicator of performance is how quickly opened tickets are resolved and closed. This will show how quickly the community is responding to opened tickets and addressing the issues and working to find a solution. Another performance metric is the ration of opened tickets to closed tickets. If open tickets far outnumber closed tickets it may be an indication that the community is not performing particularly well, or it could mean that there are just too many bugs that need fixed. If it appears that the number of open tickets is not decreasing over time it may indicate a need to allocate more resources to bugs and code reviews.
Size – The number of people participating in the community is another very helpful metric to track, but while the number of participants may indicate a large community it is the number of people who are actively contributing that is most important. Typically a small fraction of the total community will be the ones making the largest contribution.
