Evaluating Open Source Packages
Today there is an abundance of open source software packages available for use. Open source use is ubiquitous—even in proprietary systems. Surprisingly, most organizations don't have a formal process or training for determining what packages should, or should not, be used. Typically, the decision to incorporate a package is made extemporaneously, sometimes by less-experienced engineers or by engineers who do not necessarily remain with the organization long.
Incorporating an open source package is more than a momentary technical choice—it's the beginning of a long-term partnership. Without careful evaluation organizations may later experience a kind of “buyer's remorse” over the adoption of a particular package. This article outlines key considerations to help organizations make informed and strategic decisions when integrating open source packages into their products, services, and solutions.
1. License Check
Is the license acceptable and compatible with your project? It is absolutely critical to understand the differences between GPL, AGPL, MIT, ISC, and other license types before pulling a package into your commercial project. Failure to do so could result in major legal problems [ 1 ] [ 2 ] [ 3 ]. The most critical point is that “copyleft” licenses like GPL and AGPL require derivative works to be under the same license.
2. Functionality Fit
Does the package have the necessary functionality and is not overkill for the problem at hand? This may seem obvious, but I have seen multiple instances where people have started out using an open source package only to abandon it because it was not a good fit for the problem at hand. Often the issue is that a superficial understanding leads to a false assumption of good applicability. The key is to thoroughly evaluate the functionality early on to ensure it's a good fit for your needs.
3. Security Check
Are there known security vulnerabilities? It is a good idea to run tools like npm audit, pip-audit, etc., search the CVE database for known vulnerabilities and security history.
4. Documentation Check
Does the package have sufficient documentation for your needs? Evaluate the documentation that exists for a package and determine if its quality is adequate for your needs. If it is minimal or non-existent, expect to use the source code itself as your documentation.
5. Community Assessment
Understand the community developing and maintaining the package. This can range from no one (if its in need of a maintainer), one person, all the way up to several large companies with significant funding. How frequently are releases made? Can you keep up with the cadence without falling behind, or is it too slow for your needs? What is the culture like? Is it welcoming and inclusive, or elitist? When you adopt an open source package you become a member of this community. There is a good chance that you will want to submit bug requests, feature requests, and interact with the community to some degree. Even if you don't expect to submit pull requests, you may find it necessary when you become dependent on the package.
6. Tests
How has the package been tested? Does it include unit and/or integration tests? If so, how thorough are they? If the testing is spotty, you may find yourself doing that testing yourself or moving on to a better tested package.
7. Independent Test
Test the package yourself. Does it perform as expected? Again, this is probably an obvious one, but developers will often pull in a package under deadline with little to no thought about about the long term implications. It does not take very long to make a package entrenched in a product or solution.
8. Code Review
This is open source software after all—why not have a look at the code? If the package is small, review the code to get a sense for its internal design and construction. If it is large, review a sample of the code in key areas.
Conclusions
It has been said many times that free and open source software isn't free of all costs. You should not expect to download perfect working software with no security issues that meets your needs perfectly for no charge whatsoever. You should expect to sort through real software with issues large and small and make decisions with real consequences. Sometimes you have to say “this looks nice on the surface, but upon looking deeper, no, it's not for us”. And when you do say “yes”, be prepared to give back to the community in some way.
References
- Smith, B. FSF Settles Suit Against Cisco. Last modified May 20, 2009. https://www.fsf.org/news/2009-05-cisco-settlement.html
- Software Freedom Law Center. BusyBox Developers and Monsoon Multimedia Agree to Dismiss GPL Lawsuit. Last modified October 30, 2007. https://softwarefreedom.org/news/2007/oct/30/busybox-monsoon-settlement/
- Software Freedom Law Center. BusyBox Developers Agree To End GPL Lawsuit Against Verizon Last modified March 17, 2008. https://softwarefreedom.org/news/2007/oct/30/busybox-monsoon-settlement/
Acknowlegements
ChatGPT was used in the preparation of this article for light proofreading and stylistic suggestions.