Who pays free crypto developers?
Crypto or Bust
Although open source crypto software is used virtually all over the world, the projects behind it are often small and chronically underfunded. Heartbleed, however, brings a possibility of improvement.
Open source software has the advantage that anyone can inspect the code and thus discover bugs. Despite this, a glaring security hole went unnoticed in OpenSSL  for more than two years, eventually going down in history as the Heartbleed bug . Seemingly – although the OpenSSL source code is freely available – no one actually noticed the problem because hardly anyone had been looking.
Of course, a bug like this would not attract so much attention if the OpenSSL user base were not so huge. Millions of private users and countless companies rely on this cryptography software. Although it is designed to protect the security of multimillion dollar projects, its development rests in the hands of a small group of programmers. As in the case of GnuPG , sometimes only one person maintains the extensive codebase.
Security-related tests and inspections are time consuming, often requiring a dedicated tester on a full-time basis. To increase the quality of the source code, more developers and testers would be needed.
Smaller projects in particular lack the funds for full-time positions; they cover their expenditure through donations across the board. Whether OpenSSH or GnuPG, on almost every project website you can press a button to support it with your credit card or PayPal. The funds collected in this way are usually insufficient to pay one or more developers.
According to the Linux Foundation, the OpenSSL project received just US$ 2,000 in donations  per year. This is also the amount cited by Steve Marquess, who is responsible for the finances of OpenSSL . The maintainer of GnuPG (see the box "Interview with Werner Koch") received almost EUR 5,000 in donations in 2013 – before tax, mind you. PayPal grabs its share for processing fees, and the government often looks to siphon some funds.
Interview with Werner Koch
Werner Koch is the founder and maintainer of GnuPG and has worked on the free implementation of the OpenPGP standard since 1997.
Linux Magazine: How is the GnuPG project mainly financed?
Werner Koch: For the first few years, I paid for the development from my savings and a consulting job I had to take temporarily. I founded g10code in 2001 to put the development on a sound basis. The goal was to sell support contracts and acquire paid development contracts. Things initially went quite well, but there have been no earnings in recent years – obviously there are not enough support incidents.
Development for customers has earned the most money. In particular, the contract award from the BMI [Germany's Federal Ministry of the Interior -ed.] and the developments of the S/MIME part that it commissioned. As a collateral gain, I was able to fund the OpenPGP part – that is, the actual GPG – employ a full-time developer for many years and, at times, two more part-time employees. In 2011, all of this dropped off, and I had to let my long-term employees go at the end 2012, unfortunately.
I didn't even ask for donations until 2011. You can't really keep something going in this way, but there never was a real fundraising campaign. The BMWA/BMWi [Federal Ministry of Economics and Labor] promoted the port to Windows, including the creation of additional tools, as a flagship project for the new master plan for Internet Security in 1999/2000 with a total sum of DM 319,000 (equivalent to EUR 137,000).
Projects for the BSI were always by invitation to tender. The last order from the BSI came in 2009 for Gpg4win, version 2. In 2013, we received a grant of US$ 25,000 from an American NGO to carry out work on Gpg4win. In the typical style for Gpg4win projects, I did this in cooperation with partner companies.
LM: How do you use the money? Purely for operating the computers?
WK: Open iT sponsors the two GnuPG servers. The remainder is essentially wage costs. In 2013, they totaled EUR 32,000, as shown in the g10code balance sheet. You can easily work out what this leaves for me in terms of take-home pay, but I really can't go on working for this kind of money. Before June 2013, I was just about to give up the whole thing.
LM: GnuPG also launched a crowd-funding campaign. Why crowd funding and why did you choose Goteo as your platform?
WK: It was chosen by the campaign manager, who was recommended to me. Looking back, I was not satisfied. The people behind Goteo do their best, but there is much to be desired. I had to dedicate several weeks to reworking. It starts with simple things, like the fact that the platform does not let you directly query T-shirt sizes, and ends with the inconsistent and incorrect data that it delivered. But it certainly was a great success and has encouraged me to continue with my commitment to GnuPG. At least I know how I can pay my salary for the next few months.
LM: What is your experience with the various financing models? Are there any problems? Would you start another crowd-funding campaign?
WK: Donations via PayPal always come in after a release. However, PayPal is expensive and a compulsive data hoarder. That is why it is now possible to donate directly using your credit card.
Crowd-funding platforms are a good thing, but not a solution for ongoing funding, although there are now some platforms that specialize in this. They are quite expensive (Goteo takes about eight percent), which explains why our goal from the outset was to build our own donation infrastructure. I would use crowd funding again for specific, well-defined projects, but only with a manager who implements this really well and independently: that is, someone who is permanently integrated into the GnuPG project. In short, g10code is currently too small to push forward with another campaign.
LM: What would you like to see more of: money or helping hands?
WK: Both. Integrating and coordinating multiple developers is very time consuming. I, or we, can afford only this overhead, if we don't need to worry about earning a living, but it would make sense. Also in the sense of securing the future of the software – it could be that I don't feel like, or can't continue, carrying on for some reason, at some point.
LM: What would you do if you had enough money at your disposal? Do you think your software would then be more secure?
WK: Continuously maintained software is more secure. We could then also, with good conscience, write more tests and revise certain parts of the code.
OpenSSH and OpenSSL together have about same amount of code as GnuPG, with 362,000 lines of code; the latter weighed in with GPGME in 2011 at approximately 345,240 lines of code. We still actively edit the 1.4 and 2.0 branches of GnuPG – and then there are key managers, Windows tools, and so on.
Projects try different ways to boost meager donations. The OpenSSH project  sells a book to match the software  on its website. Because OpenSSH belongs to the OpenBSD project, it also benefits from its funding. This, in turn, consists of donations from several larger companies, such as the PSW Group (GmbH & Co. KG) .
The developers of the OpenSSL project offer consulting services, which start at US$ 250 per hour . That's a bargain compared with the prices of other consulting companies. Moreover, the OpenSSL developers take on commissions and offer support contracts – the latter start at US$ 20,000 per year.
This work is a blessing and a curse: Although it funds the project, it keeps the developers from working on the actual software. As Marquess points out in his blog , most developers still have a normal job and a family life, too. In his opinion, at least six full-time developers would have to work on OpenSSL.
Despite the precarious financial position, the software produced by the projects mentioned thus far enjoys great popularity with admins from all around the world, and companies that score billions in sales have relied on the code for many years. It was not until the Heartbleed bug hit home that the situation of these projects was suddenly dragged into the limelight. In response, the Linux Foundation has now founded the Core Infrastructure Initiative (CII) .
Well-heeled users of the software pay into a common pot, from which the initiative finances developments of important or security-related open source projects. Many major league players in the industry (Figure 1), including Amazon, Google, IBM, and Microsoft, are among the first donors.
The fund is helping the OpenSSL project first. In addition to additional developers, it also funds resources that help the project improve the security of its software, carry out external audits, and integrate patches faster. The projects that the initiative will help in the future are determined by a steering group, staffed by representatives of the donors.
An advisory board – including, among others, security expert Bruce Schneier, the kernel developers Ted Ts'o and Alan Cox, and Eben Moglen from the Software Freedom Law Center – advise on the choice of projects. The first projects to be funded include OpenSSL, OpenSSH, NTP, and Open Crypto Audit Project .
Buy this article as PDF
Kernel king admits his tone has alienated volunteers, but says the demands of the process require directness.
New flaw in an old encryption scheme leaves the experts scrambling to disable SSL 3
Lennart Poettering wants to change the way Linux developers talk to each other.
Enterprise giant frees itself from ink and home PCs (and visa versa).
Mozilla’s product think tank sinks silently into history.
TODO group will focus on open source tools in large-scale environments.
New tool will look like GParted but support a wider range of storage technologies.
New public key pinning feature will help prevent man-in-the-middle attacks.
Carnegie Mellon researchers say 3 million pages could fall down the phishing hole in the next year.
The US government rolls new best-practice rules for protecting SSH.