If you’re in IT, you’ve probably already heard of Log4Shell by now. It’s a serious vulnerability that impacts the Apache Log4j library, a ubiquitous little piece of software that powers logging (think: record-keeping) for all sorts of applications, from cloud computing to Minecraft. The biggest tech companies in the world use it: Apple, Amazon, Google, Microsoft, Twitter, Netflix, Tesla, Steam; just to name a few. And right now, all of them are susceptible to getting hacked because of this vulnerability in this tiny piece of software. This is, according to some, the worst vulnerability in history. And I’m inclined to agree.
But how did we get here? I mean, the biggest tech companies in the world all use this software, and none of them, not even one, decided to – oh, I dunno’ – secure it?! That’s got to be a joke, right?
ʕ·ᴥ·ʔ: Gotta’ be!
Yeah. Unfortunately, it’s not.
ʕ ಠ ᴥಠ ʔ: What?!
Big Tech is a bunch of big freeloaders.
Non-profit truly ain’t profitable
According to the Apache Software Foundation’s 2021 annual report, they support more than 350 open source software projects, including the critical Log4j library. With an annual revenue of $3 million, that means that assuming they had absolutely zero expenses to pay, they could spend an average of just under $8,600 per year on each of their software projects, including Log4j.
Imagine that: if Apache had absolutely no expenses whatsoever, they could afford to spend $8,600 per year maintaining the super-important Log4j software library that all the biggest tech companies in the world rely on, tech companies that have billions of dollars in net income. That’s net income, meaning that’s money left over after all of their expenses have been paid. Meaning: pure, unadulterated profit.
Now compare that to Apache’s revenue of $3 million. After all of their other expenses, which doesn’t even seem to include salaries for their programmers and project managers, they’re left with a little less than $1.5 million. Forget $8,600 per year on each project; they’ve only got about $4,300 per year to spend on each project, including Log4j. Well, no wonder nobody saw this critical vulnerability that has been around since 2014. Them Apache programmers got day jobs and families to feed, so why would they spend much, if any, of their time securing some of the most important software in the world?
Meanwhile, Big Tech is using their software for free, and making billions of dollars in profit. Apple alone made $51 billion in profit last year. If they had spent just $1 billion of that on funding the most critical pieces of software powering their enterprise, we might have had at least one full time programmer maintaining Log4j who would have prevented Log4Shell from ever happening.
ʕ·ᴥ·ʔ: Well, that sucks. But no point crying over spilled milk now, right? Let’s just make sure this never happens again.
Oh, but this time is the “again”. This has happened before, just when this vulnerability was first introduced, in 2014!
ʕ ಠ ᴥಠ ʔ: What?!
We never learn
In April 2014, a terrible vulnerability called Heartbleed was discovered in a tiny piece of software known as OpenSSL. OpenSSL was and still is a ubiquitous software library that powers encryption for all sorts of applications, from online banking to emailing your grandma. The biggest tech companies in the world use it: Amazon, Google, Facebook, Twitter, Reddit, Yahoo!, GitHub, Steam; just to name a few. Oh, and Minecraft uses it too. At the time, all of them were susceptible to getting hacked because of this vulnerability in this tiny piece of software. This was, according to some, the worst vulnerability in history, at the time. And it was introduced all because the OpenSSL Software Foundation, the non-profit that manages OpenSSL, had an annual revenue of $2,000 from donations by individuals to fund the maintenance of OpenSSL.
Sound familiar?
ʕ´•ᴥ•`ʔ: Hm, yeah. Feels kind of like déjà vu.
Kind of. The difference is: Log4Shell is way worse. It’s a lot easier to exploit, and a lot more effective as a hacking tool than Heartbleed.
At the time, the IT world got into a big hubbub about Heartbleed. In the same month that Heartbleed was discovered, the Linux Foundation established the Core Infrastructure Initiative (CII), a committee to identify and fund the development and securing of core internet infrastructure software projects that the world relies on. It was backed by some of the biggest tech companies in the world: Amazon, Google, Microsoft, Facebook, Intel, Cisco, IBM, Dell; just to name a few.
ʕ·ᴥ·ʔ: Great! With that much money on the table, they must’ve supported hundreds or even thousands of critical software projects with billions of dollars, right?!
Uh, no.
ʕ ಠ ᴥಠ ʔ: You have got to be kidding me.
No.
It’s currently hard to find any data on it, so take this with a grain of salt, but according to the Wikipedia page, in 2016, the CII supported only 11 software projects. Eleven. And the archived CII 2016 annual report supports this. And if that Wikipedia page is to be believed, they spent a grand total of just over $1.6 million in 2016 funding these projects. Eleven projects, $1.6 million dollars. That averages to about $150,000 per year for each project, which isn’t bad, but there are hundreds of free and open source software projects like these that are powering modern digital infrastructure, being thanklessly maintained by unpaid volunteers around the world.
Meanwhile, there were at least 18 big tech companies backing the CII. You mean to tell me they really couldn’t find more money to fund more projects in 2016? Like, say, the Apache Software Foundation for Log4j? Really?!
ʕ´•ᴥ•`ʔ: Mm. Feels bad, man. What’s going on with the CII now?
No idea. They seem to be dead. Their last announcement was in 2018.
ʕ ಠ ᴥಠ ʔ: For real?
For real. Feels bad, man.
They also failed to stop the next “worst vulnerability in history”, Shellshock. That happened in September 2014, just five months after Heartbleed was discovered.
It was the same thing all over again: the biggest tech companies in the world use this little piece of software called Bash, for free, to make huge sums of money. Meanwhile, some random guy in Ohio named Chet Ramey has been thanklessly maintaining it for years, unpaid by all the big tech companies that rely on it. And no, CII didn’t fund him either, even after Heartbleed and Shellshock.
We need to stop freeloading
Unsung heroes like Apache, OpenSSL, and Chet Ramey should be getting the support and recognition they deserve. It’s kind of hard to do that whenever we describe critical software like Log4j, OpenSSL, and Bash as “little pieces of software that power the modern internet.” So I propose we make up a word for software projects like these.
I would have liked to describe them as Critical Internet Software (CIS), but that would just be confused with the Center for Internet Security (CIS also). So let’s settle with the next best thing (that I could come up with, at least): Global Software Dependencies, or GSD’s for short. They are little pieces of software that the global internet depends on. Fitting, don’tcha’ think? And let’s call their maintainers Global Software Dependency Maintainers. You know, a dignified title to address them by.
ʕ·ᴥ·ʔ: For what it’s worth, this fictional bear likes it.
Just today, the US Senate approved the National Defense Authorization Act (NDAA), which includes $605 million for US Cyber Command to do with however it pleases. Seeing that Apache, OpenSSL, and Chet Ramey are all American, along with most other GSD Maintainers, I think it would be fitting for at least some of that money to go to securing and supporting GSD’s and their maintainers.
There is also the Open Source Security Foundation (OpenSSF), which was formed last year in 2020 and backed, once again, by big tech companies: Google, Microsoft, Intel, IBM, and several others. Let’s hope this time it doesn’t just kind of fade away like the CII did, and that they put significant resources into identifying and supporting as many GSD’s as possible before the next Heartbleed, Shellshock, or Log4Shell. I’m not impressed with their work so far, as they have an initiative dedicated to just this goal but, as of this writing, they still haven’t identified Log4j or Bash as “critical software projects” that need to be secured. Nor does it seem like they’ve really done anything to support the unsupported GSD Maintainers that they’ve already identified.
Other big tech companies, whether as part of the OpenSSF or not, should also pull their own weight: identify what GSD’s they rely on and give due support in the form of resources (including money) to those maintainers who are essentially keeping their enterprises afloat for free.
The bottom line: if you make a significant amount of money from the use of a GSD, lend its maintainer/s a hand, no matter how small. Don’t be a freeloader who profits from the taking and gives nothing back.
ʕ·ᴥ·ʔ: So, how was Christmas?
Oh, it’s been a wild one. First Log4Shell happened, then I wrote this article, and then, before I could publish it, a category-5 super typhoon cut through the midsection of my country! This article woulda’ come out 10 days ago had it not been for that super storm cutting the electricity, and the water, and cellular service, and the internet, and ripping off roofs, smashing windows, uprooting trees, causing landslides, and leaving the general populace including me scrambling for drinking water. Yeah, it’s been an interesting month, a Christmas I’ll never forget.
ʕ´•ᴥ•`ʔ: Woah! All this happened in the last 10 days?!
Yep. But things are finally beginning to settle down now. I’ve finally had time to rest, and have the time to publish this article. Maybe the next article will be about disaster preparedness. I learned a lot about that the past 10 days.
See ya’ next year, and a happy new year.
ʕ·ᴥ·ʔฅ: Stay safe out there! Happy new year!