When Christian Grobmeier went to help his son with a Minecraft problem, he found the game displaying a warning: “We are suffering from a security hole from Log4J, please be careful and update immediately.”
I stared at the screen and told my son, ‘I’m sorry, it’s my fault.’
Christian Grobmeier, Log4j maintainer
This is the untold story of how one maintainer and the Log4j team navigated a crisis that exposed critical gaps in our digital infrastructure and demonstrated the importance of open source security and sustainability. Now, initiatives like the GitHub Secure Open Source Fund are working to make sure it never happens again.
It all started a few hours earlier on a cold November day, when Christian, who is a maintainer of the open source project Log4j, planned to spend time playing games with his son. Instead, he found himself staring at his phone, watching notifications pile up in his inbox—10, then 20 emails flooding in. When he saw the words “remote code execution,” his first thought was: “Maybe I’m on the wrong mailing list.”
He wasn’t. And within hours, Christian would be at the center of what became known as Log4Shell: the most severe vulnerability in internet history, affecting billions of devices from Fortune 500 companies to Minecraft servers worldwide.
“I told my son, I will play with you in like five minutes,” Christian recalls. “But he didn’t see me for the next couple of days.”
| Watch the full interview with Christian Grobmeier and Gregg Cochran, staff program manager at GitHub, above. 👆 |
The ubiquity that made Log4Shell a perfect storm
Log4j is foundational software. This 20+ year-old Java logging library quietly powers system events in applications worldwide, like user logins and calculation results. But this small piece of software had quietly become a dependency in thousands of projects across the Java ecosystem.
Log4j is such a small, tiny library. But everybody can use it in their software.
Christian Grobmeier
That ubiquity made Log4Shell devastating. Financial services companies relied on it for compliance auditing. E-commerce systems used it to track security incidents. Insurance companies needed it to monitor their software behavior. In a 2022 Tidelift survey, 49% of open source developers reported that their organization relies on Java—and most of them were using Log4j without even knowing it.
When Christian realized the scope of the vulnerability, the weight hit him immediately: “Literally all Java applications in the world could be affected. Even 10% would be a major problem. This would be catastrophic.”
A vulnerability that scored a perfect 10
Log4Shell reveals how a seemingly innocent feature became an attack vector. Log4j used Java’s Naming and Directory Interface (JNDI) to provide flexibility, allowing developers to load software components from remote servers. But the library didn’t validate whether JNDI lookup strings were coming from trusted sources.
“How can a string break the internet?” Christian asks.
The exploitation was frighteningly simple. An attacker could input a malicious JNDI string into any application field that gets logged—a username field, a search box, even a Minecraft chat message—and execute remote code on the target system.
jndi:<protocol>://<server-name>:<port>/<path-to-object>“You don’t even need to have special knowledge,” Christian notes. “You just run around and push the string wherever you want it.”
The Common Vulnerability Scoring System (CVSS) gave Log4Shell a perfect 10: the highest possible score.
“The first time I heard about this score, I thought, maybe it’s not so bad,” Christian remembers. “And then after a couple of days, I thought, yeah, maybe we should extend this to a score of 15 or 20.“
The human cost of maintaining critical infrastructure
The personal toll on maintainers during the Log4Shell crisis reveals the hidden human cost of our software supply chain. Christian and his team, mostly volunteers, suddenly found themselves responsible for patching a vulnerability affecting half the internet. The pressure was immense and deeply personal.
Some of us stopped sleeping. We all felt that either we fix it right now in the next few days, or we close this project.
Christian Grobmeier
Fixing the initial vulnerability led to the discovery of additional issues, creating what Christian describes as “a bag of water with a hole. When you patch the hole, you see another one.”
Meanwhile, the community response was mixed. “On the one hand, you have people who really hate you, and on the other hand, you have people who are really behind you,” Christian explains.
Perhaps most telling:
Nobody stops in to check on you. They check on the project. There’s also nobody standing up and saying, ‘hey, thank you for the good work you’re doing to remediate this issue.’
Christian Grobmeier
How the GitHub Secure Open Source Fund is strengthening security
The Log4Shell incident highlighted a critical gap in open source security: Maintainers often lack the training and resources to build security into their projects from the ground up. This realization sparked initiatives like the GitHub Secure Open Source Fund, which provides both funding and security training to critical open source projects.
The fund has been effective and efficient as a form of proactive protection, pooled resources, and shared responsibility. Think of it as “insurance” for the open source supply chain—helping make the digital ecosystem safer and reducing risks that could impact billions of users.
Christian participated in the GitHub Secure Open Source Fund security training program, and the impact was transformative.
The training didn’t just provide technical knowledge—it shifted his perspective. Christian explains, “With this training, developers are no longer the weakest link. Instead, they’re the first line of defense.”
This change in mindset is crucial. As Christian puts it:
Ignorance is by far the worst and most critical security hole. It will basically break all software.
Christian Grobmeier
When asked if the GitHub Secure Open Source Fund training could have prevented Log4Shell, Christian is direct: “If this training had existed five years ago, maybe Log4Shell wouldn’t be here today.”
Technical lessons: Building security by default
The Log4Shell incident taught the industry several critical lessons about secure development practices:
1. Validate all external input: Never trust data that crosses trust boundaries, especially in foundational libraries that process user input.
2. Disable dangerous features by default: Log4j now ships with JNDI lookups disabled by default.
3. Implement defense in depth: Modern applications need multiple layers of protection, from input validation to runtime protections.
4. Automate security scanning: Tools like GitHub’s code scanning and Dependabot can catch vulnerabilities before they reach production.
5. Maintain a software bills of materials (SBOMs): When Log4Shell hit, many organizations couldn’t determine if they were affected because they didn’t know their dependencies.
“I got phone calls from colleagues, asking me: ‘Am I really affected?’ SBOMs give you a technical way to find out what dependencies you’re using in a project,” Christian explains.
Industry-wide lessons for sustainable open source
While the technical lessons from Log4Shell are crucial, technology changes aren’t enough. The deeper challenge lies in how we support the humans who maintain the open source infrastructure our world depends on. This crisis exposed several systemic issues in how we approach open source sustainability and security:
Community is crucial: “If you’re maintaining open source software, just as one single person, that’s a risk,” Christian emphasizes.
Security training needs to be accessible: Traditional security education often doesn’t reach the maintainers who need it most.
Funding alone isn’t enough: While financial support helps, Christian found that training and community were equally important. When offered funding to pay team members, many declined due to tax implications or existing jobs.
Kindness matters: “Behind every small open source library, there’s a human writing the code,” Christian reminds us. “If you find something that’s not right, help out instead of being angry.”
Security can be improved for every project: During the program, Christian implemented multiple new security improvements, including hardening GitHub Actions against script injections, developing a new threat model, and collaborating with ScanCode to identify hidden Log4j artifacts in third-party code.
Your role in securing the software supply chain
The Log4Shell story isn’t just about one vulnerability; it’s about the collective responsibility we all share in maintaining the open source ecosystem that powers the modern internet.
For maintainers: Apply to programs like the GitHub Secure Open Source Fund. Enable built-in security tools like GitHub’s code scanning and Dependabot. Export SBOMs to help downstream users understand their dependencies, and publish security advisories for all vulnerabilities found in your project.
For enterprises: Become a Funding or Ecosystem Partner of the GitHub Secure Open Source Fund. Invest engineering time in the upstream projects you depend on. Don’t just consume open source—contribute with code, documentation, security reviews, and funding.
For individual developers: Select carefully the new dependencies that you pull in, for example by checking their security posture. Consider the data you process may be attacker controlled and strictly validate untrusted inputs to prevent unintended behavior. Contribute test cases and documentation.
The path forward
Today, Log4j has an 8.3 OpenSSF score, which demonstrates good security practices.
But the broader lesson extends beyond any single project. As Christian puts it:
Learning is the only cure for ignorance. So just keep learning.
Christian Grobmeier
The Log4Shell incident showed us how quickly our digital world can be threatened by a single vulnerability. But it also demonstrated the power of the open source community to respond, adapt, and improve. The question isn’t whether the next critical vulnerability will emerge—it’s whether we’ll be ready for it.
Ready to strengthen your open source project’s security? The GitHub Secure Open Source Fund provides funding, training, and resources to help maintainers build more secure software:
- Projects and Maintainers: Apply now to the GitHub Secure Open Source Fund and help make open source safer for everyone.Funding and Ecosystem Partners: Become a Funding or Ecosystem Partner and support a more secure open source future. Join us on this mission to secure the software supply chain — at scale!
Thank you to all of our partners
We couldn’t do this without our incredible network of partners. Together, we are helping secure the open source ecosystem for everyone!
Funding Partners: Alfred P. Sloan Foundation, American Express, Chainguard, Datadog, Herodevs, Kraken, Mayfield, Microsoft, Shopify, Stripe, Superbloom, Vercel, Zerodha, 1Password
Ecosystem Partners: Ecosyste.ms, CURIOSS, Digital Data Design Institute Lab for Innovation Science, Digital Infrastructure Insights Fund, Microsoft for Startups, Mozilla, OpenForum Europe, Open Source Collective, OpenUK, Open Technology Fund, OpenSSF, Open Source Initiative, OpenJS Foundation, University of California, Santa Cruz OSPO, Sovereign Tech Agency, SustainOSS
The post Inside the breach that broke the internet: The untold story of Log4Shell appeared first on The GitHub Blog.
