What does GitHub's security team even do?

GitHub's response time on malicious repositories is often lackluster. However, from conversations with folks at GitHub, my suspicion is that this is because they're being starved for resources, not incompetency or maliciousness.

This (IMO) points to a perverse reality: things need to get worse before they can get better. In other words, Microsoft probably needs to feel more pain (in the form of negative revenue pressure) before they take their own platform responsibilities (vis a vis not distributing malware) seriously.

We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.

(Edit: to be absolutely clear, I have first-hand experience that GitHub's security folks work extremely hard, and are often doing the kinds of invisible, thankless "deck-swabbing" work that nobody even thinks about. They're just under-resourced.)

This is like US healthcare. Almost nobody in US healthcare starts of evil. Almost none of the processes start with a clear evil intent. The problem is that the only real reward is money so the processes and people that generate money stay while the processes and people that merely made things better, but didn't increase revenue, slowly fade away.

Does GH get money for taking these actions? Do they have any monetary incentive other than 'their reputation', which clearly isn't changing usage, to improve? Maybe the best question here is why is it that after a lot of black eyes on data usage, reliability and monitoring aren't people switching. What keeps you using GH after stories like this?

I have a suggestion. PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.

> Why did they stop and take no further action?

It costs them nothing to blame the user for clicking links that they don't trust, even though they're hosting the links...

I generally believe in hanlons razor (assume stupidity rather than maliciousness), it's likely that github saw the easiest possible solution rather than diving deeper into the cause of the problem to fix it permanently

GitHub today isn't GitHub from the early-mid 2010's. Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs. But, GitHub makes good money, and turns out they're a great source of training material.

They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.

They do stuff perfect for gorgeous looks that can be pushed asap.

"security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.

So far this year 40% of my contracts have been a recovery from a Shai Hulud like attack. That’s in the area of 750k profit and more spent by the companies.

The instability and security issues that come of relying on the software supply chain has been pointed out for around 30 years. Seriously, go look. Multiple articles have pointed out the problems we’re seeing.

I used to ask the same question on behalf of clients but after multiple non-answers and silence my response now is just put something between you and GitHub that you can control.

The only reason they deleted the static list of identified repositories was because of the negative PR of hitting HN. Although I have no affiliation with GitHub, I would imagine that no one with any resource allocation power is incentivized to take action to police public repos in general. The reality is that as soon as they start, it will become an arms race, and they will have to deal with false positives and the community headache that comes with that. They also have bigger fish to fry with the effect the influx of AI slop is having to their open source bread and butter. The two problems are not entirely disjoint, and I suspect solutions won't come until the problem gets much worse, and the agentic coding space stabilizes a bit more.

> Why did they stop and take no further action?

Simple: further action was not requested in the Jira ticket, and what is not in a Jira ticket is not getting done because the team responsible for doing the needful is already overloaded with other crap.

Or maybe because the higher-up whose authorization is needed to go on a few days worth of deep dive other than doing exactly what is asked and accounted for in tickets doesn't have the time for a few minutes to explain to them why it is needed, or the higher-up needs authorization from finance or legal first.

Obvious /s, but I wouldn't be surprised at all if this is exactly what happened. If I were to guess, the "legal" is my biggest suspicion - if a provider reacts on notice of illegal/harmful content, they're just fulfilling legal obligations. But if they go and actively wade through the archives to find more incriminating content, that might be construed as Github doing active moderation of their own, leading to a loss of pure content hoster legal protections.

Oh this one is so easy: nobody cares, especially investors.

We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.

Reminds me of when Windows security problems hit critical mass in the early-2000s and Microsoft went all-in on addressing it (this started in the XP days IIRC).

> Why did they stop and take no further action?

It costs them nothing to blame the user for clicking links that they don't trust, even though they're hosting the links...

They do stuff perfect for gorgeous looks that can be pushed asap.

> Why did they stop and take no further action?

Oh this one is so easy: nobody cares, especially investors.

Reminds me of when Windows security problems hit critical mass in the early-2000s and Microsoft went all-in on addressing it (this started in the XP days IIRC).

> PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.

PyPI goes out of its way to not be an arbiter of package quality or security. PyPI really doesn't want people assuming those things based on presence, since it's (1) an open index, and (2) the resources needed to make those kinds of determinations at PyPI's scale are several orders of magnitude greater than what PyPI actually has access to.

(This is different from PyPI removing malware based on user reports, which does happen. But that's a reactive task and not one that comes with any sort of blanket guarantee.)

They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.

And embrace extend extinguish lives on

"security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.

Exec Bonuses being able to be clawed back for Security issues is just about the only fix that will have real effects. Anything else will have MBAs believing it's a "next guy" problem, and a obstacle to hitting their objectives.

Though right now the US thinks it's winning the Security Vulnerability Stockpile war, so it won't change the state quo.

So far this year 40% of my contracts have been a recovery from a Shai Hulud like attack. That’s in the area of 750k profit and more spent by the companies.

I used to ask the same question on behalf of clients but after multiple non-answers and silence my response now is just put something between you and GitHub that you can control.

Hi ofjcihen, where can I learn more about your professional services?

And embrace extend extinguish lives on

Hi ofjcihen, where can I learn more about your professional services?

Though right now the US thinks it's winning the Security Vulnerability Stockpile war, so it won't change the state quo.

Europe is implementing a law that requires software made for profit to hit at least industry security standards. Punishments include the purchaser being able to sue the seller as well as jail time for execs.

At some point, we need to push back against the reality in the US that we have effectively no way to stop mass harvesting (and then breaching) of our PII -- and there's basically zero downside to companies when it happens.

And how do you enact exec bonuses being clawed back? They're often the most connected people in the company to those setting the rules of the company, the board.

Hell, some companies have a CEO that has an absolute majority of voting power, meaning they cannot be held accountable and made to implement changes like the one you suggest.

(This is different from PyPI removing malware based on user reports, which does happen. But that's a reactive task and not one that comes with any sort of blanket guarantee.)

Agreed. It is a hard job. Decisions like this make sense but likely need to be reevaluated regularly based on how the world is changing.

And how do you enact exec bonuses being clawed back? They're often the most connected people in the company to those setting the rules of the company, the board.

Hell, some companies have a CEO that has an absolute majority of voting power, meaning they cannot be held accountable and made to implement changes like the one you suggest.

Agreed. It is a hard job. Decisions like this make sense but likely need to be reevaluated regularly based on how the world is changing.

Right now, there are currently thousands of repositories on GitHub distributing malware. Any of you can find these repositories, and you don’t need any special knowledge to do so. All you have to do is use the standard search function on the GitHub website.

These repositories have been around for two years. GitHub has billions of dollars, a security team, and artificial intelligence. Why haven’t they solved this problem in two years?

First, we’ll look at the repositories we’ve already found, identify common patterns within them, and then use those patterns to find other repositories.

Take a look at these repositories; in each one, the readme contains a link to a zip archive containing a Trojan:

Link to download the zip archive containing the Trojan

If we download this zip archive and submit individual files from it to VirusTotal, we’ll see the following results:

Scanning files from the zip archive on VirusTotal

Even a quick glance at these repositories shows that they have the same structure, nearly identical headings, and every heading contains an emoji. That’s all the information we need to find other repositories.

Let’s use the heading "📥 Download" and search for that string on GitHub. But we don’t want to search the entire codebase — just the readme files. Go to github.com and enter the following in the search bar at the top:

path:readme.md "## 📥 Download"

The number of results will always vary. At first, it showed me 9k repositories, but when I went to page 2, there were only 109 repositories. After refreshing the page, it was back to 9k repositories.

In the search output, you’ll need to visually filter out the results that contain any other text besides just the heading. The heading variations will look something like this:

## 📥 Download Now
## 📥 Download Now Again
## 📥 Download the Software
## 📥 Download & Install

Results of the first search

Open these repositories. They will have a link to a zip archive containing a Trojan.

However, the search results will include many legitimate repositories. We can improve the search by adding a search for zip archives. The search string will look like this:

path:readme.md "## 📥 Download" ".zip"

This will significantly improve the results. Now, right in the search results, we can see which repositories contain the heading we need and a link to a zip archive.

Results of the second search

But these results still include legitimate repositories. How else can we refine the search query?

All links to zip archives lead to githubusercontent.com or github.com. Also, the zip archive contains a version number, for example, "Software-3.6.zip". So we just need to write a regular expression to search for such links. But as I said at the beginning, you don’t need any specialized knowledge for this. Any free AI model can handle it. We feed it 10 such links, and after a few iterations, we get this result:

path:README.md /raw\.githubusercontent\.com\/.*\d+\.\d+\.zip|github\.com\/.*\/raw\/refs\/heads\/.*\d+\.\d+\.zip/

Enter this query, and you’ll get the repositories that are distributing the zip archive containing the Trojan. The number of repositories varies. In my case, sometimes there are 111, sometimes 4.4k.

Results of the third search

But this entire search only worked because we had an initial list of repositories from which we were able to derive a general search pattern.

Maybe GitHub’s security team didn’t have these repositories?
Maybe they don’t know about this general pattern?

A month ago, I published an article in which I analyzed this scheme in detail. I wrote a script that found 10,000 such repositories. I published the list of all the repositories and the script on GitHub.

The article made it to the Hacker News front page. Other cybersecurity websites wrote about this scheme.

Here is the full list of actions GitHub took:

They deleted all 10k repositories that the script found.

That’s it. They didn’t do anything else.

What’s more, a few hours later I ran the script again; it found new repositories, and I added them to the article. They weren’t blocked for an entire month. Even though all they had to do was open my article again, grab the new links, and block those repositories. That turned out to be too difficult for them.

But there’s one conclusion we can draw: they’re well aware of this malware distribution scheme.

No matter how hard I try, I can’t find an answer to why this is happening. Microsoft is a corporation with billions in revenue. They have thousands of employees, limitless resources, and artificial intelligence. All they needed to do was allocate a few days for any regular employee so they could use Copilot to find all these repositories and block them.

I’ve never worked at a large company. And I can’t imagine how decisions are made specifically at GitHub, or what bureaucratic nightmare managers have to go through to start combating malicious repositories.

But they deleted all 10k repositories just a few hours after the first article was published.

Why did they stop and take no further action?

← Before Problems with hourly and project-based pay

RSS · Telegram

Hacker Times

Hacker Times

What does GitHub's security team even do?

Discussion

Discussion

Subscribe