Erroneous 'Spam' Flag Affected 102 npm Packages (npmjs.org) 84
There was some trouble last weekend at the world's largest package repository. An anonymous reader quotes the official npm blog:
On Saturday, January 6, 2018, we incorrectly removed the user floatdrop and blocked the discovery and download of all 102 of their packages on the public npm Registry. Some of those packages were highly depended on, such as require-from-string, and removal disrupted many users' installations... Within 60 seconds, it became clear that floatdrop was not a spammer -- and that their packages were in heavy use in the npm ecosystem. The staffer notified colleagues and we re-activated the user and began restoring the packages to circulation immediately. Most of the packages were restored quickly, because the restoration was a matter of unsetting the deleted tombstones in our database, while also restoring package data tarballs and package metadata documents. However, during the time between discovery and restoration, other npm users published a number of new packages that used the names of deleted packages. We locked this down once we discovered it, but cleaning up the overpublished packages and inspecting their contents took additional time...
In cases where the npm staff accepts a user's request to delete a package, we publish a replacement package by the same name -- a security placeholder. This both alerts those who had depended on it that the original package is no longer available and prevents others from publishing new code using that package name. At the time of Saturday's incident, however, we did not have a policy to publish placeholders for packages that were deleted if they were spam. This made it possible for other users to publish new versions of eleven of the removed packages. After a thorough examination of the replacement packages' contents, we have confirmed that none was malicious or harmful. Ten were exact replacements of the code that had just been removed, while the eleventh contained strings of text from the Bible -- and its publisher immediately contacted npm to advise us of its publication.
They're now implementing a 24-hour cooldown on republication of any deleted package names -- and are also updating their review process. "As a general rule, the npm Registry is and ought to be immutable, just like other package registries such as RubyGems and crates.io... However, there are legitimate cases for removing a package once it has been published. In a typical week, most of the npm support team's work is devoted to handling user requests for package deletion, which is more common than you might expect. Many people publish test packages then ask to have them deprecated or deleted. There also is a steady flow of requests to remove packages that contain contain private code that users have published inadvertently or inappropriately."
In cases where the npm staff accepts a user's request to delete a package, we publish a replacement package by the same name -- a security placeholder. This both alerts those who had depended on it that the original package is no longer available and prevents others from publishing new code using that package name. At the time of Saturday's incident, however, we did not have a policy to publish placeholders for packages that were deleted if they were spam. This made it possible for other users to publish new versions of eleven of the removed packages. After a thorough examination of the replacement packages' contents, we have confirmed that none was malicious or harmful. Ten were exact replacements of the code that had just been removed, while the eleventh contained strings of text from the Bible -- and its publisher immediately contacted npm to advise us of its publication.
They're now implementing a 24-hour cooldown on republication of any deleted package names -- and are also updating their review process. "As a general rule, the npm Registry is and ought to be immutable, just like other package registries such as RubyGems and crates.io... However, there are legitimate cases for removing a package once it has been published. In a typical week, most of the npm support team's work is devoted to handling user requests for package deletion, which is more common than you might expect. Many people publish test packages then ask to have them deprecated or deleted. There also is a steady flow of requests to remove packages that contain contain private code that users have published inadvertently or inappropriately."
Re: (Score:3)
Re: (Score:2)
Says the person using improper capitalization.
Re: (Score:2)
express-cocaine-service (Score:1)
What? (Score:5, Insightful)
I have no idea what this block of text is telling me. What are npm packages? Who is floatdrop and why is he relevant?
Re:What? (Score:5, Informative)
npm is "NodeJS package manager". NodeJS is run by sloppy hippies. Being sloppy hippies they deleted a user and that user's packages without checking if other packages used them. Because they're sloppy hippies.
Re: (Score:1)
Re: (Score:2)
Maybe they found a gender-specific pronoun in one of the comments.
Re: (Score:2)
Re:What? (Score:4, Interesting)
This seems as good a place to leave my NPM story. Or I could just link to the bug [github.com].
The short version is if you use NPM to install dependencies, it will install your dependencies and whatever they depend on. If you run it again, it will install your dependencies and then DELETE anything they depend on.
Why would you repeat the install? Well, if your dependencies change (e.g., you update a dependency or add a new one), you would then repeat the install to get the new dependencies. Except you can't do this without blowing away the entire existing install due to this bug.
It gets somewhat worse - starting in NPM 5, they introduced a lock file. Anyone familiar with other package managers should know what a lock file is and why they're important. If the lock file exists, it will never install indirect dependencies (probably) [github.com].
Basically, if anyone doubted that NPM was run by sloppy idiots, note that both bugs are still open and have been for months.
Maintain your own dependencies! (Score:5, Insightful)
If you have an important project and it is dependent upon a remotely-published package, it's not an important enough project to you.
There is a certain appeal to letting someone else maintain code and do bug fixes for you, but you have to give up that dream when the project becomes more than a casual "thing" you play with for fun.
Mod parent up to +5 insightful. (Score:2)
"Maintain your own dependencies!"
Bingo! Extra delicious cookie for you, sir! And a salute.
Re: (Score:2)
Re: (Score:1)
The entire concept of making one's build process dependent on some remote site being up at that very moment is so braindamaged that I don't understand how it ever got popular. I guess if a developer was a fetus when GitHub was created, they might not realize that any site can go down, any company can fail. Even GitHub.
Re: (Score:2)
Their inferior minds will cause deadlines to be missed! Contracts to be lost, Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes! The dead rising from the grave! Human sacrifice, dogs and cats living together ... mass hysteria!
What a clusterfuck (Score:5, Insightful)
They're now implementing a 24-hour cooldown on republication of any deleted package names
So make attackers wait a whole day before uploading their compromised replacements for widely-used packages. Got it!
Seriously, NPM is a shithole. "As a general rule, the npm Registry is and ought to be immutable", you think? It's not a "general rule". It's "all the time, every" you freaking amateurs.
This right here is how you brought it upon yourself, and why I have zero sympathy for your self-imposed situation. If I contribute a package to Debian, you think they'll spend "most of their week" removing it just because I asked? That's not gonna happen. Here's how you fix this:
"Effective immediately, we no longer remove packages unless they cause a clear and imminent threat to their users. If you accidentally included your GitHub password, change it. That's your problem, not ours. Next time try not to do that, OK? Also, we no longer reuse package names, ever, for any reason. If you wanted it, you should have registered it. And finally, under no circumstances, period, may you ever reuse a version number. Ten years from now, package foo-1.2.3 will be bytewise identical to the one we issued last week. We guarantee it."
Anything short of that is a joke to the rest of the industry. I'm not being idealistic or unrealistic, either: these are completely reasonable, common policies that pretty much literally every other package repo implements.
Re:What a clusterfuck (Score:5, Informative)
These are people who need libraries to left pad a string [theregister.co.uk] and to check if something is an array [npmjs.com]. Technical prowess is not their forte.
Re:What a clusterfuck (Score:5, Insightful)
Eh, I don't care about that so much. If it's the idiom in your language to let someone else write every little function like that, and that's just how it is in that ecosystem, then so be it. I wouldn't want to work that way, but everyone has their preferences.
But if you're going to foster an ecosystem where everyone's going to use the same "leftpad", then you damn well better make sure that:
If you can't guarantee all three of those conditions, I want nothing to do with it. And again, pretty much everyone else offers these guarantees. This isn't just some greybeard rant about an ideal world no one has ever lived in before.
Re: (Score:2)
> unless there's a critical security flaw
I hope you mean unless there is malicious code hidden in it.
A security flaw no matter how critical is no reason to remove something.
Sure, a repo should have a way to handle security issues, but removing insecure versions is not a sensible way of doing that.
> comes from the same author
That is too strict. You should have a mechanism to hand over maintainership of a project. But yes, it should not be some random person with 0 vetting.
Re: (Score:2)
I agree about the “same author” bit. I didn’t spell it out, but that to me means “officially designated maintainer”. Maybe that’s always the same individual maintainer, or maybe it’s “Release Manager at Foo Corp
Re: (Score:2)
There's something even more screwed up about the "leftpad" example: The package that half of the world depended on was leftpad 0.0.3. So someone had the genius to depend on code which based on its version number looked like an early alpha release.
Re: (Score:3)
There's something even more screwed up about the "leftpad" example: The package that half of the world depended on was leftpad 0.0.3. So someone had the genius to depend on code which based on its version number looked like an early alpha release.
I think what's worse is it apparently took 3 minor releases to get leftpad right.
WHAT THE FUCK
Re: (Score:2)
Even worse. If you look at the current version, there's something like 14 different releases. And it has 2 outstanding issues. All over something to left pad a string which is a CS101 entry-level coding task.
Re: (Score:2)
I wish I was.
1.2.0 is the latest of 14 releases
https://www.npmjs.com/package/... [npmjs.com]
Re: (Score:2)
One of the open issues is because it doesn't handle NaN passed as the length parameter... Holy crap JavaScript is amazing.
This unaccepted pull request to fix it is hilarious: https://github.com/stevemao/le... [github.com]
Re: (Score:2)
This unaccepted pull request to fix it is hilarious: https://github.com/stevemao/le [github.com]... [github.com]
I think the single most amazing thing about that fix is that the single line change literally modifies ~10% of the code of the package :-)
Re: (Score:2)
Re: (Score:2)
I would add one more thing -- that "leftpad-4.5.7" is from the same source as the author's source location. We've been finding that the minified code (blob, if you will) that the npm directory points to is not necessarily the exact same as what is in the author's source tree -- meaning what people would be inspecting is not necessarily what is being distributed.
It could be that leftpad is doing bad things to you code -- you just may not notice it because the source and the blob are totally different.
Re: (Score:2)
Re: (Score:2)
To be fair, the ability to left pad a string [mozilla.org] was only added to the JavaScript standard last year. Although Array.isArray [mozilla.org] has existed for quite a while now.
As for why you need a special function to determine if something is an array, MDN links to this article [mit.edu].
Note that one context in which you will never need to use Array.isArray over instanceof Array is inside a Node.js program, as it doesn't run in a browser context.
Although even then, isArray probably doesn't work the way most people would expect: certain
Re: (Score:2)
To be fair, the ability to left pad a string [mozilla.org] was only added to the JavaScript standard last year. Although Array.isArray [mozilla.org] has existed for quite a while now.
To be unfair, the ability to left pad a string is something that anyone should know how to do as an entry-level task to learning how to program.
Re: (Score:2)
Re: (Score:2)
Yeah, if more people realised why 'published interfaces are immutable' is such an important rule you wouldn't need to spend ages cleaning up when a screwup like this breaks your build environment when you do an update.
Re: (Score:2)
Re: (Score:2)
FTFY.
Re: (Score:2)
Even if you have a local version control system you still need to merge changes from upstream if you want to upgrade the environment to the latest version - e.g. to get security patches. And traditionally you need to merge the mainline into your local copy and then lock the mainline and merge back - basically it means that if the mainline has changed in an incompatible way you resolve that in your branch or local copy before merging back. In Clearcase terms you don't merge back to the mainline until the mer
Re: (Score:2)
You can add packages, you just can't delete them
It's like in C++ when you're trying not to break back compatibility - you can add interfaces, and you can add methods to interfaces but you can't delete interfaces or delete methods from interfaces.
That's how Symbian's binary compatibility worked. Now Symbian had a bunch of other fatal illnesses which eventually killed it, but a lack of interface immutability within a release wasn't one of them.
Re: (Score:2)
Append-only is probably more accurate, then. You can always add, but not delete.
Re: (Score:2)
Re: (Score:3)
A repository of shitty Javascript lbraries for incompetent Javascript monkeys.
Re: (Score:2)
Thank you so much for joining me here.
Hello, my name is CaptainDork, and I admit that I don't know one single, solitary goddam thing about npm and my ignorance is more than I can take on my own.
I need help
Funny video mocking NodeJS zealots (Score:3)
Node.js Is Bad Ass Rock Star Tech [youtube.com]
Re: (Score:2)
And not even one Amazon affiliate link in sight.
any idea what's in your dependencies? (Score:4, Informative)
I’m harvesting credit card numbers and passwords from your site. Here’s how. [hackernoon.com]
Need a mirror (Score:2)
Also, as is industry practice with maven central, it sounds like any company using npm needs to run their own caching mirror, to keep permanent copies of the artifacts your company is using. Unfortunately, I don't think npm's support for private mirrors is as good as maven's.
Anyone know of a good solution to this?