Generally our practice is to pin everything to major versions, in ruby-speak this means like `gem 'net-sftp', '~> 4.0'` which allows 4.0.0 up to 4.9999.9999 but not 5. Exceptions for non-semver such as `pg` and `rails` which we just pin to exact versions and monitor manually. This little file contains our intentions of which gems to update automatically and for any exceptions, why not.

Then we encourage aggressive performances of `bundle update` which pulls in tons of little security patches and minor bugfixes frequently, but intentionally.

Without the lockfile though, you would not be able to do our approach. Every bundle install would be a bundle update, so any random build might upgrade a gem without anyone even meaning to or realizing it, so, your builds are no longer reproducible.

So we'd fix reproducibility by reverting to pinning everything to X.Y.Z, specifically to make the build deterministic, and then count on someone to go in and update every gem's approved version numbers manually on a weekly or monthly basis. (yeah right, definitely will happen).

    > But... why would libpupa’s author write a version range that includes versions that don’t exist yet? How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.

    > For that, kids, I have no good answer.

You actually kind-of answer your own question with this bit. Semver not being a guarantee of anything is true, but I'd extend this (and hopefully it's not a stretch): package authors will republish packages with the same version number, but different package contents or dependency specs. Especially newer authors, or authors new to a language or packaging system, or with packages that are very early in their lifecycle.

There are also cases where packages get yanked! While this isn't a universally-available behaviour, many packaging systems acknolwedge that software will ship with unintentional vulnerabilities or serious stability/correctness issues, and give authors the ability to say, "I absolutely have to make sure that nobody can install this specific version again because it could cause problems." In those cases, having flexible subdependency version constraints helps.

It might be helpful to think by analogy here. If a structure is _completely rigid,_ it does have some desirable properties, not the least of which being that you don't have to account for the cascading effects of beams compressing and extending, elements of the structure coming under changing loads, and you can forget about accounting for thermal expansion or contraction and other external factors. Which is great, in a vacuum, but structures exist in environments, and they're subject to wear from usage, heat, cold, rain, and (especially for taller structures), high winds. Incorporating a planned amount of mechanical compliance ends up being the easier way to deal with this, and forces the engineers behind it to account for failure modes that'll arise over its lifetime.

Scala uses Maven repositories (where the common practice is to use fixed dependency versions) but with different resolution rules:

* When there are conflicting transitive versions, the highest number prevails (not the closest to the root).

* Artifacts declare the versioning scheme they use (SemVer is common, but there are others)

* When resolving a conflict, the resolution checks whether the chosen version is compatible with the evicted version according to the declared version scheme. If incompatible, an error is reported.

* You can manually override a transitive resolution and bypass the error if you need to.

The above has all the advantages of all the approaches advocated for here:

* Deterministic, time-independent resolution.

* No need for lock files.

* No silent eviction of a version in favor of an incompatible one.

* For compatible evictions, everything works out of the box.

* Security update in a transitive dependency? No problem, declare a dependency on the new version. (We have bots that even automatically send PRs for this.)

* Conflicting dependencies, but you know what you're doing? No problem, force an override.

Using hashes also makes it easier to distribute, fetch, proxy, etc. since there's no need for trust. In contrast, fetching code based only on (name and) version number requires more centralised repositories with a bunch of security hoops to jump through.

Also, on that note, I can plug my own post on the topic: http://www.chriswarbo.net/blog/2024-05-17-lock_files_conside...

The joke is this:

Lupa and Pupa received their paycheques, but the accountant messed up, so Lupa received payment belonging to Pupa, and Pupa — belonging to Lupa.

“To Lupa” sounds like “dick head” when translated to Russian. The ending reads as if Pupa received a dick head, which means that he didn’t receive anything.

I am not sure, but it could that the entire post intent is to get English-speaking folks to discuss “libpupa” and “liblupa”.

[1] https://imgur.com/a/q1XVDZU

As a separate thought, it seems that it would be possible to statically analyze the usage of "B" in the source code of "A" and compare it to the public API for any version of "B" to determine API compatibility. This doesn't account for package incompatibility due to side effects that occur behind the API of "B", but it seems that it would get you pretty far. I assume this would be a solution for purely functional languages.

However for NPM you will hit issues where 2 packages need a different React version and if you want to use them both you need to pick. In addition it is better for security. The lock file is a distributed checksum. Not impervious to supply chain attacks but better equipped than trusting the package author not to retrospectively bump (I guess you could have a hash for this tbat included the downloaded source code and claimed deps).

When you first take a dependency, you typically want the latest compatible version, to have all the available bug fixes (especially security fixes).

Once you’ve started building on top of a dependency you need stability and have to choose when to take updates.

It’s about validating the dependency… on first use, there’s no question you will be validating its use in your app. Later, you have to control when you take an update so you can ensure you have a chance to validate it.

BTW, of course semantics versioning isn’t perfect. It just lowers the risk of taking certain bug fixes, making it feasible to take them more frequently.

The lock file just holds the state for this mechanism.

Languages ecosystems who try to make it sane for developers usually end up with some sort of snapshot/bom system that lists that are compatible together, and that nudges lib developers to stay compatible with each other. I'm not going to pretend this is easy, because this is hard work on the side of lib devs, but it's invaluable for the community.

Compared to that, people extolling the virtues of semver always seem to miss the mark.

But it's also missing the value of hashes, even if every package used symver, then you had a script that could easily update to get recent security updates, we would still gain value from a lockfile hashes to protect against source code changing underneath the same version code.

Maven and Java is simply broken when dealing with transitive dependencies.

I've been hit so many times with running time exception "MethodNotFound" cause two libraries have the same transitive dependency and one version gets picked over the other one.

Maven, by default, does not check your transitive dependencies for version conflicts. To do that, you need a frustrating plugin that produces much worse error messages than NPM does: https://ourcraft.wordpress.com/2016/08/22/how-to-read-maven-....

How does Maven resolve dependencies when two libraries pull in different versions? It does something insane. https://maven.apache.org/guides/introduction/introduction-to....

Do not pretend, for even half a second, that dependency resolution is not hell in maven (though I do like that packages are namespaced by creators, npm shoulda stolen that).

When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).

But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.

I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.

I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.

Maven/Java does absolutely insane things, it will just compile and run programs with incompatible version dependencies and then they crash at some point, and pick some arbitrary first version of a dependency it sees. Then you start shading JARs and writing regex rules to change import paths in dependencies and your program crashes with a mysterious error with 1 google result and you spend 8 hours figuring out WTF happened and doing weird surgery on your dependencies dependencies in an XML file with terrible plugins.

This proposed solution is "let's just never use version ranges and hard-code dependency versions". Now a package 5 layers deep is unmaintained and is on an ancient dependency version, other stuff needs a newer version. Now what? Manually dig through dependencies and update versions?

It doesn't even understand lockfiles fully. They don't make your build non-reproducible, they give you both reproducible builds (by not updating the lockfile) and an easy way to update dependencies if and when you want to. They were made for the express purpose of making your build reproducible.

I wish there was a mega article explaining all the concerns, tradeoffs and approaches to dependency management - there are a lot of them.

If every dependency was a `=` and cargo allowed multiple versions of SemVer compatible packages.

The first impact will be that your build will fail. Say you are using `regex` and you are interacting with two libraries that take a `regex::Regex`. All of the versions need to align to pass `Regex` between yourself and your dependencies.

The second impact will be that your builds will be slow. People are already annoyed when there are multiple SemVer incompatible versions of their dependencies in their dependency tree, now it can happen to any of your dependencies and you are working across your dependency tree to get everything aligned.

The third impact is if you, as the application developer, need a security fix in a transitive dependency. You now need to work through the entire bubble up process before it becomes available to you.

Ultimately, lockfiles are about giving the top-level application control over their dependency tree balanced with build times and cross-package interoperability. Similarly, SemVer is a tool any library with transitive dependencies [0]

[0] https://matklad.github.io/2024/11/23/semver-is-not-about-you...

I believe Zig is also considering adopting it.

If there are any dependencies with the same major version the algorithm simply picks the newest one of them all (but not the newest in the package registry), so you don't need a lockfile to track version decisions.

Go's go.sum contains checksums to validate content, but is not required for version selection decisions.

Go's system may be worth emulating in future designs. It's not perfect (still requires some centralized elements, module identities for versions ≥2 are confusing, etc.) but it does present a way to both not depend strongly on specific centralized authorities without also making any random VCS server on the Internet a potential SPoF for compiling software. On the other hand, it only really works well for module systems that purely deal with source code and not binary artifacts, and it also is going to be the least hazardous when fetching and compiling modules is defined to not allow arbitrary code execution. Those constraints together make this system pretty much uniquely suited to Go for now, which is a bit of a shame, because it has some cool knock-on effects.

(Regarding deterministic MVS resolution: imagine a@1.0 depending on b@1.0, and c@1.0 depending on a@1.1. What if a@1.1 no longer depends on b? You can construct trickier versions of this possibly using loops, but the basic idea is that it might be tricky to give a stable resolution to version constraints when the set of constraints that are applied depends on the set of constraints that are applied. There are possible deterministic ways to resolve this of course, it's just that a lot of these edge cases are pretty hard to reason about and I think Go MVS had a lot of bugs early on.)

The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.

>How could they know that liblupa 0.7.9, whenever it will be released, will continue to work with libpupa? Surely they can’t see the future? Semantic versioning is a hint, but it has never been a guarantee.

Yes, this is a social contract. Not everything in the universe can be locked into code, and with Semantic versioning, we hope that our fellow humans won't unnecessarily break packages in non-major releases. It happens, and people usually apologize and fix, but it's rare.

This has worked successfully if you look at RubyGems which is 6 years older than npm (although Gemfile.lock was introduced in 2010, npm didn't introduce it until 2017).

RubyGems doesn't have the same reputation for dysfunction as Node does. Neither does Rust, Go, PHP, and Haskell. Even more that I probably don't use a daily basis. Node is the only language that I will come back and find a docker container that straight up won't build or a package that requires the entire dependency tree to update because one package pushed a minor-version change that ended up requiring a minor version change to Node, then that new version of Node isn't compatible with some hack that another package did in it's C extension.

In fact, I expect some Node developer to read this article and deploy yet another tool that will break _everything_ in the build process. In other languages I don't even think I've ever really thought about dependency resolution in years.

The package file (whatever your system) is communication to other humans about what you know about the versions you need.

The lockfile is the communication to other computers about the versions you are using.

What you shouldn't have needed is fully defined versions in your package files (but you do need it, in case some package or another doesn't do a good enough job following semver)

So, this:

  package1: latest

  # We're stuck on an old version b/c of X, Y, Z
  package2: ~1.2

This article appears to be talking about lockfiles for libraries - and I agree, for libraries you shouldn't be locking exact versions because it will inevitably pay havoc with other dependencies.

Or maybe I'm missing something about the JavaScript ecosystem here? I mainly understand Python.

I assume java gets around this by bundling libraries into the deployed .jar file. That this is better than a lock file, but doesn't make sense for scripting languages that don't have a build stage. (You won't have trouble convincing me that every language should have a proper build stage, but you might have trouble convincing the millions of lines of code already written in languages that don't.)

The algorithm can be deterministic, but fetching the dependencies of a package is not.

It is usually an HTTP call to some endpoint that might flake out or change its mind.

Lock files were invented to make it either deterministic or fail.

Even with Maven, deterministic builds (such as with Bazel) lock the hashes down.

This article is mistaken.

If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.

For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).

Given that, I still see some consequences:

The burden for testing if a library can use its dependency falls back on the application developer instead of the library developer. A case could be made that, while library developers should test what their libraries are compatible with, the application developer has the ultimate responsibility for making sure everything can work together.

I also see that there would need to be tooling to automate resolutions. If ranges are retained, the resolver needs to report every conflict and force the developer to explicitly specify the version they want at the top-level. Many package managers automatically pick one and write it into the lock file.

If we don’t have lock files, and we want it to be automatic, then we can have it write to the top level package manager and not the lock file. That creates its own problems.

One of those problems comes from humans and tooling writing to the same configuration file. I have seen problems with that idea pop up — most recently, letting letsencrypt modify nginx configs, and now I have to manually edit those. Letsencrypt can no longer manage them. Arguably, we can also say LLMs can work with that, but I am a pessimist when it comes to LLM capabilities.

So in conclusion, I think the article writer’s reasoning is sound, but incomplete. Humans don’t need lockfiles, but our tooling need lockfiles until it is capable of working with the chaos of human-managed package files.

However: You absolutely do need a lock file to store a cryptographic hash of each dependency to ensure that what is fetched has not been tampered with. And users are definitely not typing a hash when adding a new dependency to package.json or Cargo.toml.

This works with minimal coordination between authors of the dependencies. It becomes a big deal when you have several unrelated dependencies, each transitively requiring that libpupa. The chance they converge on the same exact version is slim. The chance a satisfying version can be found within specified ranges is much higher.

Physical things that are built from many parts have the very same limitation: they need to specify tolerances to account for the differences in production, and would be unable to be assembled otherwise.

A: You're handling problem X and then unrelated problem Y suddenly arises because you're not locking package versions thoroughly. It's not fun.

B: Now the opposite. You lock all versions of the libs you use. You use renovate or schedule time for updates periodically. You have a thorough test suite that you can automatically exercise when trying the new updates. You can apply the updates and deoy the new version to a test environment to run a final test manually. Things look good. You deploy to production and, quite often, things go smoothly.

A is the blue pill, easy to taste but things are out of your control and will bite you eventually. B is the red pill: you're in control, for the better or worst.

.NET doesn't have lock files either, and its dependency tree runs great.

Using fixed versions for dependencies is a best practice, in my opinion.

> Our dependency resolution algorithm thus is like this:

> 1. Get the top-level dependency versions

> 2. Look up versions of libraries they depend on

> 3. Look up versions of libraries they depend on

...would fail in languages like Python where dependencies are shared, and the steps 2, 3, etc. would result in conflicting versions.

In these languages, there is good reason to define dependencies in a relaxed way (with constraints that exclude known-bad versions; but without pins to any specific known-to-work version and without constraining only to existing known-good versions) at first. This way dependency resolution always involves some sort of constraint solving (with indeterminate results due to the constraints being open-ended), but then for the sake of reproducibility the result of the constraint solving process may be used as a lockfile. In the Python world this is only done in the final application (the final environment running the code, this may be the test suite in for a pure library) and the pins in the lock aren't published for anyone to reuse.

To reiterate, the originally proposed algorithm doesn't work for languages with shared dependencies. Using version constraints and then lockfiles as a two-layer solution is a common and reasonable way of resolving the dependency topic in these languages.

Perhaps that's a valid assumption for readers of his blog, but once it appears here there are going to be a lot of readers who don't have the context to know what it's about.

Can an "NPM" tag be added to the subject of this post? More generally, I encourage authors to include a bit more context at the top of an article.

https://go.dev/ref/mod#minimal-version-selection https://research.swtch.com/vgo-mvs

Instead of a "lock" file, go includes a "sum" file, which basically tells you the checksums of the versions of the modules that were used during a build happened to be, so that you can download them from a central place later and ensure you are working from the same thing (so that any surreptitious changes are identified).

A lockfile provide a specific, concrete, minimized, satisfied solution on what an application or library uses to operate.

Generally, deployed applications have and save lock files so that nothing changes without testing and interactive approval.

Libraries don't usually ship lock files to give the end user more flexibility.

What solved system package dependency hell is allowing multiple versions and configurations of side-by-side dependencies rather than demanding a single, one-size-fits-all dependency that forces exclusion and creates unsatisfiable constraints.

and as some people mentioned, if a dependency of a dependency provides an important security patch, do you want to wait for your dependency to update first? or do you rely on overrides?