Extended Courses Discount
My Go courses are discounted for the next few weeks to help out anyone who may need or want access to them. I'm also going to try to help out anyone who can't afford a course, and I will be writing posts about working from home over the next week in an attempt to help anyone new to WFH. Read more here.
Yesterday Russ Cox announced vgo, a drop in replacement for the
go tool designed to handle package versioning. While it is still an experiment, it is a pretty unexpected change given that everyone thought dep was going to become the official dependency management tool. If you haven’t already, you should start by reading Russ’ post as well as the tour he provides. It is a great overview of what
vgo is, even if many people have misinterpreted a few things in the post.
When I first read the article, I had feelings similar to what I suspect others felt. I was confused about a few points, concerned about a few potential issues, and overall unsure of how I felt. I wanted to get a better understanding of what
vgo really was, so I opted to both read the
vgo tour, and to download
vgo and experiment a bit. You can actually check out some of the repos I created during my experimentation on GitHub: https://github.com/joncalhoun?tab=repositories
After getting a better understanding of
vgo, I was surprised that a lot of the assumptions people were making were incorrect despite being clearly covered in the tour and easily double checked by using
vgo. Unfortunately I doubt everyone will go download the tool before complaining, so this post is intended to help clarify on a few of the misunderstandings I saw, while also giving me a chance to express my excitement for the project 😁
Note: I suspect Russ is working on posts to help clarify and expand on
vgo; my intention here is not to speak for him, but rather to try to address what I consider simpler questions and confusions in the hopes that by doing so Russ can focus his efforts on the bigger questions that I do now know the answers to.
Oddly enough, this was a major source of confusion yet it was made pretty clear in the tour. I suspect many people read the original post, misunderstood what Russ was saying, and then never bothered to read the tour where things would have been cleared up.
There are only three circumstances that need to be considered, and each are described below.
The first use case is incredibly simple. When a new package is added to a project as a dependency,
vgo will opt for the NEWEST version of that package unless you manually specify otherwise. I don’t see any reason why anyone would something different, and this point doesn’t seem to concern anyone except the people who incorrectly believe
vgo will opt for the oldest version, which is proven incorrect in the
Most package managers wouldn’t permit this.
vgo allows this by expecting major versions to have a different import path - one with
/v2 (or any other major version number) appended to it - which would make it a completely different package than other major versions.
It is slightly unclear to me at this time how developers are supposed to manage this, but I suspect Russ will clarify a bit over the coming week. Regardless, this is pretty much the only way to handle this (aside from banning multiple major versions), so it isn’t a huge point of contention and I would prefer to wait for more information before discussing it further.
Assuming major versions are the same, dependencies are resolved by choosing the highest version in the list of minimum versions. That is, if 3 modules all require the
turtle package we might have a set of MINIMUM versions like:
vgo would then determine that
v1.3.2 is the highest semver version in this set of minimum versions, so that is the version that would be used.
I suspect this has confused many people because Russ speaks about selecting the minimum version, but what he meant by this was that the build process WOULD NOT use
v1.3.3 even if it was available, and would instead opt for the minimum version that satisfies all of its known requirements.
Doesn't Russ mention always preferring "older" verions?
In Russ’ post there are a few instances where he uses the term “older”. Specifically, he says things like, “because no older version will be published” which lead many people to believe that
vgo wouldn’t work with a project that releases a
v1.3.0, and then later releases a
This is completely untrue, and was confirmed both in my experimenting as well as by Russ on reddit.
vgo DOES depend on is adhering to Semantic Versioning. That is, you CAN release a
v1.3.0, but your users shouldn’t be expected to “upgrade” from
v1.2.2. This would be a downgrade, even though
v1.2.2 was released later.
Now that we understand how versions are determined, let’s take a look at the few ways you can initiate a dependency upgrade.
Earlier in this post we had the
turtle example where we ended up with the following minimum version requirements from all our modules:
We are going to continue to build on this imaginary project, looking at ways that
turtle could be upgraded.
This is pretty self explanatory; if you want to start using
turtle, you can simply update your module to specify
v1.4.4 as the minimum version.
vgo provides tools to choose a specific version, upgrade to the latest version, and complete a variety of other tasks to help with this. Check out the vgo tour for info on how to do this.
The only other way a dependency can be upgraded is if you add or update an existing dependency. This happens because transitive dependencies also update, so when you upgrade package
foo (or add it), it might also cause package
turtle to update if the minimum version requirements change.
It is also important to remember that these transitive dependencies can be arbitrarily deep. That is, package
foo could require package
bar, which requires package
spaz, and so on until finally a package requires
email@example.com, which would cause our version of
turtle to upgrade from
v1.3.6 because this is the new minimum version that satisfies all of our modules.
Now that we have a proper understanding of how
vgo decides which version to use, and how to upgrade packages, let’s take a look at some of the most common concerns people had with
I saw this in a few places, but it simply isn’t true.
vgo takes the dependency requirements of all your modules into consideration when determining which version of a package to use. This also means that if you upgrade package
foo, you are guaranteed to have packages that meet or exceed the minimum version requirements specified by
The basic claim here is that if we are using
v1.2.0 of a package, we should be able to specify whether we want to automatically use the most recent patch (
v1.2.X) for security updates, or we could opt into any minor version updates automatically (
vgo, this update won’t automatically occur. Instead, you would need to manually tell
vgo to update your packages using something like
vgo get -u.
While this argument does have some merit, I’m not sure how much weight I’d give it. For starters, we aren’t going to ever update our code without creating a new build, so any security patches released while our code is deployed won’t be added until we do a new release. If that isn’t the case, I would be terrified of having different version of my dependencies in different servers at the same time, as this could be a nightmare to debug and manage.
That leaves us with this issue only really being an issue when we build, but that isn’t really an issue. With the change to what
vgo test all means, you could fairly easily automate your build process to automatically update all of your dependencies, test, and then deploy if tests pass much like you would expect to see with another dependency manager. It would be concerning if
vgo prevented this, but given that it doesn’t I’m not sure why this would be a major concern. You can customize your build pipeline, but that doesn’t mean your customizations should definitely be in the standard library’s tooling.
Russ Cox also addresses this point in a reply on the golang-nuts mailing list:
In the tour (research.swtch.com/vgo-tour) I show how “all” is redefined to be useful again. So it’s completely reasonable to try an update, go test all, and if all the tests pass (and you trust your tests), then check in the go.mod file. Someone could build a bot to do this automatically, even. I don’t think minimal version selection means you’re always stuck with old versions. It just means you don’t get new versions until you ask for them and are ready to evaluate how good they are, not just because you run ‘go get’ and a new version has appeared overnight.
The general argument here is that as a package manager, it will be harder to support your users when they aren’t automatically being upgraded to the latest version.
As a counterpoint, I’d like to point out that the reverse is also true. If I run package
turtle and it depends on package
foo, it isn’t uncommon to have to deal with issues when a new version of
foo is released, has a breaking change, and dependency managers all update to the latest version of
foo by default. When this happens, packages like
turtle will almost always get an issue submitted when in reality
turtle isn’t to blame.
This makes me wonder which would actually cause more issues for package maintainers - users automatically updating all their dependencies by default, or users generally leaning towards the minimum required versions you have certified to work with your package in your
In both scenarios you will have people using outdated versions of the package as well as users who are constantly updating all their dependencies, but at this point I’m not sure we really understand whether leaning towards one or the other causes more issues.
I also feel this isn’t an incredibly hard problem to solve at the issue level. When someone wants to submit an issue, ask them to upgrade versions first. This isn’t uncommon, especially with desktop applications like Atom, and is a completely reasonable request for a package maintainer to make.
Imagine we are working on the
foo package and accidentally added a backwards incompatible change to
v1.3.2, but then fix it in
v1.3.3. What would happen if
vgo determined that
v1.3.2 meets our minimum requirements, but that incompatible change causes our code to break? Wouldn’t it be better to automatically use
This is a very specific and rare occurrence, and I don’t see it happening frequently in practice. For it to occur, a package you depend on needs to get updated (or added) manually, and that update needs to include a transitive dependency that sets the minimum version of
v1.3.2 (the broken version). Most of the time these broken builds are fixed fairly quickly, so having another package set it as its minimum version seems extremely unlikely. Even if it does occur, I suspect the package with the bad minimum version would get updated promptly.
If this does happen, the solution is fairly simple:
v1.3.2breakage so they can release an updated
go.modto require a minimum version of
What is especially nice about
vgo in this scenario is that by its very nature, updates only occur when a developer starts them, so a developer will not only be there ready to debug the issue, but will be prepared to handle issues like this as they are initiating an upgrade. This is a logical time for issues like this to occur. Build time, on the other hand, is not when you expect issues like this to occur or be handled.
I saw a few users concerned about offline builds, and I had the question myself, but was informed by tv64738 on reddit that
vgo currently uses the
$GOPATH/src/v/ directory to store the source code allowing for offline builds once you resolve dependencies once. It is unclear if this is a permanent solution, but it does imply that this is on Russ’ mind which is a good thing.
A few users were concerned that repositories were being downloaded via regular old HTTP instead of HTTPS. As far as I can tell, the source for
vgo seems to be using HTTPS, and Russ’ use of “HTTP” in the blog post likely just meant the HTTP protocol instead of say
A few users noticed that github has some API limits that kick in and require some setup to make
vgo continue working w/out getting errors, but I suspect issues like this could be overcome with some collaboration between GitHub and Google.
While this post is mostly meant to clarify and discuss a few concerns people have with
vgo, I think it is also worth pointing out some of the perks to this approach. This is nowhere near an exhaustive list, but are some points I personally resonated with.
I definitely love that we started to get the community converging on a single tool (dep), but it is still a tool that developers need to learn and become comfortable with.
vgo on the other hand feels fairly natural so far. The hardest part might be getting people comfortable with the
go.mod files, but if that ends up being the hardest part I’d call that a success.
go test allis way more useful
vgo also makes
go test all way more useful, as it only tests your current module and all of its dependencies. While this didn’t seem to be a primary goal of
vgo, it was intentional and I am loving it.
More generally speaking, I am hoping that the move way from a
$GOPATH setup allows tools like
gorename, which (iirc) won’t work if you have any broken code in your
$GOPATH/src directory, to start working more consistently because they can be leveraged in individual module contexts, making it much more likely that all the code there is building correctly.
I doubt we can ever make builds 100% consistent, but the approach
vgo is taking should avoid any of those painful issues that occur when you go to build and a new version of
foo was released last night. That doesn’t mean you can’t get the latest version of
foo, but you won’t be forced to use it unless you explicitly state you want it.
In the words of Russ:
I don’t think minimal version selection means you’re always stuck with old versions. It just means you don’t get new versions until you ask for them and are ready to evaluate how good they are, not just because you run ‘go get’ and a new version has appeared overnight.
With a build process that can auto-upgrade a package version, any random developer could end up needing to fix the issue. Or worse yet, EVERY developer on your team could be blocked because of an issue caused by a new version of a package, and when that happens chances are 2+ devs are going to go try to fix the same problem at the same time wasting resources. With
vgo, this issue only occurs when someone sets out to upgrade a package, so the likelihood of it blocking an entire team or being tackled by multiple developers at the same time are decreased significantly.
While this often isn’t considered when discussing build consistency, I can say from experience that a breaking change like this is a huge time waster for teams and is exactly what leads to teams specifying EXACT versions of packages that they use with no wiggle room at all.
Another important point that Russ makes in the golang-nuts mailing list is that this approach allows you to completely control your own builds, but limits your control over people who use your code.
In his own words:
Yes, I’ll make that point again in tomorrow’s posts elaborating minimal version selection, but I think this is probably the most important algorithmic policy detail. You get total control over your own build. You only get limited control over other people’s builds that happen to import your code.
I suspect we will see Russ elaborate on this one a bit, but the main takeaway for me is that this limits a package’s ability to say “My code only works with
firstname.lastname@example.org!” and instead the package can only say “My code works with
email@example.com or greater.“, which is much more likely to resolve nicely with other packages you use.
This pain is summed up pretty nicely by David Anderson in the mailing list:
I’ve struggled with
depdependency hell a lot in my Go projects (especially those that use the Kubernetes client library, which abuses “I, the library author, can define byzantine constraints on my users” to the extreme). The way I’ve described it informally is that traditional “arbitrary version selection” algorithms create the wrong incentive structure for library authors, in the sense that they have all the power to lock versions arbitrarily, but bear none of the pain associated with their decisions - instead, the binary author at the apex of the dependency tree gets that pain. “Authority without responsibility” is the death of many incentive structures.
Sign up for my mailing list and I'll send you a FREE sample from my course - Web Development with Go. The sample includes the first few chapters from the book, and over 2.5 hours of screencasts.
You will also receive emails from me about upcoming courses (including FREE ones), new blog posts, and course discounts.
Jon Calhoun is a full stack web developer who also teaches about Go, web development, algorithms, and anything programming related. He also consults for other companies who have development needs. (If you need some development work done, get in touch!)
Jon is a co-founder of EasyPost, a shipping API that many fortune 500 companies use to power their shipping infrastructure, and prior to founding EasyPost he worked at google as a software engineer.
Jon's latest progress update: Writing Course Notes
Spread the word
Did you find this page helpful? Let others know about it!
Sharing helps me continue to create both free and premium Go resources.
Want to discuss the article?
©2018 Jonathan Calhoun. All rights reserved.