Every language has a problem with dependencies. In C there are problems with bumping minor shared library versions and having unexpected bugs appear in applications. Ruby and Python have problems with applications having conflicting dependencies requiring the use of bundle or virtualenv to create private library stores.
Go has a distinct advantage because it is a statically compiled language. In other words, a Go program always carries its runtime dependencies with it; no possibilities of library versions changing. However, creating a developer environment in which to compile this static binary is a bit more difficult and some tools are required.
While working on etcd, a project written in Go, we took an evolutionary path towards a dependency system that satisfied a few basic goals we had:
tl;dr We arrived at using godep but the rest of the post below has some insights on how we arrived here.
To satisfy the need for reproducible builds and zero dependencies we checked in
a copy of the GOPATH to third_party/src. However, this revealed several
problems over time:
go get github.com/coreos/etcd was broken since downstream dependencies
would change master and go get would setup a GOPATH that looked different
than our checked in version.We felt that go get wasn't a useful tool for installing etcd since it was
just a project built in Go and go get is primarily useful for easily grabbing
libraries when you are hacking on something. However, we were overwhelmed every
week with bug reports from users who wanted to simply go get github.com/coreos/etcd.
To solve the Windows problem I created third_party.go which ported the GOPATH
management shell script to Go and gave us the ability to drop the Powershell.
third_party.go worked well for a few weeks and we could remove the duplicate build logic in the Powershell scripts. The basic usage was simple:
# Bump the raft dependency in the custom GOPATH
go run third_party.go bump github.com/coreos/go-etcd
# Use third_party.go to set GOPATH to third_party/src and build
go run third_party.go build github.com/coreos/etcd
But, there was a fatal flaw with this setup: it broke cross compilation via GOOS and GOARCH.
GOOS=linux go run third_party.go build github.com/coreos/etcd
fork/exec /var/folders/nq/jrsys0j926z9q3cjp1yfbhqr0000gn/T/go-build584136562/command-line-arguments/_obj/exe/third_party: exec format error
The reason is that GOOS and GOARCH get used internally by go run. Meaning it
literally tries to build third_party.go as a Linux binary and runs it.
Running a Linux binary on an OSX machine doesn't work.
Furthermore by using third_party.go didn't get us any closer to being "go
gettable". With the continued weekly emails and bugs for this I started looking
around for better solutions and found goven.
goven achieves all of the desirable traits: reproducible builds, zero
dependencies to start developing, cross compilation, and as a bonus
go install github.com/coreos/etcd works!
The basic theory of operation is it checks all dependencies into subpackages of
your project. Instead of importing code.google.com/p/goprotobuf you import
github.com/coreos/etcd/third_party/code.google.com/p/goprotobuf. It makes the
imports uglier but it is automated by goven.
Along the way I wrote some helper tools to assist in bumping dependencies which
can be found on GitHub at philips/goven-bump. The scripts
goven-bump and goven-bump-commit grab the hg revision or git hash of the
dependency along with running goven. This makes bumping a dependency and
getting a basic commit message as easy as:
cd ${GOPATH}/github.com/coreos/etcd
goven-bump-commit code.google.com/p/goprotobuf
git commit -m 'bump(code.google.com/p/goprotobuf): 074202958b0a25b4d1e194fb8defe5d69c300774'
The way that goven re-wrote the package paths turned out to be a very
successful model for us and saved the developers of etcd a ton of headaches.
However, the goven-bump scripts were rather annoying and it forced the
developer to use git log to find the hash of the upstream dependency.
After emailing and chatting with Keith Rarick, author of goven and godep, he
told me that he was planning on adding a feature to his godep project that
rewrote import paths just like goven but had the advantages of a static
dependency manifest file. After a few months of nagging he merged
the godep save -r feature and I could finally start converting projects to
stop using goven-bump. Woo!
godep adds some additional complexity for the etcd team. But, the simplicity it
presents to regular contributors and users used to go get make it worth the
additional effort. For all of the CoreOS non-library projects that are written
in Go we now use godep and have achieved all of our goals:
If you maintain a Go project, I highly recommend you go install github.com/tools/godep.