Module Proxies and Vendoring

One of the risks of using third-party packages in your Go code is that the package repository may cease to be available. For example, the httprouter package plays a central part in our application, and if the author ever decided to delete it from GitHub it would cause us quite a headache to scramble and replace it with an alternative.

(I’m not suggesting this is likely to happen with httprouter — just using it as an example!)

Fortunately, Go provides two ways in which we can mitigate this risk: module proxies and vendoring.

Module proxies

Go supports module proxies (also known as module mirrors) by default. These are services which mirror source code from the original, authoritative, repositories (such as those hosted on GitHub, GitLab or BitBucket).

Go ahead and run the go env command on your machine to print out the settings for your Go operating environment. Your output should look similar to this:

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOENV="/home/alex/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/alex/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/alex/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/alex/Projects/greenlight/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="..."

The important thing to look at here is the GOPROXY setting, which contains a comma-separated list of module mirrors. By default it has the following value:

GOPROXY="https://proxy.golang.org,direct"

The URL https://proxy.golang.org that we see here points to a module mirror maintained by the Go team at Google, containing copies of the source code from tens of thousands of open-source Go packages.

Whenever you fetch a package using the go command — either with go get or one of the go mod * commands — it will first attempt to retrieve the source code from this mirror.

If the mirror already has a stored copy of the source code for the required package and version number, then it will return this code immediately in a zip file. Otherwise, if it’s not already stored, then the mirror will attempt to fetch the code from the authoritative repository, proxy it onwards to you, and store it for future use.

If the mirror can’t fetch the code at all, then it will return an error response and the go tool will fall back to fetching a copy directly from the authoritative repository (thanks to the direct directive in the GOPROXY setting).

Using a module mirror as the first fetch location has a few benefits:

• The https://proxy.golang.org module mirror typically stores packages long-term, thereby providing a degree of protection in case the original repository disappears from the internet.
• It’s not possible to override or delete a package once it’s stored in the https://proxy.golang.org module mirror. This can help prevent any bugs or problems which might arise if a package author (or an attacker) releases an edited version of the package with the same version number.
• Fetching modules from the https://proxy.golang.org mirror can be much faster than getting them from the authoritative repositories.

In most cases, I would generally suggest leaving the GOPROXY setting with its default values.

But if you don’t want to use the module mirror provided by Google, or you’re behind a firewall that blocks it, there are other alternatives like https://goproxy.io and the Microsoft-provided https://athens.azurefd.net that you can try instead. Or you can even host your own module mirror using the open-source Athens and goproxy projects.

For example, if you wanted to switch to using https://goproxy.io as the primary mirror, then fall back to using https://proxy.golang.org as a secondary mirror, then fall back to a direct fetch, you could update your GOPROXY setting like so:

$ export GOPROXY=https://goproxy.io,https://proxy.golang.org,direct

Or if you want to disable module mirrors altogether, you can simply set the value to direct like so:

$ export GOPROXY=direct

Vendoring

Go’s module mirror functionality is great, and I recommend using it. But it isn’t a silver bullet for all developers and all projects.

For example, perhaps you don’t want to use a module mirror provided by Google or another third-party, but you also don’t want the overhead of hosting your own mirror. Or maybe you need to routinely work in an environment without network access. In those scenarios you probably still want to mitigate the risk of a disappearing dependency, but using a module mirror isn’t possible or appealing.

You should also be aware that the default proxy.golang.org module mirror doesn’t absolutely guarantee that it will store a copy of the module forever. From the FAQS:

proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.

Additionally, if you need to come back to a ‘cold’ codebase in 5 or 10 years’ time, will the proxy.golang.org module mirror still be available? Hopefully it will — but it’s hard to say for sure.

So, for these reasons, it can still be sensible to vendor your project dependencies using the go mod vendor command. Vendoring dependencies in this way basically stores a complete copy of the source code for third-party packages in a vendor folder in your project.

Let’s demonstrate how to do this. We’ll start by adapting our make tidy rule to also call the go mod verify and go mod vendor commands, like so:

...

# ==================================================================================== #
# QUALITY CONTROL
# ==================================================================================== #

## tidy: format all .go files, and tidy and vendor module dependencies
.PHONY: tidy
tidy:
	@echo 'Formatting .go files...'
	go fmt ./...
	@echo 'Tidying module dependencies...'
	go mod tidy
	@echo 'Verifying and vendoring module dependencies...'
	go mod verify
	go mod vendor

...

Just to be clear about what’s going on behind-the-scenes here, let’s quickly step through what will happen when we run make tidy:

• The go mod tidy command will make sure the go.mod and go.sum files list all the necessary dependencies for our project (and no unnecessary ones).
• The go mod verify command will verify that the dependencies stored in your module cache (located on your machine at $GOPATH/pkg/mod) match the cryptographic hashes in the go.sum file.
• The go mod vendor command will then copy the necessary source code from your module cache into a new vendor directory in your project root.

Let’s try this out and run the new tidy rule like so:

$ make tidy
Formatting .go files...
go fmt ./...
Tidying module dependencies...
go mod tidy
Verifying and vendoring module dependencies...
go mod verify
all modules verified
go mod vendor

Once that’s completed, you should see that a new vendor directory has been created containing copies of all the source code along with a modules.txt file. The directory structure in your vendor folder should look similar to this:

$ tree -L 3 ./vendor/
./vendor/
├── github.com
│   ├── go-mail
│   │   └── mail
│   ├── julienschmidt
│   │   └── httprouter
│   └── lib
│       └── pq
├── golang.org
│   └── x
│       ├── crypto
│       └── time
├── gopkg.in
│   └── alexcesaro
│       └── quotedprintable.v3
└── modules.txt

Now, when you run a command such as go run, go test or go build, the go tool will recognize the presence of a vendor folder and the dependency code in the vendor folder will be used — rather than the code in the module cache on your local machine.

If you like, go ahead and try running the API application. You should find that everything compiles and continues to work just like before.

$ make run/api 
go run ./cmd/api -db-dsn=postgres://greenlight:pa55word@localhost/greenlight
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established"
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development

Because all the dependency source code is now stored in your project repository itself, it’s easy to check it into Git (or an alternative version control system) alongside the rest of your code. This is reassuring because it gives you complete ownership of all the code used to build and run your applications, kept under version control.

The downside of this, of course, is that it adds size and bloat to your project repository. This is of particular concern in projects that have a lot of dependencies and the repository will be cloned a lot, such as projects where a CI/CD system clones the repository with each new commit.

Let’s also take a quick look in the vendor/modules.txt file that was created. If you’ve been following along it should look similar to this:

# github.com/go-mail/mail/v2 v2.3.0
## explicit
github.com/go-mail/mail/v2
# github.com/julienschmidt/httprouter v1.3.0
## explicit; go 1.7
github.com/julienschmidt/httprouter
# github.com/lib/pq v1.10.9
## explicit; go 1.13
github.com/lib/pq
github.com/lib/pq/oid
github.com/lib/pq/scram
# github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce
## explicit
github.com/tomasen/realip
# golang.org/x/crypto v0.26.0
## explicit; go 1.17
golang.org/x/crypto/bcrypt
golang.org/x/crypto/blowfish
# golang.org/x/time v0.6.0
## explicit
golang.org/x/time/rate
# gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc
## explicit
gopkg.in/alexcesaro/quotedprintable.v3
# gopkg.in/mail.v2 v2.3.1
## explicit

This vendor/modules.txt file is essentially a manifest of the vendored packages and their version numbers. When vendoring is being used, the go tool will check that the module version numbers in modules.txt are consistent with the version numbers in the go.mod file. If there’s any inconsistency, then the go tool will report an error.

Lastly, you should avoid making any changes to the code in the vendor directory. Doing so can potentially cause confusion (because the code would no longer be consistent with the original version of the source code) and — besides — running go mod vendor will overwrite any changes you make each time you run it. If you need to change the code for a dependency, it’s far better to fork it and import the forked version instead.

Vendoring new dependencies

In the next section of the book we’re going to deploy our API application to the internet with Caddy as a reverse-proxy in-front of it. This means that, as far as our API is concerned, all the requests it receives will be coming from a single IP address (the one running the Caddy instance). In turn, that will cause problems for our rate limiter middleware which limits access based on IP address.

Fortunately, like most other reverse proxies, Caddy adds an X-Forwarded-For header to each request. This header will contain the real IP address for the client.

Although we could write the logic to check for the presence of an X-Forwarded-For header and handle it ourselves, I recommend using the realip package to help with this. This package retrieves the client IP address from any X-Forwarded-For or X-Real-IP headers, falling back to use r.RemoteAddr if neither of them are present.

If you’re following along, go ahead and install the latest version of realip using the go get command:

$ go get github.com/tomasen/realip@latest
go: downloading github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce
go get: added github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce

Then open up the cmd/api/middleware.go file and update the rateLimit() middleware to use this package like so:

package main

import (
    "errors"
    "expvar"
    "fmt"
    "net/http"
    "strconv"
    "strings"
    "sync"
    "time"

    "greenlight.alexedwards.net/internal/data"
    "greenlight.alexedwards.net/internal/validator"

    "github.com/tomasen/realip" // New import
    "golang.org/x/time/rate"
)

...

func (app *application) rateLimit(next http.Handler) http.Handler {

    ...

    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if app.config.limiter.enabled {
            // Use the realip.FromRequest() function to get the client's real IP address.
            ip := realip.FromRequest(r)

            mu.Lock()

            if _, found := clients[ip]; !found {
                clients[ip] = &client{
                    limiter: rate.NewLimiter(rate.Limit(app.config.limiter.rps), app.config.limiter.burst),
                }
            }

            clients[ip].lastSeen = time.Now()

            if !clients[ip].limiter.Allow() {
                mu.Unlock()
                app.rateLimitExceededResponse(w, r)
                return
            }

            mu.Unlock()
        }

        next.ServeHTTP(w, r)
    })
}

...

If you try to run the API application again now, you should receive an error message similar to this:

$ make run/api 
go: inconsistent vendoring in /home/alex/Projects/greenlight:
        github.com/tomasen/realip@v0.0.0-20180522021738-f0c99a92ddce: is explicitly 
            required in go.mod, but not marked as explicit in vendor/modules.txt

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor
make: *** [Makefile:24: run/api] Error 1

Essentially what’s happening here is that Go is looking for the github.com/tomasen/realip package in our vendor directory, but at the moment that package doesn’t exist in there.

To solve this, you’ll need to run the make tidy command, like so:

$ make tidy
Formatting .go files...
go fmt ./...
Tidying module dependencies...
go mod tidy
go: finding module for package github.com/tomasen/realip
go: downloading github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce
go: found github.com/tomasen/realip in github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce
Verifying and vendoring module dependencies...
go mod verify
all modules verified
go mod vendor

Once that’s done, everything should work correctly again:

$ make run/api 
go run ./cmd/api -db-dsn=postgres://greenlight:pa55word@localhost/greenlight
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="database connection pool established"
time=2023-09-10T10:59:13.722+02:00 level=INFO msg="starting server" addr=:4000 env=development

Additional Information

The ./… pattern

Most of the go tools support the ./... wildcard pattern, like go fmt ./..., go vet ./... and go test ./.... This pattern matches the current directory and all sub-directories, excluding the vendor directory.

Generally speaking, this is useful because it means that we’re not formatting, vetting or testing the code in our vendor directory unnecessarily — and our make audit rule won’t fail due to any problems that might exist within those vendored packages.