I was trying to setup a docker container for development and ran into an issue - no matter what I tried, file change events weren’t being propagated to the container, so I couldn’t get hot/live reloading to work.
As you can imagine, this was pretty frustrating; developing while having to manually stop, rebuild, and restart your application can already be time consuming enough, but toss in a docker container and a few extra steps and suddenly this process can feel like it is devouring your entire day. It just wasn’t going to work for me.
I’m a fairly pragmatic guy, so at this point my next step would typically be to minimize wasted time and revert to a tool I knew would work. In the past that has been running modd locally and not using a container for my local Go app. This has proven to work well for me because modd
can handle most prep tasks I need done and is also great for running tests after file changes. Win-win!
Unfortunately, that wasn’t going to cut it in this situation. As many of you know, I create programming courses, and one of the reasons I was so vested in getting this local docker setup to work is because I want to offer docker-compose configs for all of my new courses. This helps avoid any support issues where everything works on one OS, but has some subtle bug in another (I’m looking at you, Windows!) while also making it easier for newcomers to get things up and running quickly - you just need to have Docker installed. All of this meant that I needed a solution and couldn’t go back to my old ways. 😭
I examined my options, and eventually decided to just write a custom live reloader that uses polling. I decided that a library would be best, then I could just write a quick (~50 lines) main.go
file to use it for each project in the future.
I realize there are other live reloading tools out there, and many of the support polling, but I still felt writing my own was the best option in this particular case as it would get me exactly what I needed in a relatively short amount of time.
The rest of this article is going to document the process of writing the live reloading library, named pitstop
, and then discussing how I got it working with my Go app. After that I’ll talk about a few additional ideas and some potential issues with the library.
The first thing I did was break up the steps I thought I was going to take. History has taught me that this can be wrong, but I like having a broad-strokes idea of the steps I’m taking before starting, and I want to share that here.
The final version ended up taking a few more steps than this, but these turned out to be a pretty good starting point.
I needed a function that could recursively look at all the files in a directory looking for any that have changed since a given timestamp. The first version didn’t really need anything special, like extensions to ignore. I just needed a proof of concept so I could move forward with the rest of the steps and get something working.
Luckily, Go has a function that does most of this for me: filepath.Walk
With filepath.Walk
we simply need to provide a filepath.WalkFunc and it will walk all the files in a directory recursively while providing us with an os.FileInfo, which as luck would have it, has a ModTime()
method. That means all we really need is a way to detect changes, which can be done with a closure giving us the following code for our DidChange
function.
func DidChange(dir string, since time.Time) bool {
var changed bool
filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.IsDir() {
return nil
}
if info.ModTime().After(since) {
changed = true
}
return nil
})
return changed
}
Next up I needed a way to build and run the app. In many live reloaders this is just a bash command, but I didn’t see any real reason why I had to limit myself to just bash commands. After all, this was meant to be a library, so I could accept any function as a build or run step.
I settled on the following:
type BuildFunc func() error
type RunFunc func() (stop func(), err error)
As I mentioned earlier, many build steps tend to be bash commands like go install
so I also wanted to make this easy to achieve. To accommodate this, I added the following helpers.
// BuildCommand works similar to exec.Command, but rather than returning an
// exec.Cmd it returns a BuildFunc that can be reused.
func BuildCommand(command string, args ...string) BuildFunc {
return func() error {
cmd := exec.Command(command, args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
return fmt.Errorf("error building: \"%s %s\": %w", command, strings.Join(args, " "), err)
}
return nil
}
}
// RunCommand works similar to exec.Command, but rather than returning an
// exec.Cmd it returns a RunFunc that can be reused.
func RunCommand(command string, args ...string) RunFunc {
return func() (func(), error) {
cmd := exec.Command(command, args...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Start()
if err != nil {
return nil, fmt.Errorf("error running: \"%s %s\": %w", command, strings.Join(args, " "), err)
}
return func() {
cmd.Process.Kill()
}, nil
}
}
Now anywhere that I expect a BuildFunc
in my library I can provide a typical bash command with something like:
pitstop.BuildCommand("go", "install")
pitstop.BuildCommand("go", "test", "./...")
Similarly, I can provide RunFunc
s this way too:
pitstop.RunCommand("server", "--prod")
One of the trade-offs I made here for simplicity is that anything created by RunCommand
or BuildCommand
will write to os.Stdout
and os.Stderr
and this cannot be customized. I also opted to return an error if any build command exits with any non-zero status code, which we will see shortly means that any go test ./...
failures result in the build being halted. And finally, I opted to just use cmd.Process.Kill()
to kill a command. My reasoning here is that technically anyone using this library can write their own custom BuildFunc
that doesn’t use this default behavior, but for me this was what I wanted 90% of the time and I didn’t want to have to configure these helpers all the time.
Finally, I put this all together with a single Run
function.
// Run will run all pre BuildFuncs, then the RunFunc, and then finally the post
// BuildFuncs. Any errors encountered will be returned, and the build process
// halted. If RunFunc has been called, stop will also be called so that it is
// guaranteed to not be running anytime an error is returned.
func Run(pre []BuildFunc, run RunFunc, post []BuildFunc) (func(), error) {
for _, fn := range pre {
err := fn()
if err != nil {
return nil, err
}
}
stop, err := run()
if err != nil {
return nil, err
}
for _, fn := range post {
err := fn()
if err != nil {
stop()
return nil, err
}
}
return stop, nil
}
Sidenote: I considered a variadic param for the post
argument, but decided against this as it would make how we pass in pre
and post
different despite the fact that these are basically identical aside from when they are run.
With all of that code written and tested I was ready to put it all together and (hopefully!) start using it!
For the first version I wanted to limit the number of customization options, so I limited it to just the directory we are watching, the interval to wait between scanning for file changes, and the build/run functions we described in step 2. That left me with the Poller
type shown below.
type Poller struct {
// ScanInterval is the duration of time the poller will wait before scanning for new file changes. This defaults to 500ms.
ScanInterval time.Duration
// Dir is the directory to scan for file changes. This defaults to "." if it isn't provided.
Dir string
// Pre, Run, and Post represent the functions used to build and run our app.
// Pre functions are called first, then run, then finally the post functions.
Pre []BuildFunc
Run RunFunc
Post []BuildFunc
}
Once I had a type to work with, I wrote the Poll
method that would continuously check for file changes and try to rebuild our app. Again, I wanted to keep the code as simple as possible, so I decided not to worry about a build failing due to an issue that may resolves itself (eg a port being used). Instead, I just assume that if a build fails it won’t get fixed until another file changes. For my own purposes this tends to work well enough.
func (p *Poller) Poll() {
scanInt := p.ScanInterval
if scanInt == 0 {
scanInt = 500 * time.Millisecond
}
dir := p.Dir
if dir == "" {
dir = "."
}
var stop func()
var err error
var lastBuild time.Time
for {
if !DidChange(p.Dir, lastBuild) {
time.Sleep(scanInt)
continue
}
if stop != nil {
fmt.Println("Stopping running app...")
stop()
}
fmt.Println("Building & Running app...")
stop, err = Run(p.Pre, p.Run, p.Post)
if err != nil {
fmt.Printf("Error running: %v\n", err)
}
lastBuild = time.Now()
time.Sleep(scanInt)
}
}
Given how crude this Poller
type is, and the fact that our DidChange
and Run
functions have decent test coverage, I opted to skip writing tests at this time and instead tested this code manually. I know, I know, I’m breaking some cardinal rule of coding, but this is a dev tool that only I’m using so I’m allowed to take risks like this. 😜
I’ll probably eventually write better tests, but for now the tool is working and I’m letting it be.
In order to use the Poller
type I needed to create a main
package, import the pitstop
package, set a Poller
up, and finally call the Poll
method. For the most part that was pretty normal.
package main
import (
"fmt"
"os"
"time"
"github.com/joncalhoun/pitstop"
)
func main() {
poller := pitstop.Poller{
ScanInterval: 500 * time.Millisecond,
Dir: "./",
Pre: []pitstop.BuildFunc{
pitstop.BuildCommand("go", "test", "./..."),
pitstop.BuildCommand("go", "install", "./cmd/server"),
},
Run: pitstop.RunCommand("server"),
Post: []pitstop.BuildFunc{
func() error {
f, err := os.Create("../ui/src/last_built.js")
if err != nil {
return err
}
defer f.Close()
loc, err := time.LoadLocation("America/New_York")
if err != nil {
return err
}
now := time.Now().In(loc)
fmt.Fprintf(f, "const date =\"%s\";\n", now.Format(time.RFC1123))
fmt.Fprintln(f, "export default date;")
return err
},
},
}
poller.Poll()
}
Most of this should be pretty obvious. My Pre
functions are running go test ./...
and then installing the main
packager inside cmd/server
. My Run
function is then executing the server
command which is just a way of calling the binary that was just installed.
The only real oddity here is the Post
function I provided. Rather than running a bash command, I instead opted to provide some Go code that will create a file at ../ui/src/last_build.js
and write the following JavaScript to it:
const date ="Tue, 05 May 2020 19:43:51 EDT";
export default date;
This is just a little hack I add to some projects when I’m using React to ensure the UI reloads anytime I make an API change and I can visually see when the API was last updated using a React component like:
function LastBuilt(props) {
return (
<div className="w-full py-4 px-2 text-center bg-yellow-100 text-gray-600">
The Go API was last built: {props.date}
</div>
);
}
And there you have it - a live reloader in ~200 lines of Go code. About 150 of those lines are writing the pitstop
library, and the extra 50 come from the main
package you need to create a runnable binary.
Interestingly enough, seeing this in action ultimately inspired me to make changes to the pitstop
package. Most notably, it inspired me to allow users to provide an OnError
callback function that can then trigger neat visualizations, as we will see in the next section.
To keep things simple, I decided my OnError
callback was going to be a function that accepts an error and doesn’t return anything. After all, this is meant to handle errors and having it return its own error would be kinda weird. I’m not about to add an OnErrorError
callback 😂
type Poller struct {
// ...
// OnError is similar to Pre and Post, but is only called when Pre, Run, or
// Post encounter an error.
OnError func(error)
}
Rather than passing the error-case build functions into the Run
function, I opted to just place it in the Poll
method’s for loop. To simplify this, I do some nil checking early on in the Poll
method and assign an empty on error function if one was provided. The end result is the following code.
func (p *Poller) Poll() {
// ...
onError := p.OnError
if onError == nil {
onError = func(error) {}
}
// ...
for {
// ...
stop, err = Run(p.Pre, p.Run, p.Post)
if err != nil {
fmt.Printf("Error running: %v\n", err)
onError(err)
}
lastBuild = time.Now()
time.Sleep(scanInt)
}
}
This probably wouldn’t be ideal for testing, but I was hacking at this project at this point and was more concerned with seeing the end results. Besides, refactoring this type of code tends to be incredibly easy in Go, so I saw no need to get caught up making the first version perfect until I decided if it was worth keeping.
Next was the Go code to make use of the OnError
callback. For this I needed to head back over to the main
package I created that uses the pitstop
package. In this file I added two functions - one that writes a null error to an api_error.js
file on successful builds, and another that writes out an error on failed builds.
poller := pitstop.Poller{
Post: []pitstop.BuildFunc{
// ...
func() error {
f, err := os.Create("../ui/src/api_error.js")
if err != nil {
return err
}
defer f.Close()
fmt.Fprintf(f, "const error = null;\nexport default error;", err)
return nil
},
},
OnError: func(buildErr error) {
f, err := os.Create("../ui/src/api_error.js")
if err != nil {
fmt.Printf("failed to write error to api_error.js: %v\n", err)
return
}
defer f.Close()
fmt.Fprintf(f, "const error = `%v`;\nexport default error;", buildErr)
},
}
And finally some JavaScript and React to render this:
// import the error
import apiError from "./api_error";
// Define the React component
function Error({ error }) {
return (
<div className="w-full px-8 py-4">
<h3 className="text-2xl font-mono text-red-600">API Error:</h3>
<pre className="w-full py-4 px-2 bg-gray-200 text-red-600 border-2 border-gray-300">
{error}
</pre>
</div>
);
}
// Then where I want to use it:
{apiError ? <Error error={apiError} /> : ""}
Then I excitedly spun things up and intentionally introduced an error…
🥁 Drum roll… 🥁
And my javascript output was what you see below.
const error = `error building: "go test ./...": exit status 2`;
export default error;
And in the UI it looks like the screenshot below.
While this is neat, it wasn’t super helpful. Why did the build fail? What happened to all that output when we run go test ./...
?
I headed back to the pitstop
package and started hacking on the BuildCommand
and RunCommand
functions. In order to persist the output from commands, I wanted to use a strings.Builder and to assign it to the Stdout
and Stderr
for each command being created. This would have enabled me to take any output from my commands and append it to the error when there was one. Unfortunately, this would have also meant that anything we previously had written to the terminal would stop being written, which might be a bad idea if someone was expecting build errors to show up there.
The solution I opted for was io.MultiWriter, which is a function that accepts multiple writers and returns a single writer that will write to all of the provided writers. In short, this enabled me to have both os.Stdout
and my strings.Builder
being written to by the commands I was running.
var sb strings.Builder
cmd.Stdout = io.MultiWriter(os.Stdout, &sb)
cmd.Stderr = io.MultiWriter(os.Stderr, &sb)
err := cmd.Run()
if err != nil {
return fmt.Errorf("error building: \"%s %s\": %w\n%v", command, strings.Join(args, " "), err, sb.String())
}
The code is nearly identical for BuildCommand
and RunCommand
so I only show the important bits here. Click the source code link above to see the entire repo.
Now we are getting a slightly more useful error to work with:
const error = `error building: "go test ./...": exit status 2
# github.com/calhounio/api/admin
admin/handler.go:11:1: syntax error: non-declaration statement outside function body
`;
export default error;
Similarly, a failed test will also be much easier to read now:
const error = `error building: "go test ./...": exit status 1
? github.com/calhounio/api/admin [no test files]
? github.com/calhounio/api/cmd/server [no test files]
? github.com/calhounio/api/cmd/watcher [no test files]
--- FAIL: TestThing (0.00s)
watcher_test.go:6: Thing() = nil; want "hello"
FAIL
FAIL github.com/calhounio/api/poller 0.004s
FAIL
`;
export default error;
And in our React UI we can immediately see when a test is failing, making it pretty easy to follow TDD if that is what you prefer to do. If not, that’s cool too.
At this point I decided to call it quits. I have no idea if I’ll continue hacking on this project, but for now it solves a small problem I had while also enabling me to improve my development process ever so slightly.
I am aware that this project is far from perfect. I am okay with that. Part of my motivation in writing this article and sharing this code was to convey the point that often times our goal shouldn’t be to create a feature-complete piece of software, but simply to get something done that meets our needs.
Sign up for my mailing list and I'll send you a FREE sample from my course - Web Development with Go. The sample includes 19 screencasts and the first few chapters from the book.
You will also receive emails from me about Go coding techniques, upcoming courses (including FREE ones), and course discounts.
Jon Calhoun is a full stack web developer who teaches about Go, web development, algorithms, and anything programming. If you haven't already, you should totally check out his Go courses.
Previously, Jon worked at several statups including co-founding EasyPost, a shipping API used by several fortune 500 companies. Prior to that Jon worked at Google, competed at world finals in programming competitions, and has been programming since he was a child.
Related articles
Spread the word
Did you find this page helpful? Let others know about it!
Sharing helps me continue to create both free and premium Go resources.
Want to discuss the article?
See something that is wrong, think this article could be improved, or just want to say thanks? I'd love to hear what you have to say!
You can reach me via email or via twitter.
©2018 Jonathan Calhoun. All rights reserved.