Creating a Live Reloader in Less than 200 Lines of Go Code

I was trying to setup a docker container for development and ran into an issue - no matter what I tried, file change events weren’t being propagated to the container, so I couldn’t get hot/live reloading to work.

As you can imagine, this was pretty frustrating; developing while having to manually stop, rebuild, and restart your application can already be time consuming enough, but toss in a docker container and a few extra steps and suddenly this process can feel like it is devouring your entire day. It just wasn’t going to work for me.

I’m a fairly pragmatic guy, so at this point my next step would typically be to minimize wasted time and revert to a tool I knew would work. In the past that has been running modd locally and not using a container for my local Go app. This has proven to work well for me because modd can handle most prep tasks I need done and is also great for running tests after file changes. Win-win!

Unfortunately, that wasn’t going to cut it in this situation. As many of you know, I create programming courses, and one of the reasons I was so vested in getting this local docker setup to work is because I want to offer docker-compose configs for all of my new courses. This helps avoid any support issues where everything works on one OS, but has some subtle bug in another (I’m looking at you, Windows!) while also making it easier for newcomers to get things up and running quickly - you just need to have Docker installed. All of this meant that I needed a solution and couldn’t go back to my old ways. 😭

I examined my options, and eventually decided to just write a custom live reloader that uses polling. I decided that a library would be best, then I could just write a quick (~50 lines) main.go file to use it for each project in the future.

I realize there are other live reloading tools out there, and many of the support polling, but I still felt writing my own was the best option in this particular case as it would get me exactly what I needed in a relatively short amount of time.

The rest of this article is going to document the process of writing the live reloading library, named pitstop, and then discussing how I got it working with my Go app. After that I’ll talk about a few additional ideas and some potential issues with the library.

Breaking the code into steps

The first thing I did was break up the steps I thought I was going to take. History has taught me that this can be wrong, but I like having a broad-strokes idea of the steps I’m taking before starting, and I want to share that here.

1. A function to detect file changes.
2. A function to build and run my app.
3. An outer loop that calls these functions and sleeps as necessary using user-provided config variables.

The final version ended up taking a few more steps than this, but these turned out to be a pretty good starting point.

Step 1 - Detecting file changes

I needed a function that could recursively look at all the files in a directory looking for any that have changed since a given timestamp. The first version didn’t really need anything special, like extensions to ignore. I just needed a proof of concept so I could move forward with the rest of the steps and get something working.

Luckily, Go has a function that does most of this for me: filepath.Walk

With filepath.Walk we simply need to provide a filepath.WalkFunc and it will walk all the files in a directory recursively while providing us with an os.FileInfo, which as luck would have it, has a ModTime() method. That means all we really need is a way to detect changes, which can be done with a closure giving us the following code for our DidChange function.

func DidChange(dir string, since time.Time) bool {
	var changed bool

	filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		}
		if info.IsDir() {
			return nil
		}
		if info.ModTime().After(since) {
			changed = true
		}
		return nil
	})

	return changed
}

Step 2 - Building and running the app

Next up I needed a way to build and run the app. In many live reloaders this is just a bash command, but I didn’t see any real reason why I had to limit myself to just bash commands. After all, this was meant to be a library, so I could accept any function as a build or run step.

I settled on the following:

type BuildFunc func() error

type RunFunc func() (stop func(), err error)

As I mentioned earlier, many build steps tend to be bash commands like go install so I also wanted to make this easy to achieve. To accommodate this, I added the following helpers.

// BuildCommand works similar to exec.Command, but rather than returning an
// exec.Cmd it returns a BuildFunc that can be reused.
func BuildCommand(command string, args ...string) BuildFunc {
	return func() error {
		cmd := exec.Command(command, args...)
		cmd.Stdout = os.Stdout
		cmd.Stderr = os.Stderr
		err := cmd.Run()
		if err != nil {
			return fmt.Errorf("error building: \"%s %s\": %w", command, strings.Join(args, " "), err)
		}
		return nil
	}
}

// RunCommand works similar to exec.Command, but rather than returning an
// exec.Cmd it returns a RunFunc that can be reused.
func RunCommand(command string, args ...string) RunFunc {
	return func() (func(), error) {
		cmd := exec.Command(command, args...)
		cmd.Stdout = os.Stdout
		cmd.Stderr = os.Stderr
		err := cmd.Start()
		if err != nil {
			return nil, fmt.Errorf("error running: \"%s %s\": %w", command, strings.Join(args, " "), err)
		}
		return func() {
			cmd.Process.Kill()
		}, nil
	}
}

Now anywhere that I expect a BuildFunc in my library I can provide a typical bash command with something like:

pitstop.BuildCommand("go", "install")
pitstop.BuildCommand("go", "test", "./...")

Similarly, I can provide RunFuncs this way too:

pitstop.RunCommand("server", "--prod")

One of the trade-offs I made here for simplicity is that anything created by RunCommand or BuildCommand will write to os.Stdout and os.Stderr and this cannot be customized. I also opted to return an error if any build command exits with any non-zero status code, which we will see shortly means that any go test ./... failures result in the build being halted. And finally, I opted to just use cmd.Process.Kill() to kill a command. My reasoning here is that technically anyone using this library can write their own custom BuildFunc that doesn’t use this default behavior, but for me this was what I wanted 90% of the time and I didn’t want to have to configure these helpers all the time.

Finally, I put this all together with a single Run function.

// Run will run all pre BuildFuncs, then the RunFunc, and then finally the post
// BuildFuncs. Any errors encountered will be returned, and the build process
// halted. If RunFunc has been called, stop will also be called so that it is
// guaranteed to not be running anytime an error is returned.
func Run(pre []BuildFunc, run RunFunc, post []BuildFunc) (func(), error) {
	for _, fn := range pre {
		err := fn()
		if err != nil {
			return nil, err
		}
	}
	stop, err := run()
	if err != nil {
		return nil, err
	}
	for _, fn := range post {
		err := fn()
		if err != nil {
			stop()
			return nil, err
		}
	}
	return stop, nil
}

Sidenote: I considered a variadic param for the post argument, but decided against this as it would make how we pass in pre and post different despite the fact that these are basically identical aside from when they are run.

With all of that code written and tested I was ready to put it all together and (hopefully!) start using it!

Step 3 - Putting it all together

For the first version I wanted to limit the number of customization options, so I limited it to just the directory we are watching, the interval to wait between scanning for file changes, and the build/run functions we described in step 2. That left me with the Poller type shown below.

type Poller struct {
	// ScanInterval is the duration of time the poller will wait before scanning for new file changes. This defaults to 500ms.
	ScanInterval time.Duration

	// Dir is the directory to scan for file changes. This defaults to "." if it isn't provided.
	Dir string

	// Pre, Run, and Post represent the functions used to build and run our app.
	// Pre functions are called first, then run, then finally the post functions.
	Pre  []BuildFunc
	Run  RunFunc
	Post []BuildFunc
}

Once I had a type to work with, I wrote the Poll method that would continuously check for file changes and try to rebuild our app. Again, I wanted to keep the code as simple as possible, so I decided not to worry about a build failing due to an issue that may resolves itself (eg a port being used). Instead, I just assume that if a build fails it won’t get fixed until another file changes. For my own purposes this tends to work well enough.

func (p *Poller) Poll() {
	scanInt := p.ScanInterval
	if scanInt == 0 {
		scanInt = 500 * time.Millisecond
	}
	dir := p.Dir
	if dir == "" {
		dir = "."
	}

	var stop func()
	var err error
	var lastBuild time.Time

	for {
		if !DidChange(p.Dir, lastBuild) {
			time.Sleep(scanInt)
			continue
		}
		if stop != nil {
			fmt.Println("Stopping running app...")
			stop()
		}
		fmt.Println("Building & Running app...")
		stop, err = Run(p.Pre, p.Run, p.Post)
		if err != nil {
			fmt.Printf("Error running: %v\n", err)
		}
		lastBuild = time.Now()
		time.Sleep(scanInt)
	}
}

Given how crude this Poller type is, and the fact that our DidChange and Run functions have decent test coverage, I opted to skip writing tests at this time and instead tested this code manually. I know, I know, I’m breaking some cardinal rule of coding, but this is a dev tool that only I’m using so I’m allowed to take risks like this. 😜

I’ll probably eventually write better tests, but for now the tool is working and I’m letting it be.

Using the Poller

In order to use the Poller type I needed to create a main package, import the pitstop package, set a Poller up, and finally call the Poll method. For the most part that was pretty normal.

package main

import (
	"fmt"
	"os"
	"time"

	"github.com/joncalhoun/pitstop"
)

func main() {
	poller := pitstop.Poller{
		ScanInterval: 500 * time.Millisecond,
		Dir:          "./",
		Pre: []pitstop.BuildFunc{
			pitstop.BuildCommand("go", "test", "./..."),
			pitstop.BuildCommand("go", "install", "./cmd/server"),
		},
		Run: pitstop.RunCommand("server"),
		Post: []pitstop.BuildFunc{
			func() error {
				f, err := os.Create("../ui/src/last_built.js")
				if err != nil {
					return err
				}
				defer f.Close()
				loc, err := time.LoadLocation("America/New_York")
				if err != nil {
					return err
				}
				now := time.Now().In(loc)
				fmt.Fprintf(f, "const date =\"%s\";\n", now.Format(time.RFC1123))
				fmt.Fprintln(f, "export default date;")
				return err
			},
		},
	}
	poller.Poll()
}

Most of this should be pretty obvious. My Pre functions are running go test ./... and then installing the main packager inside cmd/server. My Run function is then executing the server command which is just a way of calling the binary that was just installed.

The only real oddity here is the Post function I provided. Rather than running a bash command, I instead opted to provide some Go code that will create a file at ../ui/src/last_build.js and write the following JavaScript to it:

const date ="Tue, 05 May 2020 19:43:51 EDT";
export default date;

This is just a little hack I add to some projects when I’m using React to ensure the UI reloads anytime I make an API change and I can visually see when the API was last updated using a React component like:

function LastBuilt(props) {
  return (
    <div className="w-full py-4 px-2 text-center bg-yellow-100 text-gray-600">
      The Go API was last built: {props.date}
    </div>
  );
}

And there you have it - a live reloader in ~200 lines of Go code. About 150 of those lines are writing the pitstop library, and the extra 50 come from the main package you need to create a runnable binary.

Interestingly enough, seeing this in action ultimately inspired me to make changes to the pitstop package. Most notably, it inspired me to allow users to provide an OnError callback function that can then trigger neat visualizations, as we will see in the next section.

Creating an OnError callback

To keep things simple, I decided my OnError callback was going to be a function that accepts an error and doesn’t return anything. After all, this is meant to handle errors and having it return its own error would be kinda weird. I’m not about to add an OnErrorError callback 😂

type Poller struct {
	// ...

	// OnError is similar to Pre and Post, but is only called when Pre, Run, or
	// Post encounter an error.
	OnError func(error)
}

Rather than passing the error-case build functions into the Run function, I opted to just place it in the Poll method’s for loop. To simplify this, I do some nil checking early on in the Poll method and assign an empty on error function if one was provided. The end result is the following code.

func (p *Poller) Poll() {
	// ...
	onError := p.OnError
	if onError == nil {
		onError = func(error) {}
	}
	// ...
	for {
		// ...
		stop, err = Run(p.Pre, p.Run, p.Post)
		if err != nil {
			fmt.Printf("Error running: %v\n", err)
			onError(err)
		}
		lastBuild = time.Now()
		time.Sleep(scanInt)
	}
}

This probably wouldn’t be ideal for testing, but I was hacking at this project at this point and was more concerned with seeing the end results. Besides, refactoring this type of code tends to be incredibly easy in Go, so I saw no need to get caught up making the first version perfect until I decided if it was worth keeping.

Next was the Go code to make use of the OnError callback. For this I needed to head back over to the main package I created that uses the pitstop package. In this file I added two functions - one that writes a null error to an api_error.js file on successful builds, and another that writes out an error on failed builds.

poller := pitstop.Poller{
	Post: []pitstop.BuildFunc{
		// ...
		func() error {
			f, err := os.Create("../ui/src/api_error.js")
			if err != nil {
				return err
			}
			defer f.Close()
			fmt.Fprintf(f, "const error = null;\nexport default error;", err)
			return nil
		},
	},
	OnError: func(buildErr error) {
		f, err := os.Create("../ui/src/api_error.js")
		if err != nil {
			fmt.Printf("failed to write error to api_error.js: %v\n", err)
			return
		}
		defer f.Close()
		fmt.Fprintf(f, "const error = `%v`;\nexport default error;", buildErr)
	},
}

And finally some JavaScript and React to render this:

// import the error
import apiError from "./api_error";

// Define the React component
function Error({ error }) {
  return (
    <div className="w-full px-8 py-4">
      <h3 className="text-2xl font-mono text-red-600">API Error:</h3>
      <pre className="w-full py-4 px-2 bg-gray-200 text-red-600 border-2 border-gray-300">
        {error}
      </pre>
    </div>
  );
}

// Then where I want to use it:
{apiError ? <Error error={apiError} /> : ""}

Then I excitedly spun things up and intentionally introduced an error…

🥁 Drum roll… 🥁

And my javascript output was what you see below.

const error = `error building: "go test ./...": exit status 2`;
export default error;

And in the UI it looks like the screenshot below.

While this is neat, it wasn’t super helpful. Why did the build fail? What happened to all that output when we run go test ./...?

Persisting output from exec.Cmd

I headed back to the pitstop package and started hacking on the BuildCommand and RunCommand functions. In order to persist the output from commands, I wanted to use a strings.Builder and to assign it to the Stdout and Stderr for each command being created. This would have enabled me to take any output from my commands and append it to the error when there was one. Unfortunately, this would have also meant that anything we previously had written to the terminal would stop being written, which might be a bad idea if someone was expecting build errors to show up there.

The solution I opted for was io.MultiWriter, which is a function that accepts multiple writers and returns a single writer that will write to all of the provided writers. In short, this enabled me to have both os.Stdout and my strings.Builder being written to by the commands I was running.

var sb strings.Builder
cmd.Stdout = io.MultiWriter(os.Stdout, &sb)
cmd.Stderr = io.MultiWriter(os.Stderr, &sb)
err := cmd.Run()
if err != nil {
	return fmt.Errorf("error building: \"%s %s\": %w\n%v", command, strings.Join(args, " "), err, sb.String())
}

The code is nearly identical for BuildCommand and RunCommand so I only show the important bits here. Click the source code link above to see the entire repo.

Now we are getting a slightly more useful error to work with:

const error = `error building: "go test ./...": exit status 2
# github.com/calhounio/api/admin
admin/handler.go:11:1: syntax error: non-declaration statement outside function body
`;
export default error;

Similarly, a failed test will also be much easier to read now:

const error = `error building: "go test ./...": exit status 1
?   	github.com/calhounio/api/admin	[no test files]
?   	github.com/calhounio/api/cmd/server	[no test files]
?   	github.com/calhounio/api/cmd/watcher	[no test files]
--- FAIL: TestThing (0.00s)
    watcher_test.go:6: Thing() = nil; want "hello"
FAIL
FAIL	github.com/calhounio/api/poller	0.004s
FAIL
`;
export default error;

And in our React UI we can immediately see when a test is failing, making it pretty easy to follow TDD if that is what you prefer to do. If not, that’s cool too.

At this point I decided to call it quits. I have no idea if I’ll continue hacking on this project, but for now it solves a small problem I had while also enabling me to improve my development process ever so slightly.

I am aware that this project is far from perfect. I am okay with that. Part of my motivation in writing this article and sharing this code was to convey the point that often times our goal shouldn’t be to create a feature-complete piece of software, but simply to get something done that meets our needs.