JSON APIs Are Just Web Applications

A common misconception, especially amongst new developers, is that learning to create an API is vastly different from learning to build an a web application that renders server side HTML. This notion is further compounded by the fact that there are many frameworks and libraries designed specifically for building APIs; Ruby has Rails::API, Python has the Django REST Framework, and there are countless others out there.

The truth is, building an API isn’t that different from building any other type of web application. Sure, there are a few differences, but in the grand scheme of things MOST of the logic powering a web application will occur behind the scenes and (if designed properly) will be agnostic of the input and output format being HTML forms, JSON, or even XML (yuck!).

The problem most people run into is that they start with an application where all of this is intermingled together. Their HTTP handler functions will parse JSON, validation fields, run SQL queries, and basically touch everything; as a result, it feels nearly impossible to go from an HTML based app to a JSON API without touching everything!

In this post I want to prove just how similar an HTML-based web application is to a JSON API. In order to do this, we are going to start off with a relatively messy, but simple, web application and refactor it using techniques like those described in Ben Johnson’s Standard Package Layout. Once the refactor is done we will update our application to be a JSON API using oauth-based authentication, and at that point it will be pretty clear just how little the JSON-specific code affects the rest of our application.

Now I’m going to be honest - a lot of these changes during the refactor are going to feel like overkill. The application we are starting with is simple out of necessity, and as a result it can often seem premature to make some of the changes we will be making. I couldn’t agree more, and I am a big advocate for avoiding big refactors like this until there is a need, but for the purpose of this article I think those changes are worth making. For starters, it can be insightful to see concrete examples of just how to apply some of these techniques, but more importantly by refactoring our code it will make the final changes - the ones that convert our app into a JSON API - more effective in demonstrating that APIs are just web applications.

Initial setup

Before we can get to the good parts, we first need to get a basic understanding of our messy demo application. This will be fairly brief, but if you want to explore the code we are starting with you can here: widgetapp on GitHub

Signing in users

While the demo application doesn’t have a real authentication system, I tried to mimic the parts that would ultimately affect both a web application and an API. Most notably, I store a user’s remember token in a cookie in order to fake the important parts of a session based authentication system.

I felt this was important because authentication is one of the areas where it is easy for someone new to web development to get confused thinking the two are drastically different, so by including it in our demo application we will see just how similar the two actually are.

Let’s take a moment to check out the source code used in this fake authentication. We are mostly going to discuss the happy path, as this is where all the interesting details are, but errors and other issues are handled in the code if you’d like to look in more detail.

func processSignin(w http.ResponseWriter, r *http.Request) {
	email := r.PostFormValue("email")
	password := r.PostFormValue("password")
	// Fake the password part
	if password != "demo" {
		http.Redirect(w, r, "/signin", http.StatusNotFound)
		return
  }

  // Lookup the user ID by their email in the DB
	email = strings.ToLower(email)
	row := db.QueryRow(`SELECT id FROM users WHERE email=$1;`, email)
	var id int
	err := row.Scan(&id)
	if err != nil {
		switch err {
		case sql.ErrNoRows:
			// Email doesn't map to a user in our DB
			http.Redirect(w, r, "/signin", http.StatusFound)
		default:
			http.Error(w, "Something went wrong. Try again later.", http.StatusInternalServerError)
		}
		return
  }

In the first portion of our processSignin handler function we get some post form values and “authenticate” the user. This is clearly faked, but the basic idea isn’t too far off. We look up a user with the provided email address and verify their password to make sure we should actually sign them in.

Once we have a valid user we need to sign them in. To do this our demo application uses a fake session token and then persists it in a cookie so that we can figure out who the user is in subsequent requests. Using session tokens is incredibly common for authenticating users in web applications, but it is important to note that I’m doing a few things “wrong” here. Most notably, I’m creating fake session tokens and not storing hashes of them in my database.

	// Create a fake session token
	token := fmt.Sprintf("fake-session-id-%d", id)
	_, err = db.Exec(`UPDATE users SET token=$2 WHERE id=$1`, id, token)
	if err != nil {
		http.Error(w, "Something went wrong. Try again later.", http.StatusInternalServerError)
		return
	}
	cookie := http.Cookie{
		Name:  "session",
		Value: token,
	}
	http.SetCookie(w, &cookie)
	http.Redirect(w, r, "/widgets", http.StatusFound)
}

Finally we redirect the users to the widgets page, and because we stored the user’s session token in a cookie any upcoming web requests can figure out who this user is by querying the database for a user with that session token. Yes, it requires a database query on every page load, but if a page starts to gain traction we can cache that very easily so it isn’t really a performance issue to worry about; it is easy to fix if it ever becomes necessary.

Verifying users via cookies

Okay, so we know that we stored a cookie to verify who the user is later, but what does that verification actually look like?

Let’s look at one of the handlers that require a user to be logged in such as the allWidget handler.

func allWidgets(w http.ResponseWriter, r *http.Request) {
	// Verify the user is signed in
	session, err := r.Cookie("session")
	if err != nil {
		http.Redirect(w, r, "/signin", http.StatusFound)
		return
	}
	row := db.QueryRow(`SELECT id FROM users WHERE token=$1;`, session.Value)
	var userID int
	err = row.Scan(&userID)
	if err != nil {
		switch err {
		case sql.ErrNoRows:
			// Email doesn't map to a user in our DB
			http.Redirect(w, r, "/signin", http.StatusFound)
		default:
			http.Error(w, "Something went wrong. Try again later.", http.StatusInternalServerError)
		}
		return
	}

In this code we first get the session cookie from the incoming web request. The name - session - is the name we gave our cookie when we created it. The value of this cookie is going to be the session token we created for the user and stored in their cookie.

We then query the database for a user with this session token. If we find one we know the user’s ID, and if we don’t we redirect them to the signin page. Easy enough. 👍

Viewing widgets

We also view widgets in the allWidgets function, so we will once again be looking at that code.

func allWidgets(w http.ResponseWriter, r *http.Request) {
  // ... removed for brevity

	// Query for this user's widgets
	rows, err := db.Query(`SELECT id, name, price, color FROM widgets WHERE userID=$1`, userID)
	if err != nil {
		http.Error(w, "Something went wrong.", http.StatusInternalServerError)
		return
	}
	defer rows.Close()
	var widgets []Widget
	for rows.Next() {
		var widget Widget
		err = rows.Scan(&widget.ID, &widget.Name, &widget.Price, &widget.Color)
		if err != nil {
			log.Printf("Failed to scan a widget: %v", err)
			continue
		}
		widgets = append(widgets, widget)
	}

I’ve removed the user verification code for brevity, so the code above starts right at the database query for widgets. From there we then scan each row returned by the query, parse it into a Widget instance, and then insert it into the widgets slice.

Next we need to render all of the widgets, so we get into some HTML-specific code.

	// Render the widgets
	tplStr := `
<!DOCTYPE html>
<html lang="en">
<h1>Widgets</h1>
<ul>
{{range .}}
	<li>{{.Name}} - {{.Color}}: <b>${{.Price}}</b></li>
{{end}}
</ul>
<p>
	<a href="/widgets/new">Create a new widget</a>
</p>
</html>`
	tpl := template.Must(template.New("").Parse(tplStr))
	err = tpl.Execute(w, widgets)
	if err != nil {
		http.Error(w, "Something went wrong.", http.StatusInternalServerError)
		return
	}
}

The last section of code in the allWidgets function defines the HTML template string and then creates a template.Template so that we can render the HTML with dynamically rendered widgets in the list. The html/template package handles escaping any script tags or other things the user might insert into fields of our widgets, so this code is pretty easy and straightforward.

Creating widgets

Creating widgets is fairly similar to the code we have seen before. There is a newWidget function that renders the HTML form (see the source code), and then there is a createWidget handler that parses the information, validates it, creates a new widget in the SQL database, and then finally redirects the user to the page with all the widgets.

The only thing I want to point out is that the createWidget function has a little bit of everything in it. We use post form values, convert strings into integers, interact with a database, and there is even some validation logic where we make sure any green widgets have an even price.

// Parse form values and validate data (pretend w/ me here)
name := r.PostFormValue("name")
color := r.PostFormValue("color")
price, err := strconv.Atoi(r.PostFormValue("price"))
if err != nil {
  http.Error(w, "Invalid price", http.StatusBadRequest)
  return
}
if color == "Green" && price%2 != 0 {
  http.Error(w, "Price must be even with a color of Green", http.StatusBadRequest)
  return
}

// Create a new widget!
_, err = db.Exec(`INSERT INTO widgets(userID, name, price, color) VALUES($1, $2, $3, $4)`, userID, name, price, color)
if err != nil {
	http.Error(w, "Something went wrong. Try again later.", http.StatusInternalServerError)
	return
}
http.Redirect(w, r, "/widgets", http.StatusFound)

What ugly handlers you have! 😱

For now our handlers may be fine, but if this application were to grow they could become an issue. There is just such a wide variety of dependencies all in one place. We interact directly with a database, authenticate users, create session tokens and cookies, and oh so much more all in our http handlers! Imagine if we needed to test this code; it would be very hard to test each of these components in isolation, and we would instead need to have a real database connection, real users in the database, and more just to verify that our code didn’t allow for green widgets with an invalid price. Yikes!

When we lump all of our logic - validation, parsing HTML forms, rendering results, etc - all into a single handler it doesn’t take long before things start to get out of hand. It works for simpler applications, but just not so well for larger applications. This is why you will see people start to isolate different parts of their code into packages that each focus on a specific dependency. For instance, you might have a psql package that handles all the Postgres specific code.

We are going to make changes like that as well with our demo widget application. I’m not expecting you to know exactly what each of these are now, but below is a rough outline of what we will be doing. After going over the outline we will jump into the code for each change and I’ll explain what is going on in more detail at that time.

1. First, we are going to define our domain types in root app package. This will give us a common set of types to work with as we build out our application and will eventually make it easier to write implementation agnostic code in many areas.
2. Next we are going to move all of our PostgreSQL specific code into its own package where we define all of our database interactions. This will make details like how a database is queried irrelevant in our handler functions and will make it much easier to write tests with fakes and mocks [1] .
3. After that we will introduce middleware with the goal of making our authentication logic easier to reuse and test. This version will include some code that uses context values in a not-so-great manner, but will address that in the next step.
4. Next we make our usage of context values a little safer. This mostly just means adding a custom context package to isolate keys and add a little bit of type-safety back into our code.
5. Finally we will move our handlers into an http package, since the primary dependency these focus on is the net/http package. In the process of making this change our main package will become much simpler, only focusing on setting things up and starting the server.

Domain types

While nearly all of our design is shaped by ideas like Ben Johnson’s Standard Package Layout, the idea behind this section comes directly from Ben’s article, so rather than trying to explain what a domain is I’m going to use a quote from the article:

Your application has a logical, high-level language that describes how data and processes interact. This is your domain. If you have an e-commerce application your domain involves things like customers, accounts, charging credit cards, and handling inventory. If you’re Facebook then your domain is users, likes, & relationships. It’s the stuff that doesn’t depend on your underlying technology.

- Ben Johnson, Standard Package Layout

As we start to build out web applications in Go, the problem we will often run into is figuring out how and where to define our domain. Many developers coming from frameworks like Rails and Django will use to the MVC organizational pattern splitting much of their domain across packages like models, views and controllers. Others will throw organization out the window and just have one mega-package that contains all domain types.

Using the single mega-package approach can be helpful when learning because it allows developers to skip the part where they learn how to properly package their code. While it is pretty clear that it won’t scale indefinitely, most newcomers to Go aren’t building massive applications. Instead, they are worried about learning the underlying tools and packages.

The MVC pattern can also be a useful learning tool; it provides more guidance than the mega-package approach in terms of organizing your code, making it better for larger applications, while also being familiar and easy to grasp. In fact, I use the MVC pattern in my course, Web Development with Go, because that familiarity and simplicity make it much easier for new developers to get up and running with.

In short, both of these approaches, and likely any other you can imagine, can be great choices depending on your circumstances, but each definitely has its own unique downsides. The mega-package is pretty clear - if we only use a single package our code is going to become hard to maintain and reuse. The MVC pattern on the other hand is less clear. While it works well at first, there are a few issues you could eventually run into. For instance you could introduce dependency cycles, run into stuttered names like controllers.UsersController, and more importantly it is hard to isolate code based on its dependencies. That is, even if your Charge model is mostly powered by Stripe, the code for it will very likely be in the models package along with all of your database specific code.

The approach Ben discusses in Standard Package Layout is to instead define a single package with all of your domain specific types and no other dependencies. Then throughout the rest of our code we create packages that use and implement the types and interfaces defined in our domain.

The benefits of this are twofold.

First, because our package with our domain types doesn’t have any other dependencies, it won’t have any imports. This means we can import these domain types from anywhere in our code without introducing a dependency cycle. This can take some time to get used to, but once you do you will notice that this design really encourages you to write implementation agnostic code without any extra effort or thought.

Second, testing and mocking become oh so much easier. We can create fakes or mocks of all of our domain types and then run tests without relying on real implementations whenever we want.

In my experience Ben’s approach is less clear to beginners, but once you get used to it you will wonder why you haven’t been using it all along.

Enough talk; let’s see this in action! We will start by moving our main package into a cmd directory and then introducing a top level app package where we define our domain types.

$ cd $GOPATH/github.com/joncalhoun/widgetapp
$ mkdir cmd
$ mv main.go cmd/server.go
$ touch models.go

Our directory structure should now look like this:

widgetapp/
	cmd/
		server.go
	models.go

Now let’s define our domain types. At this point we don’t know what those all are, but we know we have a both a user and a widget resource in our DB so we can start out with those. We will be adding these to models.go

package app

// Widget represents the widget that we create with our app.
type Widget struct {
	ID     int
	UserID int
	Name   string
	Price  int
	Color  string
}

// User represents a user in our system.
type User struct {
	ID    int
	Email string
	Token string
}

Notice that I didn’t add any struct tags for things like JSON encoding and decoding or a database package. While you could probably get away with it, I tend to define a custom Widget type in a JSON-specific package when we get to that point. This allows me to truly isolate all of the JSON encoding/decoding details from my domain types, while also giving me an opportunity to differentiate between my internal data structure and the external JSON format. It also helps prevents bugs like accidentally exposing internal fields in our JSON output because we forgot to add the json:"-" struct tag. In short, this ends up being slightly more code, but it is typically worth the extra effort.

Now that we have domain types we can go to our cmd/server.go and update it to use them. These changes are fairly trivial so you can check them out on your own: refactor/domain-types branch

PostgreSQL specific package

Just like with our User and Widget types, we are also going to want to define implementation agnostic datastores that we can use throughout our application and implement them in an isolated, database specific package. This will keep all of our SQL and Postgres specific logic from affecting the rest of our code, making it easier to change database implementations later if we need to or to switch between using an ORM and not using one.

The tricky part here is the order in which these two pieces are created. In many static languages, like Java, you would typically find yourself writing the interface first, then creating an implementation of that interface that interacts with your database. The problem with this approach is that we often end up defining more features than we need while also forcing our implementation to match an interface that might not describe actual behavior very well.

To avoid these issues, I suggest writing your code in the following order:

1. Write your implementation-specific code first. For instance, create a UserService inside a psql package that has methods used to interact directly with a Postgres database. Don’t worry about how general this code is - just write the methods you actually need and avoid adding methods “just in case”.
2. Once you have a real implementation, use it to help you decide what your implementation-agnostic domain type should look like. This will likely be an interface, but that doesn’t mean you should just copy every method in your psql.UserService. Perhaps one of those methods isn’t very general and returns sql specific data so you may need to adapt the implementation code to be a bit more generic, or you might have some setup/teardown methods that don’t really need to be included in the domain type. The goal here is to try to figure out what your implementation-agnostic interface looks like using a real implementation as a reference point.
3. Refactor the rest of your code to use the domain type. The only time you should be using the implementation specific version is when setting things up and passing it into code that expects the domain type. This is a little clearer with code, so review the source code for this section and make sure you understand it.

We will review the actual implementation code first since this is the first step. We won’t be looking at all of the code, but we’ll look at small sample to give you a feel for the changes. As always, the complete source and diff are linked at the bottom of this section.

Our first code snippet demonstrates how we create a UserService type and then add methods to it that we can use to interact with the database, doing things like querying for a user with a specific email address.

// UserService is a PostgreSQL specific implementation of the user datastore.
type UserService struct {
	DB *sql.DB
}

// ByEmail will look for a user with the same email address, or return
// app.ErrNotFound if no user is found. Any other errors raised by the sql
// package are passed through.
//
// ByEmail is NOT case sensitive.
func (s *UserService) ByEmail(email string) (*app.User, error) {
	user := app.User{
		Email: strings.ToLower(email),
	}
	row := s.DB.QueryRow(`SELECT id, token FROM users WHERE email=$1;`, user.Email)
	err := row.Scan(&user.ID, &user.Token)
	if err != nil {
		// This part is different from GitHub but we will see the final code shortly
		return nil, err
	}
	return &user, nil
}

While writing this code I noticed an issue. In this code I return any errors encountered by the row.Scan method call, and in my cmd/server.go source file I handle the sql.ErrNoRows error differently than any other error this code might produce. I do this because sql.ErrNoRows is a special error signaling that nothing went wrong, but there weren’t any records in the database that matched our query. That could mean that the user provided an invalid email address, or with other database records, like our widgets, it could mean that I need to render a 404 page notifying the user that their request was fine, but we couldn’t locate that specific widget.

What is the issue? Well, by relying on the sql.ErrNoRows error to inform us that a record wasn’t found we end up writing code that relies on an implementation detail. We wouldn’t return sql.ErrNoRows if our datastore was implemented with MongoDB, so this is a subtle hint that our UserService isn’t as implementation agnostic as we originally thought.

Luckily the fix is fairly simple; just create a domain specific error that signals when a record wasn’t located and start using that instead. I’ll walk through the source code changes below.

First let’s look at the code where we declare the error inside our app package:

package app

import (
	"errors"
)

var (
	// ErrNotFound is an implementation agnostic error that should be returned
	// by any service implementation when a record was not located.
	ErrNotFound = errors.New("app: resource requested could not be found")
)

And then we can go back to the ByEmail method and update the error handling.

func (s *UserService) ByEmail(email string) (*app.User, error) {
  // ... unchanged
	err := row.Scan(&user.ID, &user.Token)
	if err != nil {
		switch err {
		case sql.ErrNoRows:
			return nil, app.ErrNotFound
		default:
			return nil, err
		}
	}
	return &user, nil
}

Now we can use this user service to access our data layer while truly being agnostic of the implementation. Code like our processSignin function and other http handlers don’t need to think about sql-specific errors.

Little details like this might seem unimportant at this point, but they are what differentiates an app that is locked into a specific set of technologies from one where you can make changes when your requirements evolve. In this case we are focusing on the datastore, but the same is true when looking at things like whether your app is an HTML-based web app or a JSON API, and we will see that later in this article.

In the meantime, let’s assume we continued writing PostgreSQL specific services for both our widgets and our users in the psql package and we are now ready to define our domain types. To do this we would look at our implementations and decide on an interface to describe each. Those interfaces are shown below, and are defined in the app package.

type UserService interface {
	ByEmail(email string) (*User, error)
	ByToken(token string) (*User, error)
	UpdateToken(userID int, token string) error
}

type WidgetService interface {
	ByUser(userID int) ([]Widget, error)
	Create(widget *Widget) error
}

Finally we go through the rest of the codebase and refactor it to use these new domain types. This doesn’t actually look much different right now because most of our code is inside the main package, but as we continue writing code we will also be keeping this in mind.

Authentication middleware

The http handler functions in our demo still have a lot of duplicate logic, especially when it comes to validating users. While a little duplication isn’t always an issue, authentication logic is something we tend to use a lot. We just end up having quite a few pages that require a user to be logged in to access the page.

We could move the authentication code into a function and call it from all our handlers, but in my experience middleware and context values work very well for authentication logic. Not only can they lookup the user and set them as a context value, but they can also prevent your original handler from even being called if the user isn’t logged in making it much easier to write handlers that just assume a user is logged in without doing any checks.

The basic idea behind middleware is pretty straightforward - we write a function that “wraps” the original http.HandlerFunc. That is, we want an end result that looks something like this:

func timeMiddleware(w http.ResponseWriter, r *http.Request) {
	start := time.Now()
	listWidgets(w, r)
	log.Println("Duration:", time.Now().Sub(start))
}

The problem with this example code is that it is too specific. The timeMiddleware function can only be used with the listWidgets function, and what we really want is a way to use the timeMiddleware with any handler function.

To achieve this we accept an http.HandlerFunc argument which could be listWidgets or any other function, and then we write return a closure that resembles our original timeMiddleware function.

func timeMiddlware(fn http.HandlerFunc) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		fn(w, r)
		log.Println("Duration:", time.Now().Sub(start))
	}
}

We are going to add two middleware functions to our application; one to retrieve the user from the session cookie and set it in the request context, and another to redirect the user to the login page if they aren’t logged in. Splitting these two pieces of functionality up is handy because it allows us to lookup the user for both pages that require a user to be logged in as well as on pages that can optionally use the current logged in user. For instance, we might want to render the menu of our page different based on whether the user is logged in, but we don’t require a user on many pages where we do this.

Let’s look at the code first, then discuss it.

type Auth struct {
	UserService app.UserService
}

func (a *Auth) UserViaSession(next http.Handler) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		session, err := r.Cookie("session")
		if err != nil {
			// No user found - time to move on
			next.ServeHTTP(w, r)
			return
		}
		user, err := a.UserService.ByToken(session.Value)
		if err != nil {
			next.ServeHTTP(w, r)
			return
		}
		r = r.WithContext(context.WithValue(r.Context(), "user", user))
		next.ServeHTTP(w, r)
	}
}

func (a *Auth) RequireUser(next http.Handler) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		tmp := r.Context().Value("user")
		if tmp == nil {
			http.Redirect(w, r, "/signin", http.StatusFound)
			return
		}
		if _, ok := tmp.(*app.User); !ok {
			http.Redirect(w, r, "/signin", http.StatusFound)
			return
		}
		next.ServeHTTP(w, r)
	}
}

We start off with an Auth type so that we can access the user service later on in our middleware functions. We then define the two middleware functions - UserViaSession and RequireUser.

In UserViaSession we do pretty much the same thing we did in our handler functions when we looked up a user via their session token. The only real difference is that after we retrieve the user we add it to the request context so that we can access it later in our handler functions.

Now that we have the user stored in the request context, verifying that a user is logged in is incredibly easy. We just need to check to see if a user is set in the context. If one is, we are good to go. If a user isn’t set we redirect the user to the signin page.

tmp := r.Context().Value("user")
if tmp == nil {
	http.Redirect(w, r, "/signin", http.StatusFound)
	return
}

We also do a few checks to make sure the value stored actually is an *app.User, but that will get cleaned up a bit later.

Finally we write some code to make it easier to apply middleware, apply our middleware where it is appropriate, and remove any of the old code that is handled by our authentication middleware. While I won’t be posting all of that code in this article, you can find links to it all below.

Better context values

The core idea in this section is to make our usage of context values safer. I have a lengthier article that discusses this in more detail - Pitfalls of context values and how to avoid or mitigate them in Go - so I’m not going to go into the “why” part too much. Instead we will just quickly scan over the code changes. Check out the linked article if you want to understand the “why” part more. It’s a good read, I promise 😄.

First we want a key that can’t be overwritten by other packages. This will prevent us from having to worry about some random code accidentally storing something at the same key we use. To do this, we create an unexported type in our context package.

type ctxKey string

const (
	userKey ctxKey = "user"
)

Now we use that key to create a type-safe way to add users to a context.

func WithUser(ctx context.Context, user *app.User) context.Context {
	return context.WithValue(ctx, userKey, user)
}

And finally we need a function to get the user out of a context.

func User(ctx context.Context) *app.User {
	tmp := ctx.Value(userKey)
	if tmp == nil {
		return nil
	}
	user, ok := tmp.(*app.User)
	if !ok {
		// This shouldn't happen. If it does, it means we have a bug in our code.
		log.Printf("context: user value set incorrectly. type=%T, value=%#v", tmp, tmp)
		return nil
	}
	return user
}

Now we no longer need to use the standard library’s context package directly in the rest of our code to get user values. We will instead use our new context package.

http package

We have been organizing our code based on dependencies, and our HTTP handlers are no different; they all rely on the net/http package and should be in a package that reflects this.

While making these changes we will also introduce some new types and make our http.HandlerFuncs methods so that we can eliminate the usage of global variables, but if you wanted you could also use an approach like the one Mat Ryer describes in How I write Go HTTP services after seven years where you return a closure and pass in things like the UserService as an argument. I’m opting not to do this because we would need to pass those services into quite a few functions, and I find it easier to attached them to say a single UserHandler once and let it be used by multiple handler methods.

Let’s look at the UserHandler first.

type UserHandler struct {
	userService app.UserService
}

func (h *UserHandler) ShowSignin(w http.ResponseWriter, r *http.Request) {
	html := `
<!DOCTYPE html>
<html lang="en">
<form action="/signin" method="POST">
	<label for="email">Email Address</label>
	<input type="email" id="email" name="email" placeholder="you@example.com">

	<label for="password">Password</label>
	<input type="password" id="password" name="password" placeholder="something-secret">

	<button type="submit">Sign in</button>
</form>
</html>`
	fmt.Fprint(w, html)
}

The code here is basically the same as it was before, but we took our showSignin function and made it a method on the UserHandler type. This allows us to avoid global variables, so when we migrate our processSignin function we can use the userService field on our UserHandler rather than a global var.

func (h *UserHandler) ProcessSignin(w http.ResponseWriter, r *http.Request) {
	// ... mostly unchanged

	// Here is the change - we use the userService field of h, the UserHandler
	user, err := h.userService.ByEmail(email)

	// ...
}

Another benefit of using structs and methods is that we can now shorten our method names. If we had a package level function named New(...) it would be pretty vague, but when the method is attached to the WidgetHandler type it is pretty clear that this function is used to render a new widget form.

func (h *WidgetHandler) New(w http.ResponseWriter, r *http.Request) {
	html := `
		<!DOCTYPE html>
		<html lang="en">
			...
		</html>`
	fmt.Fprint(w, html)
}

Our middleware also relies on the net/http package but is currently in its own mw package. While we may have missed that before, as we start to build out our new http package it becomes pretty clear. As a result, I decided to move it into our new http package and update a few names used. You can see the code on github.

Our routes are also very http-specific. You simply cannot define a route without talking about http handler functions, http methods, and other http specifics. To address this, I opted to create a Server type in our http package and have it contain all of our handlers as well as our router. Now we simply need to add a ServeHTTP method to it and we can return it as our master handler.

func NewServer(us app.UserService, ws app.WidgetService) *Server {
	server := Server{
		authMw: &AuthMw{
			userService: us,
		},
		users: &UserHandler{
			userService: us,
		},
		widgets: &WidgetHandler{
			widgetService: ws,
		},
		router: mux.NewRouter(),
	}
	server.routes()
	return &server
}

type Server struct {
	// unexported types - do not use the zero value
	authMw  *AuthMw
	users   *UserHandler
	widgets *WidgetHandler
	router  *mux.Router
}

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	s.router.ServeHTTP(w, r)
}

func (s *Server) routes() {
	s.router.Handle("/", http.RedirectHandler("/signin", http.StatusFound))

	// User routes
	s.router.HandleFunc("/signin", s.users.ShowSignin).Methods("GET")
	s.router.HandleFunc("/signin", s.users.ProcessSignin).Methods("POST")

	// Widget routes
	s.router.Handle("/widgets", ApplyMwFn(s.widgets.Index,
		s.authMw.UserViaSession, s.authMw.RequireUser)).Methods("GET")
	s.router.Handle("/widgets", ApplyMwFn(s.widgets.Create,
		s.authMw.UserViaSession, s.authMw.RequireUser)).Methods("POST")
	s.router.Handle("/widgets/new", ApplyMwFn(s.widgets.New,
		s.authMw.UserViaSession, s.authMw.RequireUser)).Methods("GET")
}

With that we are ready to update our cmd/server.go but this introduces an interesting problem. http.ListenAndServe is a function in the http package. Didn’t we want to isolate all of our http dependencies? And won’t we have two packages we are trying to use with the name http now?

There are basically two logical ways to solve this problem:

1. Add a custom identifier for one of the http packages. Eg import stdhttp "net/http"
2. Add a ListenAndServe function to our new http package.

I opted for (2), but you could just as easily have gone for (1). I just prefer (2) because it means I can customize how that function works if I ever need to, or if I just want the default behavior I assign a variable to the original function.

package http

import "net/http"

var ListenAndServe = http.ListenAndServe

Now we can update our cmd/server.go and it becomes much simpler: server.go on refactor/http branch

I’m guessing I may have missed a few details that were changed during this phase, so definitely check out the code and the diff (linked below) to see exactly what all changed. You should also grab a copy of the code and run it locally to verify that it is the same logic; our code is just organized in a different manner.

Isolating our HTML specific code

The authentication middleware change is the easiest to explain, so let’s start there. What we are going to do is define an interface for our authentication middleware, and then take our existing implementation and make sure it implements that interface.

type AuthMw interface {
	SetUser(next http.Handler) http.HandlerFunc
	RequireUser(next http.Handler) http.HandlerFunc
}

While it isn’t necessary, I am going to move the implementation into a file named html.go where ALL of our HTML specific Go code is going to reside. Below is the updated authentication middleware which now implements the AuthMw interface.

type htmlAuthMw struct {
	userService app.UserService
}

func (mw *htmlAuthMw) SetUser(next http.Handler) http.HandlerFunc {
	// ... same as before, just a new function name
}

func (mw *htmlAuthMw) RequireUser(next http.Handler) http.HandlerFunc {
	// ... same as before
}

After that we want to isolate our HTML specifics in both the UserHandler and the WidgetHandler. To do this, I first create new fields on both that define functions used to render and parse with HTML-specific logic, such as whether the widget is being provided via an HTML form, or whether we should redirect the user instead of sending them a JSON response.

Below is what the widget handler ends up looking like.

type WidgetHandler struct {
	widgetService app.WidgetService

	renderNew func(http.ResponseWriter)

	parseWidget         func(*http.Request) (*app.Widget, error)
	renderCreateSuccess func(http.ResponseWriter, *http.Request)
	renderCreateError   func(http.ResponseWriter, *http.Request, error)

	renderIndexSuccess func(http.ResponseWriter, *http.Request, []app.Widget) error
	renderIndexError   func(http.ResponseWriter, *http.Request, error)
}

There are a lot more fields now, but with these we can go through our New, Create and Index methods and start using these methods instead of writing HTML-specific code.

func (h *WidgetHandler) New(w http.ResponseWriter, r *http.Request) {
	// Render with the new field `renderNew`
	h.renderNew(w)
}

func (h *WidgetHandler) Create(w http.ResponseWriter, r *http.Request) {
	user := context.User(r.Context())

	// Parse the widget with the new `parseWidget` field
	widget, err := h.parseWidget(r)

	// ...
}

There are several changes like this, so be sure to check out the complete source code, but the TL;DR is that we replaced ALL of our HTML-specific code in our handler methods with functions like parseWidget and renderNew in the previous two code samples.

While we were making those changes, we also added a few new errors and error types.

type validationError struct {
	fields  []string
	message string
}

func (e validationError) Error() string {
	return fmt.Sprintf("http validation error: Fields[%s] are not valid. %s", strings.Join(e.fields, ", "), e.message)
}

// Then we use it in our handlers like so...
if widget.Color == "Green" && widget.Price%2 != 0 {
	h.renderCreateError(w, r, validationError{
		fields:  []string{"price", "color"},
		message: "Price must be even with a color of Green",
	})
	return
}

I opted to make these changes because we were rendering specific errors when a user made a bad request, and I wanted a way to retain that functionality. These really are validation errors, so that’s the name I went with. Now when we render errors we can take this specific type of error into account easily.

renderCreateError: func(w http.ResponseWriter, r *http.Request, err error) {
	switch v := err.(type) {
	case validationError:
		http.Error(w, v.message, http.StatusBadRequest)
	default:
		http.Error(w, "Something went wrong. Try again later.", http.StatusInternalServerError)
	}
},

And more importantly, later if we migrated to a JSON API this error could be just as easily used to render a JSON-specific error.

Finally, we needed to implement all the new functions we added as fields on our WidgetHandler and UserHandler. These are all done in the html.go source file in the htmlUserHandler and htmlWidgetHandler functions. I won’t dive into the code any further, but a few of the code samples above come from these functions.

Once again, take a minute to review the final code from this section. At this point we are done refactoring and are ready to see just how easy it is to change our app from an HTML-based web app into a JSON API using OAuth for validation.

Migrating to a JSON API with OAuth2 authentication

We are finally ready to prove the point of this post - that APIs are just web applications. To do this we are going to add a new source file - json.go - inside the http package. In this file we will define all of our JSON specific code.

Let’s start with a simple renderJSON function that we can use to render JSON with a status code.

func renderJSON(w http.ResponseWriter, data interface{}, status int) {
	w.WriteHeader(status)
	enc := json.NewEncoder(w)
	enc.Encode(data)
}

It is also likely that we will need to render JSON specific errors, so we can use a generic jsonError type to satisfy our needs.

type jsonError struct {
	Message string `json:"error"`
	Type    string `json:"type"`
}

func (e jsonError) Error() string {
	return fmt.Sprintf("json %s error: %s", e.Type, e.Message)
}

Now we can create an AuthMw implementation. RequireUser won’t really change much, except when a user isn’t found we won’t redirect them to the signin page and will instead just give them a JSON error in response.

func (mw *jsonAuthMw) RequireUser(next http.Handler) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		user := context.User(r.Context())
		if user == nil {
			renderJSON(w, jsonError{
				Message: "Unauthorized access. Do you have a valid oauth2 token set?",
				Type:    "unauthorized",
			}, http.StatusUnauthorized)
			return
		}
		next.ServeHTTP(w, r)
	}
}

Next up is the SetUser method. This entails retrieving an "Authorization" header value, then making sure the value starts with the "Bearer" prefix. These are both fairly standard for OAuth2, so it shouldn’t shock any developers interacting with our API.

After that we trim the last portion of the authorization header - the actual oauth2 token - and then use that to lookup the user in our database. Interestingly, we don’t need to change any of our backend logic to support this flow. We can simply use session tokens as oauth2 tokens, and if our session tokens had expiration dates we could later return that information when a user signs in.

func (mw *jsonAuthMw) SetUser(next http.Handler) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		bearer := r.Header.Get("Authorization")
		if len(bearer) < len("Bearer") {
			next.ServeHTTP(w, r)
			return
		}
		token := strings.TrimSpace(bearer[len("Bearer"):])
		user, err := mw.userService.ByToken(token)
		if err != nil {
			next.ServeHTTP(w, r)
			return
		}
		r = r.WithContext(context.WithUser(r.Context(), user))
		next.ServeHTTP(w, r)
	}
}

At this point our authentication middleware is complete for our JSON API.

Next we need to write a JSON specific implementation of the UserHandler. We won’t actually need any code to render a signin form since JSON APIs don’t use those, so all we really need to implement is the ProcessSignin function in which we will parse the incoming username and password JSON and then return an OAuth2 token if they are valid, otherwise we need to render a JSON error.

func jsonUserHandler(us app.UserService) *UserHandler {
	uh := UserHandler{
		userService: us,

		parseEmailAndPassword: func(r *http.Request) (email, password string) {
			var req struct {
				Email    string `json:"email"`
				Password string `json:"password"`
			}
			dec := json.NewDecoder(r.Body)
			dec.Decode(&req)
			return req.Email, req.Password
		},
		renderProcessSigninSuccess: func(w http.ResponseWriter, r *http.Request, token string) {
			t := oauth2.Token{
				TokenType:   "Bearer",
				AccessToken: token,
			}
			renderJSON(w, t, http.StatusOK)
		},
		renderProcessSigninError: func(w http.ResponseWriter, r *http.Request, err error) {
			switch err {
			case errAuthFailed:
				renderJSON(w, jsonError{
					Message: "Invalid authentication details",
					Type:    "authentication",
				}, http.StatusBadRequest)
			default:
				renderJSON(w, jsonError{
					Message: "Something went wrong. Try again later.",
					Type:    "internal_server",
				}, http.StatusInternalServerError)
			}
		},
	}
	return &uh
}

That wasn’t so bad, was it?

Now on to the widget handler, where once again we don’t need the New endpoint that renders a form, but we will still need the Index that returns a list of widgets.

While updating this code we have to make a decision; when a user creates a widget our HTML application would redirect them to the widgets index. In an API a redirect like this doesn’t make sense, and instead an API will often do one of two things:

1. Return something like a 204 (NO CONTENT) status code with no content in the body.
2. Return something like a 201 (CREATED) status code with a JSON payload of the newly created widget.

I personally feel option (2) is best, but our WidgetHandler doesn’t pass the newly created widget into our renderCreateSuccess function. Luckily that is easy to update, and it won’t break our HTML code becuase it can just ignore that data if it doesn’t use it.

type WidgetHandler struct {
	// ...

	// Everything else is mostly unchanged. Just add the
	// *app.Widget to the end of renderCreateSuccess as an
	// additional argument.
	renderCreateSuccess func(http.ResponseWriter, *http.Request, *app.Widget)

	// ...
}

// then later in WidgetHandler's Create method:
func (h *WidgetHandler) Create(w http.ResponseWriter, r *http.Request) {
	// ...

	// This is the last line in the method.
	h.renderCreateSuccess(w, r, widget)
}

Now we need to update the html.go source file to account for this change.

renderCreateSuccess: func(w http.ResponseWriter, r *http.Request, _ *app.Widget) {
	http.Redirect(w, r, "/widgets", http.StatusFound)
},

Easy enough. Now back to writing JSON specific code. Our next step is going to be creating a type to help us render widgets in JSON.

type jsonWidget struct {
	ID    int    `json:"id"`
	Name  string `json:"name"`
	Color string `json:"color"`
	Price int    `json:"price"`
	// userID is intentionally not here.
}

func (jw *jsonWidget) read(w app.Widget) {
	jw.ID = w.ID
	jw.Name = w.Name
	jw.Color = w.Color
	jw.Price = w.Price
}

Adding this code instead of attaching it to your domain type might seem like extra work, and if you really want you can just add struct tags to your app.Widget, but I often find that how I store my data in the database and how I return a JSON object don’t match up 100%, so by having a JSON-specific widget I have a chance to make any data transformations I find useful (like nested resources in the JSON payload).

We are finally ready to implement the JSON widget handler. My first pass definitely isn’t the prettiest code in the world with code duplication and other parts that could be cleaned up, but for now it get’s the job done. Let’s start by looking at the code to parse a widget from an incoming request.

func jsonWidgetHandler(ws app.WidgetService) *WidgetHandler {
	wh := WidgetHandler{
		widgetService: ws,

		parseWidget: func(r *http.Request) (*app.Widget, error) {
			var req struct {
				Name  string `json:"name"`
				Color string `json:"color"`
				Price int    `json:"price"`
			}
			dec := json.NewDecoder(r.Body)
			err := dec.Decode(&req)
			if err != nil {
				// We could try to parse out the price validation error here if
				// we want, or return a more generic errBadRequest, but I'm going
				// to be lazy for now and just assume this is a validation error
				return nil, validationError{
					fields:  []string{"price"},
					message: "Price must be an integer",
				}
			}
			return &app.Widget{
				Name:  req.Name,
				Color: req.Color,
				Price: req.Price,
			}, nil
		},
		// ...
	}
	// ...
}

The code here defines a variable named req that is defined as an anonymous struct. These can be really handy when you just need to parse a few specific variables of a type and want to ignore all others. Eg in our case we want to parse things like a widget’s name, but we don’t want the user to be be allowed to set things like the ID or user ID, so we omit those from our struct.

It can be more code to write anonymous structs like this, especially if you have many endpoints that all accept similar, but not exactly identical, data, so be sure to consider whether it makes sense in your application.

Next up are the functions for rendering the Create endpoint. We’ll look at both the successful and error responses, as the success side of things is very simple.

func jsonWidgetHandler(ws app.WidgetService) *WidgetHandler {
	wh := WidgetHandler{
		// ...
		renderCreateSuccess: func(w http.ResponseWriter, r *http.Request, widget *app.Widget) {
			var res jsonWidget
			res.read(*widget)
			renderJSON(w, res, http.StatusCreated)
		},
		renderCreateError: func(w http.ResponseWriter, r *http.Request, err error) {
			switch v := err.(type) {
			case validationError:
				renderJSON(w, struct {
					Fields []string `json:"fields"`
					jsonError
				}{
					Fields: v.fields,
					jsonError: jsonError{
						Message: v.message,
						Type:    "validation",
					},
				}, http.StatusBadRequest)
			default:
				renderJSON(w, jsonError{
					Message: "Something went wrong. Try again later.",
					Type:    "internal_server",
				}, http.StatusInternalServerError)
			}
		},
		// ...
	}
	// ...
}

While this code may look confusing at first, it is actually pretty simple.

If we successfully create a widget, we render the JSON of that widget using a jsonWidget and return that with the http.StatusCreated status code.

If we encounter an error we have to decide how detailed we want our error message to be. In this particular case I opted to only handle one specific error type - the validationError - in a special way. Most notably, I add an additional field to the error that specifies which fields are invalid.

Long term this might not scale, and we might need to come up with a single renderJSONError function to handle all the types of errors our API responds with, but short term this felt like the best solution as it makes it very clear which types of errors are possible from each API endpoint.

Finally we get to the methods used to render the results of the Index method.

func jsonWidgetHandler(ws app.WidgetService) *WidgetHandler {
	wh := WidgetHandler{
		// ...
		renderIndexSuccess: func(w http.ResponseWriter, r *http.Request, widgets []app.Widget) error {
			res := make([]jsonWidget, 0, len(widgets))
			for _, widget := range widgets {
				var jw jsonWidget
				jw.read(widget)
				res = append(res, jw)
			}
			enc := json.NewEncoder(w)
			return enc.Encode(res)
		},
		renderIndexError: func(w http.ResponseWriter, r *http.Request, err error) {
			renderJSON(w, jsonError{
				Message: "Something went wrong. Try again later.",
				Type:    "internal_server",
			}, http.StatusInternalServerError)
		},
	}
	return &wh
}

Again these are very similar to what we had before, with the most notable exception being that we have to convert a slice of app.Widget objects into a slice of jsonWidgets.

The final change we need to make is adding routes. We have a lot of options here, so to start we are going to just take the path of least resistance. We will start by adding a way to opt out of form-specific routes.

func (s *Server) routes(webMode bool) {
	if webMode {
		s.router.Handle("/", http.RedirectHandler("/signin", http.StatusFound))
	}
	// ...
	if webMode {
		s.router.HandleFunc("/signin", s.users.ShowSignin).Methods("GET")
	}
	// ...
	if webMode {
		s.router.Handle("/widgets/new", ApplyMwFn(s.widgets.New,
			s.authMw.SetUser, s.authMw.RequireUser)).Methods("GET")
	}
}

This change is pretty obvious - we are skipping routes that are only used to render forms so that when we construct our API routes we can skip them. If it turns out you needed to skip other routes you could update the variable name to reflect what is actually happening (eg webMode), but this works for us now.

Next we update the NewServer function, both by renaming it to HTMLServer, and also by making sure it passes true into the routes method.

func HTMLServer(us app.UserService, ws app.WidgetService) *Server {
	// ... all the same

	server.routes(true)
	return &server
}

Now we need a JSON server. Seems simple enough - we can just copy the HTMLServer source code and tweak a few things.

func JSONServer(us app.UserService, ws app.WidgetService) *Server {
	server := Server{
		authMw: &jsonAuthMw{
			userService: us,
		},
		users:   jsonUserHandler(us),
		widgets: jsonWidgetHandler(ws),
		router:  mux.NewRouter(),
	}
	server.routes(false)
	return &server
}

And finally, what happens if we want to support both? Turns out that is easy as well. The code below does this with hard coded path prefixes, but we could make those dynamic by adding a few more arguments to the NewServer function.

func NewServer(us app.UserService, ws app.WidgetService) http.Handler {
	html := HTMLServer(us, ws)
	json := JSONServer(us, ws)
	mux := http.NewServeMux()
	mux.Handle("/", html)
	mux.Handle("/api/", http.StripPrefix("/api", json))
	return mux
}

The oddest part about our code now is probably the fact that we ever returned our Server type. The only method we expose on it is the ServeHTTP method, so if we wanted we could just update our HTMLServer and JSONServer functions to return http.Handlers. This represents what their goal is much better than returning Server does, and by doing this we could avoid needing to export the Server type.

Now I know, “accept interfaces, return structs” is the motto in Go, but if you ask me this is one of those cases where the interface just makes more sense. It does a much better job of describing our actual intent, which is only to return some form of http.Handler.

Testing our API

If you want to test this code out, I have provided a client/demo.go source file that will use the oauth2 package along with Go’s net/http package to send a login request, get an oauth2 token back, then use that token to make an API request to create a widget, then another to retrieve a list of all the widgets.

If you run this you will probably notice that our endpoint for creating a widget doesn’t return the correct widget ID. To fix this, look at the psql package where we implement the WidgetService. We need to add something like RETURNING id to the SQL statement to get the ID back and then scan it into the widget’s ID field.

JSON vs HTML doesn’t affect most of your web app

Now it is pretty hard to deny - the HTML and JSON specific code in this application doesn’t really affect the rest of our application very much. Sure, we had to refactor a ton to get to the final version, so it might feel like it had a major impact, but in reality the only difference between an HTML-based web app and a JSON API is contained in two files - http/html.go and http/json.go. That is pretty impressive, especially when you consider that the rest of our application continues to operate exactly the same regardless of whether the web request is via the API or an HTML web page.

The truth is, when you build larger applications a vast majority of your logic is going to be agnostic of whether you are rendering HTML pages, or server JSON (or even XML!) via an API. Those details stop mattering as soon as you parse the incoming data, and only start mattering again when we finally need to render a result to the user. It can sometimes feel like this isn’t true, especially if your application looks something like the code we started with, but in larger applications the single-source-file approach likely won’t cut it, and you’ll find yourself splitting code into packages anyway, much like we did during our refactor.

So stop worrying about learning how to build an API. Learn to build web applications in Go. Learn how to design maintainable and usable packages in Go. Just learn how to build things in Go, and the rest will slowly come, I promise!

Thank you Eno Compton for reviewing early drafts of this article! It was riddled with typos before he helped out 😀