How to determine if a JSON key has been set to null or not provided

I was recently skimming my Twitter feed for new Go articles, libraries, etc and I stumbled across a tweet with this message:

#Golang’s JSON Unmarshalling Is Bad

I’m not even sure how it managed to find its way into my feed. I wasn’t following the person who said it, so the only reason it would show up in my feed is the #Golang hashtag. Regardless, this struck me as a big cry for help. I have written about using JSON with Go in the past on the Gopher Academy’s Advent series (see https://blog.gopheracademy.com/advent-2016/advanced-encoding-decoding/) and I never personally found JSON to be handled poorly in Go. Perhaps not the easiest language in the world, but definitely not “bad”.

JSON has two types of null value

After reaching out it became clear to me that the issue here wasn’t specific to Go, but rather a shortcoming of JSON used in conjunction with any typed language. In JSON there are effectively two types of null.

1. When the key is provided, and the value is explicitly stated as null.
2. When the key is not provided, and the value is implicitly null.

Unfortunately (or fortunately?), that isn’t how Go, or really any typed languages work. We don’t declare a struct (or class) and magically lose a field when it isn’t defined. That field will always be present, and the value of that field will be nil or a valid value.

For example, let’s imagine we have a Blog struct and the PublishedAt is an optional JSON attribute, your code might look like this.

type Blog struct {
  ID          int    `json:"id"`
  Title       string `json:"title"`
  Markdown    string `json:"markdown"`
  PublishedAt *int   `json:"published_at"`
}

This leaves us with two options for PublishedAt - either it will reference an integer, or it will be nil. We don’t have any way to determine whether it being nil was implicit or explicit, we just know whether it is nil or not.

Why do we need to know the difference between implicit and explicit null values?

Most of the time the difference between a key being set to null and not being set at all isn’t relevant. If the published at value is set to null during a POST we can likely just assume that the post shouldn’t be published yet and move along.

Where this does come into play is when we are doing partial updates. For instance, if we have a JSON API and we are letting users update their blog posts via a PATCH request, an API user might send the following JSON payload to update the title of a blog post.

{"title": "some new title"}

When the published_at field isn’t provided our Blog type will set the field to null and we can assume that this means the user didn’t want to update the field. But what happens when the user does want to update this field? Specifically, what happens if the user wants to unpublish a blog post by setting the published_at field to null?

{"published_at":null}

Unfortunately, our code won’t know the difference between explicitly setting the field to null and not providing the key. So how to we handle this in Go?

Determining whether a key is provided

The first thing to note is that we can customize how JSON gets unmarshalled for any type by implementing the Unmarshaler interface. That means we can replace our PublishedAt field with a new type and write our own code to handle JSON parsing.

The second thing to remember is that if our field is a pointer type (eg PublishedAt *int), then its UnmarshalJSON() method will never be called if the value is null or if the key was never provided. That means we need to make sure we DO NOT use a pointer if we need to determine if a key has been set.

Putting those two tidbits together, we can create a custom JSONInt type that will be used to determine if a value has been set, and whether or not it is null.

type JSONInt struct {
  Value int
  Valid bool
  Set   bool
}

func (i *JSONInt) UnmarshalJSON(data []byte) error {
  // If this method was called, the value was set.
  i.Set = true

  if string(data) == "null" {
    // The key was set to null
    i.Valid = false
    return nil
  }

  // The key isn't set to null
  var temp int
  if err := json.Unmarshal(data, &temp); err != nil {
    return err
  }
  i.Value = temp
  i.Valid = true
  return nil
}

If we replace our *int type with the JSONInt type (not a pointer) the UnmarshalJSON() method will get called whenever we parse some JSON into a Blog object, giving us an opportunity to handle that portion of the unmarshalling with our custom code. Specifically, we can set a few flags to dictate whether or not the key was provided.

If you want to see this in action, we can test it out with a few JSON strings using the code below, or you can run it on the Go Playground.

func main() {
  notSet := `{}`
  setNull := `{"published_at": null}`
  setValid := `{"published_at": 123}`

  parseAndPrint(notSet)
  parseAndPrint(setNull)
  parseAndPrint(setValid)
}

func parseAndPrint(str string) {
  var b Blog
  json.Unmarshal([]byte(str), &b)
  fmt.Printf("<Value:%d> <Set:%t> <Valid:%t>\n",
    b.PublishedAt.Value, b.PublishedAt.Set, b.PublishedAt.Valid)
}

In Summary

While there are likely better API designs to avoid this issue, if you do ever run into a situation where you need to support both types of null for a JSON key you can leverage this technique to make it happen. Simply replace the Value field with whatever type you need to support and you should be set.