Does Go's range Copy a Slice Before Iterating Over It?

This week someone in my Go courses Slack asked why their for loop wasn’t working the way they expected. More specifically, they were wondering if Go’s range keyword was copying their slice before iterating over it.

I won’t use the exact same code here, but we can look at a similar example by trying to generate the Fibonacci sequence.

If you don’t know what the Fibonacci sequence is, don’t worry. It isn’t vital to understanding the code. All you really need to know is that we are appending new values to our slice as we iterate over it.

fib := []int{0, 1}
for i, f1 := range fib {
  f2 := fib[i+1]
  fib = append(fib, f1+f2)
  if f1+f2 > 100 {
    break
  }
}
fmt.Println(fib)

Run it on the Go Playground.

At first glance we might expect this code to continue adding new values until it adds a value greater than 100, at which point the for loop would exit with the break keyword. In reality, this code produces the following output:

[0 1 1 2]

On the other hand, if we were to rewrite this code with a more traditional for loop it would work as expected.

fib := []int{0, 1}
for i := 0; i < len(fib); i++ {
  // Note: I'm trying to keep this code as similar as possible to the previous
  // code, so I'm using f1 and f2 instead of just adding the numbers together
  // here.
  f1, f2 := fib[i], fib[i+1]
  fib = append(fib, f1+f2)
  if f1+f2 > 100 {
    break
  }
}
fmt.Println(fib)

Run it on the Go Playground.

When we run this code we get the output we would expect - a list of numbers that keeps growing until one reaches a value greater than 100.

[0 1 1 2 3 5 8 13 21 34 55 89 144]

All of this confusion leads to the question, “Does Go copy a slice when using the range keyword?”

The answer here is yes, but probably not how you think.

According to the Go spec, range expressions are only evaluated onces.

The range expression x is evaluated once before beginning the loop, with one exception: if at most one iteration variable is present and len(x) is constant, the range expression is not evaluated.

Go only reads the slice once, so when we use the range keyword it is like taking the following code:

for i, f1 := range fib {
  // ...
}

And translating it into code that is roughly equivalent to the following code:

var f1 int
temp := fib
for i = 0; i < len(temp); i++ {
  f1 = temp[i]
  // ...
}

If we take this knowledge and apply it to our for loop that didn’t work, it becomes a bit more clear why it wasn’t working.

fib := []int{0, 1}
// This is roughly what range results in
var f1 int
temp := fib
for i := 0; i < len(temp); i++ {
  f1 = temp[i]
  // back to our code
  f2 := fib[i+1]
  fib = append(fib, f1+f2)
  if f1+f2 > 100 {
    break
  }
}
fmt.Println(fib)

Our code is updating the fib slice, but this isn’t the same slice that our for loop is using to evaluate when to stop running.

Ideally this would be the end of the discussion, but slices can be a little confusing. Behind the scenes, a slice isn’t quite the same as an array. Instead, it has an underlying array and the slice itself stores a pointer to that array. You can read more about this here, and the code for a slice is shown below.

type slice struct {
  array unsafe.Pointer
  len   int
  cap   int
}

This is important because it means that we can alter items in our slice, even when using the range keyword, and we might see those changes as we iterate over the slice. Take the following bubble sort where we swap numbers while we iterate over the slice:

numbers := []int{5, 4, 3, 2, 1}
for x := 0; x < len(numbers); x++ {
  for i, num := range numbers {
    if i+1 >= len(numbers) {
      break
    }
    if num > numbers[i+1] {
      // Swap numbers if the current one is greater than the next
      numbers[i], numbers[i+1] = numbers[i+1], num
    }
  }
}
fmt.Println(numbers)

Run it on the Go Playground.

In this code we are swapping numbers in our slice when the number at a lower index is greater than the next number. When we do this, the next iteration of the for loop reads the number we swapped to the higher index. If we translate this code like we did before, we get the following:

numbers := []int{5, 4, 3, 2, 1}
for x := 0; x < len(numbers); x++ {
  var num int
  temp := numbers
  for i := 0; i < len(temp); i++ {
    num = temp[i]
    if i+1 >= len(numbers) {
      break
    }
    if num > numbers[i+1] {
      // Swap numbers if the current one is greater than the next
      numbers[i], numbers[i+1] = numbers[i+1], num
    }
  }
}
fmt.Println(numbers)

Running this code also produces the same result - a sorted list. This occurs because the slices temp and numbers both point to the same underlying array. Where this becomes confusing is when we start adding new values to our slice and the capacity of our slice starts to matter.

In a slice, the length is the number of values stored in the underlying array, but in reality that array might have space for additional values. That is, our slice might have a length of 5, but the underlying array might have a length of 7. Let’s look at a visual representation of this:

// Our slice might have the following fields:
array: *pointer to X below*
len: 5
cap: 7

// X is an array, not a slice. It has a fixed length of 7
X = [5,4,3,2,1,0,0]

In this case the capacity of our slice is 7, which means if we were to add a value to our slice, the underlying array could still be used. We would just need to update our slice’s len field.

// If we ran this code:
numbers = append(numbers, 8)

// Our slice would then have the following fields:
array: *pointer to X below*
len: 6
cap: 7

// X is an array, not a slice. It has a fixed length of 7
X = [5,4,3,2,1,8,0]

As long as we do not exceed the capacity of our slice, or do anything else to cause the underlying array pointer to change, both the slice read by the range keyword and the one we reference inside the for loop will point to the same underlying array, so those changes could potentially be read in the for loop.

Here is a somewhat confusing piece of code to help illustrate this further. It utilizes Go’s make keyword to create slices with a specific initial capacity.

sLen, sCap := 0, 3
slice := make([]int, sLen, sCap)
slice = append(slice, 12, 1, 4)

for i, num := range slice {
  fmt.Println("Current Number:", num)
  if num%2 == 0 {
    slice = append(slice, num/2)
    swap(i+1, len(slice)-1, slice)
  }
}
fmt.Println(slice)

// This is the swap function being used. It just swaps two values in a slice
func swap(i, j int, slice []int) {
	slice[i], slice[j] = slice[j], slice[i]
}

Run this on the Go Playground and try using different values for the slice capacity (sCap). Try 0, 3, 4, and 5. Notice how different capacity values can alter the results of the code. This occurs because different capacities will affect when our append changes the array that our slice points to.

When we call append and the underlying array has enough space for the new values the underlying array won’t change. This occurs when the number of items being appended to a slice is less than the slice capacity minus the slice length.

slice := make([]int, 0, 5)
x := []int{1,2,3}

// The underlying array changes if we add more values than can fit in it.
if len(x) > (cap(slice) - len(slice)) {
  fmt.Println("underlying array will change")
}

// When doing this
x := []int{1,2,3}
append(slice, x...)

Tips for iterating over slices

To avoid issues like this, I tend to alter the type of loop I am using depending on what I need to do. If I plan to change the slice as I iterate over it, I’ll typically prefer to use a traditional for loop with only an index, and I’ll manually check the length of the slice.

for i := 0; i < len(slice); i++ { ... }

Using this approach, we can rewrite the confusing code above to always have same output regardless of the capacity of the slice. We can also ensure that it runs on all of the numbers we add to the slice.

// sCap can be any value and we get the same results
sLen, sCap := 0, 0
slice := make([]int, sLen, sCap)
slice = append(slice, 12, 1, 4)

for i := 0; i < len(slice); i++ {
  num := slice[i]
  fmt.Println("Current Number:", num)
  if num%2 == 0 {
    slice = append(slice, num/2)
    swap(i+1, len(slice)-1, slice)
  }
}
fmt.Println(slice)

Run it on the Go Playground.

I’ll still use the range keyword, but I find it works best when we don’t alter the slice as we iterate over it. For example, we might write the following code to get the same result as the previous example, but without altering the slice we are iterating over.

sLen, sCap := 0, 0
slice := make([]int, sLen, sCap)
slice = append(slice, 12, 1, 4)

res := make([]int, 0, cap(slice))
for _, num := range slice {
  fmt.Println("Current Number:", num)
  res = append(res, num)
  for num%2 == 0 {
    num /= 2
    fmt.Println("Current Number:", num)
    res = append(res, num)
  }
}
fmt.Println(res)

Run it on the Go Playground.

I would also argue that this last version of the code is easier to understand when reading it. It is much clearer to me how many times the for loop might iterate, whereas adding values to a slice as we iterate over them might unintentionally lead to an infinite loop. The “downside” is that we need to create an additional slice to store values in. For most apps the performance difference will be negligible, but in a few special cases it could matter.

Join a Go community!

This entire discussion and deep-dive stemmed from someone joining a community and asking a question. The end result was a much deeper understanding of how slices work in Go for the person asking, and chances are several others were able to benefit from the discussion.

When learning something new, I highly recommend finding and joining a community of peers to learn with. It doesn’t need to be my Slack (though if you purchase one of my Go courses, I highly recommend joining to connect with others taking the course), but simply finding one where you can ask questions, help others, and grow is incredibly valuable. I learn new stuff all the time helping others out, and it is a great way to give back to the community regardless of whether you are a beginner or an expert.

If you need a place to start, here are a couple options:

• Gophers Slack - https://invite.slack.golangbridge.org/
• Gophers Discord - https://discord.com/invite/golang

You can also find some great communities that aren’t specific to Go. An example of this is the #100DaysOfCode Slack. There you can find others who are taking the challenge of coding every day for 100 days, and this might be a better fit depending on your goals.