Exactly a year ago (yesterday), Russ Cox submitted a proposal to add two new range capabilities to Go: range-over-int, and range-over-func. The former was added to Go 1.22. And the latter left me immediately skeptical. I spent a fair amount of time (as measured in Internet minutes) trying to understand the proposal, and eventually coming to the conslusion that I wasn’t very smart.

Assumptions

In the year since then, my co-host, Shay Nehmad, and I have talked about the range-over-func progress several times on the Cup o’ Go podcast, sharing some resources from the community, as well as our own speculation. And that’s probably a good place to start this post: My own speculations about this feature.

• Ugly interface names First, the interface names annoyed me. If you’re patient enough to actually read through the issue, you’ll come to this comment, which has this tidbit burried deep inside:

  package iter
  
  type Seq0 func(yield func() bool) bool
  type Seq[V any] func(yield func(V) bool) bool
  type Seq2[K, V any] func(yield func(K, V) bool) bool
  

  Yuck! One thing Shay and I could agree on was how terrible these names were. It reminded me of Perl’s confusingly named open2 and open3, which always required looking up the documentation to either write or read the code that used them. I was afraid these silly names would lead to the same problem when writing Go iterators.

• Steep learning curve Second, considering the effort it took to understand the range-over-func proposal, it seemed that this feature would prove to be cumbersome to use when writing an iterator. But the consensus seemed to be that it would be worth the effort, becasue it would be easier to use the iterators. That seemed like a reasonable enough trade off. I’d be willing to do the heavy lifting to make life easier on the consumers of my APIs. At least in some cases.

• New keyword Closely related to the general learning curve, the new yield keyword^* was a bit confusing and magical to me. But I’m a big boy. If I can learn to knit and crochet (and yes, I can do both), surely I can learn to use yield in Go!

  ^*Spoiler alert: not a keyword, after all.

• Backward compatibility Putting new iterators behind //go:build v1.23 tags could be annoying, and render different effective versions of a library, depending on which version of Go the consumer is using.

So that was my starting point. These were my key beliefs and assumptions about range-over-func as of 24 hours ago.

My Experience

Then I had a few hours free yesterday afternoon, and I decided it would be a good opportunity to play around with range-over-func, and learn what the fuss is really all about.

I’m the author and maintainer of an open source library that’s the perfect candidate for this feature: Kivik. TL;DR; it provides a database/sql-style interface for CouchDB. And as one might expect, when reading data from a database, you often want to iterate over that data. And Kivik has four distinct iterators, each of which uses a for thing.Next() { /* ... */ } approach.

So I upgraded to Go 1.23rc1, opened up my IDE, and a blog post by Zach Musgrave, Go range iterators demystified, and started hacking.

Now, being who I am, I jumped straight to the code examples, breezing past most of the explanations in that blog post. That’s where I found code that looks like this:

func (s Slice) ErrorIter() func(yield func(i int, e error) bool) {
	return func(yield func(i int, e error) bool) {
		for _, i := range s {
            // If there's an error getting the next element,
            // pass it into the yield function as the second parameter
			if !yield(i, nil) {
				return
			}
		}
	}
}

And I promptly adapted it to my use case in Kivik:

func (r *ResultSet) Iterator() func(yield func(*driver.Row, error) bool) {
	return func(yield func(*Row, error) {
		for r.Next() {
			row := r.iter.curVal.(*driver.Row)
			if !yield(row, row.Err) {
				break
			}
		}
	})
}

Wow. That seemed easy. Was that all there was to it? (Note: This was not the final version, and indeed isn’t even complete, in ways I didn’t realize at the time. Don’t copy this as an example of a good or valid iterator!)

Of course, I need to try using this code. Not only to see if it works, but to see if it’s ergonomic. So I wrote a little test program that I could demonstrate works, using the old iterator style. Key part was:

for rows.Next() {
	id, err := rows.ID()
	if err != nil {
		panic(err)
	}
	fmt.Println(id)
}
if err := rows.Err(); err != nil {
	panic(err)
}

Then I re-wrote it to use the new iterator I had just created:

for doc, err := range rows.Iterator() {
	if err != nil {
			panic(err)
	}
	id, err := doc.ID()
	if err != nil{
		panic(err)
	}
	fmt.Println(id)
}

And it worked! Well, mostly. I did a few tweaks that have been lost to my memory. But it only took minor tweaking.

I found that to be surprisingly easy. My proof-of-concept was ready in under an hour. Including the time it took to upgrade to Go 1.23. So far, my assumptions that learning to write range-over-func iterators would be difficult, was proving untrue.

So I created a quick PR against my open-source project. I was expecting that all the old versions of Go would start failing to compile. They didn’t. WAT! So there’s another assumption of mine, that backward compatibility would be a pain, disproven.

So I went back to my IDE, and started writing some tests. Yes, I didn’t write my tests first. I don’t see any problem with doing random experiments like this, to learn a new language or feature, without writing tests first.

But once I’m ready to switch to writing “real” code, I usually write my tests first.

So I took an existing test case, to test the old style iterator, and modified it to work with the new style iterator.

func TestResultSetIterator(t *testing.T) {
	want := []string{"a", "b", "c"}
	var idx int

	r := newResultSet(context.Background(), nil, &mock.Rows{
		NextFunc: func(r *driver.Row) error {
			if idx >= len(want) {
				return driver.EOQ
			}
			r.ID = want[idx]
			idx++
			return nil
		},
	})

	ids := []string{}

	for row, err := range r.Iterator() {
		if err != nil {
			t.Fatalf("Unexpected error: %s", err)
		}
		id, err := row.ID()
		if err != nil {
			t.Fatalf("Unexpected error: %s", err)
		}
		ids = append(ids, id)
	}
	if d := cmp.Diff(want, ids); d != "" {
		t.Errorf("Unexpected IDs: %s", d)
	}
}

I wrote similar tests for the three other iterators in my code base, then wrote iterators to match them. But when I pushed this PR to GitHub, these tests started failing in all older versions of Go. It turns out that you can write an iterator function in any version of Go, and it will compile just fine. But of course the range keyword doesn’t know what to do with such a function until Go 1.23 (or 1.22 with the rangefunc experiment enabled). So I moved all my tests to separate files, and added some build tags:

//go:build go1.23
// +build go1.23

Now my tests were passing again.

But I wasn’t done.

I had two classes of use cases to test:

• Would my iterators properly return error values?
• Would my iterators work properly when the loop was breaked (is that the proper way to conjugate a language keyword??)?

So I added some more tests:

func TestResultSetIteratorError(t *testing.T) {
	r := newResultSet(context.Background(), nil, &mock.Rows{
		NextFunc: func(*driver.Row) error {
			return errors.New("failure")
		},
	})

	for _, err := range r.Iterator() {
		if err == nil {
			t.Fatal("expected error")
		}
		return
	}
	t.Fatal("Expected an error during iteration")
}

func TestResultSetIteratorBreak(t *testing.T) {
	r := newResultSet(context.Background(), nil, &mock.Rows{
		NextFunc: func(*driver.Row) error {
			return nil
		},
	})

	for _, err := range r.Iterator() {
		if err != nil {
			t.Fatalf("Unexpected error: %s", err)
		}
		break
	}
	if r.iter.state != stateClosed {
		t.Errorf("Expected iterator to be closed")
	}
}

Two test scenarios, across 4 different iterators meant 8 new tests. And I think 6 of them failed initially. So I fixed the failures, and pushed my final PR.

First Impressions

So what do I think of range-over-func now, less than 24 hours after trying it for the first time?

In a phrase: I love it!

First, let me revisit my a priori assumptions:

• Ugly interface names. Eh who cares. I never once had to write or read those names. It seems those names exist solely as documentation. So whatever. Call them whatever you want from now on. Name them in Latin if you will. You have my blessing! I’m sure the Go team is relieved. (Note: It was pointed out to me on reddit when these names are actually useful. Please consider my Latin suggestion with the intended sarcasm.)

• Steep learning curve Well, I don’t know about you, but if I can go from zero to working, production-ready code, in just a couple of hours, I don’t think the learning curve is all that steep after all.

• New keyword This yield keyword is what I was sure would break old versions of Go. But when it didn’t, it forced me to look more closely. It’s not a keyword at all! LOL. It’s just the conventional name for a callback function passed into the iterator function.

• Backward compatibility Almost a non-issue. Just as you could add a Unwrap() []error method to an error prior to Go 1.20, or Unwrap() error prior to Go 1.13 and it would compile, it just wouldn’t be used, you can create iterator functions in any version of Go. They just won’t be used until Go 1.23. The only slight annoyance is that code that uses range-over-func (such as tests in my case) must be behind a build flag, if you need to support older versions of Go.

So all in all, a very positive experience.

Every major sticking point I had with regard to the use of range-over-func turned out to be overblown, or non-existent.

And after implementing the code, my initial thoughts are that I really like the code. I only wish I could retire the old iterator implementation. I think it would make the internals of my library simpler, as well as the use of it. But until I drop support for Go 1.22 and older (probably at least 2 years down the road, maybe with Kivik 6.x), that’s not an option.

YMMV

My intention with this post is to describe my experience and conclusions with the new range-over-func feature. No doubt my experience is not the same as everyone’s. I’ve already spoken to a few people who have had different experiences and conclusions than I have. Feel free to let me know your experience.

If you’re looking for a deeper explanation of how range-over-func works, or even a tutorial, let me point you once again to Zach Musgrave’s post.