Today I want to expand with a thought I touched on yesterday: Callbacks with iterators.
Yesterday’s context was error handling. But I’ve found callbacks with iterators to be very valuable in a slightly different case. To understand the problem callbacks have helped me solve, we need to move away from our grep example to something a bit more involved. Let’s imagine we’re querying a REST API, which returns paginated results. A typical response body might look something like this, in JSON:
{
"items": [
{
"id": 1234,
"description": "Foo",
...
},
{
"id": 2345,
"description": "Bar",
...
},
...
],
"page": {
"page_no": 2,
"total_items": 1346,
"items_per_page": 20,
"cursor": "Y3Vyc29yCg=="
}
}
Now you might just read the entire response body in at once, parse it, then return a slice of items, along with some pagination data:
func (c *client) Items(ctx context.Context, /* ... */) ([]*Item, *Page, error)
But if these items are particularly large–maybe they contain Base64-encoded images, or videos, for example. Or maybe we’re returning pages of 1,000 items at a time. Or for whatever reason, you want to parse them one at a time, as they come in over the network, rather than all at once. Now we want an iterator:
func (c *client) Items(ctx context.Context, /* ... */) iter.Seq2[*Item, error]
One problem solved nicely, but what about that pagination data? It likely cannot even be read off the network until the entire rest of the response has been read and parsed. So we need some way read and return that data to the user after processing has completed.
This is where I find a callback very useful:
func (c *client) Items(ctx context.Context, final func(*Page), /* ... */) iter.Seq2[*Item, error]
In this case, the Items method can call the final callback once iteration has completed successfully. And unlike the approach of adding a Result object with a Page method on it, there’s no possible way for the consumer to try to read the pagination data too early.
I used this technique in my Ticketmaster client library, which you’re welcome to look at for specifics. (Here I mixed this technique with functional options—a topic I should perhaps address another day.)
