Blog / Go Language Spec

Lexical elements: Tokens

January 10, 2023

Tokens

Tokens form the vocabulary of the Go language. There are four classes: identifiers, keywords, operators and punctuation, and literals. White space, formed from spaces (U+0020), horizontal tabs (U+0009), carriage returns (U+000D), and newlines (U+000A), is ignored except as it separates tokens that would otherwise combine into a single token. Also, a newline or end of file may trigger the insertion of a semicolon. While breaking the input into tokens, the next token is the longest sequence of characters that form a valid token.

In other words, after removing comments (and replacing them with either spaces or newlines, as discussed yesterday), we can treat a source file as being passed through a split function that splits any number of on space, tab, carriage return, and newline characters (or /[ \t\r\n]+/ in RegExp speak). This should seem pretty intuitive, as most modern langauges work in a similar way.

It’s not quite this simple, because of the way Go handles implicit semicolons, though, which we’ll be discussing tomorrow.

Finally, the last sentence is also intuitive, but important: “the next token is the longest sequence of characters that form a valid token.”

This just means that forward won’t be interpreted as for followed by ward during tokenization.

Quotes from The Go Programming Language Specification, Version of June 29, 2022

Share this

Direct to your inbox, daily. I respect your privacy .

Unsure? Browse the archive .

Related Content

Assignment statements

Assignment statements An assignment replaces the current value stored in a variable with a new value specified by an expression. Assignments should be familiar to you. They’re pretty fundamental to almost all programming languages. But there are a few aspects to Go’s assignments that are worth calling out. … An assignment statement may assign a single value to a single variable, or multiple values to a matching number of variables.

IncDec statements

IncDec statements The “++” and “–” statements increment or decrement their operands by the untyped constant 1. As with an assignment, the operand must be addressable or a map index expression. IncDecStmt = Expression ( "++" | "--" ) . Most commonly you’ll see ++ and -- used on a simple variable. But as explained, it’s not strictly limited to that. These operators can be used on any addressable (numeric) expression.

Send statements

Send statements A send statement sends a value on a channel. The channel expression’s core type must be a channel, the channel direction must permit send operations, and the type of the value to be sent must be assignable to the channel’s element type. SendStmt = Channel "<-" Expression . Channel = Expression . Both the channel and the value expression are evaluated before communication begins. Communication blocks until the send can proceed.