What

  • Overview of how go/ast, go/token, go/types and compilation related packages work.
  • How is it different from reflect.

AST (go/ast)

The go/ast is the package for abstract syntax tree (AST). Notice the go/ prefix in the path as opposed to a reflection library reflect. The go/ directory contains Go language related packages.

The go/ast package contains the nodes types and utility functions. There are some notable interfaces.

interfaces in go/ast

AST is a tree and ast.Node interface is the fundamental unit of a node in the tree. AST is built from parsing textual code so the Node contains the starting position Pos() and the ending position End() which are of type token.Pos.

The go/ast package depends on go/token package. The token.Pos’s underlying type is int. This is a “source position within a file set” and a file set is best to be viewed as a literal set of files. Your Go program is essentially a collection of .go files and file set is the internal representation during parsing.

The token.Pos is not a direct mapping to line/column position of your Go file but is used to get the token.Position which contains the filename, offset, line and column used for reporting.

In a tree, there’s a root and it is ast.File. Do not conflate with token.File.

Other notable interfaces are:

The generated documentation isn’t very helpful explaining what these are but the source code neatly lists all available types that implement the interfaces. For example,

1
2
3
4

Spec interface {
    Node
    specNode()
}

1
2
3
4
5
6

// specNode() ensures that only
// spec nodes can be assigned to
// a Spec.
func (*ImportSpec) specNode() {}
func (*ValueSpec) specNode()  {}
func (*TypeSpec) specNode()   {}

Readers who seek deeper understanding of Go language should visit The Go Programming Language Specification.

Traversing the tree

Given a tree, one might want to traverse the tree to find a node, for example, to check that struct tags are in a certain format. The ast.Walk provides this capability. The package level functions already define common ways to filter and/or walk the tree. The notables ones are Preorder and PreorderStack.

Parsing

In order to work with AST, code must be parsed. There’s a go/parser package but there’s a built-in tool package at golang.org/x/tools/go/packages.

The packages.Load function returns packages.Package given the config and patterns. The patterns here are the same patterns used when building Go files.

Important option in Config used in Load is the LoadMode which controls what and how to load. Depending on the use-case, one may want to select only the ones needed. It’s important to note that commonly used mode is already defined for convenience as noted in the doc:

1
2
3
4
5

[LoadFiles]     lists of files in each package
[LoadImports]   ... plus imports
[LoadTypes]     ... plus type information
[LoadSyntax]    ... plus type-annotated syntax
[LoadAllSyntax] ... for all dependencies

The packages.Packages returned from the Load are perhaps the most useful but also initially difficult to work with. The fields in the structs are set depending on the LoadMode set within the Config.

It’s effectively a collection of all AST things and type things to look up from. Also do not conflate packages.Package with types.Packages.

The TypesInfo is perhaps the most interesting field that provides mapping between the syntax tree (go/ast) and the types (go/types).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// comments are stripped for brevity
type Info struct {
	Types map[ast.Expr]TypeAndValue
	Instances map[*ast.Ident]Instance
	Defs map[*ast.Ident]Object
	Uses map[*ast.Ident]Object
	Implicits map[ast.Node]Object
	Selections map[*ast.SelectorExpr]*Selection
	Scopes map[ast.Node]*Scope
	InitOrder []*Initializer
	FileVersions map[*ast.File]string
}

Go Types (go/types)

The go/types package contains all things to resolve and validate types in Go. The types.Info, which was mentioned just above, contains all the mapping between go/ast and go/types. For example, given an AST selector expression myStruct.myField somewhere in the code, the Selections field can be used to look up the actual type (types.Selection) of the expression which contains the receiver, the type, etc.

For example, using the same example myStruct.myField, lets assume we have a reference to *ast.SelectorExpr. We can use the Info.Selections map to get the *types.Selection instance. It’s Kind() will be types.FieldVal, Recv() will be the struct type of myStruct and Type() will be the type of myField variable, for example.

An important interface in this package is an Object which is “a named language entity” which has reference back to go/ast types. The definition of this isn’t clear but documentation does mention what concrete types implement the interface.

Reflection

Reflection is a mechanism to inspect the type during runtime using reflect package. Some semantics are similar to that of go/types package but their capabilities are different. Reflection is all about runtime. Types defined in reflect and go/types are not compatible as they are two distinct unrelated packages.

For example, given the following function:

1
2
3

func Save(d any) error{
    // save d
}

Static analysis only sees that d is assignable to any type.

On the contrary, reflection loses all information about the code stripped after compilation such as filename or line of code. Comments and type arguments in generics are also stripped after compilation so they do not appear in reflection.

Final remarks

The writing lacks practical examples of how each is used and I hope to add them in the near future.