---
title: PromQL Parser Improvements
type: Proposal
menu: proposals
status: Implementation in progress
owner: slrtbtfs
---

# Improving position and error handling in the PromQL parser

## Motivation

For the planned PromQL language server (see proposal) it is desirable to use the same parser as prometheus to ensure consistency and avoid code duplication.

The current Parser is not sufficient for that use case, though.

## Technical summary

The proposed changes are intended to improve the metadata generated by the parser and provide meaningful data for expressions that cannot be fully parsed.

All proposed concepts and data structures are already successfully employed by the go compiler. 

## Proposed Changes

### Implement consistent position handling within the PromQL Parser

#### Motivation

For the Purposes of a Language Server it is important to keep track of positions in Source Files and Queries. The PromQL Parser does not yet do that in a consistent way.

For example consider the following excerpts from the parser code:

For parser errors two values are used, one for line and one for column.

    // promql/parse.go:40
    // ParseErr wraps a parsing error with line and position context.
    // If the parsing input was a single line, line will be 0 and omitted
    // from the error string.
    type ParseErr struct {
        Line, Pos int
        Err       error
    }

Inside the lexer Positions are denoted by an integer, where the first char of the input string corresponds to 0.

    // promql/lex.go:315
    // Pos is the position in a string.
    type Pos int

The AST does not keep track of positions at all.

    // promql/ast.go:38
    type Node interface {
        // String representation of the node that returns the given node when parsed
        // as part of a valid query.
        String() s
    }

For comparison one might look at the go compiler, which uses the following type to handle positions:

    // go/token/position.go:76
    type Pos int

This behaves similar to string indices, with the difference, that it encodes much more information and can be decoded into a Position, e.g. for displaying error messages.

    // go/token/position.go:20
    type Position struct {
        Filename string // filename, if any
        Offset   int    // offset, starting at 0
        Line     int    // line number, starting at 1
        Column   int    // column number, starting at 1 (byte count)
    }

#### Proposed Change

It is proposed to use the aforementioned types as defined in the `go/token` package.

Since the `go/token` package contains a lot of code specific to the go compiler, it is proposed to copy the relevant parts, which are all in the same file, to the prometheus project.

The aforementioned types of the PromQL Parser are changed to the following.

    type ParseErr struct {
        Pos       Pos
        Err       error
    }

    type Node interface {	
        // Old interface: Only kept there for backwards compatibility
        String() s // String representation of the node, 
        // New interface. Same as used by the go compiler 
        Pos() Pos // position of first character belonging to the node
        End() Pos // position of first character immediately after the node
    }

#### Scope

The necessary changes are constrained to the files `promql/(ast|lex|parse).go`.

`ast.go`: All structs that implement the Node interface needs to be changed to the new interface.
`parse.go`: All code that creates Nodes has to use the new interface. The ParseErr struct needs to be changed as proposed and functions generating errors need to be fitted to use the new struct.
`lex.go`: The lexer struct as well as functions such as `next()` and `peek()` have to use the new Pos type.

Code outside the parser won’t be broken, since ParseErr still provides the old error interface and the old  Node interface is kept for compatibility as well.

### Use an Error List

#### Motivation

Currently the parser just panics as soon as an error is encountered. For the Language Server it is desirable to still get an AST in case of certain recoverable errors, such as incomplete expressions.

#### Proposed Change

It is proposed to add a field of type ErrorList  as defined in `go/scanner/Errors.go` to the parser struct in `promql/parse.go`.

If the parser encounters an error, it appends the error to the list. If the error is not recoverable, it additionally panics.

The  `ParseExpr()` procedure returns nil if the error list is empty after parsing, else it returns the `ErrorList`. 

#### Scope

The necessary changes are constrained to `promql/parse.go` . Since ErrorList implements the  error interface, there is no need to change code that calls the parser.

The major part of the work will be making some errors recoverable. After the proposed changes are implemented this can be done without breaking things.