# Promql Langserver > Development of new client editor --- # Source: https://github.com/prometheus-community/promql-langserver/blob/master/doc/developing_editor.md Development of new client editor ============================= This documentation will give you some tips to help when you are going to support the promql-server in a new editor. ## Config There is two different config used by the promql-server. ### Cold configuration (YAML or by environment) The first one is a classic yaml file which can be used to customize the server when it is starting. It's specified by adding the `--config-file` command line flag when starting the language server. It has the following structure: ```yaml activate_rpc_log: false # It's a boolean in order to activate or deactivate the rpc log. It's deactivated by default and mainly useful for debugging the language server, by inspecting the communication with the language client. log_format: "text" # The format of the log printed. Possible value: json, text. Default value: "text" prometheus_url: "http://localhost:9090" # the HTTP URL of the prometheus server. rest_api_port: 8080 # When set, the server will be started as an HTTP server that provides a REST API instead of the language server protocol. Default value: 0 metadata_lookback_interval: 2d # Interval used to retrieve data such as label and metrics from prometheus. Default value: 12h ``` In case the file is not provided, it will read the configuration from the environment variables with the following structure: ```bash export LANGSERVER_ACTIVATERPCLOG="true" export LANGSERVER_PROMETHEUSURL="http://localhost:9090" export LANGSERVER_RESTAPIPORT="8080" export LANGSERVER_LOGFORMAT="json" export LANGSERVER_METADATALOOKBACKINTERVAL="1w" ``` Note: documentation and default value are the same for both configuration (yaml and environment) ### JSON configuration (hot configuration) There is a second configuration which is used only at the runtime and can be sent by the language client over the `DidChangeConfiguration` API. It's used to sync configuration from the text editor to the language server. It has the following structure: ```json { "promql": { "url": "http://localhost:9090", # the HTTP URL of the prometheus server. "metadataLookbackInterval": "2h" } } ``` Using this way of changing configuration requires both providing those config options in the Text editor and sending them to the language server whenever they are changed. --- # Source: https://github.com/prometheus-community/promql-langserver/blob/master/doc/progress.md # Progress ## 11/10/2019 * implemented basic text synchronization * PR for position data in promql is ready: https://github.com/prometheus/prometheus/pull/6061 ### TODO * return hover information; expected monday evening * consider adding child to implementation in languiage server * documeting funcs: use a map from func name to struct * for e2e tests: invest 1-2hrs researching gopls and check back ## 14/10/2019 * passing messages to language client but VSCode is ignoring them * found ~5-line Vim script to connect language client to language server * will test if Vim language client works ## 21/10/2019 * identified issue preventing requests from being sent successfully: request context was cancelled prematurely * can now successfully send error messages to client and display them in VSCode * will use go-bindata or statik to compile documentation strings into the binary * have working example of `label_replace` hover text * VSCode's monaco is >2mb minified, may need to find alternative language client implementation for Prometheus frontend ### TODO * finish function documentation hover in the next few days * implement lite parser ~1 week * minimum e2e tests afterwards --- --- title: Replacing the PromQL Parser with a generated parser type: Proposal menu: proposals status: published owner: slrtbtfs --- # Replacing the PromQL Parser with a Generated Parser ## Motivation ### Current State of the Parser Currently, the PromQL query engine utilizes a handwritten, recursive-descent parser, that was written five years ago and hasn't seen large changes since then. It seems that none of the active members of the Prometheus project has a deep understanding of the parser code and is willing to maintain it or review complex PRs on it (based on the discussion in ). There also seem to be some hidden features and edge cases that nobody is aware of (see and ). The code seems to be far more complex than necessary and would require significant refactoring before continuing to develop new features on it. ### Using the Parser for the Upcoming Language Server The upcoming PromQL language Server (see [proposal](https://github.com/slrtbtfs/promql-lsp/blob/master/doc/proposals/2019_promql-language-server.md)) requires a parser in order to analyze PromQL queries. For purposes of compatibility, maintainability, and code reuse, it is intended for the language server to use the same PromQL parse as Prometheus. To enable that use case, some enhancements to the existing parser have been [proposed](https://github.com/slrtbtfs/promql-lsp/blob/master/doc/proposals/2019_promql-parser-improvements.md). These mainly concern the _output_ of the parser and have already been partly implemented. However, some of the features described there, such as error recovery, would be difficult to implement in the existing parser. #### Producing a Deeper Syntax Tree Consider the following PromQL query: metric{label=="value", foo~="ba."}[5m] offset 3h The corresponding syntax tree produced by the current parser looks something like the following: |---- MatrixSelector: metric{label=="value", foo~="ba."}[5m] offset 3h This syntax tree is sufficient if the only goal is evaluating PromQL queries. In fact, the PromQL query engine expects this simplified format. However, it is insufficient for more involved use cases, such as: auto-completion that works almost anywhere in the code, generating helpful error messages, showing function signatures and documentation while editing queries, etc. Instead, a syntax tree structured as follows would be more appropriate: |---- OffsetModifier: metric{label=="value", foo~="ba."}[5m] offset 3h . . . |---- MatrixSelector: metric{label=="value", foo~="ba."}[5m] . . . . . . |---- VectorSelector: metric{label=="value", foo~="ba."} . . . . . . . . . |---- LabelMatcher: label=="value" . . . . . . . . . . . . |---- Label: label . . . . . . . . . . . . |---- Value: "value" . . . . . . . . . |---- LabelMatcher: foo~="ba." . . . . . . . . . . . . |---- Label: foo . . . . . . . . . . . . |---- Value: "ba." ### Advantages of a Generated Parser #### It is easier to maintain The complex task of recognizing the PromQL syntax is performed by the generated code. Humans only have to deal with a comparably readable formal grammar and the code that builds up a syntax tree from already well structured input. #### It explicitly documents the formal grammar of PromQL A formal grammar provides a human- and machine-readable definition of what is a correct PromQL query. This would eliminate edge cases and unknown features such as the ones described above. #### It is possible to handle incomplete expressions Consider the following incomplete PromQL input from a user: some_metric{la By adding syntax rules for incomplete expressions, the parser could figure out that it got an incomplete vector selector with a certain structure. A language server could then use this information to offer completions for all labels that occur together with `some_metric` and start with `la`. #### It is easier to provide helpful error messages All that needs to be done in order to add new error messages, is to add a grammar rule that recognizes this specific error class and add some code that handles this error when building up the syntax tree. ## Proposal This document proposes replacing the current PromQL parser with one that is generated by [goyacc](https://godoc.org/golang.org/x/tools/cmd/goyacc). Goyacc was chosen because yacc-generated parsers have been successfully used for decades. Further, of the few alternatives for generating go parsers, none seem to be on par with goyacc. The new parser should able to output two syntax tree formats: * a deeper tree, where almost every token is assigned a node, designed to be used by applications that require additional context, such as the language server; and * a simplified tree that looks almost like the current one, intended to be used by the PromQL query engine and any other application that does not need the additional context. The parser should able to parse incomplete expressions. The changes proposed in a [previous proposal](https://github.com/slrtbtfs/promql-lsp/blob/master/doc/proposals/2019_promql-language-server.md) remain valid. The changes to the syntax tree and lexer that have already been implemented can be still used with the new parser. The remaining changes, i.e. incomplete expression parsing and having an error list, will be implemented along with the newly proposed parser. ## Architecture ### lexer Yacc relies on receiving a stream of tokens from a lexer. The new parser will continue to use the logic of the existing lexer. The lexer output will be modified to satisfy the interfaces required by goyacc. To be consistent with the usual yacc conventions, token names will be capitalized. ### yacc grammar The core of the parser is a formal grammar of PromQL, which is the input for the parser generator and provides a precise and explicit definition of the PromQL syntax. An extra rule must be declared for every class of error and incomplete expressions that the parser should handle. The grammar is able to recognize all valid PromQL queries as well as common cases of incomplete or otherwise incorrect queries. As an example, the part of the grammar recognizing an instant vector selector will look roughly like: selector: IDENTIFIER {$$ = newVectorSelector(nil, $1} | IDENTIFIER LBRACE RBRACE {$$ = newVectorSelector(nil, $1, $2, nil, $3)} | IDENTIFIER LBRACE matchers RBRACE {$$ = newVectorSelector(nil, nil, $1, $2, $3} | LBRACE RBRACE {$$ = newVectorSelector(nil, nil, $1, nil, $2)} | LBRACE matchers RBRACE {$$ = newVectorSelector(nil, $1, $2, $3, $4)} | IDENTIFIER LBRACE matchers anything {$$ = newVectorSelector([]ErrCodes{ ErrNoClosingBrace, }, $1, $2, $3, $4)} | LBRACE matchers anything {$$ = newVectorSelector([]ErrCodes{ ErrNoClosingBrace, }, nil, $1, $2, $3)} matchers: matchers COMMA matcher {$$ = newMatchers(nil, $1, $2, $3)} | matcher {$$ = newMatchers(nil, $1)} matcher: IDENTIFIER matchOp STRING {$$ = newMatcher(nil, $1, $2, $3)} | IDENTIFIER matchOp anything {$$ = newMatcher([]ErrCodes{ ErrWrongType, }, $1, $2, $3)} | IDENTIFIER anything {$$ = newMatcher([]ErrCodes{ ErrNoMatchOp, }, $1, $2)} matchOp: EQ {$$ = $1} | NEQ {$$ = $1} | REGEQ {$$ = $1} | REGNEQ {$$ = $1} anything: // All valid PromQL Syntax Elements ... | EOF {$$ = $1} | error {$$ = $1} ### Building up the Abstract Syntax Tree (AST) Every match in the grammar calls a function of the following type or simply returns the token if no special action is required: func ([]ErrCodes, ...*Expr) *Expr Tokens implement the Expr interface, so no special distinction has to be made here. These functions gradually build up a syntax tree and perform all error and type checking along the way, using the hints provided by the given `ErrCodes` ### The Expr Interface All Syntax tree nodes must satisfy the `Expr` interface and implement at least the following functions: type Expr interface { // A string representation String() string // Position Pos() token.Pos EndPos() token.Pos // Errors in the respective subtree Errors() []*ParseErr // The Type of the Expr Type() ExprType // Simplifies the subtree to allow fast Evaluation by the PromQL engine Simplify() *Expr // Reports if the subtree is already simplified. If yes, the subtree is // skipped by Simplify() for performance reasons Simplified() bool } ### AST Simplifying and Backwards Compatibility The syntax tree produced by the new parser will provide an accurate and detailed representation of the query. However, when evaluating queries in the PromQL query engine, it is better to have a less complex syntax tree. In fact, the PromQL query engine expects matrix and vector selectors to be represented as a single node rather and as a complex tree. In order to avoid completely breaking the query engine and any other existing users of the current PromQL parser, the `Expr` interface will have a function `Simplify()` that recurses through the subtree, tries to reduce the depth of the tree and changes matrix and vector selector to the format that the query engine currently expects. To produce this simplified, backwards-compatible syntax tree format, it suffices to call `Simplify()` on the root node. It is possible to implement additional optimizations like pre-evaluation of scalar expressions in the `Simplify()` method. ### Scope of the change The parser will be completely rewritten. The output format and token naming conventions in the lexer will be changed to fit the conventions and interfaces of yacc. Most of the actual logic will remain unchanged. The changes to the query engine will be minimized by the AST Simplification mechanism. The changes will be confined to some name changes in types and interfaces. ### Possible Regressions By Rice's theorem, it may be impossible be certain whether the new parser accepts exactly the same syntax as the old one. Less academically, it is possible that there exist edge cases in the current parser that the new parser does not handle because no one knows about them. It is likely impossible to completely mitigate the risk of accidentally introducing a breaking incompatibility of this type. This is a consequence of the fact that no formal definition of PromQL exists yet. In order to reduce this risk, a large sampling of queries that work with the current parser should be collected and used to test the new parser. --- published here: --- --- title: PromQL Language Server type: Proposal menu: proposals status: WIP owner: slrtbtfs --- # PromQL Language Server ## Summary This document describes the motivation and design of the PromQL language server, as well as how it can be integrated into existing development environments. ## Motivation Modern IDEs and editors (tools) provide a lot of language specific features like syntax highlighting, autocompletion or hover information. Traditionally for each combination of tool and language an own integration had to be written which leads a lot of redundant work, inconsistency and especially disadvantages tools without large communities. To improve that situation Microsoft introduced the [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/) which has since been adopted by a large set of [languages](https://microsoft.github.io/language-server-protocol/implementors/servers/) and [tools](https://microsoft.github.io/language-server-protocol/implementors/tools/) and is [endorsed by Red Hat](https://developers.redhat.com/blog/2016/06/27/a-common-interface-for-building-developer-tools/). LSP is a standardized Protocol for communication between a tool (client) and a Language Server that implements all language specific logic. As a result implementing a server for a language makes it fairly easy to integrate it with most tool and vice versa. PromQL does not have such a language Server yet. Some tools, like the Prometheus expression browser do have limited support for autocompletion tough. ## Technical summary of PromQL language server To improve the general user experience of Prometheus a PromQL language server is proposed. The PromQL language Server can either be included into IDE/editor plugins or run on a server and communicate with tools like the Prometheus Web UI over a network. PromQL Queries are supported both standalone and as part of a .yaml configuration file. To provide autocompletion for labels it can optionally connect to an Prometheus Server and use it's label data. For testing purposes some IDE plugins will be developed. These will include a [TextMate Grammar](https://macromates.com/manual/en/language_grammars) to enable syntax highlighting which is not supported by the LSP itself. ## Architecture ### Implemented Server Capabilities The [LSP Specification](https://microsoft.github.io/language-server-protocol/specification) describes a rich set of server capabilities, not all of whom are useful for PromQL. It is possible for a server to only implement a subset and advertise the supported capabilities at initialization. For capabilities marked with _maybe_ it is not yet clear, wether a use case exist. An exclamation mark denotes that these capabilities are prioritized. The following capabilities will be implemented by the PromQL Language Server: #### General Capabilities: _all!_ Necessary to establish and end communication with a client. #### Window Capabilities: _all_ Enable the server to send notifications and log messages to the client. #### Telemetry Capabilities: _maybe_ Enable the server to send telemetry events to the client. Might be useful once there are published IDE integrations. #### Client Capabilities: _all!_ Enable a client to advertise it's capabilities to the server. #### Workspace Capabilities: _maybe some_ For PromQL the concept of workspaces is not relevant. Only implemented if required by another capability. #### Text Synchronization Capabilities: _all!_ Notify the server about File Changes. Mandatory to be able to inspect the content of unsaved files. #### Diagnostics Capabilities: _all!_ Send Errors and Warnings to the Client. These are used to show syntax errors and linting. The errors shown are exactly those, that would be returned by prometheus. Warnings may be sent about common errors such as `rate(sum(...))` or `http_requests_total{status="^5..$"}`. #### Language Capabilities: _some!_ The core part of the language server. Some of these, e.g. Go to (definition|typeDefinition|declaration|implementation), renaming and folding are not useful for PromQL itself. Implemented are: ##### completion!, completion_resolve! Give completions for functions, operators and, if a Prometheus server is attached, labels. ##### hover! Show documentation for functions and operators. ##### signatureHelp Show the type of expressions (`string`, `scalar`, `Instant vector`, `Range vector` and functions combining these). ##### codeAction: _maybe_ Enable QuickFixes. In case this is implemented, also some of the Workspace Capabilities would be required. ##### formatting, rangeFormatting, onTypeFormatting: _maybe_ There aren't that much formatting changes that could be done, other than ensuring there is a sane amount of white space. ### JSON-RPC The LSP Protocol is based on the [JSON-RPC 2.0 Protocol](https://www.jsonrpc.org/specification). The transport layer is not specified by the protocol. For IDEs it's usually stdin/stdout, over a Network it could be HTTP. Thus the JSON-RPC implementation used by PromQL should abstract over the transport layer. [The implementation used by from gopls](https://github.com/golang/tools/tree/master/internal/jsonrpc2) satisfies these criteria. ### Protocol Implementation The gopls Language Server for golang already implements most parts of the LSP protocol. This project is intended to reuse large parts of gopls code. ### Parsing It is intended to use the upstream PromQL parser. Some changes need to be made there to support  this use case. These are described in a separate proposal. ### Autocompletion The proposed server has the ability to optionally connect to a prometheus server. In this case the labels that are present on the prometheus can be used for autocompletion. Querying this label data can be achieved by using the HTTP API provided by prometheus. ## Open Questions * How to handle the case when promql queries are embedded in yaml files. It is desirable that the PromQL language server and the YAML Language Server can be used at the same time in some way. Possible solutions: * Connecting both servers to the tool. In this case the PromQL server has to parse some yaml. * The yaml server calls the PromQL server once a query is detected. --- --- title: PromQL Parser Improvements type: Proposal menu: proposals status: Implementation in progress owner: slrtbtfs --- # Improving position and error handling in the PromQL parser ## Motivation For the planned PromQL language server (see proposal) it is desirable to use the same parser as prometheus to ensure consistency and avoid code duplication. The current Parser is not sufficient for that use case, though. ## Technical summary The proposed changes are intended to improve the metadata generated by the parser and provide meaningful data for expressions that cannot be fully parsed. All proposed concepts and data structures are already successfully employed by the go compiler. ## Proposed Changes ### Implement consistent position handling within the PromQL Parser #### Motivation For the Purposes of a Language Server it is important to keep track of positions in Source Files and Queries. The PromQL Parser does not yet do that in a consistent way. For example consider the following excerpts from the parser code: For parser errors two values are used, one for line and one for column. // promql/parse.go:40 // ParseErr wraps a parsing error with line and position context. // If the parsing input was a single line, line will be 0 and omitted // from the error string. type ParseErr struct { Line, Pos int Err error } Inside the lexer Positions are denoted by an integer, where the first char of the input string corresponds to 0. // promql/lex.go:315 // Pos is the position in a string. type Pos int The AST does not keep track of positions at all. // promql/ast.go:38 type Node interface { // String representation of the node that returns the given node when parsed // as part of a valid query. String() s } For comparison one might look at the go compiler, which uses the following type to handle positions: // go/token/position.go:76 type Pos int This behaves similar to string indices, with the difference, that it encodes much more information and can be decoded into a Position, e.g. for displaying error messages. // go/token/position.go:20 type Position struct { Filename string // filename, if any Offset int // offset, starting at 0 Line int // line number, starting at 1 Column int // column number, starting at 1 (byte count) } #### Proposed Change It is proposed to use the aforementioned types as defined in the `go/token` package. Since the `go/token` package contains a lot of code specific to the go compiler, it is proposed to copy the relevant parts, which are all in the same file, to the prometheus project. The aforementioned types of the PromQL Parser are changed to the following. type ParseErr struct { Pos Pos Err error } type Node interface { // Old interface: Only kept there for backwards compatibility String() s // String representation of the node, // New interface. Same as used by the go compiler Pos() Pos // position of first character belonging to the node End() Pos // position of first character immediately after the node } #### Scope The necessary changes are constrained to the files `promql/(ast|lex|parse).go`. `ast.go`: All structs that implement the Node interface needs to be changed to the new interface. `parse.go`: All code that creates Nodes has to use the new interface. The ParseErr struct needs to be changed as proposed and functions generating errors need to be fitted to use the new struct. `lex.go`: The lexer struct as well as functions such as `next()` and `peek()` have to use the new Pos type. Code outside the parser won’t be broken, since ParseErr still provides the old error interface and the old Node interface is kept for compatibility as well. ### Use an Error List #### Motivation Currently the parser just panics as soon as an error is encountered. For the Language Server it is desirable to still get an AST in case of certain recoverable errors, such as incomplete expressions. #### Proposed Change It is proposed to add a field of type ErrorList as defined in `go/scanner/Errors.go` to the parser struct in `promql/parse.go`. If the parser encounters an error, it appends the error to the list. If the error is not recoverable, it additionally panics. The `ParseExpr()` procedure returns nil if the error list is empty after parsing, else it returns the `ErrorList`. #### Scope The necessary changes are constrained to `promql/parse.go` . Since ErrorList implements the error interface, there is no need to change code that calls the parser. The major part of the work will be making some errors recoverable. After the proposed changes are implemented this can be done without breaking things. --- # About [LSP](https://microsoft.github.io/language-server-protocol/) From their homepage: > The Language Server Protocol (LSP) defines the protocol used between an editor or IDE and a language server that provides language features like auto complete, go to definition, find all references etc. * Standardized protocol supported by a lot of IDEs and languages * Originally created for VS Code. * Red Hat announced adoption in 2016 * Creating an LSP server for PromQL would add these features to most development environments. # What can LSP do? Many language insight features are not really useful for a query language. ## Not useful for PromQL * Go to (definition|typeDefinition|declaration|implementation) * renaming * folding * Workspaces ## Useful for PromQL * Diagnostics (e.g. linting and syntax errors) * Auto completion * Hover information ## Maybe useful for PromQL * Find references (e.g. of a specific label) * Document Formatting Requests (e.g. beautifying) ## Some concrete examples * Complain about syntax errors before the request has been submitted * Offer autocompletition for labels and values * Complain if a request does not match anything * Warn about request such as `http_requests_total{status="^4..$"}` or `rate(sum(...))` * Show documentation of functions on hover --- # Relevant Projects [LSP itself](https://microsoft.github.io/language-server-protocol/) ## LSP implementations written in go * [gopls](https://github.com/golang/tools/tree/master/internal/lsp), for go, Written by the Go Team itself. * [sourcegraph-go](https://github.com/sourcegraph/go-langserver), for go, _deprecated in favor of gopls_ * [terraform-lsp](https://github.com/juliosueiras/terraform-lsp), for terraform, based on sourcegraph-go, still experimental ## LSP implentations for query languages * [graphql-language-service-server](graphql-language-service-server), written in js. * [gql-language-server](https://github.com/Mayank1791989/gql-language-server), also for graphql and written in js, less actively developed. * [vscode-mysql](https://github.com/Microsoft/vscode-mssql/tree/dev/src/languageservice), for SQL, written in typescript * [sparql-language-server](https://github.com/stardog-union/stardog-language-servers/tree/master/packages/sparql-language-server), written in typescript ## SDKs No golang SDKs exist so far.