Parsing arbitrary tokens

Level: 301 · deep dive

One line: Between a function-like macro's delimiters the syntax is yours: write a type per rule and impl Parse for it, take tokens with input.parse(), choose between alternatives with lookahead1 (which also writes the error), read separated lists with Punctuated, and name words and operators Rust does not have with custom_keyword! and custom_punctuation!.

A query language in a macro

filter! takes a condition written like SQL, = and <> and and included, and becomes a closure for Iterator::filter:

demo/filter_app/src/main.rs

use filter_macro::filter;

struct Order {
    id: u32,
    status: &'static str,
    total: u32, // in cents
}

const ORDERS: [Order; 5] = [
    Order { id: 1, status: "open", total: 1250 },
    Order { id: 2, status: "paid", total: 8000 },
    Order { id: 3, status: "cancelled", total: 9900 },
    Order { id: 4, status: "open", total: 5000 },
    Order { id: 5, status: "paid", total: 1999 },
];

fn main() {
    let ids = |matched: Vec<&Order>| matched.iter().map(|order| order.id).collect::<Vec<_>>();

    let open: Vec<&Order> = ORDERS.iter().filter(filter!(status = "open")).collect();
    let large: Vec<&Order> = ORDERS
        .iter()
        .filter(filter!(status <> "cancelled" and total >= 5000))
        .collect();
    let small: Vec<&Order> = ORDERS
        .iter()
        .filter(filter!(status in ["open", "paid"] and total < 2000))
        .collect();

    println!("{:<24}{:?}", "open", ids(open)); // open                    [1, 4]
    println!("{:<24}{:?}", "not cancelled, >= 5000", ids(large)); // not cancelled, >= 5000  [2, 4]
    println!("{:<24}{:?}", "open or paid, < 2000", ids(small)); // open or paid, < 2000    [1, 5]
}

Verified output of cargo run -q -p filter_app — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

open                    [1, 4]
not cancelled, >= 5000  [2, 4]
open or paid, < 2000    [1, 5]

None of status <> "cancelled" and total >= 5000 is a Rust expression. It only has to be tokens that rustc can lex, with brackets that balance; after that, what it means is the macro's business. cargo expand shows what each call became (recorded with the command underneath it, cargo rustc -- -Zunpretty=expanded):

Verified output of cargo rustc -q -p filter_app --profile=check -- -Zunpretty=expanded — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

#![feature(prelude_import)]
extern crate std;
#[prelude_import]
use std::prelude::rust_2024::*;
use filter_macro::filter;

struct Order {
    id: u32,
    status: &'static str,
    total: u32, // in cents
}

const ORDERS: [Order; 5] =
    [Order { id: 1, status: "open", total: 1250 },
            Order { id: 2, status: "paid", total: 8000 },
            Order { id: 3, status: "cancelled", total: 9900 },
            Order { id: 4, status: "open", total: 5000 },
            Order { id: 5, status: "paid", total: 1999 }];

fn main() {
    let ids =
        |matched: Vec<&Order>|
            matched.iter().map(|order| order.id).collect::<Vec<_>>();

    let open: Vec<&Order> =
        ORDERS.iter().filter(|row| row.status == "open").collect();
    let large: Vec<&Order> =
        ORDERS.iter().filter(|row|
                    row.status != "cancelled" && row.total >= 5000).collect();
    let small: Vec<&Order> =
        ORDERS.iter().filter(|row|
                    ["open", "paid"].contains(&row.status) &&
                        row.total < 2000).collect();

    // open                    [1, 4]
    // not cancelled, >= 5000  [2, 4]
    // open or paid, < 2000    [1, 5]
    { ::std::io::_print(format_args!("{0:<24}{1:?}\n", "open", ids(open))); };
    {
        ::std::io::_print(format_args!("{0:<24}{1:?}\n",
                "not cancelled, >= 5000", ids(large)));
    };
    {
        ::std::io::_print(format_args!("{0:<24}{1:?}\n",
                "open or paid, < 2000", ids(small)));
    };
}

The grammar, as types

The grammar sits at the top of the file as a comment, and each rule below it is a type with a Parse impl:

demo/filter_macro/src/grammar.rs

//! The language `filter!` accepts, as a struct per rule and a `Parse` impl
//! that reads it:
//!
//! ```text
//! filter    = condition ("and" condition)*
//! condition = field "in" "[" value ("," value)* "]"
//!           | field op value
//! op        = "=" | "<>" | "<=" | ">=" | "<" | ">"
//! ```

use syn::parse::{Parse, ParseStream};
use syn::punctuated::Punctuated;
use syn::{Ident, Lit, Token, bracketed};

mod kw {
    // `and` is an ordinary identifier to Rust; this makes it a token type.
    syn::custom_keyword!(and);
}

// `<>` is not a Rust operator, so there is no `Token![<>]` to borrow.
syn::custom_punctuation!(LtGt, <>);

pub struct Filter {
    pub conditions: Punctuated<Condition, kw::and>,
}

pub enum Condition {
    Compare { field: Ident, op: Op, value: Lit },
    In { field: Ident, values: Punctuated<Lit, Token![,]> },
}

pub enum Op {
    Eq,
    NotEq,
    Le,
    Ge,
    Lt,
    Gt,
}

impl Parse for Filter {
    fn parse(input: ParseStream) -> syn::Result<Self> {
        let conditions = Punctuated::parse_separated_nonempty(input)?;
        Ok(Filter { conditions })
    }
}

impl Parse for Condition {
    fn parse(input: ParseStream) -> syn::Result<Self> {
        let field: Ident = input.parse()?;
        if input.peek(Token![==]) {
            let token: Token![==] = input.parse()?;
            return Err(syn::Error::new_spanned(token, "`filter!` compares with `=`, not `==`"));
        }
        // Every `peek` that fails is remembered, and `lookahead.error()` lists
        // them. Two-character operators go first: `Token![<]` also matches `<=`.
        let lookahead = input.lookahead1();
        let op = if lookahead.peek(Token![in]) {
            input.parse::<Token![in]>()?;
            let content;
            bracketed!(content in input);
            let values = Punctuated::parse_terminated(&content)?;
            return Ok(Condition::In { field, values });
        } else if lookahead.peek(LtGt) {
            input.parse::<LtGt>().map(|_| Op::NotEq)?
        } else if lookahead.peek(Token![<=]) {
            input.parse::<Token![<=]>().map(|_| Op::Le)?
        } else if lookahead.peek(Token![>=]) {
            input.parse::<Token![>=]>().map(|_| Op::Ge)?
        } else if lookahead.peek(Token![<]) {
            input.parse::<Token![<]>().map(|_| Op::Lt)?
        } else if lookahead.peek(Token![>]) {
            input.parse::<Token![>]>().map(|_| Op::Gt)?
        } else if lookahead.peek(Token![=]) {
            input.parse::<Token![=]>().map(|_| Op::Eq)?
        } else {
            return Err(lookahead.error());
        };
        let value = input.parse()?;
        Ok(Condition::Compare { field, op, value })
    }
}

In the grammar	In syn
field, value	input.parse::<Ident>(), input.parse::<Lit>(): anything that implements Parse ↗, picked by the type you ask for
"in"	Token![in]: a Rust keyword, so syn already has a type for it
"and"	kw::and, from custom_keyword! ↗: and is an ordinary identifier to Rust, and this gives it a token type of its own
"<>"	LtGt, from custom_punctuation! ↗: < and > joined, an operator Rust does not have
"[" … "]"	bracketed! ↗(content in input): content becomes a stream over what is inside; braced! and parenthesized! do the same for {} and ()
value ("," value)*	Punctuated::parse_terminated ↗: reads the whole stream, and allows a trailing comma
condition ("and" condition)*	Punctuated::parse_separated_nonempty: at least one, and it stops at the first item not followed by a separator, whatever comes after
a choice between alternatives	input.lookahead1() ↗, a peek per alternative

The macro itself only turns the parsed struct back into tokens:

demo/filter_macro/src/lib.rs

//! `filter!(status <> "cancelled" and total >= 5000)`: a query language that is
//! not Rust, turned into a closure for `Iterator::filter`.

mod grammar;

use grammar::{Condition, Filter, Op};
use proc_macro::TokenStream;
use quote::quote;
use syn::parse_macro_input;

#[proc_macro]
pub fn filter(input: TokenStream) -> TokenStream {
    let filter = parse_macro_input!(input as Filter);
    let tests = filter.conditions.iter().map(|condition| match condition {
        Condition::Compare { field, op, value } => {
            let op = match op {
                Op::Eq => quote!(==),
                Op::NotEq => quote!(!=),
                Op::Le => quote!(<=),
                Op::Ge => quote!(>=),
                Op::Lt => quote!(<),
                Op::Gt => quote!(>),
            };
            quote!(row.#field #op #value)
        }
        Condition::In { field, values } => {
            let values = values.iter();
            quote!([#(#values),*].contains(&row.#field))
        }
    });
    quote!(|row| #(#tests)&&*).into()
}

parse_macro_input!(input as Filter) calls Filter::parse and, if it fails, returns the error as the macro's output, so the user sees it as a compile error. #(#tests)&&* repeats with && between the items.

peek matches a prefix

input.peek(Token![<]) looks at the next token without consuming it. What it compares is characters, and this probe, which runs syn in an ordinary program, shows what that means:

demo/peek_table/src/main.rs

//! What `peek` answers for a few operators. `syn` works outside a macro too,
//! on a `proc_macro2::TokenStream` parsed from a string.

use syn::Token;
use syn::parse::{ParseStream, Parser};

/// Runs `peek` on `source` and returns its answer. The closure has to consume
/// every token, or `parse_str` reports the rest as unexpected.
fn peeks(source: &str, peek: fn(ParseStream) -> bool) -> bool {
    let parser = |input: ParseStream| {
        let answer = peek(input);
        input.parse::<proc_macro2::TokenStream>()?;
        Ok(answer)
    };
    parser.parse_str(source).unwrap()
}

fn main() {
    println!("source   Token![=]  Token![==]  Token![<]  Token![<=]");
    for source in ["=", "==", "=>", "<", "<=", "<>", "< ="] {
        println!(
            "{:<8} {:<10} {:<11} {:<10} {}",
            source,
            peeks(source, |input| input.peek(Token![=])),
            peeks(source, |input| input.peek(Token![==])),
            peeks(source, |input| input.peek(Token![<])),
            peeks(source, |input| input.peek(Token![<=])),
        );
    }
}

Verified output of cargo run -q -p peek_table — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

source   Token![=]  Token![==]  Token![<]  Token![<=]
=        true       false       false      false
==       true       true        false      false
=>       true       false       false      false
<        false      false       true       false
<=       false      false       true       true
<>       false      false       true       false
< =      false      false       true       false

Token![=] is true for == and =>, and Token![<] for <= and <>. A multi-character operator reaches a macro as one Punct per character, each marked joint when another punctuation character follows it with no space, and syn checks that mark only between the characters it wants (Cursor::peek_punct in syn 3.0.6 ↗). So Token![<] never asks whether a = is stuck to the <, and < = with a space is not Token![<=]. Tokens and token streams covers that spacing.

That is why Condition::parse peeks <>, <= and >= before <, > and =: tried first, Token![<] would claim the < of total <= 2000 and leave = where the value should be. And it is why == gets a check of its own before the lookahead: the Token![=] branch would claim the first half of status == "open", and the error would land on the second = with no mention of ==.

What the user sees

Four mistakes in one program:

demo/filter_mistakes/src/bin/bad_syntax.rs

use filter_macro::filter;

struct Order {
    status: &'static str,
    total: u32,
}

fn main() {
    let orders = [Order { status: "open", total: 1250 }];
    let _ = orders.iter().filter(filter!(status != "cancelled"));
    let _ = orders.iter().filter(filter!(status == "open"));
    let _ = orders.iter().filter(filter!(status = "open" total > 0));
    let _ = orders.iter().filter(filter!(status in ["open" "paid"]));
}

Verified output of cargo build -q -p filter_mistakes --bin bad_syntax, which fails on purpose — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

error: expected one of: `in`, `<>`, `<=`, `>=`, `<`, `>`, `=`
  --> filter_mistakes/src/bin/bad_syntax.rs:10:49
   |
10 |     let _ = orders.iter().filter(filter!(status != "cancelled"));
   |                                                 ^

error: `filter!` compares with `=`, not `==`
  --> filter_mistakes/src/bin/bad_syntax.rs:11:49
   |
11 |     let _ = orders.iter().filter(filter!(status == "open"));
   |                                                 ^^

error: unexpected token
  --> filter_mistakes/src/bin/bad_syntax.rs:12:58
   |
12 |     let _ = orders.iter().filter(filter!(status = "open" total > 0));
   |                                                          ^^^^^

error: expected `,`
  --> filter_mistakes/src/bin/bad_syntax.rs:13:60
   |
13 |     let _ = orders.iter().filter(filter!(status in ["open" "paid"]));
   |                                                            ^^^^^^

error: could not compile `filter_mistakes` (bin "bad_syntax") due to 4 previous errors

Each message comes from a different place:

1. expected one of: … is lookahead.error(). Every peek on a Lookahead1 that returns false records what it looked for, and the error lists them in the order they were tried, pointing at the token that matched none. That is all it takes to name every operator the grammar accepts; the order of the if chain is also the order of the message.
2. `filter!` compares with `=`, not `==` is the macro's own syn::Error, spanned on the == token it parsed. Errors: from panic! to syn::Error covers building these.
3. unexpected token is the least helpful of the four, and the grammar's fault. parse_separated_nonempty returned after status = "open", as documented, because no and followed; then parse_macro_input! found tokens left over and reported the first one. A loop that parses a condition and then peeks for kw::and through a Lookahead1 would name the missing and instead.
4. expected `,` comes from parse_terminated, inside the brackets, at the value that should have had a comma before it.

All four point at a token inside the call, and none is a panic. The mistake the grammar cannot see is a field that does not exist; that one comes from rustc, after expansion:

demo/filter_mistakes/src/bin/bad_field.rs

use filter_macro::filter;

struct Order {
    status: &'static str,
    total: u32,
}

fn main() {
    let orders = [Order { status: "open", total: 1250 }];
    let count = orders.iter().filter(filter!(totl >= 5000)).count();
    println!("{count} {}", orders[0].status);
}

Verified output of cargo build -q -p filter_mistakes --bin bad_field, which fails on purpose — declared in cargo_runs.toml and regenerated by tools/run_cargo_demos.py, never hand-typed.

error[E0609]: no field `totl` on type `&&Order`
  --> filter_mistakes/src/bin/bad_field.rs:10:46
   |
10 |     let count = orders.iter().filter(filter!(totl >= 5000)).count();
   |                                              ^^^^ unknown field
   |
help: a field with a similar name exists
   |
10 |     let count = orders.iter().filter(filter!(total >= 5000)).count();
   |                                                 +

For more information about this error, try `rustc --explain E0609`.
error: could not compile `filter_mistakes` (bin "bad_field") due to 1 previous error

The error and the suggestion point at totl inside filter!(…). The macro put the user's own Ident into row.#field, and an Ident carries its span, so rustc knows where the name came from. The &&Order in the message is the closure's parameter type, which Iterator::filter fixed and the macro never wrote.

syn 3 and syn 2

This library pins syn 3; most tutorials use syn 2. The API on this page is unchanged: parse.rs, lookahead.rs, group.rs, custom_keyword.rs and custom_punctuation.rs are byte-for-byte the same in syn 2.0.119 and 3.0.6. One neighbour did change: Punctuated::pop returns the value, Option<T>, in syn 3, where syn 2 returned Option<Pair<T, P>>; the old behaviour is pop_pair.

If you are coming from another language

• C and C++. The preprocessor cannot add syntax, and C++ has no way to define any. The nearest thing is a parser generator such as yacc or bison, which turns a grammar into C at build time. The Parse impls here are the other classic style, a hand-written recursive-descent parser: one function per rule, each consuming tokens and returning a struct. The difference is where the input comes from, which is the source of the program being compiled, and where the errors go, which is the compiler's own output, pointing into the caller's file.
• Python. Python code can only be Python. A query language is a string handed to a parser at run time, and ast will not help with this one: ast.parse("status != 'cancelled' and total >= 5000", mode="eval") succeeds, but the same with <> is SyntaxError: invalid syntax (Python 3.14). A mistake in the string surfaces when that line runs; in filter! it stops the build.
• Java. No user-defined syntax either. A query is a string that a library parses when the program runs, or a chain of builder methods that javac checks as ordinary method calls, which keeps the checking at the price of the notation.
• ABAP. (Not machine-checked — CI cannot run ABAP.) ABAP is the language here with SQL built in: SELECT … WHERE status <> 'CANCELLED' AND total >= 5000 is checked by the ABAP compiler, <> and AND included. A dynamic WHERE (lv_where) takes a string, checked only when it runs. filter! is the first kind, written as a library rather than as part of the language.

See also

• Three kinds of procedural macro — a function-like macro's input only has to be tokens
• Tokens and token streams — Punct, Spacing, and why <= is two tokens
• Parsing with syn — the parsers syn already has, for Rust syntax
• Errors: from panic! to syn::Error — the error in mistake 2, in depth
• When macro_rules! runs out — the question to ask before writing a grammar like this one
• A routes! macro — this machinery used for a whole DSL, with checks that span more than one token
• syn::parse ↗ — the module documentation, with a longer worked example

Po polsku

Między nawiasami makra funkcyjnego obowiązuje twoja składnia. Wystarczy, że wejście da się podzielić na tokeny, a nawiasy się domykają. Gramatykę zapisuje się jako typy: jedna struktura albo enum na regułę i impl Parse dla każdej. input.parse() bierze kolejny element, a to, jaki, wynika z typu, o który prosisz (Ident, Lit, Token![in]). lookahead1 wybiera między wariantami i sam składa komunikat expected one of: … z listy wszystkiego, czego szukał. Punctuated czyta listy z separatorem: parse_terminated całą zawartość nawiasów, a parse_separated_nonempty tylko tyle, ile ciągnie się separator. Słowa i operatory, których Rust nie zna, dostają własne typy przez custom_keyword!(and) i custom_punctuation!(LtGt, <>).

Pułapka: peek porównuje znaki, więc Token![<] pasuje także do <= i <>, a Token![=] do ==. Dłuższe operatory trzeba sprawdzać najpierw. Nazwa pola wpisana przez użytkownika zachowuje swój span, więc literówkę w nazwie pola kompilator wskaże wewnątrz wywołania makra, razem z podpowiedzią.

Szukaj po polsku: parsowanie w makrach proceduralnych · syn::parse::Parse · rust syn lookahead1 custom_keyword · rust proc macro custom syntax DSL · syn Punctuated parse_terminated