Rlex

Rlex is a lightweight lexer utility for traversing, peeking, and extracting parts of a UTF-8 string. It operates on a Vec<char> and retains the original string to allow for accurate byte-range slicing. It is ideal for building scanners, parsers, or any tool that needs detailed and controlled inspection of characters in a string.

Installation

Install via cargo

1cargo add rlex

Features

Creating a Lexer

First, you need an enum to represent the state of your lexer and a token type:

 1#[derive(Debug, PartialEq, Eq)]
 2enum MyState {
 3    Init,
 4    Open,
 5    Closed,
 6}
 7
 8#[derive(Debug, PartialEq, Eq)]
 9enum MyToken {
10    Tok1,
11    Tok2,
12    Tok3,
13}

Then use the enums to create a new lexer:

1let r: Rlex<MyState, MyToken> = Rlex::new("hello", MyState::Init);

Using Default State / Default Token

If you don't care to collect tokens or track state, use DefaultState and DefaultToken upon initalization.

1let r: Rlex<DefaultState, DefaultToken> = Rlex::new("hello", DefaultState::Default);

State Handling

1r.state();              // Get a reference to the current state
2r.state_set(MyState::Open);  // Set a new state

Position Utilities

1r.pos();            // Current position
2r.mark();           // Mark current position
3r.goto_start();     // Go to start of input
4r.goto_end();       // Go to end of input
5r.goto_pos(2);      // Go to a specific position
6r.goto_mark();      // Go back to marked position

Navigation

1r.next();           // Move forward one
2r.next_by(3);       // Move forward by n
3r.prev();           // Move backward one
4r.prev_by(2);       // Move backward by n
5r.next_until('x');  // Advance until char
6r.prev_until('x');  // Rewind until char

Peeking

1r.peek();            // Look at next char
2r.peek_by(2);        // Look ahead by n
3r.peek_back();       // Look behind one
4r.peek_back_by(3);   // Look back by n

Char Checks

1r.char();             // Get current char
2r.next_is('x');       // Check if next char is x
3r.next_by_is('x', 2); // Check if x is n chars ahead
4r.prev_is('x');       // Check if previous char is x
5r.prev_by_is('x', 3); // Check if x is n chars behind

Position Queries

1r.at_start();     // At beginning?
2r.at_end();       // At end?
3r.at_mark();      // At previously marked spot?

String Extraction

1r.src() // Read the Lexer source
2r.toks() // Get a reference to the collected tokens
3r.str_from_mark();  // Slice from mark to current
4r.str_from_start(); // Slice from start to current
5r.str_from_end();   // Slice from current to end
6r.str_from_collection(); // Convert the collection into a slice
7r.str_from_rng(0, 2); // Index-based slice from source 

Quote Detection

1r.is_in_quote(); // Returns true if current position is inside a quote block

Collecting Characters

1r.collect(); // Collect the character at the current position
2r.collect_pop(); // Get the newest character added to the collection
3r.collect_push('a'); // Push a character of your choice into the collection
4r.collect_clear(); // Clears the current collection

Working With Tokens

1r.token_push(MyToken::Tok1); // Push a token into the collection
2r.token_pop(); // Remove and obtain the last token in the collection
3r.token_prev(); // Peek at the last token in the collection
4r.token_consume(); // Consumes the lexer and outputs the collected tokens

Tracing

1r.trace_on() // Turn on the trace system
2r.trace_off() // Turn off the trace system
3r.trace_emit() // Get the trace as a String
4r.trace_clear() // Clear the trace

License

This project is licensed under the MIT License.