1import{_ as e,c as s,e as a,o as t}from"./app-CjUkjRdW.js";const i="/assets/lexer-BnwTc3N8.png",l="/assets/run-jflex-BkcvrdSa.png",p={};function o(c,n){return t(),s("div",null,n[0]||(n[0]=[a(`<h1 id="lexer" tabindex="-1"><a class="header-anchor" href="#lexer"><span>Lexer</span></a></h1><p>The role of the lexer is to tokenize the input text into a stream of tokens. Typically, this stream of tokens is used by the parser to build the Abstract Syntax Tree (AST) of the code, or in the case of an IntelliJ language plugin, the <abbr title="Program Structure Interface">PSI</abbr> tree, which we will get to in a later section. Another use for the lexer is to provide syntax highlighting in the editor, which gives different colors to different tokens depending on their type.</p><h2 id="the-lexer-interface" tabindex="-1"><a class="header-anchor" href="#the-lexer-interface"><span>The Lexer Interface</span></a></h2><p>The IntelliJ Platform provides an abstract class called <code>Lexer</code> that language plugins can extend to implement their own lexer.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">class</span> <span class="token class-name">Lexer</span> <span class="token punctuation">{</span></span> 2<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">void</span> <span class="token function">start</span><span class="token punctuation">(</span><span class="token class-name">CharSequence</span> buffer<span class="token punctuation">,</span> <span class="token keyword">int</span> startOffset<span class="token punctuation">,</span> <span class="token keyword">int</span> endOffset<span class="token punctuation">,</span> <span class="token keyword">int</span> initialState<span class="token punctuation">)</span><span class="token punctuation">;</span></span> 3<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">int</span> <span class="token function">getState</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 4<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token class-name">IElementType</span> <span class="token function">getTokenType</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 5<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">int</span> <span class="token function">getTokenStart</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 6<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">int</span> <span class="token function">getTokenEnd</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 7<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">void</span> <span class="token function">advance</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 8<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token class-name">LexerPosition</span> <span class="token function">getCurrentPosition</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 9<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">void</span> <span class="token function">restore</span><span class="token punctuation">(</span><span class="token class-name">LexerPosition</span> position<span class="token punctuation">)</span><span class="token punctuation">;</span></span> 10<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token class-name">CharSequence</span> <span class="token function">getBufferSequence</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 11<span class="line"> <span class="token keyword">public</span> <span class="token keyword">abstract</span> <span class="token keyword">int</span> <span class="token function">getBufferEnd</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 12<span class="line"><span class="token punctuation">}</span></span> 13<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Typically, the lexer works in a pull mode, where the caller repeatedly calls the <code>getTokenType</code> and <code>advance</code> in a loop to get the next token in the stream, until <code>getTokenType</code> returns <code>null</code> to indicate the end of the stream. The <code>getTokenStart</code> and <code>getTokenEnd</code> methods return the start and end offsets of the current token in the input buffer. The other methods are used to manage the lexer's state and position in the input.</p><p>We <em>can</em> implement our own lexer from scratch by extending the <code>Lexer</code> class, but this is rarely done in practice. Instead, a lexer generator is typically used to generate the lexer code from a lexer specification. The lexer specification is a set of pattern matching rules that define the tokens in the language. Fortunately, the Grammar-Kit plugin supports the <a href="https://www.jflex.de/" target="_blank" rel="noopener noreferrer">JFlex</a> lexer generator out of the box, so we can use it to generate the lexer for Nim.</p><h2 id="lexer-components" tabindex="-1"><a class="header-anchor" href="#lexer-components"><span>Lexer Components</span></a></h2><p>Let's first understand the different classes and interfaces that are involved in the lexer. The following diagram shows the relationships between these classes and interfaces.</p><p><img src="`+i+`" alt="Lexer and Tokens"></p><ul><li>Given a lexer specification file, here called <code>Nim.flex</code>
13, JFlex generates a lexer class called <code>NimLexer</code> that implements the <code>FlexLexer</code> interface.</li><li>However, the platform expects a lexer class that extends the <code>Lexer</code> abstract class, so an adapter class is needed to bridge the two. In our case, this adapter class is called <code>NimLexerAdapter</code>, and it extends the <code>FlexAdapter</code> base class, which does the actual work of adapting the <code>FlexLexer</code> interface to the <code>Lexer</code> interface. This is the class that will be used by the parser, not the generated <code>NimLexer</code>.</li><li>The <code>NimTokenType</code> class is a specialization of the <code>IElementType</code> class, which is used to represent the different types of tokens in the language. The lexer will return instances of <code>NimTokenType</code> (which we will create later), or one of the built-in token types defined in the <code>TokenType</code> class, according to the matching rules in the lexer specification.</li><li>In some cases, we may need to refer to a set of tokens as a single entity. For example, the set of all keywords in the language, or the set of all operators. This is where the <code>TokenSet</code> class comes in, which is a holder of <code>IElementType</code> instances. We will declare all token sets in the <code>NimTokenSets</code> class for easy access.</li></ul><h2 id="lexer-specification" tabindex="-1"><a class="header-anchor" href="#lexer-specification"><span>Lexer Specification</span></a></h2><p>JFlex comes from a family of lexer generators that has roots in the <code>lex</code> tool that was developed to generate lexical analyzers under Unix. Given a lexer specification file, JFlex generates a lexer class in Java that can be used to tokenize the input text.</p><p>The lexer specification file includes three sections separated by <code>%%</code>:</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token comment">// User code</span></span> 14<span class="line"></span> 15<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 16<span class="line"></span> 17<span class="line"><span class="token comment">// Options and declarations</span></span> 18<span class="line"></span> 19<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 20<span class="line"></span> 21<span class="line"><span class="token comment">// Lexical rules</span></span> 22<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><dl><dt><strong>User code</strong></dt><dd>This part is copied verbatim to the generated lexer class. It typically contains package declarations, imports, and other user-defined code.</dd><dt><strong>Options and declarations</strong></dt><dd>This part contains JFlex options and declarations. Options are used to configure the behavior of the lexer generator, while declarations are used to define variables and methods that are used in the lexical rules.</dd><dt><strong>Lexical rules</strong></dt><dd>This part contains the pattern matching rules that define the tokens in the language. Each rule consists of a regular expression pattern followed by a Java action that is executed when the pattern is matched. The action typically returns the token type of the matched token.</dd></dl><p>Let's start by creating a minimal lexer specification file, <code>Nim.flex</code>, which contains the bare minimum to tokenize only whitespace for now (anything else will be considered a bad character).</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token comment">// src/main/kotlin/khaledh/nimjet/Nim.flex</span></span> 23<span class="line"></span> 24<span class="line"><span class="token keyword">package</span> <span class="token namespace">khaledh<span class="token punctuation">.</span>nimjet</span><span class="token punctuation">;</span></span> 25<span class="line"></span> 26<span class="line"><span class="token keyword">
26import</span> <span class="token import"><span class="token namespace">com<span class="token punctuation">.</span>intellij<span class="token punctuation">.</span>lexer<span class="token punctuation">.</span></span><span class="token class-name">FlexLexer</span></span><span class="token punctuation">;</span></span> 27<span class="line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">com<span class="token punctuation">.</span>intellij<span class="token punctuation">.</span>psi<span class="token punctuation">.</span></span><span class="token class-name">TokenType</span></span><span class="token punctuation">;</span></span> 28<span class="line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">com<span class="token punctuation">.</span>intellij<span class="token punctuation">.</span>psi<span class="token punctuation">.</span>tree<span class="token punctuation">.</span></span><span class="token class-name">IElementType</span></span><span class="token punctuation">;</span></span> 29<span class="line"></span> 30<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 31<span class="line"></span> 32<span class="line"><span class="token operator">%</span><span class="token keyword">class</span> <span class="token class-name">NimLexer</span></span> 33<span class="line"><span class="token operator">%</span><span class="token keyword">implements</span> <span class="token class-name">FlexLexer</span></span> 34<span class="line"><span class="token operator">%</span>function advance</span> 35<span class="line"><span class="token operator">%</span>type <span class="token class-name">IElementType</span></span> 36<span class="line"><span class="token operator">%</span><span class="token keyword">char</span></span> 37<span class="line"><span class="token operator">%</span>unicode</span> 38<span class="line"></span> 39<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 40<span class="line"></span> 41<span class="line"><span class="token generics"><span class="token punctuation"><</span>YYINITIAL<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 42<span class="line"> <span class="token punctuation">[</span> \\t\\n\\r<span class="token punctuation">]</span><span class="token operator">+</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">TokenType</span><span class="token punctuation">.</span><span class="token constant">WHITE_SPACE</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 43<span class="line"></span> 44<span class="line"> <span class="token comment">// error fallback</span></span> 45<span class="line"> <span class="token punctuation">[</span><span class="token operator">^</span><span class="token punctuation">]</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">TokenType</span><span class="token punctuation">.</span><span class="token constant">BAD_CHARACTER</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 46<span class="line"><span class="token punctuation">}</span></span> 47<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><ul><li><p>In the first section, we declare the package and imports that will be copied to the generated lexer
47class.</p></li><li><p>In the second section, we specify the options and declarations:</p><ul><li>we specify the name to be used for the generated lexer class: <code>NimLexer</code>,</li><li>specify that it implements the <code>FlexLexer</code> interface,</li><li>speicfy the name of the function that advances the lexer to the next token as <code>advance</code>,</li><li>specify the return type of the <code>advance</code> function as <code>IElementType</code>,</li><li>specify that the lexer should count characters (so that the current offset is available to the parser), and</li><li>specify that the lexer should use the full Unicode input character set.</li></ul></li><li><p>In the third section, we define the lexical rules. The rules are defined within the scope of <em>states</em>. The lexer can be in one of several states, depending on the context (e.g. inside a string literal, inside a comment, etc.). The initial state is called <code><YYINITIAL></code>. Here, we define the following rules under the initial state:</p><ul><li>The <code>[ \\t\\n\\r]+</code> regex matches one or more whitespace characters and returns the built-in <code>TokenType.WHITE_SPACE</code> token type.</li><li>The <code>[^]</code> is an error fallback regex that matches any character that was not matched by any of the previous rules, and returns the built-in <code>TokenType.BAD_CHARACTER</code> token type. We're using this expression instead of <code>.</code> since it matches any character, including newlines and unpaired Unicode surrogate characters.</li></ul><p>Later, we will add more rules to match the different tokens in the language.</p></li></ul><h2 id="generating-the-lexer" tabindex="-1"><a class="header-anchor" href="#generating-the-lexer"><span>Generating the Lexer</span></a></h2><p>The Grammar-Kit plugin provides an action to generate the lexer from the lexer specification file. To do that, right-click on the <code>Nim.flex</code> file in the project view or the editor, and select <strong>Run JFlex Generator</strong> from the context menu.</p><img src="`+l+`" style="display:block;margin-left:auto;margin-right:auto;width:60%;"><p>This will generate the lexer class under the <code>src/main/gen</code> directory that looks like the following:</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token comment">// Generated by JFlex 1.9.2 http://jflex.de/ (tweaked for IntelliJ platform)</span></span> 48<span class="line"><span class="token comment">// source: src/main/kotlin/khaledh/nimjet/Nim.flex</span></span> 49<span class="line"></span> 50<span class="line"><span class="token keyword">package</span> <span class="token namespace">khaledh<span class="token punctuation">.</span>nimjet</span><span class="token punctuation">;</span></span> 51<span class="line"></span> 52<span class="line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">com<span class="token punctuation">.</span>intellij<span class="token punctuation">.</span>lexer<span class="token punctuation">.</span></span><span class="token class-name">FlexLexer</span></span><span class="token punctuation">;</span></span> 53<span class="line"><span class="token keyword">import</span> <span class="token import"><span class="token namespace">com<span class="token punctuation">.</span>intellij<span class="token punctuation">.</span>psi<span class="token punctuation">.</span></span><span class="token class-name">TokenType</span></span><span class="token punctuation">;</span></span> 54<span class="line"></span> 55<span class="line"></span> 56<span class="line"><span class="token keyword">class</span> <span class="token class-name">NimLexer</span> <span class="token keyword">implements</span> <span class="token class-name">FlexLexer</span> <span class="token punctuation">{</span></span> 57<span class="line"> <span class="token comment">// ...</span></span> 58<span class="line"><span class="token punctuation">}</span></span> 59<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>In order to include the generated lexer in the project, we need to add the <code>src/main/gen</code> directory to the <code>sourceSets</code> in the <code>build.gradle.kts</code> file:</p><div class="language-kts line-numbers-mode" data-highlighter="prismjs" data-ext="kts" data-title="kts"><pre><code><span class="line"><span class="token comment">// build.gradle.kts</span></span> 60<span class="line"><span class="token operator">..</span><span class="token punctuation">.</span></span> 61<span class="line"></span> 62<span class="line">sourceSets <span class="token punctuation">{</span></span> 63<span class="line"> main <span class="token punctuation">{</span></span> 64<span class="line"> java <span class="token punctuation">{</span></span> 65<span class="line">
65 <span class="token function">srcDirs</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"src/main/gen"</span></span><span class="token punctuation">)</span></span> 66<span class="line"> <span class="token punctuation">}</span></span> 67<span class="line"> <span class="token punctuation">}</span></span> 68<span class="line"><span class="token punctuation">}</span></span> 69<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Let's also mark the <code>src/main/gen</code> directory as a generated source directory in IntelliJ. To do that, right-click on the <code>src/main/gen</code> directory in the project view, and select <strong>Mark Directory as | Generated Sources Root</strong>.</p><h2 id="automating-the-lexer-generation" tabindex="-1"><a class="header-anchor" href="#automating-the-lexer-generation"><span>Automating the Lexer Generation</span></a></h2><p>There's one inconvenience though: every time we change the lexer specification file, we need to manually run the JFlex generator to update the lexer. To automate this process, we can use the <code>generateLexer</code> task that comes with the Grammar-Kit Gradle plugin. Let's add the plugin to the build file and configure the task:</p><div class="language-kts line-numbers-mode" data-highlighter="prismjs" data-ext="kts" data-title="kts"><pre><code><span class="line"><span class="token comment">// build.gradle.kts</span></span> 70<span class="line"></span> 71<span class="line">plugins <span class="token punctuation">{</span></span> 72<span class="line"> <span class="token function">id</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"org.jetbrains.intellij.platform"</span></span><span class="token punctuation">)</span> version <span class="token string-literal singleline"><span class="token string">"2.0.1"</span></span></span> 73<span class="line"> <span class="token function">id</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"org.jetbrains.kotlin.jvm"</span></span><span class="token punctuation">)</span> version <span class="token string-literal singleline"><span class="token string">"2.0.20"</span></span></span> 74<span class="line highlighted"> <span class="token function">id</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"org.jetbrains.grammarkit"</span></span><span class="token punctuation">)</span> version <span class="token string-literal singleline"><span class="token string">"2022.3.2.2"</span></span></span> 75<span class="line"><span class="token punctuation">}</span></span> 76<span class="line"></span> 77<span class="line"><span class="token operator">..</span><span class="token punctuation">.</span></span> 78<span class="line"></span> 79<span class="line highlighted">tasks <span class="token punctuation">{</span></span> 80<span class="line highlighted"> generateLexer <span class="token punctuation">{</span></span> 81<span class="line highlighted"> sourceFile <span class="token operator">=</span> <span class="token function">file</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"src/main/kotlin/khaledh/nimjet/Nim.flex"</span></span><span class="token punctuation">)</span></span> 82<span class="line highlighted"> targetOutputDir <span class="token operator">=</span> <span class="token function">file</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"src/main/gen/khaledh/nimjet"</span></span><span class="token punctuation">)</span></span> 83<span class="line highlighted"> purgeOldFiles <span class="token operator">=</span> <span class="token boolean">true</span></span> 84<span class="line highlighted"> <span class="token punctuation">}</span></span> 85<span class="line highlighted"></span> 86<span class="line highlighted"> compileJava <span class="token punctuation">{</span> <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateLexer<span class="token punctuation">)</span> <span class="token punctuation">}</span></span> 87<span class="line highlighted"> compileKotlin <span class="token punctuation">{</span> <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateLexer<span class="token punctuation">)</span> <span class="token punctuation">}</span></span> 88<span class="line highlighted"><span class="token punctuation">}</span></span> 89<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>This configures the <code>generateLexer</code> task and makes it a dependency of the <code>compileJava</code> and <code>compileKotlin</code> tasks, so that the lexer is generated every time the project is compiled.</p><p>At this point it would be good to test the lexer to make sure it's working as expected. Unfortunately, we can't register the lexer directly. We'll need to create a parser definition and register that first, which provides access to the lexer. We'll get to that in the next section.</p>`,32)]))}const d=e(p,[["render",o],["__file","04-lexer.html.vue"]]),u=JSON.parse('{"path":"/nimjet/04-lexer.html","title":"Lexer","lang":"en-US","frontmatter":{},"headers":[{"level":2,"title":"The Lexer Interface","slug":"the-lexer-interface","link":"#the-lexer-interface","children":[]},{"level":2,"title":"Lexer Components","slug":"lexer-components","link":"#lexer-components","children":[]},{"level":2,"title":"Lexer Specification","slug":"lexer-specification","link":"#lexer-specification","children":[]},{"level":2,"title":"Generating the Lexer","slug":"generating-the-lexer","link":"#generating-the-lexer","children":[]},{"level":2,"title":"Automating the Lexer Generation","slug":"automating-the-lexer-generation","link":"#automating-the-lexer-generation","children":[]}
89],"git":{"updatedTime":1725233521000},"filePathRelative":"nimjet/04-lexer.md","excerpt":"\\n<p>The role of the lexer is to tokenize the input text into a stream of tokens. Typically,\\nthis stream of tokens is used by the parser to build the Abstract Syntax Tree (AST) of the\\ncode, or in the case of an IntelliJ language plugin, the\\n<abbr title=\\"Program Structure Interface\\">PSI</abbr> tree, which we will get to in a\\nlater section. Another use for the lexer is to provide syntax highlighting in the\\neditor, which gives different colors to different tokens depending on their type.</p>"}');export{d as comp,u as data};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.