1import{_ as n,c as s,e as a,o as i}from"./app-CjUkjRdW.js";const t={};function l(r,e){return i(),s("div",null,e[0]||(e[0]=[a(`<h1 id="grammar-kit" tabindex="-1"><a class="header-anchor" href="#grammar-kit"><span>Grammar-Kit</span></a></h1><p>Grammar-Kit is a JetBrains plugin for generating a parser (and a lexer, if needed) from a grammar specification file that uses a BNF-like syntax. Not only does it generate the parser, but it also generates the PSI classes of the language. This is a huge time-saver, as it allows us to focus on the language syntax and semantics rather than the parsing details.</p><h2 id="the-bnf-file" tabindex="-1"><a class="header-anchor" href="#the-bnf-file"><span>The BNF File</span></a></h2><p>The grammar specification file is written in a BNF-like syntax. The file two types of sections: an <strong>attributes</strong> section and a <strong>grammar rules</strong> section. The attributes section allows us to customize the generated parser and PSI classes, while the grammar rules section defines the language syntax. Here's a simple example of such a BNF file:</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line">// Attributes</span> 2<span class="line"><span class="token operator">{</span></span> 3<span class="line"> parserClass=<span class="token string">"generated.MyParser"</span></span> 4<span class="line"><span class="token operator">}</span></span> 5<span class="line"></span> 6<span class="line">// Grammar</span> 7<span class="line">Root <span class="token operator">::=</span> BEGIN <span class="token operator">(</span>MyStmt SEMI<span class="token operator">)</span><span class="token operator">*</span> END</span> 8<span class="line">MyStmt <span class="token operator">::=</span> PrintStmt</span> 9<span class="line"> <span class="token operator">|</span> <span class="token operator">...</span></span> 10<span class="line">PrintStmt <span class="token operator">::=</span> PRINT STRING_LIT</span> 11<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>In the attributes section we specified the fully qualified name of the parser class that will be generated. In the grammar section, we defined the syntax of a simple language that consists of a <code>BEGIN</code> keyword followed by a sequence of statements separated by semicolons (the <code>SEMI</code> token), and ending with an <code>END</code> keyword. The <code>PrintStmt</code> rule defines a statement that consists of a <code>PRINT</code> keyword followed by a string literal.</p><p>There are no restrictions on the names of the rules or the tokens, but here I'm adopting the following convention:</p><ul><li>Rule names are in <code>PascalCase</code> (those are intermediate nodes in the AST)</li><li>Token names are in <code>UPPER_SNAKE_CASE</code> (those are leaf nodes in the AST)</li></ul><h2 id="generating-the-parser" tabindex="-1"><a class="header-anchor" href="#generating-the-parser"><span>Generating the Parser</span></a></h2><p>To generate the parser, we can right-click on the grammar file and select the <strong>Generate Parser Code</strong> action. This will generate the parser and lexer classes under the <code>src/main/gen</code> directory (as opposed to the <code>src/main/kotlin</code> directory where we have our handwritten code). By default, Grammar-Kit generates the following set of classes:</p><div class="language-tree line-numbers-mode" data-highlighter="prismjs" data-ext="tree" data-title="tree"><pre><code><span class="line">src/main/gen</span> 12<span class="line">âââ generated</span> 13<span class="line"> âââ psi</span> 14<span class="line"> â âââ impl</span> 15<span class="line"> â â âââ MyStmtImpl.java</span> 16<span class="line"> â â âââ PrintStmtImpl.java</span> 17<span class="line"> â âââ MyStmt.java</span> 18<span class="line"> â âââ PrintStmt.java</span> 19<span class="line"> â âââ Visitor.java</span> 20<span class="line"> âââ MyParser.java</span> 21<span class="line"> âââ GeneratedTypes.java</span> 22<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>For PSI, Grammar-Kit generates pairs of interfaces and implementation classes for each rule in the grammar. For example, the <code>PrintStmt</code> rule will have an interface <code>PrintStmt</code> and an implementation class <code>PrintStmtImpl</code>. The <code>Visitor</code> interface is used to traverse the PSI tree, and the <code>GeneratedTypes</code> class contains the token and rule element types (extending <code>IElementType</code>). It also contains an inner <code>Factory</code> class with a single method <code>createElement</code> that creates the appropriate PSI element for a given AST node (used by the <code>ParserDefinition.createElement()</code> method to create PSI elements).</p><p>We can also ask Grammar-Kit to generate a lexer for us by right-clicking on the grammar file and selecting the <strong>Generate JFlex Lexer</strong> action. This will generate the lexer specification file, but we have to select the location where we want to save it - let's say we save it under <code>src/main/gen/generated</code> as <code>MyLexer.flex</code> (alongside the generated parser and PSI). Now we can right-click on the lexer file and select the <strong>Run JFlex Generator</strong> action, which will generate the lexer class <code>MyLexer</code> under the <code>src/main/gen/generated</code> directory. So, in addition to the above files, we will have:</p><div class="language-tree line-numbers-mode" data-highlighter="prismjs" data-ext="tree" data-title="tree"><pre><code><span class="line">src/main/gen</span> 23<span class="line">âââ generated</span> 24<span class="line"> âââ ...</span> 25<span class="line"> âââ MyLexer.flex</span> 26<span class="line"> âââ MyLexer.java</span> 27<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>While generating the lexer is convenient, it is not flexible enough for complex languages that require custom lexing rules (like Nim). So, we will keep our handwritten lexer and tokens, and tell Grammar-Kit to use them instead.</p><p>We also won't need a visitor class for now, so we'll tell Grammar-Kit not to generate it as well.</p><h2 id="a-simple-nim-bnf" tabindex="-1"><a class="header-anchor" href="#a-simple-nim-bnf"><span>A Simple Nim BNF</span></a></h2><p>Let's create a BNF file for Nim that would parse the <code>echo "hello, world"</code> statement that we used in the previous sections.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">{</span></span> 28<span class="line"> generate=<span class="token operator">[</span>tokens=<span class="token string">"no"</span> visitor=<span class="token string">"no"</span><span class="token operator">]</span></span> 29<span class="line"></span> 30<span class="line"> parserClass=<span class="token string">"khaledh.nimjet.parser.NimParser"</span></span> 31<span class="line"> parserImports=<span class="token string">"static khaledh.nimjet.lexer.NimToken.*"</span></span> 32<span class="line"></span> 33<span class="line"> elementTypeClass=<span class="token string">"khaledh.nimjet.parser.NimElementType"</span></span> 34<span class="line"> elementTypeHolderClass=<span class="token string">"khaledh.nimjet.parser.NimElement"</span></span> 35<span class="line"></span> 36<span class="line"> psiPackage=<span class="token string">"khaledh.nimjet.psi"</span></span> 37<span class="line"> psiImplPackage=<span class="token string">"khaledh.nimjet.psi.impl"</span></span> 38<span class="line"><span class="token operator">}</span></span> 39<span class="line"></span> 40<span class="line">
40Root <span class="token operator">::=</span> !<<span class="token rule"><span class="token punctuation"><</span>eof<span class="token punctuation">></span></span>> NimStmt</span> 41<span class="line">NimStmt <span class="token operator">::=</span> IDENTIFIER STRING_LIT</span> 42<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>In the attributes section, we specified:</p><ul><li>that we don't want Grammar-Kit to generate tokens or a visitor class</li><li>the fully qualified name of the <code>NimParser</code> class that will be generated</li><li>the parser imports that we need, which are the token types from our lexer</li><li>the fully qualified name of the element type class <code>NimElementType</code> that we created previously; Grammar-Kit will use this class to create instances of the AST element types</li><li>the class that will hold the instances of the element types: <code>NimElement</code></li><li>the package names for the PSI and PSI implementation classes</li></ul><p>In the grammar section, we defined the syntax of a simple Nim file that consists of a single statement, which is an identifier followed by a string literal.</p><p>Notice the <code>!<<eof>></code> syntax in the <code>Root</code> rule. The <code><<...>></code> syntax is used to invoke an external rule defined in parser, which in this case is the built-in <code>eof</code> rule that matches the end of the file. The <code>!</code> operator negates the rule, so having <code>!<<eof>></code> at the beginning of the <code>Root</code> rule won't match an empty file. This prevents the parser from generating an error when the file is empty.</p><h2 id="generating-the-nim-parser" tabindex="-1"><a class="header-anchor" href="#generating-the-nim-parser"><span>Generating the Nim Parser</span></a></h2><p>Now we can right-click on the Nim BNF file and select the <strong>Generate Parser Code</strong> action to generate the parser classes. We will end up with the following set of classes:</p><div class="language-tree line-numbers-mode" data-highlighter="prismjs" data-ext="tree" data-title="tree"><pre><code><span class="line">src/main/gen</span> 43<span class="line">âââ khaledh/nimjet</span> 44<span class="line"> âââ parser</span> 45<span class="line"> â âââ NimElement.java</span> 46<span class="line"> â âââ NimParser.java</span> 47<span class="line"> âââ psi</span> 48<span class="line"> âââ impl</span> 49<span class="line"> â âââ NimStmtImpl.java</span> 50<span class="line"> âââ NimStmt.java</span> 51<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Our handwritten part of the parser should be structured as follows, under the <code>src/main/kotlin</code> directory:</p><div class="language-tree line-numbers-mode" data-highlighter="prismjs" data-ext="tree" data-title="tree"><pre><code><span class="line">src/main/kotlin</span> 52<span class="line">âââ khaledh/nimjet</span> 53<span class="line"> ...</span> 54<span class="line"> âââ parser</span> 55<span class="line"> â âââ Nim.bnf</span> 56<span class="line"> â âââ NimElementType.kt</span> 57<span class="line"> â âââ NimParserDefinition.kt</span> 58<span class="line"> âââ psi</span> 59<span class="line"> âââ NimFile.kt </span> 60<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>And to complete the organization, we should have the lexer and the various language plugin classes also tidied up as follows:</p><div class="language-tree line-numbers-mode" data-highlighter="prismjs" data-ext="tree" data-title="tree"><pre><code><span class="line">src/main/kotlin</span> 61<span class="line">âââ khaledh/nimjet</span> 62<span class="line"> âââ lang</span> 63<span class="line"> â âââ NimFileType.kt</span> 64<span class="line"> â âââ NimIcons.kt</span> 65<span class="line"> â âââ NimLanguage.kt</span> 66<span class="line">
66 âââ lexer</span> 67<span class="line"> â âââ Nim.flex</span> 68<span class="line"> â âââ NimLexerAdapter.java</span> 69<span class="line"> â âââ NimToken.kt</span> 70<span class="line"> â âââ NimTokenType.kt</span> 71<span class="line"> âââ parser</span> 72<span class="line"> â ...</span> 73<span class="line"> âââ psi</span> 74<span class="line"> ...</span> 75<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>If we test the plugin now, we should have the same functionality as before, but with the parser generated by Grammar-Kit in this case. As a final step, let's automate generating the parser using the Gradle build script, as we did with the lexer.</p><div class="language-kts line-numbers-mode" data-highlighter="prismjs" data-ext="kts" data-title="kts"><pre><code><span class="line"><span class="token comment">// build.gradle.kts</span></span> 76<span class="line"><span class="token operator">..</span><span class="token punctuation">.</span></span> 77<span class="line"></span> 78<span class="line">tasks <span class="token punctuation">{</span></span> 79<span class="line"> generateLexer <span class="token punctuation">{</span></span> 80<span class="line"> <span class="token operator">..</span><span class="token punctuation">.</span></span> 81<span class="line"> <span class="token punctuation">}</span></span> 82<span class="line"></span> 83<span class="line highlighted"> generateParser <span class="token punctuation">{</span></span> 84<span class="line highlighted"> sourceFile <span class="token operator">=</span> <span class="token function">file</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"src/main/kotlin/khaledh/nimjet/parser/Nim.bnf"</span></span><span class="token punctuation">)</span></span> 85<span class="line highlighted"> targetRootOutputDir <span class="token operator">=</span> <span class="token function">file</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"src/main/gen"</span></span><span class="token punctuation">)</span></span> 86<span class="line highlighted"> pathToParser <span class="token operator">=</span> <span class="token string-literal singleline"><span class="token string">"khaledh/nimjet/parser/NimParser.java"</span></span></span> 87<span class="line highlighted"> pathToPsiRoot <span class="token operator">=</span> <span class="token string-literal singleline"><span class="token string">"khaledh/nimjet/psi"</span></span></span> 88<span class="line highlighted"> purgeOldFiles <span class="token operator">=</span> <span class="token boolean">true</span></span> 89<span class="line highlighted"> <span class="token punctuation">}</span></span> 90<span class="line"></span> 91<span class="line"> compileJava <span class="token punctuation">{</span></span> 92<span class="line"> <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateLexer<span class="token punctuation">)</span></span> 93<span class="line highlighted"> <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateParser<span class="token punctuation">)</span></span> 94<span class="line"> <span class="token punctuation">}</span></span> 95<span class="line"> compileKotlin <span class="token punctuation">{</span></span> 96<span class="line">
96 <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateLexer<span class="token punctuation">)</span></span> 97<span class="line highlighted"> <span class="token function">dependsOn</span><span class="token punctuation">(</span>generateParser<span class="token punctuation">)</span></span> 98<span class="line"> <span class="token punctuation">}</span></span> 99<span class="line"><span class="token punctuation">}</span></span> 100<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>This should take care of generating the parser whenever we build the project, so we don't have to manually run the Grammar-Kit action every time we change the BNF file (unless we need to inspect the generated code, of course).</p>`,33)]))}const c=n(t,[["render",l],["__file","07-grammarkit.html.vue"]]),o=JSON.parse('{"path":"/nimjet/07-grammarkit.html","title":"Grammar-Kit","lang":"en-US","frontmatter":{},"headers":[{"level":2,"title":"The BNF File","slug":"the-bnf-file","link":"#the-bnf-file","children":[]},{"level":2,"title":"Generating the Parser","slug":"generating-the-parser","link":"#generating-the-parser","children":[]},{"level":2,"title":"A Simple Nim BNF","slug":"a-simple-nim-bnf","link":"#a-simple-nim-bnf","children":[]},{"level":2,"title":"Generating the Nim Parser","slug":"generating-the-nim-parser","link":"#generating-the-nim-parser","children":[]}],"git":{"updatedTime":1727020961000},"filePathRelative":"nimjet/07-grammarkit.md","excerpt":"\\n<p>Grammar-Kit is a JetBrains plugin for generating a parser (and a lexer, if needed)\\nfrom a grammar specification file that uses a BNF-like syntax. Not only does it\\ngenerate the parser, but it also generates the PSI classes of the language. This is a\\nhuge time-saver, as it allows us to focus on the language syntax and semantics rather\\nthan the parsing details.</p>"}');export{c as comp,o as data};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.