1import{_ as s,c as a,e,o as p}from"./app-CjUkjRdW.js";const t="/assets/expression-DDdG0MZY.png",o="/assets/sigil-like-op-ekiKrrib.png",l="/assets/parens-expression-CP2dxRNQ.png",i="/assets/right-assoc-op-Chs32Fow.png",c="/assets/expression-indentation-BpwsE5F5.png",r={};function d(u,n){return p(),a("div",null,n[0]||(n[0]=[e(`<h1 id="expressions" tabindex="-1"><a class="header-anchor" href="#expressions"><span>Expressions</span></a></h1><p>Now that we have support for identifiers and literals, we can start adding support for expressions. We cannot talk about expressions without talking about operators and operator precedence. Nim's approach to operators is unique, as it treats operators as regular procedures. This means that we can define our own operators using any combination of operator symbols. There are three types of operators:</p><ul><li>Unary operators: operators that take one argument and can be written in prefix notation (e.g., <code>-a</code>).</li><li>Binary operators: operators that take two arguments and can be written in infix notation (e.g., <code>a + b</code>).</li><li>Operators that take more than two arguments, which can only be used as a proc call (e.g. <code>\`*+\`(a, b, c)</code>).</li></ul><p>Examples:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token keyword">let</span></span> 2<span class="line"> a <span class="token operator">=</span> <span class="token number">40</span> <span class="token comment"># literal</span></span> 3<span class="line"> b <span class="token operator">=</span> a <span class="token comment"># identifier</span></span> 4<span class="line"> c <span class="token operator">=</span> a <span class="token operator">+</span> <span class="token number">2</span> <span class="token comment"># binary expression</span></span> 5<span class="line"> d <span class="token operator">=</span> <span class="token operator">-</span>c <span class="token comment"># unary expression</span></span> 6<span class="line"> e <span class="token operator">=</span> <span class="token number">2</span> <span class="token operator">+</span> <span class="token number">4</span> <span class="token operator">*</span> <span class="token number">10</span> <span class="token comment"># operator precedence (* has higher precedence than +)</span></span> 7<span class="line"> f <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token number">2</span> <span class="token operator">+</span> <span class="token number">4</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">10</span> <span class="token comment"># parentheses can be used to change precedence</span></span> 8<span class="line"> g <span class="token operator">=</span> true <span class="token comment"># boolean literal</span></span> 9<span class="line"> h <span class="token operator">=</span> <span class="token operator">not</span> g <span class="token comment"># unary boolean expression</span></span> 10<span class="line"> i <span class="token operator">=</span> g <span class="token operator">and</span> h <span class="token comment"># binary boolean expression</span></span> 11<span class="line"> j <span class="token operator">=</span> i <span class="token function">or</span> <span class="token punctuation">(</span>e <span class="token operator"><</span> f<span class="token punctuation">)</span> <span class="token comment"># boolean expression using relational operators</span></span> 12<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>In addition to typical expressions like arithmetic and logical operations, Nim supports block expressions, which are expressions that contain a block of code, where the value of the block is the value of the last expression in the block.</p><p>Examples:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token keyword">let</span></span> 13<span class="line"> a <span class="token operator">=</span> <span class="token keyword">block</span><span class="token operator">:</span>
13 <span class="token comment"># block expression</span></span> 14<span class="line"> echo <span class="token string">"Hello"</span></span> 15<span class="line"> <span class="token number">42</span></span> 16<span class="line"></span> 17<span class="line"> b <span class="token operator">=</span> <span class="token keyword">if</span> a <span class="token operator">></span> <span class="token number">50</span><span class="token operator">:</span> a <span class="token keyword">else</span><span class="token operator">:</span> a <span class="token operator">+</span> <span class="token number">10</span> <span class="token comment"># if expression</span></span> 18<span class="line"></span> 19<span class="line"> c <span class="token operator">=</span> <span class="token keyword">case</span> a<span class="token operator">:</span> <span class="token comment"># case expression</span></span> 20<span class="line"> <span class="token operator">of</span> <span class="token number">0</span><span class="token operator">:</span> <span class="token string">"zero"</span></span> 21<span class="line"> <span class="token operator">of</span> <span class="token number">42</span><span class="token operator">:</span> <span class="token string">"forty-two"</span></span> 22<span class="line"> <span class="token keyword">else</span><span class="token operator">:</span> <span class="token string">"not forty-two"</span></span> 23<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>We'll come back to block expressions later, but for now, let'
23s start with the basics.</p><h2 id="nim-operators" tabindex="-1"><a class="header-anchor" href="#nim-operators"><span>Nim Operators</span></a></h2><p>Operators in Nim can be defined using any combination of the following operator symbols:</p><div class="language-text line-numbers-mode" data-highlighter="prismjs" data-ext="text" data-title="text"><pre><code><span class="line">= + - * / < ></span> 24<span class="line">@ $ ~ & % |</span> 25<span class="line">! ? ^ . : \\</span> 26<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>An exception is made for the following set since these are used for other purposes in the language:</p><ul><li><code>.</code> (member access, method call syntax)</li><li><code>=</code> (assignment, initialization)</li><li><code>:</code> (type declaration, block start)</li><li><code>::</code> (seems to be reserved for future use)</li><li><code>..</code> (slice or range operator)</li></ul><p>There's also a predefined set of keyword operators:</p><div class="language-text line-numbers-mode" data-highlighter="prismjs" data-ext="text" data-title="text"><pre><code><span class="line">and or not xor shl shr div mod in notin is isnot of as from</span> 27<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div></div></div><p>Special patterns of operators affect the operator precedence:</p><ul><li>operators that start with a <code>.</code> are called dot-like operators</li><li>operators that start with a <code>@</code> are called sigil-like operators</li><li>operators that end with <code>-></code>, <code>~></code> or <code>=></code> are called arrow-like operators</li><li>operators that end with <code>=</code> and their first character is not one of <code><</code>, <code>></code>, <code>!</code>, <code>=</code>, <code>~</code>, <code>?</code> are called assignment operators</li></ul><p>Here's a modified version of the operator precedence table from the Nim manual:</p><div style="font-size:85%;"><table><thead><tr><th>Precedence<br>Level</th><th>Operators</th><th>First<br>character</th><th>Trailing<br>character(s)</th><th>Terminal<br>symbol</th></tr></thead><tbody><tr><td>10 (highest)</td><td></td><td><code>$</code> <code>^</code></td><td></td><td>OP10</td></tr><tr><td>9</td><td><code>*</code> <code>/</code> <code>%</code> <code>div</code> <code>mod</code> <code>shl</code> <code>shr</code></td><td><code>*</code> <code>/</code> <code>%</code> <code>\\</code></td><td></td><td>OP9</td></tr><tr><td>8</td><td><code>+</code> <code>-</code></td><td><code>+</code> <code>-</code>, <code>|</code>, <code>~</code></td><td></td><td>OP8</td></tr><tr><td>7</td><td><code>&</code></td><td><code>&</code></td><td></td><td>OP7</td></tr><tr><td>6</td><td><em>dot-like operator</em> (e.g. <code>.?</code>) <code>..</code></td><td><code>.</code></td><td></td><td>OP6</td></tr><tr><td>5</td><td><code>==</code> <code><=</code> <code><</code> <code>>=</code> <code>></code> <code>!=</code><br><code>in</code> <code>notin</code> <code>is</code> <code>isnot</code> <code>not</code> <code>of</code> <code>as</code> <code>from</code></td><td><code>=</code> <code><</code> <code>></code> <code>!</code></td><td></td><td>OP5</td></tr><tr><td>4</td><td><code>and</code></td><td></td><td></td><td>OP4</td></tr><tr><td>3</td><td><code>or</code> <code>xor</code></td><td></td><td></td><td>OP3</td></tr><tr><td>2</td><td></td><td><code>@</code> <code>:</code> <code>?</code></td><td></td><td>OP2</td></tr><tr><td>1</td><td><em>assignment operator</em> (e.g. <code>+=</code>, <code>*=</code>)</td><td>not one of:<br><code><</code> <code>></code> <code>!</code> <code>=</code> <code>~</code> <code>?</code></td><td><code>=</code></td><td>OP1</td></tr><tr><td>0 (lowest)</td><td><em>arrow-like operator</em> (e.g. <code>-></code>, <code>=></code>)</td><td></td><td><code>-></code> <code>~></code> <code>=></code></td><td>OP0</td></tr></tbody></table></div><h2 id="lexing-operators" tabindex="-1"><a class="header-anchor" href="#lexing-operators"><span>Lexing Operators</span></a></h2><p>Let's apply the above operator rules to recognize operators. We'll classify them into the 11 precedence levels mentioned above, from <code>OP0</code> to <code>OP10</code>. There's also a couple of things we need to address:</p><ul><li>we'll need to make sure that the five exceptions mentioned above are handled separately, since they cannot be combined with other operator symbols</li><li>we'll need to consider the assignment operators when matching certain cases, since it changes the precedence level of the operator to <code>OP1</code></li></ul><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token comment">// src/main/kotlin/khaledh/nimjet/lexer/Nim.flex</span></span> 28<span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 29<span class="line"></span> 30<span class="line"><span class="token punctuation">{</span><span class="token operator">%</span></span> 31<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 32<span class="line"></span> 33<span class="line"> <span class="token keyword">private</span> <span class="token class-name">IElementType</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">IElementType</span> type<span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 34<span class="line"> <span class="token comment">// if the last character is '=', then it's an assignment operator</span></span> 35<span class="line"> <span class="token keyword">return</span> <span class="token function">yycharat</span><span class="token punctuation">(</span><span class="token function">yylength</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token char">'='</span> <span class="token operator">?</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP1</span> <span class="token operator">:</span> type<span class="token punctuation">;</span></span> 36<span class="line"> <span class="token punctuation">}</span></span> 37<span class="line"><span class="token punctuation">}</span><span class="token operator">%</span></span> 38<span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 39<span class="line"></span> 40<span class="line"><span class="token class-name">Opr</span>
40 <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token operator">=</span><span class="token operator">+</span>\\<span class="token operator">-</span><span class="token operator">*</span><span class="token operator">/</span><span class="token generics"><span class="token punctuation"><</span><span class="token punctuation">></span></span>@$<span class="token operator">~</span><span class="token operator">&</span><span class="token operator">%</span><span class="token operator">|</span><span class="token operator">!</span><span class="token operator">?</span>\\<span class="token operator">^</span>\\<span class="token punctuation">.</span><span class="token punctuation">:</span>\\\\<span class="token punctuation">]</span></span> 41<span class="line"><span class="token class-name">NotOpr</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token operator">^=</span><span class="token operator">+</span>\\<span class="token operator">-</span><span class="token operator">*</span><span class="token operator">/</span><span class="token generics"><span class="token punctuation"><</span><span class="token punctuation">></span></span>@$<span class="token operator">~</span><span class="token operator">&</span><span class="token operator">%</span><span class="token operator">|</span><span class="token operator">!</span><span class="token operator">?</span>\\<span class="token operator">^</span>\\<span class="token punctuation">.</span><span class="token punctuation">:</span>\\\\<span class="token punctuation">]</span></span> 42<span class="line"></span> 43<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 44<span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 45<span class="line"></span> 46<span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 47<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 48<span class="line"></span> 49<span class="line"> <span class="token comment">// special tokens: \`=\`, ':', '::', '.', '..'</span></span> 50<span class="line"> <span class="token string">"="</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">EQ</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 51<span class="line"> <span class="token string">":"</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COLON</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 52<span class="line"> <span class="token string">"::"</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COLONCOLON</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 53<span class="line"> <span class="token string">"."</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span>
53 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">DOT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 54<span class="line"> <span class="token string">".."</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">DOTDOT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 55<span class="line"></span> 56<span class="line"> <span class="token comment">// operators have 11 precendence levels (0 to 10)</span></span> 57<span class="line"> <span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span><span class="token operator">?</span><span class="token punctuation">[</span>\\<span class="token operator">-</span><span class="token operator">~</span><span class="token operator">=</span><span class="token punctuation">]</span><span class="token string">">"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP0</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 58<span class="line"> <span class="token punctuation">[</span>@<span class="token operator">:</span>\\<span class="token operator">?</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP2</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 59<span class="line"> <span class="token punctuation">[</span><span class="token operator">=</span><span class="token generics"><span class="token punctuation"><</span><span class="token punctuation">></span></span><span class="token operator">!</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 60<span class="line"> \\<span class="token punctuation">.</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">+</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP6</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 61<span class="line"> \\<span class="token operator">&</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span>
61 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP7</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 62<span class="line"> \\<span class="token operator">~</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP8</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 63<span class="line"> <span class="token punctuation">[</span><span class="token operator">+</span>\\<span class="token operator">-</span><span class="token operator">|</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP8</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 64<span class="line"> <span class="token punctuation">[</span><span class="token operator">*</span><span class="token operator">%</span><span class="token operator">/</span>\\\\<span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP9</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 65<span class="line"> <span class="token punctuation">[</span>$\\<span class="token operator">^</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP10</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 66<span class="line"></span> 67<span class="line"> <span class="token comment">// keyword operators</span></span> 68<span class="line"> <span class="token string">"or"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP3</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 69<span class="line"> <span class="token string">"xor"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP3</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 70<span class="line"> <span class="token string">"and"</span>
70 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP4</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 71<span class="line"> <span class="token string">"in"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 72<span class="line"> <span class="token string">"notin"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 73<span class="line"> <span class="token string">"is"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 74<span class="line"> <span class="token string">"isnot"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 75<span class="line"> <span class="token string">"not"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 76<span class="line"> <span class="token string">"of"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 77<span class="line"> <span class="token string">"as"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 78<span class="line"> <span class="token string">"from"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP5</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 79<span class="line"> <span class="token string">"div"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP9</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 80<span class="line"> <span class="token string">"mod"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP9</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 81<span class="line"> <span class="token string">"shl"</span>
81 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP9</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 82<span class="line"> <span class="token string">"shr"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP9</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 83<span class="line"></span> 84<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 85<span class="line"><span class="token punctuation">}</span></span> 86<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>The <code>Opr</code> regex matches any operator symbol, while <code>NotOpr</code> does the opposite. We use the <code>NotOpr</code> regex in first set of rules (using the lookahead <code>/</code> operator), the five special cases, to ensure that these tokens are not followed by another operator symbol (which would make them part of a different operator).</p><p>In the second set of rules, we use the <code>Opr</code> regex to match the different kinds of operators based on leading and/or trailing symbols, and return their precedence level token. Some of these rules call the <code>considerAssign</code> utility function to check if the last character is an equals sign, which makes the operator an assignment operator (thus returning <code>OP1</code> instead of the original precedence level). Notice that we split the case of <code>OP8</code> into two rules: the one with a <code>~</code> prefix shouldn't consider assignment operators, while the one with the <code>+</code>, <code>-</code>, or <code>|</code> prefixes should (as per the Nim manual).</p><p>In the third set of rules, we match the predefined keyword operators and return their corresponding precedence level.</p><p>We have a bit of a problem though. The <code>..</code> operator should be recognized as an <code>OP6</code> token, since it's an operator with that precedence level. However, it's also going to be used in other places in the grammar. Since we can't use two different tokens for the same lexeme, we'll have to handle this in the parser. Instead of returning <code>OP6</code> from the lexer, we'll define an <code>OP6</code> rule in the grammar that matches any token from that precedence level. The only other lexer rule with that precedence level is the dot-like operator rule, so we'll give it a different token name, <code>DOT_LIKE</code>. Then we can combine <code>DOTDOT</code> and <code>DOT_LIKE</code> in the parser under the <code>OP6</code> rule.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 87<span class="line"></span> 88<span class="line"> <span class="token comment">// turned into OP6 in the grammar</span></span> 89<span class="line"> <span class="token string">".."</span> <span class="token operator">/</span> <span class="token punctuation">{</span><span class="token class-name">NotOpr</span><span class="token punctuation">}</span>
89 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">DOTDOT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 90<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 91<span class="line"></span> 92<span class="line"> <span class="token comment">// turned into OP6 in the grammar</span></span> 93<span class="line"> \\<span class="token punctuation">.</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">+</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">DOT_LIKE</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 94<span class="line"><span class="token punctuation">}</span></span> 95<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Now, let's turn our attention to the grammar.</p><h2 id="parsing-expressions" tabindex="-1"><a class="header-anchor" href="#parsing-expressions"><span>Parsing Expressions</span></a></h2><p>Parsing expressions in recursive descent parsers usually requires implementing a function for each precedence level, which delegates to the next higher precedence level to parse its operands, and keeps parsing until it encounters an operator with a lower precedence level. This has a couple of issues: first, the grammar must not be left-recursive, otherwise the parser will go into an infinite loop; and second, recursive implementation can be slow for deeply nested expressions (not to mention the risk of stack overflow).</p><p>Fortunately, Grammar-Kit has built-in support for <a href="https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md#24-compact-expression-parsing-with-priorities" target="_blank" rel="noopener noreferrer">parsing expressions</a> using an optimized approach that avoids these issues. All we need to do is define our expression rules in the order of precedence levels (from lowest to highest), and have each rule extend the root expression rule.</p><p>Let's start by adding the new expression rules to the grammar.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">{</span></span> 96<span class="line"> <span class="token operator">...</span></span> 97<span class="line"> extends<span class="token operator">(</span><span class="token string">".+Expr"</span><span class="token operator">)</span>=SimpleExpr</span> 98<span class="line"><span class="token operator">}</span></span> 99<span class="line"><span class="token operator">...</span></span> 100<span class="line"></span> 101<span class="line">// Expressions</span> 102<span class="line"></span> 103<span class="line">Expr <span class="token operator">::=</span> SimpleExpr</span> 104<span class="line"></span> 105<span class="line">SimpleExpr <span class="token operator">::=</span> Op0Expr</span> 106<span class="line"> <span class="token operator">|</span> Op1Expr</span> 107<span class="line"> <span class="token operator">|</span> Op2Expr</span> 108<span class="line"> <span class="token operator">|</span> Op3Expr</span> 109<span class="line">
109 <span class="token operator">|</span> Op4Expr</span> 110<span class="line"> <span class="token operator">|</span> Op5Expr</span> 111<span class="line"> <span class="token operator">|</span> Op6Expr</span> 112<span class="line"> <span class="token operator">|</span> Op7Expr</span> 113<span class="line"> <span class="token operator">|</span> Op8Expr</span> 114<span class="line"> <span class="token operator">|</span> Op9Expr</span> 115<span class="line"> <span class="token operator">|</span> Op10Expr</span> 116<span class="line"> <span class="token operator">|</span> PrimaryExpr</span> 117<span class="line"></span> 118<span class="line">Op0Expr <span class="token operator">::=</span> SimpleExpr OP0 SimpleExpr</span> 119<span class="line">Op1Expr <span class="token operator">::=</span> SimpleExpr OP1 SimpleExpr</span> 120<span class="line">Op2Expr <span class="token operator">::=</span> SimpleExpr OP2 SimpleExpr</span> 121<span class="line">Op3Expr <span class="token operator">::=</span> SimpleExpr OP3 SimpleExpr</span> 122<span class="line">Op4Expr <span class="token operator">::=</span> SimpleExpr OP4 SimpleExpr</span> 123<span class="line">Op5Expr <span class="token operator">::=</span> SimpleExpr OP5 SimpleExpr</span> 124<span class="line">Op6Expr <span class="token operator">::=</span> SimpleExpr OP6 SimpleExpr</span> 125<span class="line">Op7Expr <span class="token operator">::=</span> SimpleExpr OP7 SimpleExpr</span> 126<span class="line">Op8Expr <span class="token operator">::=</span> SimpleExpr OP8 SimpleExpr</span> 127<span class="line">Op9Expr <span class="token operator">::=</span> SimpleExpr OP9 SimpleExpr</span> 128<span class="line">Op10Expr <span class="token operator">::=</span> SimpleExpr OP10 SimpleExpr</span> 129<span class="line">PrimaryExpr <span class="token operator">::=</span> Literal <span class="token operator">|</span> IdentRef</span> 130<span class="line"></span> 131<span class="line">// Operators</span> 132<span class="line"></span> 133<span class="line">private OP6 <span class="token operator">::=</span> DOTDOT <span class="token operator">|</span> DOT_LIKE</span> 134<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>A few notes on the rules:</p><ul><li>The starting rule is <code>Expr</code>, which is the root rule for all expressions. Currently, it only includes <code>SimpleExpr</code>, but it will include block expressions later.</li><li>The <code>SimpleExpr</code> rule matches any of the 11 precedence level rules, as well as primary expressions.</li><li>Each precedence level rule is a binary expression that matches two operands and an operator of the corresponding precedence level.</li><li>Notice that we use the <code>SimpleExpr</code> rule as the operand type, which allows for recursive parsing of expressions. This leads to left recursion, which is handled by Grammar-Kit's optimized expression parsing. For this to work properly, we make all rules that end in <code>Expr</code> extend <code>SimpleExpr</code> using the <code>extends</code> directive.</li><li>The <code>PrimaryExpr</code> rule simply matches either a literal or an identifier reference (for now).</li><li>The <code>OP6</code> rule combines the <code>DOTDOT</code> and <code>DOT_LIKE</code> tokens into a single rule, as mentioned earlier.</li></ul><p>Time to test everything out.</p><p><img src="`+t+`" alt="Expression"></p><p>For testing, I used an expression that includes most of the operators and precedence levels. The result looks good, with the expression being parsed correctly. Since it's hard to see the entire expression tree from the screenshot, I have included a textual representation below.</p><div class="language-text line-numbers-mode" data-highlighter="prismjs" data-ext="text" data-title="text"><pre><code><span class="line"> âââââââââââââââââââââââ -> ââââ</span> 135<span class="line"> âââââââ or âââ â += âââ</span> 136<span class="line"> â + âââ â > âââââââââââââ 8 â @ âââââââââââ</span> 137<span class="line"> 1 â ^ â 4 ââââââââ .. ââ 9 âââââ and ââ</span> 138<span class="line"> 2 3 â .? âââ 11 ââ == ââ 12</span> 139<span class="line"> 5 â & â 10 11</span> 140<span class="line"> 6 7</span> 141<span class="line"> </span> 142<span class="line"> x = 1 + 2 ^ 3 or 4 > 5 .? 6 & 7 .. 11 -> 8 += 9 @ 10 == 11 and 12</span> 143<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><h2 id="unary-operators" tabindex="-1"><a class="header-anchor" href="#unary-operators"><span>Unary Operators</span></a></h2><p>Nim also supports unary operators, or prefix operators as they're called in the manual:</p><blockquote><p>Whether an operator is used as a prefix operator is also affected by preceding whitespace.</p></blockquote><p>So, a unary operator has leading whitespace, but no trailing whitespace; e.g., in <code>echo $foo</code> the <code>$</code> is a unary operator, while in <code>let c = a $ b</code> the <code>$</code> is a binary operator. Note that, in <code>a$ b</code> and <code>a$b</code> the <code>$</code> is also a binary operator.</p><p>While we can recognize unary operators in the lexer, this will double the number of operator tokens we have to handle in the parser. A simpler approach is to customize the parser to recognize if an operator is unary based on the surrounding whitespace.</p><p>Fortunately, Grammar-Kit supports adding custom methods to the parser as an escape hatch, which can be invoked using the <code><<method>></code> notation in the grammar. Those are called external rules. We can use this feature to call into a method that checks if an operator is unary. Let's start by adding a parser utility class that will handle this.</p><div class="language-kotlin line-numbers-mode" data-highlighter="prismjs" data-ext="kt" data-title="kt"><pre><code><span class="line"><span class="token comment">// src/main/kotlin/khaledh/nimjet/parser/NimParserUtil.kt</span></span> 144<span class="line"><span class="token operator">..</span><span class="token punctuation">.</span></span> 145<span class="line"></span> 146<span class="line"><span class="token keyword">class</span> NimParserUtil <span class="token operator">:</span> <span class="token function">GeneratedParserUtilBase</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 147<span class="line"> <span class="token keyword">companion</span> <span class="token keyword">object</span> <span class="token punctuation">{</span></span> 148<span class="line">
148 <span class="token operator">..</span><span class="token punctuation">.</span></span> 149<span class="line"></span> 150<span class="line"> <span class="token annotation builtin">@JvmStatic</span></span> 151<span class="line"> <span class="token annotation builtin">@Suppress</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"UNUSED_PARAMETER"</span></span><span class="token punctuation">)</span></span> 152<span class="line"> <span class="token keyword">fun</span> <span class="token function">isUnary</span><span class="token punctuation">(</span>builder<span class="token operator">:</span> PsiBuilder<span class="token punctuation">,</span> level<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> Boolean <span class="token punctuation">{</span></span> 153<span class="line"> <span class="token keyword">var</span> leading <span class="token operator">=</span> <span class="token punctuation">(</span></span> 154<span class="line"> builder<span class="token punctuation">.</span>currentOffset <span class="token operator">></span> <span class="token number">0</span> <span class="token operator">&&</span></span> 155<span class="line"> builder<span class="token punctuation">.</span>originalText<span class="token punctuation">[</span>builder<span class="token punctuation">.</span>currentOffset <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">isWhitespace</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 156<span class="line"> <span class="token punctuation">)</span></span> 157<span class="line"></span> 158<span class="line"> <span class="token keyword">var</span> trailing <span class="token operator">=</span> builder<span class="token punctuation">.</span>tokenText<span class="token operator">?</span><span class="token punctuation">.</span><span class="token function">let</span> <span class="token punctuation">{</span></span> 159<span class="line"> builder<span class="token punctuation">.</span>currentOffset <span class="token operator">+</span> it<span class="token punctuation">.</span>length <span class="token operator"><</span> builder<span class="token punctuation">.</span>originalText<span class="token punctuation">.</span>length <span class="token operator">&&</span></span> 160<span class="line"> builder<span class="token punctuation">.</span>originalText<span class="token punctuation">[</span>builder<span class="token punctuation">.</span>currentOffset <span class="token operator">+</span> it<span class="token punctuation">.</span>length<span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">isWhitespace</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 161<span class="line"> <span class="token punctuation">}</span> <span class="token operator">==</span> <span class="token boolean">true</span></span> 162<span class="line"></span> 163<span class="line"> <span class="token keyword">return</span> leading <span class="token operator">&&</span> <span class="token operator">!</span>trailing</span> 164<span class="line"> <span class="token punctuation">}</span></span> 165<span class="line"> <span class="token punctuation">}</span></span> 166<span class="line"><span class="token punctuation">}</span></span> 167<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Note: The <code>@Suppress("UNUSED_PARAMETER")</code> annotation is used to suppress the warning about the unused <code>level</code> parameter (this is just the parser level, which we don't need). We use the <code>builder</code> parameter to get the current token type and advance the lexer when needed.</p><p>The method should be self-explanatory. We make use of the <code>currentOffset</code> and <code>originalText</code> properties of the <code>PsiBuilder</code> to check if the character before and after the token is whitespace. We then return <code>true</code> if the operator has leading whitespace but no trailing whitespace.</p><p>Let's add the external rule to the grammar and use it to handle unary operators.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">{</span></span> 168<span class="line"> <span class="token operator">...</span> </span> 169<span class="line highlighted"> parserUtilClass=<span class="token string">"khaledh.nimjet.parser.NimParserUtil"</span></span> 170<span class="line"> <span class="token operator">...</span></span> 171<span class="line"><span class="token operator">}</span></span> 172<span class="line"><span class="token operator">...</span></span> 173<span class="line"></span> 174<span class="line">
174PrimaryExpr <span class="token operator">::=</span> Literal</span> 175<span class="line"> <span class="token operator">|</span> IdentRef</span> 176<span class="line highlighted"> <span class="token operator">|</span> PrefixExpr</span> 177<span class="line"></span> 178<span class="line highlighted">PrefixExpr <span class="token operator">::=</span> OP_UNARY PrimaryExpr</span> 179<span class="line"></span> 180<span class="line">// Operators</span> 181<span class="line"></span> 182<span class="line">private OP6 <span class="token operator">::=</span> DOTDOT <span class="token operator">|</span> DOT_LIKE</span> 183<span class="line">private OP <span class="token operator">::=</span> OP0 <span class="token operator">|</span> OP1 <span class="token operator">|</span> OP2 <span class="token operator">|</span> OP3 <span class="token operator">|</span> OP4 <span class="token operator">|</span> OP5 <span class="token operator">|</span> OP6 <span class="token operator">|</span> OP7 <span class="token operator">|</span> OP8 <span class="token operator">|</span> OP9 <span class="token operator">|</span> OP10</span> 184<span class="line highlighted">private OP_UNARY <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>is_unary<span class="token punctuation">></span></span>> OP</span> 185<span class="line"><span class="token operator">...</span></span> 186<span class="line"></span> 187<span class="line">// external rules</span> 188<span class="line"></span> 189<span class="line highlighted">external is_unary <span class="token operator">::=</span> isUnary</span> 190<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>We first tell Grammar-Kit to use the <code>NimParserUtil</code> class for the parser utilities (i.e. external rules). The external rules are declared at the end using the <code>external</code> directive by assigning the rule name to the corresponding method in the parser utility class.</p><p>We added a private <code>OP</code> rule that matches any operator, and an <code>OP_UNARY</code> rule that uses the <code><<is_unary>></code> external rule to check if the operator is unary.</p><p>The <code>PrefixExpr</code> rule matches a unary operator followed by a <code>PrimaryExpr</code>. This rule itself is a case of the <code>PrimaryExpr</code> rule, since it binds stronger than binary expressions.</p><h3 id="sigil-like-operators" tabindex="-1"><a class="header-anchor" href="#sigil-like-operators"><span>Sigil-like Operators</span></a></h3><p>Another special case for unary operators is the sigil-like operators, i.e. those that start with the <code>@</code> symbol. From the Nim manual:</p><blockquote><p>If a unary operator's first character is <code>@</code> it is a sigil-like operator which binds stronger than a <code>primarySuffix</code>: <code>@x.abc</code> is parsed as <code>(@x).abc</code> whereas <code>$x.abc</code> is parsed as <code>$(x.abc)</code>.</p></blockquote><p>We are not tackling the <code>primarySuffix</code> rule yet, but we can still handle those sigil-like operators in order to be ready for when we do. Even though the Nim manual refers to them as unary operators, in the actual implementation they are allowed to have trailing whitespace for some reason (maybe it was an oversight?).</p><p>Let's split the <code>OP2</code> token into two tokens, <code>OP2_SIGIL_LIKE</code> and <code>OP2_OTHER</code>, and add a rule in the parser to group them under an <code>OP2</code> rule.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre class="has-diff language-java"><code><span class="line diff remove"> <span class="token punctuation">[</span>@<span class="token operator">:</span>\\<span class="token operator">?</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span>
190 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP2</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 191<span class="line diff add"> @<span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP2_SIGIL_LIKE</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 192<span class="line diff add"> <span class="token punctuation">[</span><span class="token operator">:</span>\\<span class="token operator">?</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP2_OTHER</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 193<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Sigil-like operators are allowed to appear before identifier and literals. Let's update the grammar to reflect this.</p><div class="language-text line-numbers-mode" data-highlighter="prismjs" data-ext="text" data-title="text"><pre><code><span class="line">...</span> 194<span class="line"></span> 195<span class="line">Op0Expr ::= SimpleExpr OP0 <<expr_ind>> SimpleExpr</span> 196<span class="line">Op1Expr ::= SimpleExpr OP1 <<expr_ind>> SimpleExpr</span> 197<span class="line highlighted">Op2Expr ::= SimpleExpr OP2_OTHER <<expr_ind>> SimpleExpr</span> 198<span class="line">...</span> 199<span class="line"></span> 200<span class="line highlighted">PrimaryExpr ::= OP2_SIGIL_LIKE? Literal</span> 201<span class="line highlighted"> | OP2_SIGIL_LIKE? IdentRef</span> 202<span class="line"> | PrefixExpr</span> 203<span class="line"> | ParenExpr</span> 204<span class="line">...</span> 205<span class="line"></span> 206<span class="line">// Operators</span> 207<span class="line"></span> 208<span class="line highlighted">private OP2 ::= OP2_SIGIL_LIKE | OP2_OTHER</span> 209<span class="line">...</span> 210<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p><img src="`+o+`" alt="Sigil-like Operator"></p><p>The sigil-like operator is parsed correctly. It is treated as a unary operator, even if it has trailing whitespace.</p><h2 id="parentheses" tabindex="-1"><a class="header-anchor" href="#parentheses"><span>Parentheses</span></a></h2><p>Parentheses are used to group expressions and change the order of evaluation. They have higher precedence than any binary operator, and can be nested. They're usually considered a form of primary expression.</p><p>Let's first add the opening and closing parentheses tokens to the lexer.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 211<span class="line"></span> 212<span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 213<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 214<span class="line"> </span> 215<span class="line"> <span class="token string">"("</span>
215 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">LPAREN</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 216<span class="line"> <span class="token string">")"</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">RPAREN</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 217<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 218<span class="line"><span class="token punctuation">}</span></span> 219<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Now, let's add the parentheses rule to the grammar.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">...</span></span> 220<span class="line"></span> 221<span class="line">PrimaryExpr <span class="token operator">::=</span> Literal</span> 222<span class="line"> <span class="token operator">|</span> IdentRef</span> 223<span class="line"> <span class="token operator">|</span> PrefixExpr</span> 224<span class="line highlighted"> <span class="token operator">|</span> ParenExpr</span> 225<span class="line"></span> 226<span class="line">PrefixExpr <span class="token operator">::=</span> OP_UNARY PrimaryExpr</span> 227<span class="line highlighted">ParenExpr <span class="token operator">::=</span> LPAREN SimpleExpr RPAREN</span> 228<span class="line"><span class="token operator">...</span></span> 229<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p><img src="`+l+`" alt="Parentheses Expression"></p><p>Looks good. We have two nested parentheses expressions, and they are parsed with the correct precedence, overriding the default precedence of the operators.</p><h2 id="right-associativity" tabindex="-1"><a class="header-anchor" href="#right-associativity"><span>Right Associativity</span></a></h2><p>One final thing to consider is right-associative operators. In Nim, the only right-associative operator starts with the <code>^</code> symbol. From the manual:</p><blockquote><p>Binary operators whose first character is <code>^</code> are right-associative, all other binary operators are left-associative.</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token keyword">proc</span> <span class="token function">\`^/\`</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> y<span class="token operator">:</span> float<span class="token punctuation">)</span><span class="token operator">:</span> float <span class="token operator">=</span></span> 230<span class="line"><span class="token comment"># a right-associative division operator</span></span> 231<span class="line">result <span class="token operator">=</span> x <span class="token operator">/</span> y</span> 232<span class="line">echo <span class="token number">12</span> <span class="token operator">^/</span> <span class="token number">4</span> <span class="token operator">^/</span> <span class="token number">8</span> <span class="token comment"># 24.0 (4 / 8 = 0.5, then 12 / 0.5 = 24.0)</span></span> 233<span class="line">echo <span class="token number">12</span> <span class="token operator">/</span> <span class="token number">4</span> <span class="token operator">/</span> <span class="token number">8</span> <span class="token comment"># 0.375 (12 / 4 = 3.0, then 3 / 8 = 0.375)</span></span> 234<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div></blockquote><p>Fortunately, Grammar-Kit supports right-associative operators through the <code>rightAssociative</code> rule attribute. Right now, operators starting with <code>^</code> are grouped with those starting with <code>$</code> under the same precedence level, <code>OP10</code>. We'll need to update the lexer to recognize them separately, and then use each in a separate rule in the grammar.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre class="has-diff language-java"><code><span class="line diff remove"> <span class="token punctuation">[</span>$\\<span class="token operator">^</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span>
234 <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP10</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 235<span class="line diff add"> <span class="token punctuation">[</span>$<span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP10_DOLLAR</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 236<span class="line diff add"> <span class="token punctuation">[</span>\\<span class="token operator">^</span><span class="token punctuation">]</span><span class="token punctuation">{</span><span class="token class-name">Opr</span><span class="token punctuation">}</span><span class="token operator">*</span> <span class="token punctuation">{</span> <span class="token keyword">return</span> <span class="token function">considerAssign</span><span class="token punctuation">(</span><span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">OP10_CARET</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> </span> 237<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Let's update the grammar to reflect this change.</p><div class="language-text line-numbers-mode" data-highlighter="prismjs" data-ext="text" data-title="text"><pre class="has-diff language-text"><code><span class="line">...</span> 238<span class="line"></span> 239<span class="line">SimpleExpr ::= Op0Expr</span> 240<span class="line"> | Op1Expr</span> 241<span class="line"> ...</span> 242<span class="line diff remove"> | Op10Expr</span> 243<span class="line diff add"> | Op10LeftExpr</span> 244<span class="line diff add"> | Op10RightExpr</span> 245<span class="line"> | PrimaryExpr</span> 246<span class="line"></span> 247<span class="line">Op0Expr ::= SimpleExpr OP0 <<expr_ind>> SimpleExpr</span> 248<span class="line">Op1Expr ::= SimpleExpr OP1 <<expr_ind>> SimpleExpr</span> 249<span class="line">...</span> 250<span class="line diff remove">Op10Expr ::= SimpleExpr OP10 <<expr_ind>> SimpleExpr</span> 251<span class="line diff add">Op10LeftExpr ::= SimpleExpr OP10_DOLLAR <<expr_ind>> SimpleExpr</span> 252<span class="line diff add">Op10RightExpr ::= SimpleExpr OP10_CARET <<expr_ind>> SimpleExpr {rightAssociative=true}</span> 253<span class="line"></span> 254<span class="line">...</span> 255<span class="line"></span> 256<span class="line">// Operators</span> 257<span class="line"></span> 258<span class="line">private OP2 ::= OP2_SIGIL_LIKE | OP2_OTHER</span> 259<span class="line">private OP6 ::= DOTDOT | DOT_LIKE</span> 260<span class="line diff add">private OP10 ::= OP10_DOLLAR | OP10_CARET</span> 261<span class="line">private OP ::= OP0 | OP1 | OP2 | OP3 | OP4 | OP5 | OP6 | OP7 | OP8 | OP9 | OP10</span> 262<span class="line">private OP_UNARY ::= <<is_unary>> OP</span> 263<span class="line"></span> 264<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Let's see if it works.</p><p><img src="`+i+`" alt="Right Associative Operator"></p><p>The right-associative operator is parsed correctly. The expression <code>(5 + x) ^ 2 ^ 3</code> is parsed as <code>(5 + x) ^ (2 ^ 3)</code>, as expected (the second <code>^</code> operator has higher precedence than the first one).</p><h2 id="handling-indentation" tabindex="-1"><a class="header-anchor" href="#handling-indentation"><span>Handling Indentation</span></a></h2><p>Nim supports a flexible indentation format within expressions. Unlike structured indentation of code blocks (which requires every indent to have a corresponding dedent), expressions can have indents, equal indents, and dedents after any operator in an expression. For example, the above expression can be written as follows:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token keyword">let</span></span> 265<span class="line"> x <span class="token operator">=</span> <span class="token number">1</span> <span class="token operator">+</span> <span class="token number">2</span> <span class="token operator">^</span></span> 266<span class="line"> <span class="token number">3</span> <span class="token operator">or</span> <span class="token number">4</span> <span class="token operator">></span> <span class="token number">5</span> <span class="token operator">.?</span> <span class="token number">6</span> <span class="token operator">&</span></span> 267<span class="line">
267 <span class="token number">7</span> <span class="token operator">..</span> <span class="token number">11</span> <span class="token operator">-></span></span> 268<span class="line"> <span class="token number">8</span> <span class="token operator">+=</span> <span class="token number">9</span> <span class="token operator">@</span></span> 269<span class="line"> <span class="token number">10</span> <span class="token operator">==</span> <span class="token number">11</span> <span class="token operator">and</span></span> 270<span class="line"> <span class="token number">12</span></span> 271<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>This poses a challenge for us, since there's no pattern for the indentation inside an expression that we can capture through a grammar rule. A naive approach would be to update the expression rules to allow any number of indentation tokens (including none) to appear after an operator. The issue with this approach is that, after an expression ends, we can't tell how many dedent tokens we need to skip to get back to the correct indentation level. We need a way to track the indentation level in the parser.</p><p>Let's add a few external rules to the parser utility class to handle this.</p><div class="language-kotlin line-numbers-mode" data-highlighter="prismjs" data-ext="kt" data-title="kt"><pre><code><span class="line"><span class="token operator">..</span><span class="token punctuation">.</span></span> 272<span class="line"></span> 273<span class="line"><span class="token keyword">class</span> NimParserUtil <span class="token operator">:</span> <span class="token function">GeneratedParserUtilBase</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 274<span class="line"></span> 275<span class="line"> <span class="token keyword">companion</span> <span class="token keyword">object</span> <span class="token punctuation">{</span></span> 276<span class="line"> <span class="token keyword">private</span> <span class="token keyword">var</span> exprIndentLevel <span class="token operator">=</span> <span class="token number">0</span></span> 277<span class="line"></span> 278<span class="line"> <span class="token annotation builtin">@JvmStatic</span></span> 279<span class="line"> <span class="token annotation builtin">@Suppress</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"UNUSED_PARAMETER"</span></span><span class="token punctuation">)</span></span> 280<span class="line"> <span class="token keyword">fun</span> <span class="token function">exprIndentStart</span><span class="token punctuation">(</span>builder<span class="token operator">:</span> PsiBuilder<span class="token punctuation">,</span> level<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> Boolean <span class="token punctuation">{</span></span> 281<span class="line"> exprIndentLevel <span class="token operator">=</span> <span class="token number">0</span></span> 282<span class="line"> <span class="token keyword">return</span> <span class="token boolean">true</span></span> 283<span class="line"> <span class="token punctuation">}</span></span> 284<span class="line"></span> 285<span class="line"> <span class="token annotation builtin">@JvmStatic</span></span> 286<span class="line"> <span class="token annotation builtin">@Suppress</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"UNUSED_PARAMETER"</span></span><span class="token punctuation">)</span></span> 287<span class="line"> <span class="token keyword">fun</span> <span class="token function">exprIndent</span><span class="token punctuation">(</span>builder<span class="token operator">:</span> PsiBuilder<span class="token punctuation">,</span>
287 level<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> Boolean <span class="token punctuation">{</span></span> 288<span class="line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>builder<span class="token punctuation">.</span>tokenType <span class="token operator">==</span> NimToken<span class="token punctuation">.</span>IND<span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 289<span class="line"> exprIndentLevel<span class="token operator">++</span></span> 290<span class="line"> builder<span class="token punctuation">.</span><span class="token function">advanceLexer</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 291<span class="line"> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span></span> 292<span class="line"> <span class="token keyword">while</span> <span class="token punctuation">(</span>builder<span class="token punctuation">.</span>tokenType <span class="token operator">==</span> NimToken<span class="token punctuation">.</span>DED <span class="token operator">&&</span> exprIndentLevel <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 293<span class="line"> exprIndentLevel<span class="token operator">--</span></span> 294<span class="line"> builder<span class="token punctuation">.</span><span class="token function">advanceLexer</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 295<span class="line"> <span class="token punctuation">}</span></span> 296<span class="line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>builder<span class="token punctuation">.</span>tokenType <span class="token operator">==</span> NimToken<span class="token punctuation">.</span>
296EQD<span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 297<span class="line"> builder<span class="token punctuation">.</span><span class="token function">advanceLexer</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 298<span class="line"> <span class="token punctuation">}</span></span> 299<span class="line"> <span class="token punctuation">}</span></span> 300<span class="line"> <span class="token keyword">return</span> <span class="token boolean">true</span></span> 301<span class="line"> <span class="token punctuation">}</span></span> 302<span class="line"></span> 303<span class="line"> <span class="token annotation builtin">@JvmStatic</span></span> 304<span class="line"> <span class="token annotation builtin">@Suppress</span><span class="token punctuation">(</span><span class="token string-literal singleline"><span class="token string">"UNUSED_PARAMETER"</span></span><span class="token punctuation">)</span></span> 305<span class="line"> <span class="token keyword">fun</span> <span class="token function">exprIndentEnd</span><span class="token punctuation">(</span>builder<span class="token operator">:</span> PsiBuilder<span class="token punctuation">,</span> level<span class="token operator">:</span> Int<span class="token punctuation">)</span><span class="token operator">:</span> Boolean <span class="token punctuation">{</span></span> 306<span class="line"> <span class="token keyword">while</span> <span class="token punctuation">(</span>builder<span class="token punctuation">.</span>tokenType <span class="token keyword">in</span> <span class="token function">setOf</span><span class="token punctuation">(</span>NimToken<span class="token punctuation">.</span>DED<span class="token punctuation">,</span> NimToken<span class="token punctuation">.</span>
306EQD<span class="token punctuation">)</span> <span class="token operator">&&</span> exprIndentLevel <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 307<span class="line"> <span class="token keyword">if</span> <span class="token punctuation">(</span>builder<span class="token punctuation">.</span>tokenType <span class="token operator">==</span> NimToken<span class="token punctuation">.</span>DED<span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 308<span class="line"> exprIndentLevel<span class="token operator">--</span></span> 309<span class="line"> <span class="token punctuation">}</span></span> 310<span class="line"> builder<span class="token punctuation">.</span><span class="token function">advanceLexer</span><span class="token punctuation">(</span><span class="token punctuation">)</span></span> 311<span class="line"> <span class="token punctuation">}</span></span> 312<span class="line"> <span class="token keyword">return</span> <span class="token boolean">true</span></span> 313<span class="line"> <span class="token punctuation">}</span></span> 314<span class="line"> </span> 315<span class="line"> <span class="token operator">..</span><span class="token punctuation">.</span></span> 316<span class="line"> <span class="token punctuation">}</span></span> 317<span class="line"></span> 318<span class="line"><span class="token punctuation">}</span></span> 319<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>We declare an <code>exprIndentLevel</code> variable to track the current indentation level in an expression. We define three methods:</p><ul><li><code>exprIndentStart</code>: resets the <code>exprIndentLevel</code> to 0 at the start of an expression</li><li><code>exprIndent</code>: has two cases: <ul><li>If the current token is <code>IND</code>, it increments the <code>exprIndentLevel</code> and advances the lexer to the next token</li><li>Otherwise, it consumes any <code>DED</code> tokens (if any) while decrementing the <code>exprIndentLevel</code> and advancing the lexer. After consuming any <code>DED</code> tokens, it checks if the next token is <code>EQD</code> and simply advances the lexer if it is. This consumes any equal indents that might appear after an operator. Remember that dedents can be followed by an equal indent if the dedents bring the indentation level to the same level as a previous indentation.</li></ul></li><li><code>exprIndentEnd</code>: similar to the second case of <code>exprIndent</code>, but it consumes any remaining <code>DED</code> tokens (if any) after the expression ends. It doesn't consume <code>EQD</code> (if any) since such a token would belong to the parent block, not the expression.</li></ul><p>The idea here is to call <code>exprIndentStart</code> at the start of an expression, <code>exprIndent</code> after any operator within the expression, and <code>exprIndentEnd</code> at the end of the expression. Let's modify the grammar to use these external rules.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">...</span></span> 320<span class="line"></span> 321<span class="line">
321Expr <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>expr_ind_start<span class="token punctuation">></span></span>> SimpleExpr <<span class="token rule"><span class="token punctuation"><</span>expr_ind_end<span class="token punctuation">></span></span>></span> 322<span class="line"><span class="token operator">...</span></span> 323<span class="line"></span> 324<span class="line">Op0Expr <span class="token operator">::=</span> SimpleExpr OP0 <<span class="token rule"><span class="token punctuation"><</span>expr_ind<span class="token punctuation">></span></span>> SimpleExpr</span> 325<span class="line">Op1Expr <span class="token operator">::=</span> SimpleExpr OP1 <<span class="token rule"><span class="token punctuation"><</span>expr_ind<span class="token punctuation">></span></span>> SimpleExpr</span> 326<span class="line"><span class="token operator">...</span></span> 327<span class="line">Op10Expr <span class="token operator">::=</span> SimpleExpr OP10 <<span class="token rule"><span class="token punctuation"><</span>expr_ind<span class="token punctuation">></span></span>> SimpleExpr</span> 328<span class="line"></span> 329<span class="line"><span class="token operator">...</span></span> 330<span class="line"></span> 331<span class="line">// external rules</span> 332<span class="line"><span class="token operator">...</span></span> 333<span class="line">external expr_ind_start <span class="token operator">::=</span> exprIndentStart</span> 334<span class="line">external expr_ind <span class="token operator">::=</span> exprIndent</span> 335<span class="line">external expr_ind_end <span class="token operator">::=</span> exprIndentEnd</span> 336<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>We then use the <code><<expr_ind_start>></code> and <code><<expr_ind_end>></code> rules at the start and end of the <code>Expr</code> rule, respectively. This ensures that the indentation level is reset at the beginning of an expression and that any leftover dedent tokens are consumed at the end.</p><p>Finally, we use the <code><<expr_ind>></code> rule after each operator in the expression rules. This takes care of tracking indentation changes within the expression.</p><p>Let's try it out.</p><p><img src="`+c+'" alt="Expression with Indentation"></p><p>It seems to work as expected. The expression is parsed correctly, and the declaration following it are also parsed correctly.</p><p>This has been a long section. We've covered a lot of ground: operator precedence, binary and unary operators (including some special cases), right-associative operators, and handling indentation within expressions. We still need to handle block expressions, but for now we've got basic expressions under our belt.</p>',99)]))}const m=s(r,[["render",d],["__file","15-expressions.html.vue"]]),v=JSON.parse(`{"path":"/nimjet/15-expressions.html","title":"Expressions","lang":"en-US","frontmatter":{},"headers":[{"level":2,"title":"Nim Operators","slug":"nim-operators","link":"#nim-operators","children":[]},{"level":2,"title":"Lexing Operators","slug":"lexing-operators","link":"#lexing-operators","children":[]},{"level":2,"title":"Parsing Expressions","slug":"parsing-expressions","link":"#parsing-expressions","children":[]},{"level":2,"title":"Unary Operators","slug":"unary-operators","link":"#unary-operators","children":[{"level":3,"title":"Sigil-like Operators","slug":"sigil-like-operators","link":"#sigil-like-operators","children":[]}]},{"level":2,"title":"Parentheses","slug":"parentheses","link":"#parentheses","children":[]},{"level":2,"title":"Right Associativity","slug":"right-associativity","link":"#right-associativity","children":[]},{"level":2,"title":"Handling Indentation","slug":"handling-indentation","link":"#handling-indentation","children":[]}
336],"git":{"updatedTime":1729346661000},"filePathRelative":"nimjet/15-expressions.md","excerpt":"\\n<p>Now that we have support for identifiers and literals, we can start adding support for\\nexpressions. We cannot talk about expressions without talking about operators and operator precedence.\\nNim's approach to operators is unique, as it treats operators as regular procedures. This\\nmeans that we can define our own operators using any combination of operator symbols.\\nThere are three types of operators:</p>"}`);export{m as comp,v as data};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.