1import{_ as s,c as a,e,o as t}from"./app-CjUkjRdW.js";const p="/assets/line-comments-B0hbeKEl.png",l="/assets/unterminated-ml-comment-error-BDfWQieW.png",o="/assets/multiline-comments-DgowVidF.png",i="/assets/doc-comments-Cm-0TRJw.png",c={};function u(r,n){return t(),a("div",null,n[0]||(n[0]=[e(`<h1 id="comments" tabindex="-1"><a class="header-anchor" href="#comments"><span>Comments</span></a></h1><p>Nim comments are similar to those in Python. They start with a <code>#</code> and continue to the end of the line. Nim also supports multi-line comments that start with <code>#[</code> and end with <code>]#</code>, which can be nested. Nim is very flexible with comments, allowing them to appear almost anywhere in the code.</p><p>Examples:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token comment"># a single-line comment</span></span> 2<span class="line"></span> 3<span class="line"><span class="token comment"># line comments can span multiple</span></span> 4<span class="line"><span class="token comment"># consecutive lines</span></span> 5<span class="line"></span> 6<span class="line"> <span class="token comment"># comments can be preceeded by whitespace</span></span> 7<span class="line"> <span class="token comment"># wihtout producing indentation errors</span></span> 8<span class="line"></span> 9<span class="line"><span class="token comment">#[ a multi-line comment on a single line ]#</span></span> 10<span class="line"></span> 11<span class="line"><span class="token comment">#[ a multi-line comment that</span></span> 12<span class="line"> spans multiple lines</span> 13<span class="line"><span class="token punctuation">]</span><span class="token comment">#</span></span> 14<span class="line"></span> 15<span class="line"><span class="token comment">#[ a multi-line comment that contains another #[ nested ]# comment ]#</span></span> 16<span class="line"></span> 17<span class="line"><span class="token keyword">let</span> msg <span class="token operator">=</span> <span class="token string">"hello"</span> <span class="token comment"># a comment at the end of a line</span></span> 18<span class="line"></span> 19<span class="line"><span class="token keyword">let</span> </span> 20<span class="line"> <span class="token comment"># a comment interspersed with declarations</span></span> 21<span class="line"> foo <span class="token operator">=</span> <span class="token string">"foo"</span> <span class="token comment"># another comment</span></span> 22<span class="line"> <span class="token comment"># yet another comment</span></span> 23<span class="line"> bar <span class="token operator">=</span> <span class="token string">"bar"</span></span> 24<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>One thing to note from the Nim <a href="https://nim-lang.org/docs/manual.html#lexical-analysis-comments" target="_blank" rel="noopener noreferrer">language manual</a>:</p><blockquote><p>Comments consist of a concatenation of comment pieces. A comment piece starts with <code>#</code> and runs until the end of the line. The end of line characters belong to the piece. If the next line only consists of a comment piece with no other tokens between it and the preceding one, it does not start a new comment:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line">i <span class="token operator">=</span> <span class="token number">0</span> <span class="token comment"># This is a single comment over multiple lines.</span></span> 25<span class="line"><span class="token comment"># The lexer merges these two pieces.</span></span> 26<span class="line"><span class="token comment"># The comment continues here.</span></span> 27<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div></blockquote><p>It'
27s not clear to me what purpose does this merging serve, as comments are ignored by the parser anyway. That said, we'll follow this behavior in our lexer.</p><p>Nim also supports documentation comments that start with <code>##</code>. Unlike regular comments, which are ignored by the parser, documentation comments are part of the syntax tree and are typically used to generate documentation and provide hints in the IDE. We'll get to documentation comments later in the section.</p><h2 id="lexing-comments" tabindex="-1"><a class="header-anchor" href="#lexing-comments"><span>Lexing Comments</span></a></h2><p>Since multiline comments can be nested, they require a dedicated state in the lexer, with a way to track the nesting level. This means that we need to treat line comments differently from multiline comments.</p><p>Typically, we'd use a regex to match line comments. However, I wasn't able to find a single regex that would allow matching merged line comments (as described in the Nim manual) but reject multiline comments. JFlex does not support negative lookahead, which makes it difficult to implement this behavior. So, I'm going to use a separate state for line comments as well, which provides more flexibility in driving the lexer behaviour.</p><p>Let's introduce a <code>L_COMMENT</code> state that will be entered when a <code>#</code> character is encountered. In this state, we'll consume the rest of the line, and any subsequent lines as long as they start with a <code>#</code> (optionally preceded by whitespace), but not <code>#[</code> (which would start a multiline comment).</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token comment">// src/main/kotlin/khaledh/nimjet/lexer/Nim.flex</span></span> 28<span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 29<span class="line"></span> 30<span class="line highlighted"><span class="token operator">%</span>
30state <span class="token constant">L_COMMENT</span></span> 31<span class="line"></span> 32<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 33<span class="line"></span> 34<span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 35<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 36<span class="line highlighted"> <span class="token string">"#"</span> <span class="token punctuation">{</span> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">L_COMMENT</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 37<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 38<span class="line"><span class="token punctuation">}</span></span> 39<span class="line"></span> 40<span class="line highlighted"><span class="token generics"><span class="token punctuation"><</span>L_COMMENT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 41<span class="line highlighted"> <span class="token punctuation">{</span><span class="token constant">EOL</span><span class="token punctuation">}</span><span class="token punctuation">[</span> \\t<span class="token punctuation">]</span><span class="token operator">*</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">^</span>#<span class="token punctuation">]</span> <span class="token operator">|</span> #<span class="token punctuation">[</span>#\\<span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">// next line is not a line comment</span></span> 42<span class="line highlighted"> <span class="token function">yypushback</span><span class="token punctuation">(</span><span class="token function">yylength</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 43<span class="line highlighted"> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">DEFAULT</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COMMENT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 44<span class="line highlighted"> <span class="token generics"><span class="token punctuation"><</span><span class="token punctuation"><</span>EOF<span class="token punctuation">></span><span class="token punctuation">></span></span> <span class="token punctuation">{</span> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">AT_EOF</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COMMENT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 45<span class="line highlighted"> <span class="token punctuation">[</span><span class="token operator">^</span><span class="token punctuation">]</span> <span class="token punctuation">{</span> <span class="token comment">/* consume all other character */</span> <span class="token punctuation">}</span></span> 46<span class="line highlighted"><span class="token punctuation">}</span></span> 47<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>The <code>L_COMMENT</code> state keeps matching lines, and only exits when it encounters a line that doesn't start with a <code>#</code> character, or one that starts with either <code>##</code> or <code>#[</code> (which would start a documentation comment or a multiline comment, respectively). The <code><<EOF>></code> rule ensures that a comment at the end of file (without a newline) is also matched.</p><p>Now that we have a way to recognize line comments, let's tell our <code>NimParserDefinition</code> to treat them as comments (so they can be ignored by the parser).</p><p>First, let's add a <code>TokenSet</code> for comments in the <code>NimTokenSets</code> interface.</p><div class="language-kotlin line-numbers-mode" data-highlighter="prismjs" data-ext="kt" data-title="kt"><pre><code><span class="line"><span class="token comment">
47// src/main/kotlin/khaledh/nimjet/lexer/NimTokenSets.kt</span></span> 48<span class="line"></span> 49<span class="line"><span class="token keyword">interface</span> NimTokenSet <span class="token punctuation">{</span></span> 50<span class="line"></span> 51<span class="line"> <span class="token keyword">companion</span> <span class="token keyword">object</span> <span class="token punctuation">{</span></span> 52<span class="line"> <span class="token operator">..</span><span class="token punctuation">.</span></span> 53<span class="line highlighted"> <span class="token annotation builtin">@JvmField</span> <span class="token keyword">val</span> COMMENTS <span class="token operator">=</span> TokenSet<span class="token punctuation">.</span><span class="token function">create</span><span class="token punctuation">(</span>NimToken<span class="token punctuation">.</span>COMMENT<span class="token punctuation">)</span></span> 54<span class="line"> <span class="token punctuation">}</span></span> 55<span class="line"><span class="token punctuation">}</span></span> 56<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Now we can update the <code>NimParserDefinition</code> to use this token set for comments.</p><div class="language-kotlin line-numbers-mode" data-highlighter="prismjs" data-ext="kt" data-title="kt"><pre><code><span class="line"><span class="token comment">// src/main/kotlin/khaledh/nimjet/parser/NimParserDefinition.kt</span></span> 57<span class="line"></span> 58<span class="line"><span class="token keyword">class</span> NimParserDefinition <span class="token operator">:</span> ParserDefinition <span class="token punctuation">{</span></span> 59<span class="line"> <span class="token operator">..</span><span class="token punctuation">.</span></span> 60<span class="line"></span> 61<span class="line"> <span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">getFileNodeType</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> IFileElementType <span class="token operator">=</span> NIM_FILE</span> 62<span class="line"> <span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">getWhitespaceTokens</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> TokenSet <span class="token operator">=</span> TokenSet<span class="token punctuation">.</span>WHITE_SPACE</span> 63<span class="line highlighted"> <span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">getCommentTokens</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> TokenSet <span class="token operator">=</span> NimTokenSet<span class="token punctuation">.</span>COMMENTS</span> 64<span class="line"> <span class="token keyword">override</span> <span class="token keyword">fun</span> <span class="token function">getStringLiteralElements</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">:</span> TokenSet <span class="token operator">=</span> NimTokenSet<span class="token punctuation">.</span>STR_LITERALS</span> 65<span class="line"> <span class="token operator">..</span><span class="token punctuation">.</span></span> 66<span class="line"><span class="token punctuation">}</span></span> 67<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Let's test this out.</p><p><img src="`+p+'" alt="Line Comments"></p><p>All seems to be working as expected! The lexer correctly recognizes line comments and merges them when they are on consecutive lines.</p><h2 id="multiline-comments" tabindex="-1"><a class="header-anchor" href="#multiline-comments"><span>Multiline Comments</span></a></h2><p>Multiline comments pose a bit of a challenge for a couple of reasons:</p><ul><li>Since they can be nested, we need to keep track of the nesting level, and only close the comment when all nested levels have been closed.</li><li>We need to handle the case where the comment is not closed properly, which would be an error.</li></ul><p>To solve the first issue, we need to introduce a variable, <code>multilineCommentLevel</code>, and increment it when we encounter an opening <code>[#</code> sequence, and decrement it when we encounter a closing <code>]#</code> sequence. We'll only switch back to the <code>DEFAULT</code> state when <code>multilineCommentLevel</code> reaches zero.</p><p>The second issue is a bit more tricky. The lexer doesn't have a way to report errors; all it can do is return a token. We've already seen this with the <code>BAD_CHARACTER</code> token, which is used to represent any character that doesn't match any lexer rule. But what does the parser do with this token? Well, it'
67s not expecting such token, so it reports an error at that location about what it was expecting.</p><p>The issue with unterminated multiline comments (as well as unterminated triple-quoted strings, as we'll see later) is that they cause all subsequent text to be treated as part of the comment. This would show up in the editor as the entire block of code following the unterminated comment being highlighted as an error. Here's what it looks like:</p><p><img src="'+l+`" alt="Unterminated Multiline Comment Error" width="600"></p><p>While this is technically correct, it's not very user-friendly. A better user experience would be to show an error at the end of the file with a more descriptive error message, like <code>"Unterminated multiline comment"</code>. Let's put this aside for now, and we'll come back to it later.</p><p>Let's introduce a new lexer state for multiline comments, <code>ML_COMMENT</code>, which would enter when it encounters a <code>#[</code>. Let's also add a new class field, <code>multilineCommentLevel</code>, to track the nesting level.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 68<span class="line"></span> 69<span class="line"><span class="token operator">%</span><span class="token punctuation">{</span></span> 70<span class="line highlighted"> <span class="token keyword">private</span> <span class="token keyword">int</span> multilineCommentLevel <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span></span> 71<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 72<span class="line"><span class="token operator">%</span><span class="token punctuation">}</span></span> 73<span class="line"></span> 74<span class="line highlighted"><span class="token operator">%</span>
74state <span class="token constant">LCOMMENT</span> <span class="token constant">ML_COMMENT</span></span> 75<span class="line"></span> 76<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 77<span class="line"></span> 78<span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 79<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 80<span class="line highlighted"> <span class="token string">"#["</span> <span class="token punctuation">{</span> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">ML_COMMENT</span><span class="token punctuation">)</span><span class="token punctuation">;</span> multilineCommentLevel<span class="token operator">++</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 81<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 82<span class="line"><span class="token punctuation">}</span></span> 83<span class="line"></span> 84<span class="line highlighted"><span class="token generics"><span class="token punctuation"><</span>ML_COMMENT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 85<span class="line highlighted"> <span class="token string">"#["</span> <span class="token punctuation">{</span> multilineCommentLevel<span class="token operator">++</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 86<span class="line highlighted"> <span class="token string">"]#"</span> <span class="token punctuation">{</span> multilineCommentLevel<span class="token operator">--</span><span class="token punctuation">;</span></span> 87<span class="line highlighted"> <span class="token keyword">if</span> <span class="token punctuation">(</span>multilineCommentLevel <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 88<span class="line highlighted"> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">DEFAULT</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 89<span class="line highlighted"> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COMMENT</span><span class="token punctuation">;</span></span> 90<span class="line highlighted"> <span class="token punctuation">}</span></span> 91<span class="line highlighted"> <span class="token punctuation">}</span></span> 92<span class="line highlighted"> <span class="token generics"><span class="token punctuation"><</span><span class="token punctuation"><</span>EOF<span class="token punctuation">></span><span class="token punctuation">></span></span> <span class="token punctuation">{</span> <span class="token keyword">if</span> <span class="token punctuation">(</span>multilineCommentLevel <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span> <span class="token punctuation">{</span></span> 93<span class="line highlighted"> <span class="token comment">// emit a comment token first to treat everything</span></span> 94<span class="line highlighted"> <span class="token comment">// until eof as a comment</span></span> 95<span class="line highlighted"> multilineCommentLevel <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span></span> 96<span class="line highlighted">
96 <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">COMMENT</span><span class="token punctuation">;</span></span> 97<span class="line highlighted"> <span class="token punctuation">}</span></span> 98<span class="line highlighted"> <span class="token comment">// then emit the unterminated comment token, </span></span> 99<span class="line highlighted"> <span class="token comment">// which we will higlight using an annotation</span></span> 100<span class="line highlighted"> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">AT_EOF</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 101<span class="line highlighted"> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">ML_COMMENT_UNTERMINATED</span><span class="token punctuation">;</span></span> 102<span class="line highlighted"> <span class="token punctuation">}</span></span> 103<span class="line highlighted"> <span class="token punctuation">[</span><span class="token operator">^</span><span class="token punctuation">]</span> <span class="token punctuation">{</span> <span class="token comment">/* consume all other character */</span> <span class="token punctuation">}</span></span> 104<span class="line highlighted"><span class="token punctuation">}</span></span> 105<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>Once we encounter a <code>#[</code>, we enter the <code>ML_COMMENT</code> state and increment the nesting level (which is initially zero). When we encounter a <code>]#</code>, we decrement the nesting level, and if it reaches zero, we exit the <code>ML_COMMENT</code> state and return a comment token.</p><p>If we reach the end of the file without closing the comment, we emit a comment token for everything until the end of the file. We also set <code>multilineCommentLevel</code> to zero, and we stay in the <code>ML_COMMENT</code> state. When the lexer is advanced once more, we enter the same <code><<EOF>></code> rule, but this time we emit a special <code>UNTERMINATED_ML_COMMENT</code> token. We also switch to the <code>AT_EOF</code> state to let it call <code>processEof()</code> to return any pending <code>DED</code> tokens.</p><p>We'll leave the handling of the <code>ML_COMMENT_UNTERMINATED</code> for later when we introduce annotations. For now, let's test multiline comments and see if the nesting works as expected.</p><p><img src="`+o+`" alt="Multiline Comments" width="600"></p><p>Great! We can insert multiline comments anywhere in the code, even if they are nested, and they are correctly recognized.</p><h2 id="documentation-comments" tabindex="-1"><a class="header-anchor" href="#documentation-comments"><span>Documentation Comments</span></a></h2><p>Documentation comments are similar to regular comments, except they're not ignored by the parser. Let's add a new state, <code>DOC_COMMENT</code>, to handle documentation comments.</p><div class="language-java line-numbers-mode" data-highlighter="prismjs" data-ext="java" data-title="java"><pre><code><span class="line"><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 106<span class="line"></span> 107<span class="line highlighted"><span class="token operator">%</span>
107state <span class="token constant">L_COMMENT</span> <span class="token constant">ML_COMMENT</span> <span class="token constant">DOC_COMMENT</span></span> 108<span class="line"></span> 109<span class="line"><span class="token operator">%</span><span class="token operator">%</span></span> 110<span class="line"></span> 111<span class="line"><span class="token generics"><span class="token punctuation"><</span>DEFAULT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 112<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 113<span class="line highlighted"> <span class="token string">"##"</span> <span class="token punctuation">{</span> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">DOC_COMMENT</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 114<span class="line"> <span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span></span> 115<span class="line"><span class="token punctuation">}</span></span> 116<span class="line"></span> 117<span class="line highlighted"><span class="token generics"><span class="token punctuation"><</span>DOC_COMMENT<span class="token punctuation">></span></span> <span class="token punctuation">{</span></span> 118<span class="line highlighted"> <span class="token punctuation">{</span><span class="token constant">EOL</span><span class="token punctuation">}</span><span class="token punctuation">[</span> \\t<span class="token punctuation">]</span><span class="token operator">*</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">^</span>#<span class="token punctuation">]</span> <span class="token operator">|</span> #<span class="token punctuation">[</span><span class="token operator">^</span>#<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">// next line is not a doc comment</span></span> 119<span class="line highlighted"> <span class="token function">yypushback</span><span class="token punctuation">(</span><span class="token function">yylength</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></span> 120<span class="line highlighted"> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">DEFAULT</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">D_COMMENT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 121<span class="line highlighted"> <span class="token generics"><span class="token punctuation"><</span><span class="token punctuation"><</span>EOF<span class="token punctuation">></span><span class="token punctuation">></span></span> <span class="token punctuation">{</span> <span class="token function">yybegin</span><span class="token punctuation">(</span><span class="token constant">AT_EOF</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token keyword">return</span> <span class="token class-name">NimToken</span><span class="token punctuation">.</span><span class="token constant">D_COMMENT</span><span class="token punctuation">;</span> <span class="token punctuation">}</span></span> 122<span class="line highlighted"> <span class="token punctuation">[</span><span class="token operator">^</span><span class="token punctuation">]</span>
122 <span class="token punctuation">{</span> <span class="token comment">/* consume all other character */</span> <span class="token punctuation">}</span></span> 123<span class="line highlighted"><span class="token punctuation">}</span></span> 124<span class="line highlighted"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>The <code>DOC_COMMENT</code> state is similar to the <code>L_COMMENT</code> state, except it starts with <code>##</code>, and it returns a different token, <code>D_COMMENT</code>, which we'll use in the parser to capture documentation comments in the syntax tree.</p><p>Now, let's update the grammar to include documentation comments. Like regular comments, doc comments can appear almost anywhere. However, from a semantic point of view, they are meaningful only in certain places. A good source of information on where documentation comments are meaningful semantically is the <a href="https://github.com/nim-lang/Nim/blob/devel/tests/parser/tdoc_comments.nim" target="_blank" rel="noopener noreferrer"><code>tdoc_comments.nim</code></a> test file from the Nim repository.</p><p>Here's a summary of where documentation comments are meaningful:</p><ul><li>In a statement list, as a comment statement.</li><li>For each of the following, the documentation comment can appear at the end of the declaration line, or indented right below it: <ul><li>Routines (procs, methods, templates, etc.)</li><li>Type, type field, and enum value declarations</li><li>Vars and constants declaration</li></ul></li></ul><p>Let's start by updating the grammar to include documentation comments in places where we expect them to show up. We'll add a rule <code>DocComment</code> to be used in the meaningful places mentioned above. In all other places we'll use the <code>D_COMMENT</code> token directly.</p><div class="language-bnf line-numbers-mode" data-highlighter="prismjs" data-ext="bnf" data-title="bnf"><pre><code><span class="line"><span class="token operator">...</span></span> 125<span class="line"></span> 126<span class="line">Module <span class="token operator">::=</span> !<<span class="token rule"><span class="token punctuation"><</span>eof<span class="token punctuation">></span></span>> StmtList</span> 127<span class="line"></span> 128<span class="line">StmtList <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>list Stmt EQD<span class="token punctuation">></span></span>></span> 129<span class="line"></span> 130<span class="line">private Stmt <span class="token operator">::=</span> ConstSection</span> 131<span class="line"> <span class="token operator">|</span> VarSection</span> 132<span class="line"> <span class="token operator">|</span> LetSection</span> 133<span class="line"> <span class="token operator">|</span> Command</span> 134<span class="line"> <span class="token operator">|</span> BlockStmt</span> 135<span class="line highlighted"> <span class="token operator">|</span> CommentStmt</span> 136<span class="line"></span> 137<span class="line">ConstSection <span class="token operator">::=</span> CONST <<span class="token rule"><span class="token punctuation"><</span>section IdentDef<span class="token punctuation">></span></span>></span> 138<span class="line">LetSection <span class="token operator">::=</span> LET <<span class="token rule"><span class="token punctuation"><</span>section IdentDef<span class="token punctuation">></span></span>></span> 139<span class="line">VarSection <span class="token operator">::=</span> VAR <<span class="token rule"><span class="token punctuation"><</span>section IdentDef<span class="token punctuation">></span></span>></span> 140<span class="line"></span> 141<span class="line highlighted">
141IdentDef <span class="token operator">::=</span> IdentDecl EQ STRING_LIT DocComment<span class="token operator">?</span></span> 142<span class="line"></span> 143<span class="line">Command <span class="token operator">::=</span> IdentRef IdentRef</span> 144<span class="line"></span> 145<span class="line">BlockStmt <span class="token operator">::=</span> BLOCK COLON D_COMMENT<span class="token operator">?</span> <<span class="token rule"><span class="token punctuation"><</span>indented StmtList<span class="token punctuation">></span></span>></span> 146<span class="line"></span> 147<span class="line highlighted">CommentStmt <span class="token operator">::=</span> D_COMMENT</span> 148<span class="line"></span> 149<span class="line highlighted">DocComment <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>optind D_COMMENT<span class="token punctuation">></span></span>></span> 150<span class="line"></span> 151<span class="line">IdentDecl <span class="token operator">::=</span> IDENT</span> 152<span class="line">IdentRef <span class="token operator">::=</span> IDENT</span> 153<span class="line"></span> 154<span class="line">// meta rules</span> 155<span class="line"></span> 156<span class="line">private meta list <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>p1<span class="token punctuation">></span></span>> <span class="token operator">(</span><<span class="token rule"><span class="token punctuation"><</span>p2<span class="token punctuation">></span></span>> <
156<span class="token rule"><span class="token punctuation"><</span>p1<span class="token punctuation">></span></span>><span class="token operator">)</span><span class="token operator">*</span></span> 157<span class="line">private meta indented <span class="token operator">::=</span> IND <<span class="token rule"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span>> DED</span> 158<span class="line highlighted">private meta section <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span>> <span class="token operator">|</span> D_COMMENT<span class="token operator">?</span> <<indented <<list <span class="token operator">(</span><<span class="token rule"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span>> <span class="token operator">|</span> D_COMMENT<span class="token operator">)</span> EQD>>>></span> 159<span class="line">private meta optind <span class="token operator">::=</span> <<span class="token rule"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span>> <span class="token operator">|</span> <<indented <<span class="token rule"><span class="token punctuation"><</span>p<span class="token punctuation">></span></span>>>></span> 160<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>The <code>CommentStmt</code> rule is a new rule that allows a documentation comment to appear as a statement in a statement list. The <code>DocComment</code> rule is a new rule that allows a doc comment to appear right after an element, or indented below it (we use the <code>optind</code> meta rule for that). To support doc comments for let/var/constant sections, we updated the <code>IdentDef</code> rule to include an optional <code>DocComment</code> at the end. Finally, we updated the <code>section</code> rule to allow a doc comment token to optionally appear interspersed with the declarations.</p><p>Let's look at an example:</p><div class="language-nim line-numbers-mode" data-highlighter="prismjs" data-ext="nim" data-title="nim"><pre><code><span class="line"><span class="token comment">## this is a doc comment</span></span> 161<span class="line"><span class="token comment">## that spans multiple lines</span></span> 162<span class="line"></span> 163<span class="line"><span class="token keyword">block</span><span class="token operator">:</span> <span class="token comment">## not meaningful</span></span> 164<span class="line"> <span class="token keyword">let</span> <span class="token comment">## not meaningful</span></span> 165<span class="line"> msg <span class="token operator">=</span> <span class="token string">"hello"</span> <span class="token comment">## documents \`msg\`</span></span> 166<span class="line">
166 <span class="token comment">## continues to document \`msg\`</span></span> 167<span class="line"></span> 168<span class="line"> <span class="token comment">## not meaningful</span></span> 169<span class="line"> foo <span class="token operator">=</span> <span class="token string">"foo"</span></span> 170<span class="line"> <span class="token comment">## documents \`foo\`</span></span> 171<span class="line"></span> 172<span class="line"> echo msg</span> 173<span class="line"> <span class="token comment">## not meaningful</span></span> 174<span class="line"> echo foo</span> 175<span class="line"></span></code></pre><div class="line-numbers" aria-hidden="true" style="counter-reset:line-number 0;"><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div><div class="line-number"></div></div></div><p>As you can see, although doc comments can appear almost anywhere, they are usually used to document declarations. I'm not sure why they are allowed in other places, but typically they are not used there (a regular comment would suffice in those places).</p><p>Let's test this out.</p><p><img src="`+i+'" alt="Documentation Comments" width="750"></p><p>Looks good. Notice that the case for the <code>msg</code> variable declaration, where the doc comment starts at the end of the line, and continues on the next line, is handled as a single doc comment.</p><p>We now have full support for all kinds of Nim comments.</p>',54)]))}const m=s(c,[["render",u],["__file","13-comments.html.vue"]]),k=JSON.parse('{"path":"/nimjet/13-comments.html","title":"Comments","lang":"en-US","frontmatter":{},"headers":[{"level":2,"title":"Lexing Comments","slug":"lexing-comments","link":"#lexing-comments","children":[]},{"level":2,"title":"Multiline Comments","slug":"multiline-comments","link":"#multiline-comments","children":[]},{"level":2,"title":"Documentation Comments","slug":"documentation-comments","link":"#documentation-comments","children":[]}],"git":{"updatedTime":1728173307000}
175,"filePathRelative":"nimjet/13-comments.md","excerpt":"\\n<p>Nim comments are similar to those in Python. They start with a <code>#</code> and continue to the end\\nof the line. Nim also supports multi-line comments that start with <code>#[</code> and end with <code>]#</code>,\\nwhich can be nested. Nim is very flexible with comments, allowing them to appear\\nalmost anywhere in the code.</p>"}');export{m as comp,k as data};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.