1<meta charset="utf-8" emacsmode="-*- markdown -*-"> 2<link rel="stylesheet" href="style.css"> 3<link rel="stylesheet" href="https://casual-effects.com/markdeep/latest/journal.css"> 4 5 6 **Testorial: An executable Tutorial** 7 By [Alejandro Garcia](mailto:[email protected]) 8 9 10 11When you're learning a new technology, library, or programming language have you felt the excitement of finding the perfect tutorial. 12The one that explains what you want to learn in a voice that you can understand with an example that almost perfectly matches your needs. 13Only to feel the frustration of discovering that many instructions in the tutorial are obsolete no longer works or maybe they never did. 14 15Well, as a Content Engineer let me tell you that it's never my intention to write documentation that goes stale. 16There are many moving parts, 17So invariably a section of a good tutorial will always become obsolete almost the moment we publish it. 18It's our continuous fight against [bit rot](https://en.wikipedia.org/wiki/Software_rot). 19 20However, from the field of DevOps we have techniques of Continuous Integration that helps fight bit rot in Software. 21And we can use the same techniques in our documentation. 22By monitoring our source testorial, and detecting changes we can anticipate parts of our tutorial that will be need to be updated. 23 24And if this wasn't enough reason to justify the effort to write testorials. 25We need to consider a new Learner: Large Language Models (LLMs) 26LLMS will learn from the documentation that we write. 27So our documentation *can't be a hallucination* it *must be correct* so that the LLM, 28Can learn the correct way to do things. 29 30This tutorial is for Authors, Content Engineers and Technical Writers, 31that want to 32fight bit rot, 33don't waste their Learner's time, 34and provide more value to their organization by doing integration tests 35that their software must run. 36 37Testorial 38: A Testable Tutorial 39: Is a Tutorial written so that humans can comprehend, 40: that also works as a test in a Continuous Integration (CI) pipeline. 41: Becoming valuable for both your Learners and Quality Assurance (QA) Team. 42: synonym: executable documentation 43: related: literate programming, DevOps 44 45Tutorial 46: A goal oriented article, written so that a Learner can acquire a new skill. 47: Every Testorial is a Tutorial, but not every Tutorial is a Testorial. 48 49# Why aren't Testorials (Docs-as-test) more popular? 50 51Given that the movement Docs-as-Code has been active since at least 2015[fn:1]. 52And it's historical precedent Literate Programming, was invented by Donald Knuth in 1984[fn:2] . 53It should be more popular and it isn't. 54 55If we search online: Why isn't literate programming more popular? 56We would a find that common reason is that: 57 58> The intersection of people that are good programmers 59> and good writers at the same time is pretty small. 60 61I find this is a good point, for general programmers. 62 63But for technical writers, it's the starting position! 64 65Traditional Technical Writing has been seen as a person that can wear two hats: 66 67- Programmer: 68 - Read code, libraries, products 69 - Experiments and learns 70- Writer 71 - Writes documentation 72 - Edits documentation 73 74 75 76But Even though a Technical Writer has the skills of Programmer and a Writer. 77We don't see frequently executable documentation or testorials. 78And I think it's because 79Testorial asks the Author to wear a *third hat*: 80 81- Tester 82 - Write tests that run automatically 83 - Monitor when things break 84 85If writing is difficult, 86writing, programming, and testing is as difficult as finding a unicorn. 87 88 89 90 91But now it's 2026 And we can count on the help of LLMs! 92which has dramatically increased the size of the intersection. 93 94So doesn't matter how you started, 95If you were writer fascinated by computers and became a technical writer. 96Or you were a DevOps engineer that you learned you love of writing later in life, like me. 97LLMs can help us become the unicorn at the intersection of the three hats needed to do Docs-as-Tests 98 99 100 101## What you're reading 102 103According to my records it seems that for writing this Testorial 104I had to 34 conversations with [Claude Code](https://claude.com/product/claude-code). 105And with 18 conversations witch [ChatGpt](https://chatgpt.com) 106 107Those conversations varied from couple of minutes to a few days. 108 109With Claude it was technical conversations like: 110+ How can execute a command with `sudo` inside a script? 111+ How to install `act` in Ubuntu? 112 113And with ChatGpt the conversations where more like an editor: 114+ What is a better name for the technique of using tutorials as tests: Testorials or DocOps? 115+ Here is the first draft of my article. What are it's 5 weakest spots? 116 117 118So having LLMs at my disposal was key in bring the article that you're reading to fruition. 119It was like having a team with a Senior Developer and a tireless Editor helping me write it. 120 121In the past when I have tried to make Testorials a common practice it has no been possible, 122Because the level of expertise required in so many areas made it impractical. 123The Deadline to deliver the article had already passed. 124 125> "LLMs make it economically feasible, practical, and even fun, 126> to write testorials" 127 128However as a big proponent of using the help of LLMs to write documents. 129There are two areas where I don't think they should be used: 130 131### Deterministic Execution 132As you will see next the whole point of testorial is that it can be executed repeatedly as an integration test. 133And although in theory it could be possible to tell an LLM: "Read this testorial and execute the code, in the examples" 134That would be recipe for failure. 135We want the execution of the test to be identical on every iteration! 136In order to trust our test. 137 138Once a testorial fails, you can ask the LLM to update it and make it run again. 139But once it's correct again it's execution again must be deterministic. 140 141In this tutorial that Determinism comes from `clitest` to execute the testorial and from using the `act` container. 142 143### Your goal with the article 144 145When you're writing a tutorial you have a *goal* an approach an innovation, 146that you want to communicate to the Learner. 147That goal is *yours* and the LLM cannot, give you a goal. 148So in my opinion LLMs are best used as tireless editors. 149That can read as many drafts of your piece as necessary 150That can give clear direction. 151But you shouldn't use them to replace your voice. 152 153You're the writer, the LLM is a great Editor. 154 155For example with article my goal where to: 156 1571. Explain how to write documentation as test, Testorials. 1582. Introduce the Nine Windows framework for writing to make it easy for the Author to think about this pieces. 159 160And with that, let's learn about the Nine Windows 161 162# A map of the terrain, the Nine Windows framework 163 164We have always had to consider the audience when writing a piece. 165Now we need to consider three different audiences: 166 167A Learner 168: A human or LLM using your article to develop a new skill. 169 170A DevOps engineer 171: A person in your organization that doesn't care about your testorial, except that it will help him with QA for the whole project. 172 173An Author (yourself) 174: You need consider your own environment, deadlines, etc. and the ones of your audience. 175 176And for each element of our audience we need to consider distinct levels of understanding. 177 178Context level 179: Is under what circumstances (outside its control) are we writing. 180 181Execution level 182: the actual thing (tutorial, testorial, workflow) that the person can manipulate. 183 184Storage level 185: When executing many commands, they will have an impact in the storage or in subparts of the tutorial. 186: We must consider the impact of changing the Storage for each tutorial. 187 188 189Thinking this way, gives us a map of the terrain that we're going to traverse while writing. 190 191## Nine windows to write a testorial 192 193Considering our Audiences and the Levels of context we need to understand we can map our article as: 194 195********************************************************************************************************* 196* ââââââââââââââââââââââââââââââââ Audience ââââââââââââââââââââââââââââââââââââ * 197* * 198* Author DevOps Learner * 199* â¡ âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 200* ⢠â â â â * 201* ⢠â â â â * 202* ⢠â â â â * 203* ⢠Context â â â â * 204* ⢠â â â â * 205* ⢠â â â â * 206* ⢠â â â â * 207* ⢠â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 208* ⢠â â â â * 209* ⢠â â â â * 210* ⢠â â â â * 211* Level Execution â â â â * 212* ⢠â â â â * 213* ⢠â â â â * 214* ⢠â â â â * 215* ⢠â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 216* ⢠â â â â * 217* ⢠â â â â * 218* ⢠â â â â * 219* ⢠Storage â â â â * 220* ⢠â â â â * 221* ⢠â â â â * 222* ⢠â â â â * 223* ⣠âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 224* * 225********************************************************************************************************* 226 227Now with this map we can make sense of what we're writing for whom, when. 228 229# 1st Level: Context 230 231Let's start by understanding the Context on which each of our Audience operate 232The context is important since most of the time we can't control it. 233It's a constraint. 234 235 236 237And the main Context that we need to focus on is the one for you the Author: 238 239## A Learner's Context 240This is the most obvious one, we need to be mindful of our Learners Context. 241+ What's their previous experience? 242+ What constrains could their employer have? 243+ What operating system are they using? 244and so on. 245 246 247********************************************************************************************************* 248* * 249* * 250* Author DevOps Learner * 251* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 252* â â â â * 253* â â â â * 254* Context â â â Previous Experience? â * 255* â â â Job Constraints? â * 256* â â â Operating System? â * 257* â â â â * 258* â â â â * 259* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 260* â â â â * 261* â â â â * 262* Execution â â â â * 263* â â â â * 264* â â â â * 265* â â â â * 266* â â â â * 267* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 268* â â â â * 269* â â â â * 270* â â â â * 271* Storage â â â
271 â * 272* â â â â * 273* â â â â * 274* â â â â * 275* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 276* * 277********************************************************************************************************* 278[Figure [learner_context]: The Learners context, guides the depth and tone of our tutorial] 279 280## DevOps Context 281 282On this context we need to understand that the DevOps and QA Team, have setup a Continuous Integration (CI) pipeline. 283The CI on every change, takes the content of the source code repository, 284executes tests on it and if the tests are correct, 285publishes a new version of our software into production. 286 287All of this happens inside virtual computers called Containers, so that we don't have to 288This pipeline is crucial, since we will integrate the testorials as parts of the pipeline in the following sections. 289 290********************************************************************************************************* 291* * 292* * 293* Author DevOps Learner * 294* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 295* â â â â * 296* â â â â * 297* Context â â What is the CI pipeline? â Previous Experience? â * 298* â â â Job Constraints? â * 299* â â GitHub Runner (Container)â Operating System? â * 300* â â â â * 301* â â â â * 302* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 303* â â â â * 304* â â â â * 305* Execution â â â â * 306* â â â â * 307* â â â â * 308* â â â â * 309* â â â â * 310* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 311* â â â â * 312* â â â â * 313* â â â â * 314* Storage â â â â * 315* â â â â * 316* â â â â * 317* â â â â * 318* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 319* * 320********************************************************************************************************* 321[Figure [devops_context]: A DevOps context, informs what is the CI pipeline and the publishing process] 322 323## An Author's Context 324 325Now lets talk about you, the Author and your Context. 326 327Probably your organization already has setup a Content Management System (CMS). 328And you need to understand the workflows that will transform (publish) the testorial to a tutorial. 329 330Since a testorial is also code, and code is plain text 331Any CMS that works with plain text formats like Markdown, AsciiDoc, Org-mode will do. 332This category is commonly called Static Site Generators. 333 334Some static site generators, that work well are: 335[Docusaurus](https://docusaurus.io), 336[GitHub Pages](https://docs.github.com/en/pages/quickstart) Or my personal favorite
337[Markdeep](https://casual-effects.com/markdeep/) 338 339In this tutorial we will assume your using GitHub Pages as your CMS. 340Since it's pretty common and it integrates with GitHub Actions, that's a Continuous Integration pipeline. 341 342Probably your organization already has configured GitHub Pages. 343You need to understand on which repository on which folder you need to save your testorial markdown document. 344But, in case it's not configured you can check the section [Appendix: Configuring GitHub Pages] Configuring GitHub Pages. 345 346 347What's important to understand is the publishing process of GitHub Pages 348 349### GitHub Pages publishing process 350 351In GitHub Pages the process of publishing is: 352 3531. Create or Modify Markdown document. 3542. git commit & git push 3553. Wait for GitHub Workflow to finish. 3564. Review modifications on your website. 357 358You can monitor the state the of the GitHub Workflow on the page. 359 360Observe that the workflow it's on yellow while it's running. 361 362You can even check the detail instructions that it's executing. 363 364When it finally turns to green means your new page is published. 365 366 367 368### Tracing the publishing process 369 370Now that we understand the GitHub Pages publishing process. 371We can trace how it works in our map of writing. 372 373********************************************************************************************************* 374* * 375* * 376* Author DevOps Learner * 377* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 378* â â â â * 379* Context â CMS - GitHub Pages â â â * 380* â | â â â * 381* â | â â â * 382* â dictates â â â * 383* â organization â â â * 384* â ââââââââ|âââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 385* â | â â â * 386* Execution â v â â â * 387* â Markdown Document ---git push--> GH Workflow ---publish--> Tutorial â * 388* â â â â * 389* â â â â * 390* â â â â * 391* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 392* â â â â * 393* â â â â * 394* â â â â * 395* Storage â â â â * 396* â â â â * 397* â â â â * 398* â â â â * 399* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 400* * 401********************************************************************************************************* 402[Figure [gh_pages_workflow]: GitHub Pages publishing workflow] 403 404# 2nd Level the Execution 405 406In the previous diagram we saw that at the Execution level works like a single pipeline. 407Now on this section we're going to cover that pipeline in more detail. 408 409## A Learners Execution: The tutorial 410 411Let's start with the end in mind. 412Let's look at the tutorial that our Learner will read: 413 414 415<div class="example"> 416<div class="view"> 417Learner 418<div class="window browser"> 419<div class="buttons"></div> 420<div class="tab">Example</div> 421<div class="menu"> <- -> â³ <div class="url">hello_world.md.html</div></div> 422<iframe src="hello_world.md.html" content="text/html;charset=UTF-8"></iframe> 423</div></div></div> 424 425 426In our 9 windows framework we're here: 427 428********************************************************************************************************* 429* * 430* * 431* Author DevOps Learner * 432* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 433* â â â â * 434* â â â â * 435* â â â â * 436* Context â â â â * 437* â â â â * 438* â â â â * 439* â â â â * 440* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 441* â â â âââââââââââââââââââââ® â * 442* â â â â Browser â â * 443* Execution â â â ââââââââââââââââââââ⤠â * 444* â â â â â â * 445* â â â â Tutorial â â * 446* â â â â â â * 447* â â â â°âââââââââââââââââââ⯠â * 448* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 449* â â â â * 450* â â â â * 451* â â â â * 452* Storage â â â â * 453* â â â â * 454* â â â â * 455* â â â â * 456* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 457* * 458********************************************************************************************************* 459[Figure [learner_artifact]: A Learner reads the tutorial] 460 461## Author's Execution: the Testorial 462 463In the previous section we saw what a Student would read. 464For you the the Testorial looks like a markdown source code. 465 466~~~ markdown 467(insert hello_world.md.html here) 468~~~ 469 470Notice that we have marked the instructions we expect the student to type like `@localhost $` 471How can we make sure that the instructions are correct? 472 473### Snapshot testing 474 475Snapshot testing tools run your commands and check the results. They compare the actual output with the expected output to make sure everything works correctly. 476
477In our case we will use `clitest for posix` 478 479### Installing clitest 480 481Installing `clitest` is as easy as: 482 483~~~ bash 484@author $ curl -sOL https://raw.githubusercontent.com/aureliojargas/clitest/master/clitest 485@author $ chmod +x clitest 486@author $ mv clitest ~/.local/bin 487@author $ clitest --version 488@author clitest 0.5.0 489~~~ 490 491### Listing the commands in hello_world.md.html 492 493Now lets use `clitest` to view what commands are in our tutorial 494 495~~~ bash 496@author $ clitest --list --prefix '@learner ' ./hello_world.md.html 497@author #1 echo "hello world" 498@author #2 echo "hello world" | tr '[:lower:]' '[:upper:]' 499~~~ 500 501+ Use `--list` to show the commands that can execute in the tutorial. 502+ The parameter `--prefix '@learner'` specifies how the executable lines are prefixed. 503+ The #1 and #2 show what are the commands that `clitest` identified as possible. 504+ #1 and #2 are the test cases in our testorial /test suite/ 505 506!!! Tip 507 Is important to make sure that `--list` finds all the commands that you expect the testorial to execute. 508 509To execute the tests, remove the `--list` flag 510 511~~~{.bash} 512@author $ clitest --prefix '@learner ' ./hello_world.md.html 513@author #1 echo "hello world" 514@author #2 echo "hello world" | tr '[:lower:]' '[:upper:]' 515@author OK: 2 of 2 tests passed 516@author $ 517~~~ 518 519Now we get the report `OK: 2 of 2 tests passed`. 520 521This testorial is /pure/, it doesn't change any files in our computer. 522So you can repeat it as many times as you wish. 523 524Try it, execute `clitest` again. 525 526~~~{.bash} 527@author $ clitest --prefix '@learner ' ./hello_world.md.html #=> --exit 0 528~~~ 529 530It will run again the tests. 531The comment `exit 0`, means that the command executed correctly, 532and we don't care about the literal output, 533 534 535### Where does `clitest` fit in our nine windows framework? 536 537********************************************************************************************************* 538* * 539* * 540* Author DevOps Learner * 541* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 542* â â â â * 543* â â â â * 544* Context â â â â * 545* â â â â * 546* â â â â * 547* â â â â * 548* â â â â * 549* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 550* â .--- clitest <--. â â âââââââââââââââââââââ® â * 551* â | | â â â Browser â â * 552* Execution â executes fix â â ââââââââââââââââââââ⤠â * 553* â | | â â â â â * 554* â v | â â â Tutorial â â * 555* â testorial.markdown â â â â â * 556* â â â â°âââââââââââââââââââ⯠â * 557* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 558* â â â â * 559* â â â â * 560* â â â â * 561* Storage â â â â * 562* â â â â * 563* â â â â * 564* â â â â * 565* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 566* * 567********************************************************************************************************* 568[Figure [author_artifact]: An author makes its testorial executable by clitest] 569 570## DevOps Engineer Execution 571 572Now that the test can be reproduce locally, 573we need to think, How it's going to be executed on the Continuous Integration server? 574 575### GitHub Actions 576 577Is a Continuous Integration server provided by GitHub. 578It has two main components: 579
580+ Runners: Virtual containers that execute your code. 581+ Workflows: instructions on how to configure said containers and how to execute your code. 582 583### GitHub Workflow 584 585We will configure a workflow that can execute the testorial as we can now do locally with `clitest`. 586 587Look at this suggested workflow 588 589(embed execute_testorials.yml height=450px here) 590 591[Listing [execute_testorial_yml]: .github/workflows/execute_testorials.yml] 592 593 594(1) Name of our workflow. 595 596(2) This is the event that will trigger the execution of our workflow. 597 598(3) In particular we want to trigger the execution from the GitHub User Interface. 599 600(4) This is the Operating system where the tests are going to run. 601 602This workflow will execute on every commit, but for demonstration purposes go to where you have this tutorial stored. 603Then GitHub Workflows click on Execute wait a few minutes. 604And see that it's in *green* meaning all the tests executed correctly. 605 606 607### Commit your execute_testorials workflow 608 609Now we're ready to include this tutorial as part of the CI pipeline. 610This will need approval from your DevOps department 611Your `testorial.markdown` and `execute_tasteless.yml` workflow is all the documentation they need to include the testorials execution as part of the CI process. 612 613Perhaps the DevOps team will decide to change the trigger so that instead of on every commit, it's on every pull request. 614 615!!! Warning 616 Now you will get notified when a code change, or a library update breaks your testorial. 617 So now you have a new responsibility, to keep your testorials updated. 618 But better be you, the Author, than your frustrated readers. 619 620 621 622### Tracing the CI pipeline in the nine Windows 623 624********************************************************************************************************* 625* * 626* * 627* Author DevOps Learner * 628* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 629* â â â â * 630* â â â â * 631* â â â â * 632* Context â â â â * 633* â â â â * 634* â â â â * 635* â â â â * 636* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 637* â .--- clitest <--. â .--> GH Workflow â âââââââââââââââââââââ® â * 638* â | | â | | â â Browser â â * 639* â executes fix â | execute â ââââââââââââââââââââ⤠â * 640* Execution â | | â | | â â â â * 641* â v | â | v â â Tutorial â â * 642* â testorial.markdown---gh push clitest? ---publish-->â â â * 643* â â â â°âââââââââââââââââââ⯠â * 644* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 645* â â â â * 646* â â â â * 647* â â â â * 648* Storage â â â â * 649* â â â â * 650* â â â â * 651* â â â â * 652* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 653* * 654********************************************************************************************************* 655[Figure [devops_artifact]: How a testorial gets executed and becomes a tutorial] 656 657 658# 3rd level: Storage 659 660In the previous level we completed the "Hello World" tutorial. 661And we made it executable with `clitest` but we "cheated a little" since we used as example a Testorial, 662that was pure, meaning it didn't change the state in any of the computers. 663So it was easy to execute as many times as we wanted. 664 665However most testorials, in fact most programs aren't like that. 666Most programs change something in the computer that executes them. 667They add records to a database, store files in the hard drive, even make transactions on a blockchain! 668 669For our next example we're going to evolve our "hello_World.md.html" tutorial. 670It will create a file that gets stored in the computer. 671This side-effect doesn't let us run and re-run the tutorial with `clitest` as easy as before. 672Instead we need to create *isolation*. 673 674## As a Learner 675
676Now let's modify our Tutorial, so that learner now has to install a program (`cowsay`) in their computer. 677 678The tutorial would look something like this: 679 680<div class="example"> 681<div class="view"> 682Learner 683<div class="window browser"> 684<div class="buttons"></div> 685<div class="tab">Example</div> 686<div class="menu"> <- -> â³ <div class="url">cowsay_hello_world.md.html</div></div> 687<iframe src="cowsay_hello_world.md.html" content="text/html;charset=UTF-8"></iframe> 688</div></div></div> 689 690### Orienting ourselves in the map 691 692The important thing to know is that this tutorial asks the Learner to install something on their local computer. 693Therefore it modifies the storage. 694 695 696********************************************************************************************************* 697* * 698* * 699* Author DevOps Learner * 700* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 701* â â â â * 702* â â â â * 703* â â â â * 704* Context â â â â * 705* â â â â * 706* â â â â * 707* â â â â * 708* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 709* â â â âââââââââââââââââââââ® â * 710* â â â â Browser â â * 711* Execution â â â ââââââââââââââââââââ⤠â * 712* â â â â Tutorial â â * 713* â â â â°âââââââââââââââââââ⯠â * 714* â â â | â * 715* â â â apt install â * 716* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââ|âââââââââââââââââââ⣠* 717* â â â | â * 718* â â â v â * 719* â â â packages â * 720* Storage â â â â * 721* â â â â * 722* â â â â * 723* â â â â * 724* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 725* * 726********************************************************************************************************* 727 728 729 730## Storage Level as an Author 731 732Now we know that the Testorials looks like this 733 734~~~ markdown 735 (include ./cowsay_hello_world.md.html here) 736~~~ 737
738and we can execute it in our computer like this: 739 740~~~bash 741@author $ sudo -v && clitest --first --diff-options '--ignore-space-change' --prefix '@learner ' ./cowsay_hello_world.md.html #=> --exit 0 742~~~ 743 744Now we're using a lot more parameters so let's explain them: 745 746`sudo -v` 747: Before execution, `sudo -v` asks for your local sudo password so the tutorial can install packages. 748`&&` 749: means *and* only execute the next command if the previous one executed correctly. 750`clitest` 751: We already familiar, in the following textile, process lines tagged with the prefix as a script. 752`--first` 753: stop executing at the first test that fails. No need to continue executing if we have an error. 754`--prefix '@learner'` 755: execute the lines that start with `@learner` 756`--diff-options '--ignore-space-change'` 757: When comparing, ignore space differences between the output and the expected value. 758` ./cowsay_hello_world.md.html` 759: the name of the markdown file (`.md.html`) we're going to execute with clitest. 760 761 762The problem is that if we execute the script again we will get an error! 763Since the `cowsay` package was installed on first run, the second time we will get an error 764 765~~~ bash 766@author $ sudo -v && clitest --first --diff-options '--ignore-space-change' --prefix '@learner ' ./cowsay_hello_world.md.html #=> --exit 1 767~~~ 768 769So now we need to clean after the execution of the tests. 770 771## Testorial tear-down 772 773When doing unit testing _tear-down_ is the method that gets called after each test execution. 774
775In our case, we need to clean what we stored in the computer so that our testorials are repeatable. 776 777in `clitest` that's achieved with the `--post-flight COMMAND` parameter. 778 779For example 780 781~~~ bash 782@author $ sudo -v && clitest --post-flight 'sudo apt remove -y cowsay' --first --prefix '@learner ' --diff-options '--ignore-space-change' ./cowsay_hello_world.md.html #=> --exit 0 783~~~ 784 785`--post-flight 'sudo apt remove -y cowsay' ` 786: After the last test, execute the command `apt remove` so that the cowsay that we installed in the testorial, gets removed 787 788 789### How does the Storage level looks from the point of view of the Author 790 791********************************************************************************************************* 792* * 793* * 794* Author DevOps Learner * 795* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 796* â â â â * 797* â â â â * 798* â â â â * 799* Context â â â â * 800* â â â â * 801* â â â â * 802* â â â â * 803* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 804* â clitest <ââââââ® â â âââââââââââââââââââââ® â * 805* â â â â â â Browser â â * 806* ââ pre-flight apt remove â â ââââââââââââââââââââ⤠â * 807* Execution ââ â â â â â Tutorial â â * 808* ââ execute testorial â â â°âââââââââââââââââââ⯠â * 809* ââ | ^ â â | â * 810* ââ apt install â â â apt install â * 811* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââ|âââââââââââââââââââ⣠* 812* ââ â â â â | â * 813* ââ v â â â v â * 814* ââ°â>packages ââââââ⯠â â packages â * 815* Storage â â â â * 816* â â â â * 817* â â â â * 818* â â â â * 819* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 820* * 821********************************************************************************************************* 822 823### There different ways to handle isolation of tests 824Now there are several ways we could handle this: 825 8261. We could have a pre-flight command requiring that cowsay it's not installed. That's called a `set-up` method in unit testing 8272. Make the tutorial clean after it's execution. That's the `tear-down` in unit testing 8283. And finally make our testorial execute on an *isolated* environment, like a docker container. And we will do that in the next phase. 829 830 831## As a DevOps Engineer impact on Storage 832 833Now the impact of storing on Devops Engineer is *minimal* their map is the same as before: 834 835********************************************************************************************************* 836* * 837* * 838* Author DevOps Learner * 839* âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ * 840* â â â â * 841* â â â â * 842* â â â â * 843* Context â â â â * 844* â â â â * 845* â â â â * 846* â â â â * 847* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 848* â clitest <ââââââ® â âââ> GH Workflow â âââââââââââââââââââââ® â * 849* â â â â â â â â Browser â â * 850* â â pre-flight apt remove â â execute â ââââââââââââââââââââ⤠â * 851* Execution â â â â â â â â â â â * 852* â â execute testorial âgit push v â â Tutorial â â * 853* â â | ^ â clitest? ---publish-->â â â * 854* â â apt install â â â â â°âââââââââââââââââââ⯠â * 855* â ââââââââââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠* 856* â â â â â â â â â * 857* â â v â â v â v â * 858* â â°â>packages ââââââ⯠â packages â packages â * 859* Storage â â â â * 860* â â â â * 861* â â â â * 862* â â â â * 863* âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ * 864* * 865********************************************************************************************************* 866[Figure [devops_storage]: Because of container isolation, DevOps don't worry about storage changes] 867 868This is because a DevOps Engineer is already working with an *isolated container* so each push starts a new computer, clean of previous modified state. 869 870Wouldn't it be nice if we could do the same in our Author column? 871 872 873# Revisiting the Storage level of an Author with containers 874 875To have the same level of isolation, we will use containers. 876But not any container, a container specifically designed to imitate the containers that DevOps engineer use. 877So that if our testorial works in our container, we can be certain that it will run on the Continuous Integration pipeline. 878 879Fortunately for such a tool exits *act*
880With act, it will seem like our Author Column has a new Context, Execution, and Storage level, 881That, completely matches the one a DevOps Engineer uses. 882 883*********************************************************************************************************************************** 884* 885* 886* Author DevOps Learner 887* âââââââââ¤âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ 888* âReal Container â â â 889* âCompu â â â â 890* âter â â â 891* Context â â Act ----inspired by ----> GH Runners â â 892* â | â â â 893* â â list â â â 894* â | â â â 895* â ââââââââªââââââââ|ââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠896* â .--> GH Workflow â .--> GH Workflow â âââââââââââââââââââââ® â 897* â â | | â | | â â Browser â â 898* â | execute â | execute â ââââââââââââââââââââ⤠â 899* Execution â â | | â | | â â â â 900* â | v â | v â â Tutorial â â 901* âtestorial-' clitest? ---git push---' clitest? ---publish-->â â â 902* â | â | â â°âââââââââââââââââââ⯠â 903* â ââââââââªââââââââ|ââââââââââââââââââ¬ââââââââ|ââââââââââââââââââ¬âââââ|ââââââââââââââââââ⣠904* â | â | â | â 905* â â v â v â v â 906* â packages â packages â packages â 907* Storage â â â â â 908* â â â â 909* â â â â â 910* â â â â 911* âââââââââ§âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ 912* 913***************************************************************************************************************************************** 914[Figure [author_act_execution]: An Act container creates a new Execution environment] 915 916 917### Using `act` 918 919To install act (run GitHub Actions locally) on Ubuntu: 920 921~~~ bash 922@author $ sudo snap install act #=> --exit 0 923~~~ 924you can verify that you have act installed by: 925 926 927~~~ bash 928@author $ act --version 929@author act version 0.2.67 930~~~ 931 932Now there are three different kinds of machines using act: 933 934Docker image sizes (used by act): 935 936 - Micro (node:16-buster-slim): ~200MB - minimal, may not work for all workflows 937 - Medium (default): ~500MB - good balance, works for most workflows 938 - Large (catthehacker/ubuntu:full-latest): ~18GB - includes most tools 939 940For our case we will use the medium machine. 941So to execute our workflows we will use the workflow we already have in section [GitHub Workflow]. 942
943First let's see if `act` can find our workflow with `--list`. 944 945~~~bash 946@author $ act --list #=> --exit 0 947@author INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' 948@author Stage Job ID Job name Workflow name Workflow file Events 949@author 0 run-testorial run-testorial execute testorials execute_testorials.yml push,pull_request,workflow_dispatch 950~~~ 951 952Now let's execute it: 953 954~~~bash 955@author $ act --platform ubuntu-latest=catthehacker/ubuntu:act-latest --job run-testorial #=> --exit 0 956~~~ 957 958`--platform ubuntu-latest=catthehacker/ubuntu:act-latest` 959: In our workflow when it uses the `ubuntu-latest` runner, use the docker image `catthehacker/ubuntu:act-latest` 960`--job run-testorial` 961: A workflow can have multiple jobs, we only care about the `run-testorial` one. 962 963And now that our workflows run in an isolated environment, we can run them repeatedly 964 965~~~bash 966@author $ act --platform ubuntu-latest=catthehacker/ubuntu:act-latest --job run-testorial #=> --exit 0 967~~~ 968 969Congratulations! 970 971 972### Our updated map 973 974Observe that now the flow of information between the our Author container with `act` is the same as the one for the DevOps Engineer. 975Which is exactly what we want to avoid surprises! 976 977*********************************************************************************************************************************** 978* 979* 980* Author DevOps Learner 981* âââââââââ¤âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââââ¦âââââââââââââââââââââââââ 982* âReal Container â â â 983* âCompu â â â â 984* âter â â â 985* Context â â Act ----inspired by ----> GH Runners â â 986* â | â â â 987* â â list â â â 988* â | â â â 989* â ââââââââªââââââââ|ââââââââââââââââââ¬âââââââââââââââââââââââââââ¬ââââââââââââââââââââââââ⣠990* â .--> GH Workflow â .--> GH Workflow â âââââââââââââââââââââ® â 991* â â | | â | | â â Browser â â 992* â | execute â | execute â ââââââââââââââââââââ⤠â 993* Execution â â | | â | | â â â â 994* â | v â | v â â Tutorial â â 995* âtestorial-' clitest? ---git push---' clitest? ---publish-->â â â 996* â | â | â â°âââââââââââââââââââ⯠â 997* â ââââââââªââââââââ|ââââââââââââââââââ¬ââââââââ|ââââââââââââââââââ¬âââââ|ââââââââââââââââââ⣠998* â | â | â | â 999* â â v â v â v â 1000* â packages â packages â packages â 1001* Storage â â â â â 1002* â â â â 1003* â â â â â 1004* â â â â 1005* âââââââââ§âââââââââââââââââââââââââââ©âââââââââââââââââââââââââââ©âââââââââââââââââââââââââ 1006* 1007***************************************************************************************************************************************** 1008[Figure [author_act_execution]: An Act container creates a new Execution environment] 1009 1010 1011# What have we learned? 1012
1013And with that we have written a Testorial that covers a wide range of problems you might encounter when putting this ideas into practice. 1014 1015+ There are 9 windows: We need to consider all of them when writing executable documentation. 1016+ 3 belong to our audience that now includes *DevOps Engineer* that will integrate our testorials as port of the Continuous Integration workflow. 1017+ And we need to consider 3 levels of context, for each one of the audiences. 1018+ We need to consider all the 9 windows, 1019 But *one by one* if we think of them all at once we get confused. We can decide the following paragraph. 1020 For whom is it? Author, Learner, or DevOps; 1021 and on what level is it working? Context, Execution, or Storage. 1022 1023 1024+ Writing executable documentation is hard, we need to be at the same time: Writer, a Programmer and a Tester. 1025 But LLMs helps us in our weak spots. 1026 1027+ There are things we shouldn't delegate to LLMs 1028 One is the actual execution of the testorial, they can hallucinate the execution. But they can help you debug the testorial. 1029 The other is the writing of the testorial. It has to have your voice and care, but they can help you edit and review it. 1030 1031+ There's no single tool to do executable documentation. 1032 - clitest, to execute our tutorial. 1033 - GitHub Actions to define the CI workflow. 1034 - act to simulate locally, how our tutorial would run on GitHub Action servers. 1035 - Markdeep to publish the tutorial 1036 - plus our standard text editor! 1037 1038 1039 But your selection of tools is context-specific. 1040 For example if your team uses another Continuous Integration tool not github actions, then you would need to change the workflow.yml file. To orient yourself you just need to keep in mind the 9 windows. 1041 1042## Is it worth it? 1043 1044If you want to have always correct, and up-to-date documentation and help your QA team with *real* test cases. 1045It is definitively worth it, now that it has become cheaper to do it, with the help of LLMs 1046 1047 1048> because ... 1049> "The reward for work well done is the opportunity to do more." 1050> 1051> -- Jonas Salk, inventor of the polio vaccine 1052 1053 1054 1055# Footnotes 1056 1057[fn:7] codex exec "Where is stored the workflow that publishes a markdown to github pages?" 1058 1059[fn:6] codex exec "How can I take a screenshot from a website from the command line ?" 1060 1061[fn:5] A linter is program that reads the instructions in a program and gives you suggestions on how to improve it's style. Or points coding that could be errors. 1062 1063[fn:4] https://github.com/aureliojargas/clitest 1064 1065[fn:3] [[https://en.wikipedia.org/wiki/Nine_windows][Wikipedia: Nine Windows]] 1066 1067[fn:2] [[https://en.wikipedia.org/wiki/Literate_programming][wikipedia: Literate Programming]] 1068 1069[fn:1] MacNamara, Riona: 1070[[https://www.youtube.com/watch?v=EnB8GtPuauw][Documentation, Disrupted: How Two Technical Writers Changed Google Engineering Culture]] 1071 Write the Docs conference 2015 1072 1073 1074# Appendix: Configuring GitHub Pages 1075 1076## Install GitHub Pages 1077 1078For simplicity we will use a a CMS GitHub pages. 1079 1080To do that you just need to follow the instructions [https://docs.github.com/en/pages/quickstart](in the quickstart). 1081 1082When you're finish you can see your webpages here: https://$USERNAME.github.io/ 1083 1084And it will look something like this [fn:6]: 1085 1086 1087 1088## Modify a GitHub Page 1089 1090Now let's modify the default `readme.md` document to show that the update mechanism works. 1091 1092Clone your repository to your local machine: 1093 1094~~~{.bash .invisible} 1095 @author:$ export $USERNAME='jag-academy' 1096 @author: exit 0 1097~~~ 1098 1099~~~ bash 1100 @author:$ gh repo clone $USERNAME/$USERNAME.github.io #=> --exit 0 1101~~~ 1102 1103That would create your local copy of your repository. 1104 1105Now you can modify the `readme.md` file. 1106 1107This script appends the current date and time to the readme file to make sure it's modified. 1108 1109~~~ 1110 @author:$ cd $USERNAME.github.io 1111 @author:$ echo "modified on: $(date +'%Y%m%d %H:%M:%S')" >> README.md 1112~~~ 1113 1114Now do commit and push 1115 1116 1117~~~ 1118 @author:$ git commit -am "Append current date and time to trigger regeneration" 1119 @author:$ git push 1120~~~ 1121 1122this will start the republishing of the page. 1123 1124## Check the publishing pipeline 1125 1126Now go to [https://github.com/jag-academy/jag-academy.github.io/actions](github.com/$USERNAME/$REPO/actions) to see your page as it's being generated. 1127 1128Observe that the workflow it's on yellow while it's running. 1129 1130You can even check the detail process that is following. 1131 1132When it finally turns to green means your new website is finished. 1133 1134 1135 1136Got your link again and should see the document with the new date time. 1137 1138https://$USERNAME.github.io/ 1139
1140<script>markdeepOptions={tocStyle:'long'};</script>
1140 1141<!-- Markdeep: --><style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style>
1142<script src="https://morgan3d.github.io/markdeep/latest/markdeep.min.js" charset="utf-8"></script>
vendor: 1 bytes, line 1142
1142
1143<script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>
vendor: 1 bytes, line 1143
1143
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.