1<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Damian Brady]]></title><description><![CDATA[cloud, devops and developer process]]></description><link>https://damianbrady.com.au/</link><image><url>https://damianbrady.com.au/favicon.png</url><title>Damian Brady</title><link>https://damianbrady.com.au/</link></image><generator>Ghost 6.68</generator><lastBuildDate>Thu, 01 Oct 2026 10:21:18 GMT</lastBuildDate><atom:link href="https://damianbrady.com.au/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[MLOps [12] - What is Azure Machine Learning?]]></title><description><![CDATA[Azure Machine Learning is one of the many offerings in the Azure ecosystem, specifically aimed at teams working on machine learning projects. It contains everything you need to create, manage, and deploy your predictive models.]]></description><link>https://damianbrady.com.au/2020/12/01/what-is-aml/</link><guid isPermaLink="false">5fab760dffa752003930727b</guid><category><![CDATA[One Dev Question]]></category><category><![CDATA[MLOps]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 01 Dec 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/12-what-is-aml.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/12-what-is-aml.png" alt="MLOps [12] - What is Azure Machine Learning?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p><a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a> is one of the many offerings in the Azure ecosystem, specifically aimed at teams working on machine learning projects. It contains everything you need to create, manage, and deploy your predictive models.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-is-Azure-Machine-Learning--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What is Azure Machine Learning? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>In short, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a> is an enterprise-grade cloud-based machine learning service that helps you build, train, manage, track, and deploy models more reliably and faster. Ultimately, it's a DevOps tool specifically for machine learning projects.</p><p>From an MLOps perspective, it covers the vast majority of the things you care about - especially those aspects that are different to traditional software projects. Handling data, long-running experiments and training runs, tracking metrics during training, and using powerful and scalable compute are just some examples.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-pixabay-158826.jpg" class="kg-image" alt="MLOps [12] - What is Azure Machine Learning?" loading="lazy" width="1280" height="856" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/12/pexels-pixabay-158826.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/12/pexels-pixabay-158826.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-pixabay-158826.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>An image of a lot of red lights in a grid. Image from <strong><a href="https://www.pexels.com/?ref=damianbrady.com.au">Pexels</a></strong>.</figcaption></figure><p>One of the most important features of Azure Machine Learning is the concept of <a href="https://docs.microsoft.com/en-us
1/azure/machine-learning/concept-ml-pipelines?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure ML Pipelines</a>. As you can probably guess, Azure ML Pipelines are a parallel to traditional continuous integration pipelines in that they define what has to happen to move from code to a potentially deployable model. The big difference is they're specifically designed to be good at ML workflows rather than software builds - in other words, they <em>should</em> focus on machine learning tasks like data preparation and processing, training and validating, saving and registering the resulting models.</p><p>Azure ML Pipelines can also be used for deployment, however there are advantages to orchestrating deployments <em>outside</em> Azure Machine Learning, while still using the deployment capabilities in the product itself. For example, Azure DevOps could be used to manage rollout using proven techniques commonly used in good DevOps practices. Blue-green deployments, gradual rollouts, or canary deployments could be used to more safely roll a change out to users. The actual deployment step could simply be a <a href="https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-train-deploy-model-cli?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#deploy-the-model">call to Azure Machine Learning using the CLI</a> to make use of the powerful deployment capabilities of the service.</p><p><strong>Recently, <a href="https://techcommunity.microsoft.com/t5/azure-ai/azure-machine-learning-s-native-support-for-mlflow/ba-p/1737491?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure Machine Learning announced a public preview of native support for MLflow</a></strong>, meaning (amongst other things) that you can take your existing on-prem workloads and MLOps processes and move them to the cloud. MLFlow is an open source platform for managing the lifecycle of an ML project, which nicely decouples the specification from the implementation. </p><p>Importantly, this means that just as we recommend <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-get-started?view=azure-devops&WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#define-pipelines-using-yaml-syntax">defining CI/CD pipelines in code</a> (<a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions?ref=damianbrady.com.au">using YAML</a>), we can now define our ML pipelines in code the same way!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-spacex-586019.jpg" class="kg-image" alt="MLOps [12] - What is Azure Machine Learning?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/12/pexels-spacex-586019.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/12/pexels-spacex-586019.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-spacex-586019.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A collection of steel pipelines attached to a wall. Image from <strong><a href="https://www.pexels.com/photo/gray-steel-tubes-586019/?ref=damianbrady.com.au">Pexels</a></strong>.</figcaption></figure><p>Pipelines are a huge part of what makes Azure Machine Learning a fantastic tool for MLOps, but there are some other really powerful features it has for data scientists and ML engineers as well.</p><p>A large number of the features in Azure Machine Learning are focused around the artifacts you use and produce. Capabilities like <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-data?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">managing data</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">tracking experiments</a>
1, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#register-package-and-deploy-models-from-anywhere">versioning and managing models</a>, and <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">deploying operationalized models</a> in <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#deploy">various forms including containers on Azure Container Instances or Azure Kubernetes Service</a>.</p><p>There are also features that help you manage some of the benefits of the cloud, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#train">in particular scalable compute for training runs</a> so you only pay for all those souped-up machines and GPUs when you actually need them. There are even specific <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-distributed-training?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">capabilities around distributed training and data parallelism </a>to really speed things up.</p><p>Finally, there are fantastic cloud-based tools specifically for machine learning engineers in the <a href="https://docs.microsoft.com/en-us/azure/machine-learning/overview-what-is-machine-learning-studio?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure Machine Learning Studio</a>. <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-run-jupyter-notebooks?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Hosted Jupyter Notebooks</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-labeling-projects?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">data labeling capabilities</a>, the <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-designer?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Azure Machine Learning Designer</a> (drag and drop pipelines!), and even <a href="https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-first-experiment-automated-ml?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">Automated Machine Learning capabilities</a> so you don't have to do all the busywork of comparing algorithms and tuning hyperparameters.</p><p><strong>There's a lot, and so much of it is mind-blowing.</strong></p><p>As a bonus, Azure Machine Learning comes with a <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-responsible-ml?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">number of features focused on ethics and responsible machine learning</a>, which is becoming more and more important as our models become better and easier to create.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-mati-mango-5952651.jpg" class="kg-image" alt="MLOps [12] - What is Azure Machine Learning?" loading="lazy" width="1280" height="854" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/12/pexels-mati-mango-5952651.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/12/pexels-mati-mango-5952651.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/12/pexels-mati-mango-5952651.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A man in a hoodie with numbers projected on his face. Photo by <strong><a href="https://www.pexels.com/@mati?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Mati Mango</a></strong> from <strong><a href="https://www.pexels.com/photo/text-5952651/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>If you want to get started with Azure Machine Learning, I highly recommend the "
1<a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">Build AI solutions with Azure Machine Learning</a>" learning path on Microsoft Learn. It's an awesome in-depth resource.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [11] - How can MLOps improve my predictive models?]]></title><description><![CDATA[Changing the way you work doesn't necessarily improve the work you produce. However, by focusing on the right things, the final result can be improved.]]></description><link>https://damianbrady.com.au/2020/11/19/how-can-mlops-improve-my-models/</link><guid isPermaLink="false">5fab74fdffa752003930725a</guid><category><![CDATA[One Dev Question]]></category><category><![CDATA[MLOps]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Thu, 19 Nov 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/11-improve-predictive-models.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/11-improve-predictive-models.png" alt="MLOps [11] - How can MLOps improve my predictive models?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>It might seem that changing the way you work doesn't necessarily improve the work you produce. However, by focusing on the right things, the final result can be improved simply because you're spending the time on the product rather than peripheral issues.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/How-can-MLOps-improve-my-predictive-models--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="How can MLOps improve my predictive models? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>In a <a href="https://damianbrady.com.au/2020/11/12/why-care-about-mlops/">previous post</a>, I talked about how implementing MLOps meant that all the peripheral tasks around actually creating a predictive model fall away, allowing your data scientists to focus on creating the best solution.</p><blockquote>Let experts focus on what they're best at.</blockquote><p>In short, if all of the effort of the machine learning experts and data scientists is going towards producing better models, there's much less wasted effort. It's focused on the end goal.</p><p>A good MLOps process can take care of most of the heavy listing when it comes to things like <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">managing engineering environments</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">managing production environments</a>, <a href="https://github.com/?ref=damianbrady.com.au">versioning and collaborating on code</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">executing experiments and training runs</a>, testing, validating, and v<a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au#register-package-and-deploy-models-from-anywhere">ersioning predictive models</a>
1, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?WT.mc_id=devops-9710-dabrady&tabs=azcli&ref=damianbrady.com.au#define-an-inference-configuration">packaging those models into a deployable form</a>, and <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au#deploy-your-model">deploying them in safe ways</a>.</p><p><a href="https://twitter.com/realgenekim/?ref=damianbrady.com.au">Gene Kim</a> talks about <a href="https://www.infoq.com/articles/unicorn-project/?ref=damianbrady.com.au">"focus, flow, and joy" as the second of his DevOps ideals</a>. What this means in practice is that the more you can focus on a single task, the better the outcome!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-941555.jpg" class="kg-image" alt="MLOps [11] - How can MLOps improve my predictive models?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-andrea-piacquadio-941555.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-andrea-piacquadio-941555.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-941555.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A woman focusing on her laptop. Photo by <strong><a href="https://www.pexels.com/@olly?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Andrea Piacquadio</a></strong> from <strong><a href="https://www.pexels.com/photo/photo-of-a-woman-thinking-941555/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>Just as important is being able to know <em>what </em>is valuable. <a href="https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops?WT.mc_id=devops-9710-dabrady&ref=damianbrady.com.au">DevOps is all about delivering value to end users</a>, but without measuring what is and isn't valuable, you can't know whether the work you're putting in is positive change. This also goes for MLOps.</p><p>For example, your data scientists may have created a better predictive model by adding extra layers and nodes to the neural network. It now makes a correct prediction 95% of the time instead of 90%. Great, right?</p><p>But this extra 5% <em>may not be valuable</em>. If that model takes twice as long to make predictions, that 5% gained by users may be more than offset by the pain of waiting for a response. Where the model was supposed to make (for example) filling out a form faster, it might now be faster just to type.</p><p>If there was no (or slow) feedback telling the team that performance needed to be a priority, there's a good chance all that effort is being poured into eking out another couple of percentage points of accuracy on a model that's too slow to use.</p><p>So just as we do with agile software development, it's important to look at where the effort should go, and objectively evaluate (as much as possible) whether a change has provided value or not.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3760810.jpg" class="kg-image" alt="MLOps [11] - How can MLOps improve my predictive models?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-andrea-piacquadio-3760810.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-andrea-piacquadio-3760810.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3760810.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A man at a laptop looking at his watch. Photo by <strong><a href="https://www.pexels.com/@olly?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Andrea Piacquadio</a></strong> from <strong><a href="https://www.pexels.com/photo/confused-businessman-checking-time-on-wristwatch-3760810/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>So an organisation with good MLOps practices can ultimately produce better predictive models in two ways:</p><ol><li>They let the machine learning experts and data scientists focus on what they're good at - creating better models</li><li>They encourage fast feedback so that focus can be directed to the right place</li></ol>]]></content:encoded></item><item><title><![CDATA[MLOps [10] - When should I think about MLOps?]]></title><description>
1<![CDATA[Learning a new way of working can be challenging. But it's important to applying MLOps practices and techniques at the right time. That time is as early as possible!]]></description><link>https://damianbrady.com.au/2020/11/17/when-should-i-think-about-mlops/</link><guid isPermaLink="false">5fab7447ffa7520039307240</guid><category><![CDATA[One Dev Question]]></category><category><![CDATA[MLOps]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 17 Nov 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/10-when-should-i-think.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/10-when-should-i-think.png" alt="MLOps [10] - When should I think about MLOps?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>
1Learning a new way of working can be challenging. But it's important to apply MLOps practices and techniques at the right time. That time is as early as possible!</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/When-should-I-think-about-MLOps--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="When should I think about MLOps? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>There's a common anti-pattern I see in software projects, and I've seen it for years. <em>Leaving the deployment of your application until there's something meaningful to deploy.</em></p><p>On the surface, it sounds like a reasonable decision. After all, time is limited, and if the application doesn't even do the minimum yet, surely spending time trying to deploy it is a waste of cycles? Better to work on features, right?</p><p>The problem with this argument is that by deferring deployment to the end, you're building up software that's more and more complex, and therefore harder and harder to deploy. In many cases, delaying means you'll never get a great result because the system won't have been designed for easy deployment.</p><p>It's similar to software testing. If you're doing <a href="https://docs.microsoft.com/en-us/visualstudio/test/quick-start-test-driven-development-with-test-explorer?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">test-driven development (TDD)</a> or even just writing tests when you write the code, your code <em>necessarily</em> becomes testable. If write some code and can't write a test for it, you immediately know it's going to be hard to diagnose if there's a problem.</p><p>Similarly, if you only think about how you're going to deploy your software once it's large and complex, you miss the opportunity to <a href="https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/moving-cloud-cadence?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">optimize for a fast cadence</a>. Being able to continuously and quickly deploy small changes is one of the goals of DevOps, and leads to much better software delivery outcomes.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-ruiyang-zhang-3717291.jpg" class="kg-image" alt="MLOps [10] - When should I think about MLOps?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-ruiyang-zhang-3717291.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-ruiyang-zhang-3717291.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-ruiyang-zhang-3717291.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>Blurred cars moving fast on a road. Photo by <strong><a href="https://www.pexels.com/@ruiyang-zhang-915467?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Ruiyang Zhang</a></strong> from <strong><a href="https://www.pexels.com/photo/time-lapse-photo-of-cars-in-asphalt-road-3717291/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>So how does this relate to ML projects? <strong>In exactly the same way.</strong></p><p>If you only think about how you're going to <a href="https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-1st-experiment-sdk-train?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">train</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-train-deploy-model-cli?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au#deploy-the-model">package, and deploy</a> your models towards the <em>end</em> of your project, it's going to be much harder to do.</p><p>For example, let's imagine you have a team of data scientists working on a <a href="https://azure.microsoft.com/en-us
1/blog/building-recommender-systems-with-azure-machine-learning-service/?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">recommendation engine</a> for your product catalogue. The ultimate goal is to use this model on your company's ecommerce site to increase sales.</p><p>You and your team work relatively independently with your own <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-run-jupyter-notebooks?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">Jupyter notebooks</a> and database extracts. Maybe you pass notebooks to each other by email and have regular meetings and pairing sessions to make sure you're all working together. Training is performed by running your notebooks on one of a few shared <a href="https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/overview?WT.mc_id=devops-9708-dabrady&ref=damianbrady.com.au">Data Science Virtual Machines</a> in the cloud.</p><p>At some point, you get a solution you think will work. It's producing good recommendations - at least from the data extracts you've been using! You have a great spreadsheet with plenty of examples and proof, pulled right from your notebook.</p><p>So, time to move to the next step. But how do you do that?</p><ul><li>How do you get from those few cells in Jupyter to a production ecommerce site?</li><li>How can the web application use your model?</li><li>Can your team reproduce your experiment and work on tweaking it for better results?</li><li>How will you know if the model is performing in the real world?</li></ul><p>This can be the hardest part. Modifying what you've done in order to create something you can operationalize can be really tricky. If you don't consider it until the end of your project, you might be in for a world of frustration and rework!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3807738.jpg" class="kg-image" alt="MLOps [10] - When should I think about MLOps?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-andrea-piacquadio-3807738.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-andrea-piacquadio-3807738.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3807738.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A woman at a laptop looking frustrated. Photo by <strong><a href="https://www.pexels.com/@olly?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Andrea Piacquadio</a></strong> from <strong><a href="https://www.pexels.com/photo/woman-in-white-shirt-showing-frustration-3807738/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>In my consulting days, we had a process for new projects. Right at the start, we'd build a full end-to-end pipeline for the absolute minimum application. For example, if it was a web app, it might literally be a "<a href="https://en.wikipedia.org/wiki/%22Hello,_World!%22_program?ref=damianbrady.com.au">Hello World</a>". We'd have all the code in source control, a continuous integration build that ran an empty test suite, and continuous deployment to "test", "staging", then "production".</p><p>Even though very little code would survive to the end, and the CI/CD processes built, tested, and deployed almost nothing to servers <em>pretending</em> to be test/staging/production, it meant that we had our pipeline and processes already working. We got into the habit of using it, tweaking it, and trusting it.</p><p>Importantly, when we added a feature, a dependency, or something that required a change to the pipeline, it was just that - a <em>change</em>. Not something we have to work out in 6 months, or an unexpected, "oh I forgot about that" surprise right before project delivery date. Just like code, <em>small frequent changes are easier to manage than big infrequent ones.</em></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-taryn-elliott-3889661.jpg" class="kg-image" alt="MLOps [10] - When should I think about MLOps?" loading="lazy" width="1280" height="1600" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-taryn-elliott-3889661.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-taryn-elliott-3889661.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-taryn-elliott-3889661.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>Small huts on the beach. Photo by <strong><a href="https://www.pexels.com/@taryn-elliott?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Taryn Elliott</a></strong> from <strong><a href="https://www.pexels.com/photo/cozy-optimist-change-rooms-on-seashore-beach-in-winter-3889661/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>DevOps is similar to design, accessibility, testing, security, and a host of other practices in that it should be considered right from the start. It's very difficult to get a good result if you only think about these things once most of the work is done!</p><p>The same goes for MLOps. </p><p>The best time to think about MLOps is day one. The second best time is now.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [9] Why should I care about MLOps?]]></title><description><![CDATA[If you're working on machine learning projects, MLOps is not just another buzzword that can safely be ignored. Applying MLOps can mature your projects and lead to better outcomes.]]></description><link>https://damianbrady.com.au/2020/11/12/why-care-about-mlops/</link><guid isPermaLink="false">5fab71a0ffa752003930721a</guid><category><![CDATA[One Dev Question]]></category><category><![CDATA[MLOps]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Thu, 12 Nov 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/9-why-care-about-mlops.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/9-why-care-about-mlops.png" alt="MLOps [9] Why should I care about MLOps?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>If you're delivering software, DevOps has become a must-have. It's not just something you <em>should</em> do, but something you <em>must do </em>if you want to compete.</p><p>Along those lines, if you're working on machine learning projects, MLOps is not just another buzzword that can safely be ignored. Applying MLOps can mature your projects and lead to better outcomes.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/Why-should-I-care-about-MLOps--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="Why should I care about MLOps? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>It's becoming a <a href="https://damianbrady.com.au/2020/10/27/devops-practices-applied-to-ml-projects/">recurring theme in this series</a>, but it's true - it's easy to think of ML projects as significantly different than traditional software projects. The day to day work is different, so surely the surrounding processes can be different!</p><p>For example, the work of producing a predictive model is often done by a smaller team or even a single person, rather than a team of engineers. So why bother with source control?</p><p>It's also really common for the deployment cadence to be much slower in an ML project. Teams just don't deploy that often and it can take some time to get a result you're happy to release.</p><p>All this begs the question - why care about MLOps at all? The work doesn't mirror what happens in traditional development, so why are we trying to force ML projects into the DevOps mold?</p><p>First, it'
1s worth revisiting <a href="https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">Microsoft's definition for DevOps</a>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png" class="kg-image" alt="MLOps [9] Why should I care about MLOps?" loading="lazy" width="1660" height="936" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/devops-definition.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/devops-definition.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/devops-definition.png 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png 1660w" sizes="(min-width: 720px) 720px"><figcaption>Microsoft's DevOps definition, written by Donovan Brown</figcaption></figure><blockquote>DevOps is the union of people, process, and products to enable continuous delivery of <strong>value</strong> to the end user.</blockquote><p>As I mentioned <a href="https://damianbrady.com.au/2020/10/20/what-is-mlops/">earlier in the series</a>, the real focus of DevOps is delivering value. Not code, not bug fixes, not scalability - but <strong>value</strong>. So the definition absolutely includes ML projects. There is value in a better predictive model.</p><p>So while we're saying "MLOps", we really mean "DevOps for ML", and the same benefits should apply. As long as we're conscious of <a href="https://damianbrady.com.au/2020/10/29/how-does-mlops-differ-from-devops/">the differences between the two specialties</a>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-thisisengineering-3862627.jpg" class="kg-image" alt="MLOps [9] Why should I care about MLOps?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-thisisengineering-3862627.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-thisisengineering-3862627.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-thisisengineering-3862627.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A team looking at a laptop on a car factory floor. Photo by <strong><a href="https://www.pexels.com/@thisisengineering?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">ThisIsEngineering</a></strong> from <strong><a href="https://www.pexels.com/photo/engineers-in-workshop-3862627/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>I'm often asked about the tangible benefits of DevOps practices (and MLOps practices). Sure, things can be deployed quickly, but what's <em>really </em>wrong with a hard deployment if you only do it every few months?</p><p>I could speak for <em>hours </em>on that topic, but it all boils down to one thing:</p><blockquote>Let experts focus on what they're best at.</blockquote><p>The way to do this is to make sure all the surrounding aspects of their job <em>just work</em>. That's things like <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">managing engineering environments</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">managing production environments</a>, <a href="https://github.com/?ref=damianbrady.com.au">versioning and collaborating on code</a>
1, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">executing experiments and training runs</a>, testing, validating, and v<a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au#register-package-and-deploy-models-from-anywhere">ersioning predictive models</a>, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?WT.mc_id=devops-9707-dabrady&tabs=azcli&ref=damianbrady.com.au#define-an-inference-configuration">packaging those models into a deployable form</a>, and <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au#deploy-your-model">deploying them in safe ways</a>.</p><p>If all that stuff just falls away, data scientists can focus on the data science. Machine learning experts can focus on creating good predictive models, and know that their experiments will run consistently, they'll <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">get the information they need to make decisions</a>, and if they produce something good, it can be deployed without a fight.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3808018.jpg" class="kg-image" alt="MLOps [9] Why should I care about MLOps?" loading="lazy" width="1280" height="823" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-andrea-piacquadio-3808018.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-andrea-piacquadio-3808018.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-andrea-piacquadio-3808018.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A woman wearing headphones focusing on a laptop. Photo by <strong><a href="https://www.pexels.com/@olly?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Andrea Piacquadio</a></strong> from <strong><a href="https://www.pexels.com/photo/woman-in-yellow-shirt-wearing-white-headphones-3808018/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>Sound familiar? It should - these are all exactly the same benefits as DevOps. There's <a href="https://www.amazon.com.au/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339?ref=damianbrady.com.au">plenty of research out there</a> by experts such as <a href="https://nicolefv.com/?ref=damianbrady.com.au">Dr Nicole Forsgren</a>, <a href="https://itrevolution.com/faculty/gene-kim/?ref=damianbrady.com.au">Gene Kim</a>, and <a href="https://itrevolution.com/faculty/jaz-humble/?ref=damianbrady.com.au">Jez Humble</a> that shows there are measurable, tangible benefits to doing DevOps.</p><p>DevOps overlaps heavily with MLOps. <a href="https://medium.com/mlops-community/mlops-is-just-glorified-devops-bccd5d86bfba?ref=damianbrady.com.au">You could even argue (and I do) that <em>it's the same thing</em>, just with a slightly specialised focus</a>. MLOps is solving the same problem as DevOps - delivering value to end users.</p><p>One tool that's extremely good at managing the MLOps lifecycle is <a href="https://azure.microsoft.com/en-au/services/machine-learning/?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a>. As well as features that make creating models easier than ever, it can manage pipelines, track experiment and model versions, and even package and deploy to production.</p><p>If you want to get started with Azure Machine Learning, the best place to get hands-on experience (for free) is <a href="https://docs.microsoft.com/learn?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">Microsoft Learn</a>. The "
1<a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">Build AI solutions with Azure Machine Learning</a>" learning path is an awesome in-depth resource.</p><p>And of course, there's the usual collection of <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9707-dabrady&ref=damianbrady.com.au">comprehensive documentation on Microsoft Docs </a>to answer any additional questions you have along the way.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [8] Can I use DevOps tooling for Machine Learning projects?]]></title><description><![CDATA[As a software developer, it can be tempting to use or recommend the tools you're used to when implementing MLOps. In some cases, the tools you already use are great, but in other cases they're really not fit for purpose.]]></description><link>https://damianbrady.com.au/2020/11/10/mlops-8-can-i-use-devops-tooling-for-machine-learning-projects/</link><guid isPermaLink="false">5fa8a1b9ffa752003930708e</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 10 Nov 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/8-existing-devops-tooling.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/8-existing-devops-tooling.png" alt="MLOps [8] Can I use DevOps tooling for Machine Learning projects?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>As a software developer, it can be tempting to use or recommend the tools you're used to when implementing MLOps. In some cases, the tools you already use are great, but in other cases they're really not fit for purpose.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/Can-I-use-DevOps-tooling-for-Machine-Learning-projects--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="Can I use DevOps tooling for Machine Learning projects? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>There are countless tools people use for managing their software delivery lifecycle. We all have our favourites (<a href="https://code.visualstudio.com/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">VS Code</a>, <a href="https://github.com/?ref=damianbrady.com.au">GitHub</a>, <a href="https://azure.microsoft.com/en-us/services/devops/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure DevOps</a>, and <a href="https://azure.microsoft.com/en-us/services/monitor/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure Monitor</a> are mine), but they (and the majority) are focused around traditional software development.</p><p>That's not to say that these tools can't be used for machine learning projects, but many aren't designed for those workflows.</p><h3 id="source-control">Source Control</h3><p><a href="https://damianbrady.com.au/2020/11/02/parts-of-an-ml-project-in-source-control/">In the last post</a>, we talked about source control, so let's use that as a simple example.</p><p>Keeping all your python or R code in a git repository makes sense, along with your training and inferencing environment specifications and any other infrastructure-as-code. But as we saw, data is a different story.</p><p>If you'
1re using <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a>, tools like the built-in <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-version-track-datasets?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">versioning capabilities of datasets</a> can be exceedingly useful. If used properly, this can enable you to effectively version your training data each time it changes or as needed.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-min-an-962317.jpg" class="kg-image" alt="MLOps [8] Can I use DevOps tooling for Machine Learning projects?" loading="lazy" width="1280" height="851" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-min-an-962317.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-min-an-962317.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-min-an-962317.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>Silhouette showing many branches of a tree. Photo by <strong><a href="https://www.pexels.com/@minan1398?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Min An</a></strong> from <strong><a href="https://www.pexels.com/photo/monochrome-photography-of-bare-tree-962317/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><h3 id="continuous-integration-training">Continuous Integration / Training</h3><p>Similarly, in <a href="https://damianbrady.com.au/2020/10/29/how-does-mlops-differ-from-devops/">part 6 of this series</a>, we noted that traditional "continuous integration" strategies were unlikely to work due to the data- and computationally-intensive training process. If every change leads to a full training run, you better have extremely deep pockets.</p><p>Even if money is no object, the fact that these training runs usually take much longer than traditional software projects means the CI/CD tooling is unlikely to handle it well.</p><p>For example, GitHub Actions has <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration?ref=damianbrady.com.au">a limit of 72 hours for a workflow run</a>. This sounds like a lot (and it is considering <a href="https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/githubs-products?ref=damianbrady.com.au#github-free-for-user-accounts">so much is free</a>), but if you're using GitHub-hosted runners, <a href="https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration?ref=damianbrady.com.au">each job in that workflow can only run for 6 hours</a>. That's plenty of time for all but the most extreme software builds, but it's unlikely to be enough for serious machine learning training runs. Of course, if you can get your training done in <a href="https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners?ref=damianbrady.com.au#usage-limits">under 24 hours, you might be able to get away with self-hosted runners</a> - but then you're managing the infrastructure yourself.</p><p>But let's assume you <em>can </em>effectively run your training in that time. Because these systems aren't built for working with large amounts of data, you'll have to consider how you get access to and use that data efficiently. You'll also probably want to <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-runs?tabs=python&WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">monitor the progress of your experiment</a> visually rather than trying to parse through logs.</p><p>Tools like Azure Machine Learning are <a href="https://docs.microsoft.com/en-us
1/azure/machine-learning/how-to-create-register-datasets?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">built around these specific requirements</a>, so they're better suited to the task.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/aml_metrics.png" class="kg-image" alt="MLOps [8] Can I use DevOps tooling for Machine Learning projects?" loading="lazy" width="1329" height="501" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/aml_metrics.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/aml_metrics.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/aml_metrics.png 1329w" sizes="(min-width: 720px) 720px"><figcaption>An example of two graphs of metrics (accuracy and loss) produced while training in Azure Machine Learning</figcaption></figure><p>But even if it's not advisable to do the actual training there, <a href="https://github.com/features/actions?ref=damianbrady.com.au">GitHub Actions</a> is a great place to <em>trigger</em> a workflow when something changes - especially code.</p><p>For example, you might modify your code to try a different activation function. You run a few tests locally with a subset of data and are confident it's worth a try. So you commit and push the change.</p><p>GitHub Actions can react to that commit by kicking off a workflow - passing control to . That workflow might use the <a href="https://github.com/Azure/actions?ref=damianbrady.com.au#train-and-deploy-machine-learning-models">actions for Azure Machine Learning</a> to update and run a new training pipeline (<a href="https://github.com/machine-learning-apps/ml-template-azure?ref=damianbrady.com.au">learn how here</a>), and send a message to a Teams channel to let the team know there's a new training run, complete with links to the experiment itself.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/hal-experiments.png" class="kg-image" alt="MLOps [8] Can I use DevOps tooling for Machine Learning projects?" loading="lazy" width="1189" height="577" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/hal-experiments.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/hal-experiments.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/hal-experiments.png 1189w" sizes="(min-width: 720px) 720px"><figcaption>An example of experiment runs in Azure Machine Learning.</figcaption></figure><h3 id="deployment-and-monitoring">Deployment and Monitoring</h3><p>Controlling the actual deployment of a predictive model - ideally in an isolated, portable format - is a task that can absolutely be done with traditional DevOps tools. In terms of portability, one common delivery technique is wrapping the predictive model in a docker container, so it can be deployed anywhere and scaled as required.</p><p>In fact, most of the behaviours you'd want to emulate when going to production are the bread and butter of standard deployment automation tools. Gradual rollout techniques like <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/kubernetes/canary-demo?view=azure-devops&tabs=yaml&WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">canary deployments</a>, <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops&WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au#rolling-deployment-strategy">rolling deployments</a>, using <a href="https://docs.microsoft.com/en-us/azure/devops/migrate/phase-rollout-with-rings?view=azure-devops&WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">deployment rings</a>, and even <a href="https://azure.github.io/AppService/2020/08/03/ab_testing_app_service.html?ref=damianbrady.com.au">A/B testing</a> of models or <a href="https://twitter.com/damovisa/status/1295853224386154496?ref=damianbrady.com.au">shadow deployments</a> can be handled beautifully by tools like Azure Pipelines. After all, at this point, you have a piece of software that needs to be deployed. Where it came from is not relevant if the deployment method is the same.</p><p>Similarly, monitoring how your model is being used in production can be handled effectively by existing DevOps tooling. Provided the right logs and events are being produced by your model (or any software), <a href="https://azure.microsoft.com/en-us
1/services/monitor/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure Monitor</a> can give you all the insights you need.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/azure-monitor.png" class="kg-image" alt="MLOps [8] Can I use DevOps tooling for Machine Learning projects?" loading="lazy" width="756" height="606" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/azure-monitor.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/azure-monitor.png 756w" sizes="(min-width: 720px) 720px"><figcaption>An example of some metrics tracked in Azure Monitor</figcaption></figure><p>In summary, you <em>can </em>use many of the tools you're used to for traditional software development:</p><ul><li>Git is great for any text-based file versioning and collaboration.</li><li><a href="https://github.com/features/actions?ref=damianbrady.com.au">GitHub Actions</a> and <a href="https://azure.microsoft.com/en-us/services/devops/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure Pipelines</a> are great at kicking off pipelines</li><li><a href="https://azure.microsoft.com/en-us/services/devops/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure Pipelines</a> is fantastic at carefully controlling rollout of your model</li><li><a href="https://azure.microsoft.com/en-us/services/monitor/?WT.mc_id=devops-9706-dabrady&ref=damianbrady.com.au">Azure Monitor</a> can be used to observe what your model is actually doing in production</li></ul><p><strong>BUT </strong>the actual training should be done using specialist tools, and you'll need to consider how you're going to version and access data both day-to-day and during training.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [7] - What parts of an ML project should be in source control?]]></title><description><![CDATA[With traditional software development, it's important that everything goes in source control. The same can be said for ML projects, but there are some important considerations!]]></description><link>https://damianbrady.com.au/2020/11/02/parts-of-an-ml-project-in-source-control/</link><guid isPermaLink="false">5f7dab4331ce320039f0e7a9</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Mon, 02 Nov 2020 16:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/7-ml-project-source-control.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/7-ml-project-source-control.png" alt="MLOps [7] - What parts of an ML project should be in source control?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>With traditional software development, it's important that everything goes in source control. The same can be said for ML projects, but there are some important considerations!</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-parts-of-an-ML-project-should-be-in-source-control--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What parts ofan ML project should be in source control? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>Any DevOps evangelist will tell you <em>everything</em>
1 should go in source control. Your code, infrastructure definitions, dev environment setup, database schema, even lookup data. And this is <em>generally</em> good advice.</p><p>The problem comes when this data is too large to be reasonably stored in traditional source control.</p><p><a href="https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">When Microsoft moved the Windows codebase to a git repository</a>, they ultimately had to write a whole new "virtual file system" (<a href="https://github.com/Microsoft/VFSForGit?ref=damianbrady.com.au">VFS for Git</a>) to handle the size. As it stood, git couldn't handle the 300 GB size of the repository. Many commands would take upwards of an hour to run.</p><p>300 GB is not a particularly large amount of data for many ML projects. As <a href="https://damianbrady.com.au/2020/10/29/how-does-mlops-differ-from-devops/">mentioned previously in this series</a>, machine learning projects rely very heavily on data as well as the code. That data is often massive. Like, petabytes massive. This much data just won't work in a Git repository. Even with a virtual file system, you'll still have to pull all those files down to do a training run. It's just not feasible.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-pixabay-158826.jpg" class="kg-image" alt="MLOps [7] - What parts of an ML project should be in source control?" loading="lazy" width="1280" height="856" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-pixabay-158826.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-pixabay-158826.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-pixabay-158826.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A seemingly infinite grid of red lights. Image from <a href="https://pexels.com/?ref=damianbrady.com.au">Pexels</a>.</figcaption></figure><h3 id="everything-except-the-data">Everything except the data</h3><p>There are solutions for data - which we'll get to - but to be clear, <em>everything else</em> should be in source control.</p><p>That means:</p><ul><li>All the code required to train your model</li><li>Code for testing and validation of your code and trained models</li><li>Definition of your data and training pipelines</li><li>Definition and configuration of your training environment</li><li>Definition and configuration of your inferencing/production environment</li><li><em>Maybe</em> a subset of the data so data scientists can prove out ideas locally</li></ul><p>Ultimately, your source repository should contain everything a data scientist could reasonably require to start working on your project and be productive.</p><p>Another (darker) way of thinking about it is what you'd lose if your local machine, training environments, and production environments all stopped working. Could you get started again quickly? Or at all?</p><p>Keeping everything in source control also allows you to go back in time. This is important for <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-machine-learning-interpretability?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">explainability</a>
1 or diagnosing issues, and you may not know that there's a problem with a model in production until well after deployment. At that point, your code and data likely look very different than what's currently. If there is an issue - for example with bias, harmful predictions, or even just unexpected failures - it's important to be able to identify what caused that issue.</p><h3 id="what-about-the-data">What about the data?</h3><p>As I mentioned, there are solutions for storing and versioning large amounts of data.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-pixabay-164686.jpg" class="kg-image" alt="MLOps [7] - What parts of an ML project should be in source control?" loading="lazy" width="1280" height="850" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/11/pexels-pixabay-164686.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/11/pexels-pixabay-164686.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/11/pexels-pixabay-164686.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A hand-written leger of transactions. Image from <a href="https://pexels.com/?ref=damianbrady.com.au">Pexels</a>.</figcaption></figure><p>From a pure data-versioning standpoint, products like <a href="https://dvc.org/?ref=damianbrady.com.au">DVC </a>and <a href="https://www.pachyderm.com/?ref=damianbrady.com.au">Pachyderm</a> are good options. They allow you to specifically attach versions to your data at specific points in time.</p><p>If you're using Databricks (on Azure or elsewhere), you can make use of <a href="https://docs.microsoft.com/en-us/azure/databricks/delta/?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">Delta Lake</a> to give you "time travel".</p><p>If you're using <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a>, you can also use the built-in <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-version-track-datasets?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">versioning capabilities of datasets</a> - a native part of Azure ML.</p><p>And of course if your data is stored elsewhere in Azure, you can make use of built-in versioning capabilities like <a href="https://docs.microsoft.com/en-us/azure/storage/blobs/versioning-overview?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">Blob versioning</a> or <a href="https://docs.microsoft.com/en-us/azure/azure-sql/database/recovery-using-backups?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au#point-in-time-restore">Point-in-time Restore</a> for Azure SQL.</p><p>One final note - many of these options require some thought when it comes to data engineering in the first place. One common strategy is making your data immutable and append-only. You can't modify, you only write new records and timestamp them appropriately. That allows you to run queries to identify the state of the data at any point in time. <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing?WT.mc_id=devops-9705-dabrady&ref=damianbrady.com.au">Event Sourcing</a> is one popular way to implement this, where the focus is on ordered state changes rather than the current state.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [6] - How does MLOps differ from DevOps?]]></title><description><![CDATA[While machine learning projects and traditional software development have many similarities, there are some important differences. This means blindly applying the practices of one to the other is unlikely to be successful.]]></description><link>https://damianbrady.com.au/2020/10/29/how-does-mlops-differ-from-devops/</link><guid isPermaLink="false">5f7da7c231ce320039f0e781</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><category><![CDATA[DevOps]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Thu, 29 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/5-MLOps-differ-from-DevOps.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/5-MLOps-differ-from-DevOps.png" alt="MLOps [6] - How does MLOps differ from DevOps?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>While machine learning projects and traditional software development have many similarities, there are some important differences. This means blindly applying the practices of one to the other is unlikely to be successful.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/How-does-MLOps-differ-from-DevOps--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="How does MLOps differ from DevOps? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>In the <a href="https://damianbrady.com.au/2020/10/27/devops-practices-applied-to-ml-projects/">last blog post</a> I talked about how the day-to-day work of a data scientist is usually <em>very </em>different than the day-to-day work of a traditional software engineer. Even so, the processes <em>around</em> that work can benefit greatly from DevOps practices, techniques, and ideas.</p><p>To understand where MLOps differs from DevOps, it's worth drilling into that first point - what is the difference when it comes to the day-to-day work?</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/women-coders-working-resize.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1080" height="720" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/women-coders-working-resize.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/women-coders-working-resize.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/women-coders-working-resize.jpg 1080w" sizes="(min-width: 720px) 720px"><figcaption>Two female developers collaborating in the office</figcaption></figure><h3 id="experimentation">Experimentation</h3><p>The first difference is to do with experimentation. While it's not unusual for a traditional software engineer to experiment and try things out, these "<a href="https://en.wikipedia.org/wiki/Spike_(software_development)?ref=damianbrady.com.au">spikes</a>" are usually relatively short-lived and isolated from the main project. It's rare that a developer will do a week's worth of work, realise it's not the right direction, then abandon it. More importantly, these spikes generally don't need to be kept around or tracked. They're not pertinent to the rest of the work going on.</p><p>That's not the case in machine learning projects. It's quite common for a data scientist to experiment with solving a problem a particular way, only to stop and try another method several days or weeks down the track.</p><p>Of course there are tools to help with this experimentation process. Azure Machine Learning has an <a href="https://docs.microsoft.com/en-us
1/azure/machine-learning/concept-automated-ml?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">automated machine learning feature</a> allowing you to run a whole lot of experiments in parallel and compare them to get the best performing models quicker.</p><p>Regardless of how you run all these experiments, the very point of experimentation is to find those methods that work and those that don't. <strong>The "failures" are very rarely mistakes</strong>, rather they represent the reality of machine learning. You're trying to solve a problem where you can't personally envisage the solution. </p><p>It actually represents a more agile way of working than we usually see with traditional software development. You frequently don't know what you'll find until you try something, and it's important to be able to change course if it's the sensible thing to do.</p><p>But if it might take weeks of experimentation to find a feasible path, does that mean all the failed attempts should be discarded? Well, no. It's important that these experiments are still tracked for a few reasons. It can be helpful to the rest of your team to see what's been tried and what went wrong. Alternatively, these experiments might be more successful than you originally thought and worth another look later in the project. <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">Ultimately, an "experiment" represents a method used to find a solution, and they're all important in exactly the same way as any other scientific field.</a></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-thisisengineering-3861968.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1280" height="853" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-thisisengineering-3861968.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-thisisengineering-3861968.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-thisisengineering-3861968.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A female engineer in a sound studio. Photo by <strong><a href="https://www.pexels.com/@thisisengineering?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">ThisIsEngineering</a></strong> from <strong><a href="https://www.pexels.com/photo/female-engineer-in-sound-studio-3861968/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><h3 id="working-with-data">Working with data</h3><p>As the name suggests, data scientists work primarily with data. And data is unquestionably the most important component of a machine learning project. Erroneous, insufficient, or unrepresentative data leads to bad predictive models. In other words, data that doesn't accurately reflect the real world will ultimately lead to models that don't work in the real world. It's also important to realise and acknowledge <a href="https://www.microsoft.com/en-us/ai/responsible-ai?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">that the machine learning process can perpetuate and even exaggerate biases that exist in that data</a>.</p><p>Yes, there is code. But if you put all the code required for an ML project alongside all the data required, it's likely to be a fraction of a fraction of a percent.</p><p>However, with traditional software development, it's all about the code. There may be data involved, but all the work is focused around the code. And managing code and managing data are quite different.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/collaboration-data-resize.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1080" height="720" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/collaboration-data-resize.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/collaboration-data-resize.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/collaboration-data-resize.jpg 1080w" sizes="(min-width: 720px) 720px"><figcaption>A team working with and analysing data</figcaption></figure><p>Aspects like versioning and reproducibility are much easier to handle with code than they are with data. Source control for a even several hundred megabytes of text-based files is comparatively trivial when contrasted with terabytes of data for an ML project.</p><p>What this means in practice is easier provenance. <a href="https://docs.microsoft.com/en-us
1/azure/devops/boards/queries/link-work-items-support-traceability?view=azure-devops&tabs=new-web-form&WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">With established source control and good CI/CD practices, it's relatively easy to say <em>this code </em>created <em>this artefact</em> which was deployed to <em>this environment</em></a>. A commit SHA can be all you need to identify that whole chain from code change to running application.</p><p>But it's not quite so easy when there's huge amounts of data in the mix. <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-version-track-datasets?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">You also have to account for what the data looked like at the time you trained your model.</a> Getting a snapshot of the <em>code </em>(including infrastructure) at a point in time can be as simple as doing a <code>git checkout [sha]</code>, but getting a snapshot of data at a point in time depends very much on the way
1it's been stored. If you're lucky and have considered this with your up-front data engineering, it can be a simple query. If not, it can be borderline impossible. This makes provenance harder, and <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-machine-learning-interpretability-aml?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">explainability of a model</a> is extremely important in many implementations.</p><blockquote>With established source control and good CI/CD practices, it's relatively easy to say <em><strong>this code</strong> </em>created <em><strong>this artefact</strong></em> which was deployed to <em><strong>this environment</strong></em>... But it's not quite so easy when there's huge amounts of data in the mix.</blockquote><p>With DevOps, using source control and CI/CD to enable collaboration, reproducibility, and provenance is standard practice.</p><p>With MLOps, a lot more thought and planning may need to go into data engineering if you truly want these same benefits.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/hot-aisle-green-resize.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1080" height="720" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/hot-aisle-green-resize.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/hot-aisle-green-resize.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/hot-aisle-green-resize.jpg 1080w" sizes="(min-width: 720px) 720px"><figcaption>An aisle in a data centre</figcaption></figure><h3 id="ci-cd">CI/CD</h3><p>I mentioned good CI/CD practices, and CI/CD is one of the other areas where MLOps can differ greatly from DevOps - particularly the continuous integration part.</p><p>A typical build for a traditional software engineering project might take a few minutes. Maybe an hour for very large projects. Additionally, most builds can be performed on <a href="https://github.com/actions/virtual-environments?ref=damianbrady.com.au">relatively low-spec machines or docker containers</a>.</p><p>At that speed and with those resource requirements, it's feasible to do a full build for every commit that's made - <em>continuous</em> integration. Every change can be committed and there's a very fast feedback loop telling the engineer whether the change they made is "good".</p><p>With a machine learning project, a training run may take days or weeks, and might require a powerful machine with multiple GPUs. <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-attach-compute-cluster?tabs=python&WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">Even with massively scalable compute offered by the cloud</a>, full training runs can be extremely expensive and time-consuming. It's simply not feasible to do a full training run for every code or data change. Even if money was no object, getting feedback on a change you made would be far from immediate. You lose the connection between cause and effect.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-pixabay-257736.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1280" height="851" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-pixabay-257736.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-pixabay-257736.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-pixabay-257736.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>
1A person probing a circuit board with a lot of wires. Image from <a href="https://www.pexels.com/photo/close-up-of-telephone-booth-257736/?ref=damianbrady.com.au">Pexels</a>.</figcaption></figure><p>The good news is you can still stay true to the fundamental goals of DevOps without doing a full training run on every change. The idea of "<a href="https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/shift-left-make-testing-fast-reliable?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">shifting left</a>" - where you try to discover issues as early as possible in the lifecycle of a change - can still be applied pragmatically.</p><p>For example, you might want to implement continuous integration that runs quality checks and tests over your code, and maybe does a mini training run with a very limited set of data - just to catch any glaring issues. If that succeeds, the next stage might be a training run over a larger set of data that you abandon after an hour, ensuring that the metrics are still heading in the right direction. Then a nightly training run that takes a few hours with a full evaluation, then a weekly run over all the data and so on.</p><p>Ultimately, <em>you </em>can control the balance between fast feedback and the cadence for your production-ready models.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/woman-working-at-night-resize.jpg" class="kg-image" alt="MLOps [6] - How does MLOps differ from DevOps?" loading="lazy" width="1080" height="720" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/woman-working-at-night-resize.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/woman-working-at-night-resize.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/woman-working-at-night-resize.jpg 1080w" sizes="(min-width: 720px) 720px"><figcaption>A programmer working on multiple monitors at night.</figcaption></figure><p>For me, these are the three key areas to be aware of when applying DevOps practices to machine learning projects:</p><ol><li>Treating experimentation as a necessary and important part of the work</li><li>Being conscious of data engineering and versioning to cater for effective collaboration, repeatability, and explainability</li><li>Applying continuous integration pragmatically to avoid unreasonable cost and time delays</li></ol><p>One tool that's extremely good at managing the MLOps lifecycle is <a href="https://azure.microsoft.com/en-au/services/machine-learning/?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a>. As well as features that make creating models easier than ever, it can manage pipelines, track experiment and model versions, and even package and deploy to production.</p><p>If you want to get started with Azure Machine Learning, the best place to get hands-on experience (for free) is <a href="https://docs.microsoft.com/learn?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">Microsoft Learn</a>. The "<a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">Build AI solutions with Azure Machine Learning</a>" learning path is an awesome in-depth resource.</p><p>And of course, there's the usual collection of <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9703-dabrady&ref=damianbrady.com.au">comprehensive documentation on Microsoft Docs </a>to answer any additional questions you have along the way.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [5] - What DevOps practices can be applied to Machine Learning projects?]]></title><description><![CDATA[There are many lessons and good practices we've learned from decades of software development and more than a decade of "DevOps". Most (but not all) of these can be applied to machine learning projects.]]></description><link>https://damianbrady.com.au/2020/10/27/devops-practices-applied-to-ml-projects/</link><guid isPermaLink="false">5f7da6ab31ce320039f0e763</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[DevOps]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 27 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/5-DevOps-applied-to-ML.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/5-DevOps-applied-to-ML.png" alt="MLOps [5] - What DevOps practices can be applied to Machine Learning projects?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>There are many lessons and good practices we've learned from decades of software development and more than a decade of "DevOps". Most (but not all) of these can be applied to machine learning projects.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-DevOps-practices-can-be-applied-to-Machine-Learning-projects--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What DevOps practices can be applied to Machine Learning projects? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>The day-to-day work of data scientists for machine learning projects is usually very different from the day-to-day work of traditional software developers. While there's still (usually) code, there's more experimentation, a focus on data, longer cycle times, etc.</p><p>Because of this, it can be tempting to think all the process <em>around</em> that work must also be different. Data scientists are (usually) not developers after all.</p><p>But DevOps is broad, and although many practices can be prescriptive, <a href="https://azure.microsoft.com/en-us
1/overview/devops-tutorial/?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">the fundamental ideas are not</a>. Even the practices, when tweaked and applied appropriately can fit perfectly well into an ML world.</p><p>The <a href="https://www.mlflow.org/?ref=damianbrady.com.au">open source MLflow platform</a> is a perfect example of how some of these DevOps concepts align nicely to machine learning projects.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/mlflow-website.png" class="kg-image" alt="MLOps [5] - What DevOps practices can be applied to Machine Learning projects?" loading="lazy" width="1791" height="1170" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/mlflow-website.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/mlflow-website.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/mlflow-website.png 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/mlflow-website.png 1791w" sizes="(min-width: 720px) 720px"><figcaption>The <a href="https://mlflow.org/?ref=damianbrady.com.au">mlflow.org</a> website</figcaption></figure><p><strong>Earlier this month, <a href="https://techcommunity.microsoft.com/t5/azure-ai/azure-machine-learning-s-native-support-for-mlflow/ba-p/1737491?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">Azure Machine Learning announced a public preview of native support for MLflow</a></strong>, meaning (amongst other things) that you can take your existing on-prem workloads and MLOps processes and move them to the cloud. There's support for MLflow Tracking, Projects, Models, and Registry.</p><p>Let's look again at the <a href="https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">definition for DevOps that we use at Microsoft</a>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png" class="kg-image" alt="MLOps [5] - What DevOps practices can be applied to Machine Learning projects?" loading="lazy" width="1660" height="936" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/devops-definition.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/devops-definition.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/devops-definition.png 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png 1660w" sizes="(min-width: 720px) 720px"><figcaption>Microsoft's DevOps definition, written by Donovan Brown</figcaption></figure><blockquote>DevOps is the union of people, process, and products to enable continuous delivery of <strong>value</strong> to the end user.</blockquote><p>What we're really talking about is coordination in order to deliver value. That also fits perfectly with what MLOps is trying to achieve. The people, process and products may differ, but the focus towards delivering value in a continuous way does not.</p><p>So let's look briefly at some of the fundamental practices in DevOps to see how they can align with MLOps:</p><ol><li><strong>Planning and tracking work </strong>- <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">keeping track of current and historical experiments</a> gives us continuity and provenance. We can easily look back at what was tried, and the results of that experiment.</li><li><strong>Source control </strong>- Keeping all ML code in source control means <a href="https://github.com/features?ref=damianbrady.com.au#code-review">team members can collaborate more easily</a>. If practices like <a href="https://trunkbaseddevelopment.com/?ref=damianbrady.com.au">trunk-based development</a>
1 and <a href="https://docs.microsoft.com/en-us/azure/devops/repos/git/git-branching-guidance?view=azure-devops&WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">appropriate branching strategies</a> are used, it also means there's a central repository of the main production-candidate code, alongside any work in progress.</li><li><strong>Infrastructure as code </strong>- By <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">defining the training environment requirements in code</a>, resources can be spun up and torn down efficiently and consistently. When combined with cloud-first training, this means effective elastic compute that you only pay for when you need. And by <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-environments?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au#use-environments-for-web-service-deployment">defining the production or inferencing environments in code</a>, you can scale up and down easily, and even spin up test environments that you know will mirror production.</li><li><strong>Continuous Integration </strong>- While the actual "build" (read: training run) is almost always more CPU-, time-, and data-intensive than a traditional software build, that doesn't mean some of the same concepts can't be applied. Quality gates and tests can be run, and <a href="https://github.com/Azure/azureml-examples?ref=damianbrady.com.au">executing training runs from the main branch in source control means consistency and reproducibility</a>.</li><li><strong>Continuous Delivery</strong> - Following on from an automated training run, <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au#register-package-and-deploy-models-from-anywhere">deployable artefacts in the form of packaged models can be versioned and tracked</a>. When combined with defined production environments definitions in code, new models can be deployed at a moment's notice.</li><li><strong>Continuous Delivery</strong> - <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-continuously-deploy?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">Automated deployment of portable, packaged models is the next step</a>. Controlling the rollout of these new models is also something DevOps does really well. Gradual rollout techniques like <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/kubernetes/canary-demo?view=azure-devops&tabs=yaml&WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">canary deployments</a>, <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops&WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au#rolling-deployment-strategy">rolling deployments</a>, using <a href="https://docs.microsoft.com/en-us/azure/devops/migrate/phase-rollout-with-rings?view=azure-devops&WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">deployment rings</a>, and even <a href="https://azure.github.io/AppService/2020/08/03/ab_testing_app_service.html?ref=damianbrady.com.au">A/B testing</a> of models or <a href="https://twitter.com/damovisa/status/1295853224386154496?ref=damianbrady.com.au">shadow deployments</a> can be extremely useful.</li><li><strong>Production Monitoring </strong>- Watching what happens in production is key to closing the loop. <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-enable-app-insights?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">Production monitoring is more than just watching for errors, it's about knowing whether your hypotheses about what's "valuable" are actually correct</a>. This is even more true for machine learning projects. Often it's difficult to really evaluate "value" until the model has been running in production for some time.</li></ol><p>These are just some of the practices in DevOps that can be applied very effectively to machine learning projects. Of course, consideration must be given to the differences in the flow of work, but the core ideas are still present.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-thisisengineering-3862140.jpg" class="kg-image" alt="MLOps [5] - What DevOps practices can be applied to Machine Learning projects?" loading="lazy" width="1280" height="854" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-thisisengineering-3862140.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-thisisengineering-3862140.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-thisisengineering-3862140.jpg 1280w" sizes="(min-width: 720px) 720px"><figcaption>A woman working with graphs on a computer. Photo by <strong>
1<a href="https://www.pexels.com/@thisisengineering?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">ThisIsEngineering</a></strong> from <strong><a href="https://www.pexels.com/photo/woman-using-computer-on-table-3862140/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>I'd strongly encourage you to check out <a href="https://techcommunity.microsoft.com/t5/azure-ai/azure-machine-learning-s-native-support-for-mlflow/ba-p/1737491?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">Azure Machine Learning's support for MLflow</a>. In particular, we have some great <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow?WT.mc_id=devops-9702-dabrady&ref=damianbrady.com.au">docs that walk you through how to use Azure ML to track experiments</a>.</p><p>If you want to get straight into running code, there are some excellent <a href="https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/track-and-monitor-experiments/using-mlflow?ref=damianbrady.com.au">MLflow examples in the Azure Machine Learning Notebooks repository on GitHub</a>.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [4] - What is AIOps and how does it differ from MLOps?]]></title><description><![CDATA[MLOps and AIOps both sit at the union of DevOps and AI. They may sound like the same thing, but they represent completely different ideas. So what is AIOps?]]></description><link>https://damianbrady.com.au/2020/10/22/what-is-aiops/</link><guid isPermaLink="false">5f7d9be831ce320039f0e749</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Thu, 22 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/What-is-AIOps.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/What-is-AIOps.png" alt="MLOps [4] - What is AIOps and how does it differ from MLOps?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>Another term you may have heard in this space is "AIOps". It would be easy to assume it was another word for the same thing, but that's not the case.</p><p>MLOps and AIOps both sit at the union of DevOps and AI. They may sound like the same thing, but they represent completely different ideas. So what is AIOps?</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-is-AIOps-and-how-does-it-differ-from-MLOps--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What is AIOps and how does it differ from MLOps? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p><a href="https://en.wikipedia.org/wiki/Artificial_Intelligence_for_IT_Operations?ref=damianbrady.com.au">AI Ops, or "Artificial Intelligence for IT Operations"</a> is the reverse of MLOps in one respect - it's the application of ML to DevOps, rather than the application of DevOps to ML.</p><p>AIOps is often described as artificial intelligence applied to IT operations, implying that it's limited to the day to day management of configuration, infrastru
1cture, networking, and other IT operations aspects. But applied AIOps can also involve automated continuous delivery of change to those systems.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-christina-morillo-1181354.jpg" class="kg-image" alt="MLOps [4] - What is AIOps and how does it differ from MLOps?" loading="lazy" width="1920" height="1281" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-christina-morillo-1181354.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-christina-morillo-1181354.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/pexels-christina-morillo-1181354.jpg 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-christina-morillo-1181354.jpg 1920w" sizes="(min-width: 720px) 720px"><figcaption>A woman in a server room working on a laptop. Photo by <strong><a href="https://www.pexels.com/@divinetechygirl?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Christina Morillo</a></strong> from <strong><a href="https://www.pexels.com/photo/woman-standing-while-carrying-laptop-1181354/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>Some examples of applications of AIOps include:</p><ul><li>You might have a <a href="https://docs.microsoft.com/en-us/learn/paths/create-no-code-predictive-models-azure-machine-learning/?WT.mc_id=devops-9701-dabrady&ref=damianbrady.com.au">predictive model</a> to analyse a set of code changes to determine the probability that this change will introduce an unexpected issue so you can give them an extra look before going live. You could base that prediction on all your previous releases and their success or failure.</li><li>You could use <a href="https://azure.microsoft.com/en-us/services/cognitive-services/anomaly-detector/?WT.mc_id=devops-9701-dabrady&ref=damianbrady.com.au">anomaly detection</a> over your network traffic and automatically modify firewall rules to mitigate unusual behaviour.</li><li>You could use a <a href="https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/ai/real-time-recommendation?WT.mc_id=devops-9701-dabrady&ref=damianbrady.com.au">recommendation engine</a> to identify related log entries across multiple sources of data to help identify and remediate a complex issue.</li></ul><p>When two broad fields collide (DevOps/IT Operations and Machine Learning), it opens up a world of possibilities. Machine Learning can be used to help solve those tricky problems where a solution isn't obvious, and there are plenty of those in DevOps!</p>]]></content:encoded></item><item><title><![CDATA[MLOps [3] - What is MLOps?]]></title><description><![CDATA[MLOps sounds like just another buzzword, but it presents some important concepts and practices for maturing your machine learning projects.]]></description><link>https://damianbrady.com.au/2020/10/20/what-is-mlops/</link><guid isPermaLink="false">5f7d692331ce320039f0e728</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 20 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/3-What-is-MLOps.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/3-What-is-MLOps.png" alt="MLOps [3] - What is MLOps?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>Now we get to the core question! What even is "MLOps"? Have we invented a new term for something everyone is already doing? Is it just DevOps applied blindly to ML projects? Surely there are some important differences!?</p><p>MLOps sounds like just another buzzword, but it presents some important concepts and practices for maturing your machine learning projects.</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-is-MLOps--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What is MLOps? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>MLOps, or "DevOps for Machine Learning" is really just the practice of applying DevOps strategies, techniques, and ideas to machine learning projects rather than traditional software projects.</p><p>Over the past decade or so, DevOps has represented a way of ensuring high-quality software that is released into production systems as quickly and safely as possible. It's meant an increase in maturity in so many organisations, and has been shown to dramatically improve not just software delivery, but the an organisations own </p><p>But before talking about how we apply DevOps principles to ML projects, let'
1s have a look at <a href="https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">Microsoft's definition for DevOps</a>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png" class="kg-image" alt="MLOps [3] - What is MLOps?" loading="lazy" width="1660" height="936" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/devops-definition.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/devops-definition.png 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/devops-definition.png 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/devops-definition.png 1660w" sizes="(min-width: 720px) 720px"><figcaption>Microsoft's DevOps definition, written by Donovan Brown</figcaption></figure><blockquote>DevOps is the union of people, process, and products to enable continuous delivery of <strong>value</strong> to the end user.</blockquote><p>The emphasis is mine, but value is the core of this definition. Without knowing whether you're delivering value, you're just guessing whether all the work your doing is improving your product or not.</p><p><strong>It's <em>exactly</em> the same for a predictive model you're deploying to a production system.</strong></p><p>Let's take an image recognition model as an example. If you've made a change you think might improve the accuracy of that model in production, it's extremely powerful being able to make those changes, commit them to source control, and have an automated pipeline that kicks off a training run, model validation, packaging into a portable format, and deployment to a test or production system. </p><p>Most importantly, as a data scientist, your day to day focus can be on the changes needed to improve that model. You don't need to do the "how do we get this into production" dance!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-andrea-piacquadio-845451-1.jpg" class="kg-image" alt="MLOps [3] - What is MLOps?" loading="lazy" width="1920" height="1280" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-andrea-piacquadio-845451-1.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-andrea-piacquadio-845451-1.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/pexels-andrea-piacquadio-845451-1.jpg 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-andrea-piacquadio-845451-1.jpg 1920w" sizes="(min-width: 720px) 720px"><figcaption>A man wearing headphones focusing on work. Photo by <strong><a href="https://www.pexels.com/@olly?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Andrea Piacquadio</a></strong> from <strong><a href="https://www.pexels.com/photo/man-with-headphones-facing-computer-monitor-845451/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>But it's not just about automation. Having all your code in source control means you can work more closely with your team (especially if everyone works from home!), and with the right tooling, you have a historical record of what changed, and the results of that change.</p><p>Once a training run has finished, you can use techniques commonly used in traditional software development to ensure your changes actually provide value in a production environment. Gradual rollout techniques like <a href="https://docs.microsoft.com/en-us
1/azure/devops/pipelines/ecosystems/kubernetes/canary-demo?view=azure-devops&tabs=yaml&WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">canary deployments</a>, <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops&WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au#rolling-deployment-strategy">rolling deployments</a>, using <a href="https://docs.microsoft.com/en-us/azure/devops/migrate/phase-rollout-with-rings?view=azure-devops&WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">deployment rings</a>, and even <a href="https://azure.github.io/AppService/2020/08/03/ab_testing_app_service.html?ref=damianbrady.com.au">A/B testing</a> of models or <a href="https://twitter.com/damovisa/status/1295853224386154496?ref=damianbrady.com.au">shadow deployments</a> can be extremely useful.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-david-dibert-1117210.jpg" class="kg-image" alt="MLOps [3] - What is MLOps?" loading="lazy" width="1920" height="1280" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/pexels-david-dibert-1117210.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/pexels-david-dibert-1117210.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/pexels-david-dibert-1117210.jpg 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/pexels-david-dibert-1117210.jpg 1920w" sizes="(min-width: 720px) 720px"><figcaption>"Evergreen" container ship which is pretty much the best analogy ever. Photo by <strong><a href="https://www.pexels.com/@dibert?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">David Dibert</a></strong> from <strong><a href="https://www.pexels.com/photo/green-and-gray-evergreen-cargo-ship-1117210/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels">Pexels</a></strong></figcaption></figure><p>Of course, not everything we do in DevOps for traditional software engineering can be applied blindly to ML projects. There are some significant differences between ML projects and traditional projects that need some consideration. We'll talk about them in more depth in a future post.</p><p>One tool that's extremely good at managing the MLOps lifecycle is <a href="https://azure.microsoft.com/en-au/services/machine-learning/?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">Azure Machine Learning</a>. As well as features that make creating models easier than ever, it can manage pipelines, track experiment and model versions, and even package and deploy to production.</p><p>If you want to get started with Azure Machine Learning, the best place to get hands-on experience (for free) is <a href="https://docs.microsoft.com/learn?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">Microsoft Learn</a>. The "<a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">Build AI solutions with Azure Machine Learning</a>" learning path is an awesome in-depth resource.</p><p>And of course, there's the usual collection of <a href="https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=devops-9700-dabrady&ref=damianbrady.com.au">comprehensive documentation on Microsoft Docs </a>to answer any additional questions you have along the way.</p>]]></content:encoded></item><item><title><![CDATA[MLOps [2] - What's the difference between Machine Learning and Artificial Intelligence?]]></title><description><![CDATA[The terms "Artificial Intelligence" and "Machine Learning" are often used synonymously, but they don't mean the same thing. So what's the difference?]]></description><link>https://damianbrady.com.au/2020/10/15/ml-vs-ai/</link><guid isPermaLink="false">5f7d34c831ce320039f0e706</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Thu, 15 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/2-Difference-between-ML-AI.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/2-Difference-between-ML-AI.png" alt="MLOps [2] - What's the difference between Machine Learning and Artificial Intelligence?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>In this post, we'll look at the difference between two of the buzzwords you'll hear in this space - Artificial Intelligence (AI) and Machine Learning (ML).</p><p>The terms "Artificial Intelligence" and "Machine Learning" are often used synonymously, but they don't mean the same thing. So what's the difference?</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/Whats-the-difference-between-Machine-Learning-and-Artificial-Intelligence--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What's the difference between Machine Learning and Artificial Intelligence? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>I mentioned at the end of the video that artificial intelligence can be thought of as anything that used to require human intelligence, but can now be done by a machine.</p><p>Under this definition, a calculator was once "artificial intelligence". And it's true! A "calculator" was originally a person employed to do calculations. With the advent of the mechanical calculator in the 17th century, this previously human-only task was now being performed by a machine. It must have seemed like magic in much the same way as some recent AI breakthroughs.</p><p>An aside: If you want your mind blown, I can highly recommend the "Two Minute Papers" YouTube channel. Video walkthroughs of the newest AI and technology papers:</p><figure class="kg-card kg-bookmark-card"><a class="kg-bookmark-container" href="https://www.youtube.com/user/keeroyz?ref=damianbrady.com.au"><div class="kg-bookmark-content"><div class="kg-bookmark-title">Two Minute Papers</div><small><div class="kg-bookmark-description">What a time to be alive!</div></small></div></a></figure><p>Back on topic.</p><p>Part of the reason that "Artificial Intelligence" and "Machine Learning" are used interchangeably is because when you're talking about one, you're usually talking about the other. Sometimes saying AI is more powerful (e.g. marketing), and sometimes ML is more powerful (e.g. recruiting)!</p><figure class="kg-card kg-embed-card"><blockquote class="twitter-tweet" data-width="550"><p lang="en" dir="ltr">Difference between machine learning and AI:<br><br>
1If it is written in Python, it's probably machine learning<br><br>If it is written in PowerPoint, it's probably AI</p>— Mat Velloso (@matvelloso) <a href="https://twitter.com/matvelloso/status/1065778379612282885?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">November 23, 2018</a></blockquote>
2<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
2 3</figure><p>To be more accurate, machine learning is a subset of artificial intelligence. You can think of it as one way of <em>implementing</em> AI. Most of the breakthroughs in AI in the last few decades have been because of machine learning. Deep learning is a particular type of machine learning that's provided many of those breakthroughs. So while AI and ML might be synonymous, they're not the same.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/AI-vs-ML.png" class="kg-image" alt="MLOps [2] - What's the difference between Machine Learning and Artificial Intelligence?" loading="lazy" width="947" height="948" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/AI-vs-ML.png 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/AI-vs-ML.png 947w" sizes="(min-width: 720px) 720px"><figcaption>Deep learning is a subset of machine learning which is subset of AI</figcaption></figure><p>In short, if you're actually <em>creating </em>AI, you're probably doing it with Machine Learning.</p><p>If ML interests you but you don't know where to start, there are plenty of learning paths and modules available on <a href="https://docs.microsoft.com/en-us/learn/?WT.mc_id=devops-9699-dabrady&ref=damianbrady.com.au">Microsoft Learn</a>.</p><p>If you're not familiar with <a href="https://docs.microsoft.com/en-us/learn/?WT.mc_id=devops-9699-dabrady&ref=damianbrady.com.au">MS Learn</a>, it's a site that gives you FREE hands-on guided learning across a huge number of topics. It's the best way to learn about Azure and Microsoft products and services!</p><ul><li><a href="https://docs.microsoft.com/en-us/learn/paths/create-machine-learn-models/?WT.mc_id=devops-9699-dabrady&ref=damianbrady.com.au">Create Machine Learning Models</a></li><li><a href="https://docs.microsoft.com/en-us/learn/paths/create-no-code-predictive-models-azure-machine-learning/?WT.mc_id=devops-9699-dabrady&ref=damianbrady.com.au">Create no-code Predictive Models with Azure Machine Learning</a></li><li><a href="https://docs.microsoft.com/en-us/learn/paths/build-ai-solutions-with-azure-ml-service/?WT.mc_id=devops-9699-dabrady&ref=damianbrady.com.au">Build AI solutions with Azure Machine Learning</a></li></ul><p></p>]]></content:encoded></item><item><title><![CDATA[MLOps [1] - What is Machine Learning?]]></title><description><![CDATA[Machine Learning is a term many of us have heard, but what does it actually mean, and when should it be used?]]></description><link>https://damianbrady.com.au/2020/10/13/what-is-machine-learning/</link><guid isPermaLink="false">5f7d31e731ce320039f0e6dd</guid><category><![CDATA[MLOps]]></category><category><![CDATA[One Dev Question]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 13 Oct 2020 15:15:00 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/1-What-is-ML.png" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/1-What-is-ML.png" alt="MLOps [1] - What is Machine Learning?"><p><em>This video and post are part of a <a href="https://channel9.msdn.com/Blogs/One-Dev-Minute?WT.mc_id=devops-9697-dabrady&ref=damianbrady.com.au">One Dev Question</a> series on MLOps - DevOps for Machine Learning. See the <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHj-DSzz0JlW4H_0L4wQWliy&ref=damianbrady.com.au">full video playlist here</a>, and the rest of the <a href="https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/">blog posts here</a>.</em></p><p>To understand what MLOps (DevOps for Machine Learning) is, we first need to know what Machine Learning is.</p><p>Machine Learning is a term many of us have heard, but what does it actually mean, and when should it be used?</p><!--kg-card-begin: html--><iframe src="https://channel9.msdn.com/Blogs/One-Dev-Minute/What-is-Mac
3hine-Learning--One-Dev-Question/player" width="960" height="540" allowfullscreen frameborder="0" title="What is Machine Learning? | One Dev Question - Microsoft Channel 9 Video"></iframe><!--kg-card-end: html--><p>As usual Microsoft Docs has some great resources, including <a href="https://azure.microsoft.com/en-us/overview/what-is-machine-learning-platform/?WT.mc_id=devops-9698-dabrady&ref=damianbrady.com.au">an explanation of what machine learning is and how it works</a>, but I like to think of it in more simple terms.</p><p>In traditional software development, you have a human specifying the logic to solve a problem. They write the loops, if statements, and structure to code so when you supply input, you get (hopefully) the correct output.</p><p>For example you might, as an engineer, be asked to write a function to calculate the tax due on a given transaction. You might need to do some research around state or federal taxes, or what transactions are exempt from GST or VAT. But essentially it's a thing you can work out. You know what that logic will look like.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/SEC20_Security_029.jpg" class="kg-image" alt="MLOps [1] - What is Machine Learning?" loading="lazy" width="2000" height="1380" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/SEC20_Security_029.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/SEC20_Security_029.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/SEC20_Security_029.jpg 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w2400/2020/10/SEC20_Security_029.jpg 2400w" sizes="(min-width: 720px) 720px"><figcaption>A man at a desk programming with 4 screens</figcaption></figure><p>By contrast, think about a problem like, "What's the probability this person will repay a loan of $10,000?". Writing a function to solve that problem is <em>much</em> harder.</p><p>These are the kinds of problems Machine Learning is good at. Rather than trying to come up with the logic yourself, you can supply a whole lot of data - in this case probably details of previous applicants and loans including whether they were repaid - and let the machine learn the patterns. You're using examples to train a computer what's important and what's not, and how to get the "right" answer given some input data.</p><p>What you get out the other end is a "predictive model", and that model can be used in place of a function that you might have written yourself. Understanding the algorithm and what the logic inside it does is often difficult or impossible, but it's still input -> output. You can feed it new, unseen data, and get a prediction on the answer.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/10/CLO19_azureKinectDK_008.jpg" class="kg-image" alt="MLOps [1] - What is Machine Learning?" loading="lazy" width="2000" height="1333" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/10/CLO19_azureKinectDK_008.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1000/2020/10/CLO19_azureKinectDK_008.jpg 1000w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w1600/2020/10/CLO19_azureKinectDK_008.jpg 1600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w2400/2020/10/CLO19_azureKinectDK_008.jpg 2400w" sizes="(min-width: 720px) 720px"><figcaption>An Azure Kinect webcam</figcaption></figure><p>One final point - because the model is trained <em>from</em> data, it goes without saying that if your data is bad, your model will be bad. Actually scratch that - <strong>it definitely needs to be said</strong>.</p><p>Far too often, models will be trained on biased data, ultimately building in those biases. Microsoft is putting a lot of effort into this broader field of <a href="https://www.microsoft.com/en-us
3/ai/responsible-ai?WT.mc_id=devops-9698-dabrady&ref=damianbrady.com.au">Responsible AI</a>, to make sure we're using machine learning and AI ethically and responsibly.</p><p><strong>Check out <a href="https://docs.microsoft.com/en-us/azure/?product=ai-machine-learning&WT.mc_id=devops-9698-dabrady&ref=damianbrady.com.au">Azure Machine Learning on Microsoft Docs</a>, or learn how to <a href="https://docs.microsoft.com/en-us/learn/paths/create-machine-learn-models/?WT.mc_id=devops-9698-dabrady&ref=damianbrady.com.au">create your own predictive models on Microsoft Learn</a>.</strong></p>]]></content:encoded></item><item><title><![CDATA[MLOps - One Dev Question video series]]></title><description><![CDATA[Over the coming weeks, we'll be releasing a series of short "One Dev Question" videos talking about MLOps and surrounding topics.]]></description><link>https://damianbrady.com.au/2020/10/06/machine-learning-and-mlops-one-dev-question/</link><guid isPermaLink="false">5f7bd3cd31ce320039f0e69d</guid><category><![CDATA[MLOps]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 06 Oct 2020 05:36:07 GMT</pubDate><content:encoded><![CDATA[<p>Over the coming weeks, we'll be releasing a series of short "<a href="https://www.youtube.com/playlist?list=PLLasX02E8BPCkCMbNDPT-hUfNsnP_RSBm&ref=damianbrady.com.au">One Dev Question</a>" videos talking about MLOps and surrounding topics.</p><figure class="kg-card kg-embed-card kg-card-hascaption"><iframe width="612" height="344" src="https://www.youtube.com/embed/_JuJU3wfcGc?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe><figcaption>MLOps - One Dev Question series intro video</figcaption></figure><p>As they get released, I'll be releasing accompanying blog posts exploring the ideas behind each video in more depth, and providing links for further information. You can find the currently-released posts below:</p><!--kg-card-begin: html--><div style="border:1px solid #444;padding:5px;background-color:#ffe;"> 4 <h3>The complete list of posts:</h3> 5 <ol> 6 <li><a href="https://damianbrady.com.au/2020/10/13/what-is-machine-learning/" alt="What is Machine Learning?">What is Machine Learning?</a></li> 7 <li><a href="https://damianbrady.com.au/2020/10/15/ml-vs-ai/" alt="What's the difference between Machine Learning and Artificial Intelligence?">What's the difference between Machine Learning and Artificial Intelligence?</a></li> 8 <li><a href="https://damianbrady.com.au/2020/10/20/what-is-mlops/" alt="What is MLOps?">What is MLOps?</a></li> 9 <li><a href="https://damianbrady.com.au/2020/10/22/what-is-aiops/" alt="What is AIOps and how does it differ from MLOps?">What is AIOps and how does it differ from MLOps?</a></li> 10 <li><a href="https://damianbrady.com.au/2020/10/27/devops-practices-applied-to-ml-projects/" alt="What DevOps practices can be applied to Machine Learning projects?">What DevOps practices can be applied to Machine Learning projects?</a></li> 11 <li><a href="https://damianbrady.com.au/2020/10/29/how-does-mlops-differ-from-devops/" alt="How does MLOps differ from DevOps?">How does MLOps differ from DevOps?</a></li> 12 <li><a href="https://damianbrady.com.au/2020/11/02/parts-of-an-ml-project-in-source-control/" alt="What parts of an ML project should be in source control?">What parts of an ML project should be in source control?</a></li> 13 <li><a href="https://damianbrady.com.au/2020/11/10/mlops-8-can-i-use-devops-tooling-for-machine-learning-projects/" alt="Can I use DevOps tooling for Machine Learning projects?">Can I use DevOps tooling for Machine Learning projects?</a></li> 14 <li><a href="https://damianbrady.com.au/2020/11/12/why-care-about-mlops/" alt="Why should I care about MLOps?">Why should I care about MLOps?</a></li> 15 <li><a href="https://damianbrady.com.au/2020/11/17/when-should-i-think-about-mlops/" alt="When should I think about MLOps?">When should I think about MLOps?</a></li> 16 <li><a href="https://damianbrady.com.au/2020/11/19/how-can-mlops-improve-my-models/" alt="How can MLOps improve my predictive models?">How can MLOps improve my predictive models?</a></li> 17 <li><a href="https://damianbrady.com.au/2020/12/01/what-is-aml/" alt="What is Azure Machine Learning?">What is Azure Machine Learning?</a></li> 18 </ol> 19</div><!--kg-card-end: html--><p>
19MLOps (or DevOps for Machine Learning) is a new and sometimes confusing field. It can be hard to pin down exactly what's involved, who's responsible for what, and the relationship with traditional software delivery practices.</p><p>I'm pretty excited about this series! The videos are all around 2 minutes or less, and I've tried to really drill into the key points.</p><p>Hopefully the short, digestible videos help explain some of the buzzwords, and encourage you to find out more. I'm looking forward to hearing what you think!</p>]]></content:encoded></item><item><title><![CDATA[Tracking my Internet with a Raspberry Pi (Part 2)]]></title><description><![CDATA[In part 1, I set up a regular ping and speed test on a Raspberry Pi. I saw enough to be concerned, so it was time to write some code!]]></description><link>https://damianbrady.com.au/2020/09/13/tracking-my-internet-with-a-raspberry-pi-part-2/</link><guid isPermaLink="false">5f5ea46664110a0039a6e0fb</guid><category><![CDATA[Internet]]></category><category><![CDATA[NBN]]></category><category><![CDATA[RaspberryPi]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Sun, 13 Sep 2020 23:36:55 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/internet-test-1013-09-2020-1.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/internet-test-1013-09-2020-1.jpg" alt="Tracking my Internet with a Raspberry Pi (Part 2)"><p>I've been having Internet trouble.</p><p><em><a href="https://damianbrady.com.au/2020/09/08/tracking-my-internet-with-a-raspberry-pi/"><strong>In part 1</strong></a>, I set up a regular ping and speed test on a Raspberry Pi using cron and the Ookla Speed test CLI. After running it for just under 24 hours, I saw enough to be concerned, so I dug a little deeper.</em></p><h2 id="more-testing-">More Testing!</h2><p>I'm in a Teams chat with a bunch of other Microsofties based in Australia. Topics range from PC builds to software architecture to Lego to (of course) Internet performance.</p><p>Given it's a chat full of software developers, we predictably explored how to dramatically over-engineer a solution for testing an Internet connection before paring it back to something a bit more reasonable.</p><p>In short, after a bit of hacking around with Python, I now have a script that does a simple HTTP GET request every few seconds.</p><p><a href="https://github.com/Damovisa/http-test?ref=damianbrady.com.au"><strong>The source is here</strong></a>, under an MIT license. Feel free to submit a PR!</p><h2 id="more-results-">More Results!</h2><p>There were three tests running over the weekend. The ping and speed test from about 9:45am Thursday and the HTTP test from about 10:50am Friday. All three finished at around 2:45pm Sunday (the Pi seemed to freeze after that... I should probably look into that particular issue). The tests were:</p><ul><li>A 65kB <strong>ping</strong> to my provider's website every <strong>1 minute</strong></li><li>An Ookla <strong>Speed test</strong> via the CLI every <strong>10 minutes</strong></li><li>An <strong>HTTP GET</strong> to my provider's website every <strong>10 seconds</strong> (about 108kB)</li></ul><p>I won't bore you with the raw data, but the graphs below show the results. The tests started and finished at slightly different times, but I've tried to line up the graphs... believe it or not.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/internet-test-1013-09-2020.jpg" class="kg-image" alt="Tracking my Internet with a Raspberry Pi (Part 2)" loading="lazy" width="911" height="638" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/09/internet-test-1013-09-2020.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/internet-test-1013-09-2020.jpg 911w" sizes="(min-width: 720px) 720px"><figcaption>Three graphs showing regular pings, speed tests, and HTTP GET requests. Observations are described below.</figcaption></figure><h3 id="some-observations-">Some observations:</h3><p>Just by looking, there seems to be very little correlation between a failed ping, a slow speed test, or a slow HTTP GET.</p><p>Pings seem to fail in blocks, while performance drops with the speed test and HTTP GET seem far more regular.</p><p>Pings either fail completely, or return fairly quickly. Meanwhile, there's a huge variation in response times in the speed test and HTTP tests. The best performing speed tests are over 100 mbps (which is great), but there are plenty in the 50-60 mbps range. HTTP responses are most commonly 200-300 ms, but there are plenty that take a second or more.</p><h2 id="so-what-now">So, what now?</h2><p>Now I speak to Aussie Broadband again. I'm hopeful something will come of it - they've been excellent in the past - but there's always a very good chance NBN will take a look and decide that data looks fine to them. Fingers crossed again.</p>]]></content:encoded></item><item><title><![CDATA[Tracking my Internet with a Raspberry Pi]]></title><description><![CDATA[I've been having Internet outages and I wanted something pretty simple to track my connection. A Raspberry Pi, the Ookla Speedtest CLI, and a standard ping were the solution.]]></description><link>https://damianbrady.com.au/2020/09/08/tracking-my-internet-with-a-raspberry-pi/</link><guid isPermaLink="false">5f56be8e4e17030039d34963</guid><category><![CDATA[Internet]]></category><category><![CDATA[NBN]]></category><category><![CDATA[RaspberryPi]]></category><dc:creator><![CDATA[Damian Brady]]></dc:creator><pubDate>Tue, 08 Sep 2020 01:34:54 GMT</pubDate><media:content url="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/tweet-nbn-outage.jpg" medium="image"/><content:encoded><![CDATA[<img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/tweet-nbn-outage.jpg" alt="Tracking my Internet with a Raspberry Pi"><p>I've been having Internet troubles. Why yes, I am in Australia! Funny you ask!</p><p><em><strong>Update</strong>: I wrote a little script to do a regular HTTP GET and ran the tests again over a weekend. <a href="https://damianbrady.com.au/2020/09/13/tracking-my-internet-with-a-raspberry-pi-part-2/">See Part 2 for more</a>!</em></p><figure class="kg-card kg-embed-card"><blockquote class="twitter-tweet" data-width="550"><p lang="en" dir="ltr">*sigh*<br>Time to get out a Raspberry Pi and set up some stuff to continuously test my Internet.<br>Why?<br>Because I'm getting dropouts... but apparently not enough visible ones that <a href="https://twitter.com/NBN_Australia?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">@NBN_Australia</a> will do anything. I need to prove it.</p>— Damian Brady 🥑 #BLM (@damovisa) <a href="https://twitter.com/damovisa/status/1301339666151399428?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">September 3, 2020</a></blockquote>
20<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
20 21</figure><p>In summary, I've been getting really short drop-outs semi-regularly. Only a handful of times per day, and usually only for a few minutes at a time. Just enough to stop me in my tracks and force me onto a tethered mobile phone connection. I think Aussie Broadband (my provider) calls them "flaps", and according to their app, I've had 150 of them in the last 8 days.</p><p>The trouble is, I'm told NBN (Australia's National Broadband Network) doesn't see them as serious enough to trigger a response. An "outage" is something longer, and it's really only outages that matter.</p><p>So what can I do?</p><p>After a swathe of other troubleshooting attempts, one of the representatives at Aussie Broadband (and I want to be clear - I <em>love</em> Aussie) suggested I keep my own records. They can track outages, but there's only so much they can see from their end. They know NBN won't do anything without proof. This suggestion was echoed by a number of friends and colleagues.</p><h2 id="my-solution">My solution</h2><p>Suggestions ranged from just keeping a ping running on a laptop, to setting up an end-to-end tracing solution using half a dozen <a href="https://azure.microsoft.com/?WT.mc_id=devops-0000-dabrady&ref=damianbrady.com.au">Azure</a> resources and client applications.</p><p>Instead, I implemented something in the middle - albeit a bit closer to the former.</p><figure class="kg-card kg-embed-card"><blockquote class="twitter-tweet" data-width="550"><p lang="en" dir="ltr">Thanks to <a href="https://twitter.com/vaughanknight?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">@vaughanknight</a> and others, I now have a raspberry pi set up to do a speed test every 15min, and a ping every 1min.<a href="https://twitter.com/hashtag/JustAustraliaThings?src=hash&ref_src=twsrc%5Etfw&ref=damianbrady.com.au">#JustAustraliaThings</a><br><br>Presumably in a couple of days I'll be able to hand the files to <a href="https://twitter.com/Aussie_BB?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">@Aussie_BB</a> so they can hound <a href="https://twitter.com/NBN_Australia?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">@NBN_Australia</a> about mini dropouts.</p>— Damian Brady 🥑 #BLM (@damovisa) <a href="https://twitter.com/damovisa/status/1302807741166018560?ref_src=twsrc%5Etfw&ref=damianbrady.com.au">September 7, 2020</a></blockquote>
22<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
22 23</figure><p><strong>I wanted something pretty simple, so I just set up a couple of cron jobs on a Raspberry Pi - using the Ookla Speedtest CLI, and a standard ping.</strong></p><p>First, I needed the Speedtest CLI. On my Pi running Rasbian, I installed that fairly simply with easy_install:</p><!--kg-card-begin: markdown--><pre><code class="language-shell">sudo easy_install speedtest-cli 24</code></pre> 25<!--kg-card-end: markdown--><p>Once installed, if you run <code>speedtest</code> from your commandline, you get a nice result... but it's not very parsable if you want to keep track over a period of time.</p><p>Thankfully, the CLI has a couple of arguments that will help populate a CSV, and you can use standard shell features to append to a file.</p><p>First, I created a new CSV file with headers using the <code>--csv-header</code> argument and piping to a new file:</p><!--kg-card-begin: markdown--><pre><code class="language-shell">speedtest-cli --csv-header > speedtest-track.csv 26</code></pre> 27<!--kg-card-end: markdown--><p>Then, to test it out, I ran a speedtest with the <code>--csv</code> argument, appending to that file:</p><!--kg-card-begin: markdown--><pre><code class="language-shell">speedtest-cli --csv >> speedtest-track.csv 28</code></pre> 29<!--kg-card-end: markdown--><p>Now to set up a cron job to do it every 15 minutes. You can set this up with the <code>crontab -e</code> command. One sticking point I encountered (thanks <a href="https://twitter.com/vaughanknight?ref=damianbrady.com.au">Vaughan Knight</a> for helping diagnose) - you'll need the full path to both the CLI and the output file in your cron job.</p><!--kg-card-begin: markdown--><pre><code class="language-shell"># run this every 15min 300,15,30,45 * * * * /usr/local/bin/speedtest --csv >> /home/pi/speedtest-track.csv 31</code></pre> 32<!--kg-card-end: markdown--><p>The result (after running it for a while) is a nicely parseable csv file!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/speedtest-raw.jpg" class="kg-image" alt="Tracking my Internet with a Raspberry Pi" loading="lazy" width="781" height="334" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/09/speedtest-raw.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/speedtest-raw.jpg 781w" sizes="(min-width: 720px) 720px"><figcaption>A screenshot of a spreadsheet showing details from an Ookla speed test every 15min</figcaption></figure><p>I mentioned, however, that the dropouts were only for a few minutes at a time, so there's a good chance I'll miss some if I'm only testing every 15min. I could do a speed test more often (and I might if I'm not getting the details I expect), but I figured a ping every minute would be enough to at least see those additional drops.</p><p>Again, a straight <code>ping</code> will give me plenty of details, but they're not terribly parseable if appended to a file.</p><p>A bit of google-binging led me to this solution:</p><!--kg-card-begin: markdown--><pre><code class="language-shell"># large ping every minute 33* * * * * ping -c 1 -s 65507 aussiebroadband.com.au | head -n 2 | tail -n 1 | perl -nli 'print scalar(localtime), ": ", $_' >> /home/pi/pingtest.txt 34</code></pre> 35<!--kg-card-end: markdown--><p>In short, I'm running a single ping of maximum size against aussiebroadband.com.au. I'm pulling out the second line of the output (the important one), prepending the current time, and appending it to the pingtest.txt file. Simples!</p><p>This is the result, and you can already see some outages in the preview strip in the right-hand side in <a href="https://code.visualstudio.com/?WT.mc_id=devops-0000-dabrady&ref=damianbrady.com.au">Visual Studio Code</a>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/vscode-pingtest.jpg" class="kg-image" alt="Tracking my Internet with a Raspberry Pi" loading="lazy" width="810" height="842" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/09/vscode-pingtest.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/vscode-pingtest.jpg 810w" sizes="(min-width: 720px) 720px"><figcaption>A screenshot of a text file showing the results of a ping every minute. The preview bar on the right shows gaps where no ping response was received.</figcaption></figure><h2 id="the-results">The results</h2><p>First the speed test. I let this run for just under 24 hours and charted the result:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/speedtest-24hr.jpg" class="kg-image" alt="Tracking my Internet with a Raspberry Pi" loading="lazy" width="7
3573" height="464" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/09/speedtest-24hr.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/speedtest-24hr.jpg 773w" sizes="(min-width: 720px) 720px"><figcaption>Chart showing download speeds usually around 93Mbps but drops down to around 55Mbps about 17 times on a regular basis. Uploads are consistently at 38Mbps with a few drops to around 22Mbps and one to 0Mbps towards the end of the graph.</figcaption></figure><p>It's not horrible, but you can see it's very inconsistent. There's even a drop with no upload at all for half an hour or so.</p><p>What about the ping? This one has run for about 21 hours so far, and you can see the results below.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/ping-outages.jpg" class="kg-image" alt="Tracking my Internet with a Raspberry Pi" loading="lazy" width="786" height="444" srcset="https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/size/w600/2020/09/ping-outages.jpg 600w, https://storage.ghost.io/c/c6/d9/c6d9bac7-bb8f-4562-9e1f-eea92e52f81b/content/images/2020/09/ping-outages.jpg 786w" sizes="(min-width: 720px) 720px"><figcaption>Chart showing a consistent low ping with only 4 exceptions, and two periods of Internet outage. One for two minutes, and one with a number of drops over 25 minutes, both in the first quarter of the graph.</figcaption></figure><p>You can see a couple of legitimate outages, one for only a couple of minutes, and one on and off for about 25 minutes.</p><p>What's interesting is that these traces are over approximately the same time period. If you overlay those graphs, the outages don't line up. At all. So sometimes the ping shows a failure, and sometimes the speedtest shows a failure.</p><p><strong>If you add them up, that's at least 3 periods of serious instability over about 21 hours. Not insignificant.</strong></p><p>My plan is to keep this running over the next couple of days, then present the results to Aussie Broadband with the hope that there'll be an escalation. Fingers crossed!</p>]]></content:encoded></item></channel></rss>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.