https://earl2021-aru.netlify.app/

1<!DOCTYPE html>
2<html lang="" xml:lang="">
3  <head>
4    <title>Shiny apps and aiR quality</title>
5    <meta charset="utf-8" />
6<!-- This site is hosted on Netlify. Anyone can build and deploy a site
7     like this one for free: https://netlify.new/?utm_campaign=loops&utm_source=ai-legible&utm_medium=owned&utm_content=comment&utm_id=5128dadf-03db-47ee-b8cb-2e4fdd1b28a0
8     Netlify hosting facts for this site: static/SSR served via Netlify Edge. -->
9    <meta name="author" content="Adithi R. Upadhya" />
10    

10<script src="libs/header-attrs/header-attrs.js"></script>

10
11    <link href="libs/remark-css/default.css" rel="stylesheet" />
12    <link href="libs/remark-css/default-fonts.css" rel="stylesheet" />
13    <link href="libs/tile-view/tile-view.css" rel="stylesheet" />
14    

14<script src="libs/tile-view/tile-view.js"></script>

14
15  </head>
16  <body>
17    <textarea id="source">
18class: center, middle, title-slide
19
20# Shiny apps and aiR quality
21## EARL 2021
22### Adithi R. Upadhya
23### ILK Labs
24### 10 September, 2021
25
26---
27
28
29
30
31# Hi!
32
33My name is Adithi, and I am a Geospatial Data Analyst at ILK Labs in Bengaluru, India. I will now be presenting about how I use shiny to build tools to provide a workflow to analyse air quality data. 
34
35---
36
37# Who all R there?
38
39![Multidisciplinary team](img/team.png)
40
41---
42
43# R, shiny everywhere!
44
45&lt;img src="WWW/R.png" width="100px" /&gt;
46
47- R and shiny are a powerful combination. 
48
49- With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring) it has become necessary to use management platforms.
50
51- So here we present two shiny applications.
52
53
54???
55R and shiny are a powerful couple which can be used to build interactive platforms to manage and work with the data collected.
56
57With the ever increasing global measurements of air pollutants (through stationary, mobile, low-cost, and satellite monitoring), the amount of data being collected is huge and it necessitates the use of management platforms.
58
59In an effort to address this issue, we developed two Shiny applications to analyse and visualise air quality data.
60
61---
62# Why shiny apps?
63
64- Shiny apps can be deployed 
65
66- The user needs no programming knowledge
67
68- Help available
69
70---
71# 2 types of Air Quality Measurements
72.left-column[
73
74###Staionary monitoring
75![Stationary reference grade monitor](WWW/SM.png)
76
77]
78
79--
80
81.right-column[
82###Mobile monitoring
83![Sensors in a car](img/google-street-view.png)
84]
85???
86Typically, we have two main types of air quality measurements. 
87
88---
89# Stationary Monitoring
90
91![Network of air quality sensors in Begaluru, India](img/pa.png)
92
93???
94This is what stationary monitoring network looks like. Multiple sensors recording data at real time ~ 1 min frequency. Gives information about trends at the city scale
95---
96
97# Mobile Monitoring
98![Individual instruments for each parameter](img/setup.png)
99
100???
101For a neighborhood level measurement, we use mobile monitoring. This schematic shows the many instruments that go in the mobile platform. You can imagine the complexity, diff instruments, have different download methods, from the point of view of analysis - different data formats, different time stamps, etc. 
102
103---
104.left-column[
105###Our set-up in Bengaluru, India
106
107
108]
109
110.right-column[
111![Internal set up in the mobile monitoring platform](img/cng.jpg)
112
113]
114???
115We are looking at the inside of the car now. We have a laptop as a logger for some instruments. And all instruments are secured in the blue tray with inlets outside the vehicle. This is what looks like in practise.
116---
117.left-column[
118###but sometimes our ride is unexpected
119]
120
121.right-column[
122![Unplanned and unanticipated road blocks](img/challenges.jpg)
123
124
125]
126
127???
128All that to say that many times, things are out of our control in the field. 
129
130---
131# Our next step
132
133- Build an app to make public use the open source air quality data available and think about science. 
134
135- Build another app which helps team at ILK to perform quality checks to the high frequency data. 
136
137???
138Once we saw that stationary monitors can sometimes need cleaning and also the freely available data was for public use. To facilitate public use why not try to make an app which navigates a user to use the open source air quality data easily and think about science. 
139
140And we at ILK Labs we waiting to understand how to perform quality checks on the high dimensional data we were collecting everyday. 
141
142---
143.left-column[
144#Data Cleaning - Important!!!
145![](img/tidyverse_logo.png)
146
147]
148.right-column[
149![](img/tidyverse.png)
150
151]
152???
153We use all tidyverse packages for data cleaning. Ggplot for highly flexible plotting, purrr and map functions for more efficient and faster code instead of for loops, and forcats for functions. Readr, read_csv automatically parses date-time objects, that is is very helpful
154---
155# Working with datetime using `lubridate`
156.left-column[
157![](img/lubridate.jpeg)
158
159]
160.right-column[
161Date for different instruments can have different
162![](img/data_issues.png)
163
164- Parsing date-times with `ymd_hms()`, `dmy_hms()`...
165- Assigning time zone with `with_tz()`
166]
167
168
169???
170Step 1 is consistent time stamps so that we can join all data sets together. Some instruments are either in UTC time zone, or depending on the country of origin 

170in a diff time zone. 
171
172
173---
174class: center
175
176# Stationary Monitoring - pollucheck
177
178<img src="WWW/SM.png" width="200px" />
179
180.center[CSTEP, Begaluru, India]
181
182???
183
184The second shiny application "pollucheck" helps processing open source air quality data usually called stationary monitoring. 
185
186There are several platforms which provide open source air quality data. We built this application for users of these platforms who can have quick analysis and basic plots of air quality easily generated and think about science. 
187
188---
189
190class: center
191
192# [pollucheck](https://aruapps.shinyapps.io/OpenSourceAirQualityApp)
193
194
195<img src="WWW/pollucheck_app.png" width="800px" />
196
197???
198
199This is how pollucheck looks like. 
200
201---
202
203# [pollucheck](https://github.com/adithirgis/pollucheck)
204<img src="WWW/PolluCheck.png" width="100px" />
205
206- sources compatible - [CPCB (specific to India)](https://app.cpcbccr.com/ccr/#/caaqm-dashboard-all/caaqm-landing), [OpenAQ](https://openaq.org/#/countries/IN?_k=5ecycz), and
207[AirNow](https://www.airnow.gov/international/us-embassies-and-consulates/#India)
208
209- Data processing options available along with different summary statistics
210
211- Generates statistical and summary plots
212
213- Implements linear or multilinear regression 
214
215- Allows users to compare from two sites
216
217???
218
219"pollucheck" can be used for data downloaded from CPCB (specific to India), OpenAQ and AirNow, it aims at generating a range of statistical plots and summary statistics with several data processing options. 
220
221There are options for different averaging periods as well. It also checks for normality, generates density, Q-Q plots along with all these checks for trends in the time series for the selected parameter. 
222
223It can implement linear or multilinear regression. pollucheck allows users to upload another set of data to compare selected parameters and generate plots. 
224
225It also implements two plots from the openair package. 
226
227This is made into a package. Don't forget to check out air quality of your city using our application! 
228
229---
230
231class: center
232
233# Mobile Monitoring - mmaqshiny
234
235<img src="img/google-street-view.png" width="400px" />
236
237.center[Google Street View car]
238
239
240---
241
242class: center
243
244# [mmaqshiny](https://aruapps.shinyapps.io/mmaqshiny/)
245
246
247<img src="WWW/mmaqshiny_app.png" width="800px" />
248
249???
250
251This is how mmaqshiny looks like. 
252
253The first shiny application is called "mmaqshiny" which helps in processing high resolution air quality data collected on a moving platform usually called mobile monitoring. 
254
255So we take multiple sensors in a car, and take repeated measurements of each road in an area to generate stable high resolution air pollutant maps (usually daily maps are generated). 
256
257There is superior performance in estimating long-term mean concentrations when multiple repeated drives are possible.
258
259---
260
261# [mmaqshiny](https://github.com/meenakshi-kushwaha/mmaqshiny)
262<img src="WWW/mmaqshiny.png" width="100px" />
263
264- Handles high frequency data (~ 1 Hz)
265
266- Only mandatory file is GPS and multiple inputs possible
267
268- Visualise air pollution hot spots
269
270- Pre-processes for various instrumental sensitivities
271
272- Provides unit of analysis for further study
273
274- Joins output files from different instruments
275
276- Reduces computational labour
277
278- Alarms user on instrumental errors
279
280- Near real time quality check of the data (usually after the ride)
281
282
283
284???
285
286mmaqshiny can handle data of the order 1000's every day from each of the instrument, we had nearly 5 instruments for this study. 
287
288It reduces the time consumed for analysing each pollutant individually, helps in visualising the data collected on field each day, it can also be used to look for pollution hotspots, locations that are relatively more polluted than neighboring areas. 
289
290Each pollutant or sensor data requires specific kind of pre-processing which depends on the principle on which it operates or its mechanical setup. Eventually, it joins different instrument data into one single corrected file.
291
292
293This really helps us in achieving the unit of analysis for the rest of the study. 
294
295This application has reduced computational labour. Since mobile data contains a huge amount of spatial data, GPS data file is mandatory but other files from different instruments are not necessary. 
296
297Since Alarm tab is also present in the application, it helps the user to give a near real-time check on the health of all the instruments used. This application is available as a package on CRAN.  
298
299---
300# Coming up next
301
302- Working with models for low cost sensor data
303
304- Working with satellite images 
305
306
307---
308# Resources used
309.pull-left[
310- R packages
311  - [xaringan](https://github.com/yihui/xaringan)
312  - [xaringanextra](https://github.com/gadenbuie/xaringanExtra)
313  
314- Images from [rawpixel](https://www.rawpixel.com/)
315
316- Logos from [The Noun Project](https://thenounproject.com/)
317]
318
319
320---
321 
322![](img/Team_ILK.PNG)
323
324---
325.left-column[
326
327##Thank You everyone and EARL 2021!
328
329Website:
330[Adithi R. Upadhya](https://adithirugis.rbind.io/)
331
332Twitter:  
333[AdithiUpadhya](https://twitter.com/AdithiUpadhya)
334
335Github:
336[@adithirgis](https://github.com/adithirgis)
337
338Email:
339[[email protected]]()
340
341]
342
343.right-column[
344![](img/code_hero.jpg)
345]
346
347???
348Please let me know if you all have any further questions. I am also available in the Lounge if you want to discu

348ss. Have a great day ahead! Enjoy EARL!
349
350
351
352    </textarea>
353<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>

354<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>

354

355<script>var slideshow = remark.create({
356"highlightStyle": "github",
357"highlightLines": true,
358"countIncrementalSlides": false
359});
360if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
361  window.dispatchEvent(new Event('resize'));
362});
363(function(d) {
364  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
365  if (!r) return;
366  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
367  d.head.appendChild(s);
368})(document);
369
370(function(d) {
371  var el = d.getElementsByClassName("remark-slides-area");
372  if (!el) return;
373  var slide, slides = slideshow.getSlides(), els = el[0].children;
374  for (var i = 1; i < slides.length; i++) {
375    slide = slides[i];
376    if (slide.properties.continued === "true" || slide.properties.count === "false") {
377      els[i - 1].className += ' has-continuation';
378    }
379  }
380  var s = d.createElement("style");
381  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
382  d.head.appendChild(s);
383})(document);
384// delete the temporary CSS (for displaying all slides initially) when the user
385// starts to view slides
386(function() {
387  var deleted = false;
388  slideshow.on('beforeShowSlide', function(slide) {
389    if (deleted) return;
390    var sheets = document.styleSheets, node;
391    for (var i = 0; i < sheets.length; i++) {
392      node = sheets[i].ownerNode;
393      if (node.dataset["target"] !== "print-only") continue;
394      node.parentNode.removeChild(node);
395    }
396    deleted = true;
397  });
398})();
399(function() {
400  "use strict"
401  // Replace 

401<script> tags in slides area to make them executable
402  var scripts = document.querySelectorAll(
403    '.remark-slides-area .remark-slide-container script'
404  );
405  if (!scripts.length) return;
406  for (var i = 0; i < scripts.length; i++) {
407    var s = document.createElement('script');
408    var code = document.createTextNode(scripts[i].textContent);
409    s.appendChild(code);
410    var scriptAttrs = scripts[i].attributes;
411    for (var j = 0; j < scriptAttrs.length; j++) {
412      s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
413    }
414    scripts[i].parentElement.replaceChild(s, scripts[i]);
415  }
416})();
417(function() {
418  var links = document.getElementsByTagName('a');
419  for (var i = 0; i < links.length; i++) {
420    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
421      links[i].target = '_blank';
422    }
423  }
424})();
425// adds .remark-code-has-line-highlighted class to <pre> parent elements
426// of code chunks containing highlighted lines with class .remark-code-line-highlighted
427(function(d) {
428  const hlines = d.querySelectorAll('.remark-code-line-highlighted');
429  const preParents = [];
430  const findPreParent = function(line, p = 0) {
431    if (p > 1) return null; // traverse up no further than grandparent
432    const el = line.parentElement;
433    return el.tagName === "PRE" ? el : findPreParent(el, ++p);
434  };
435
436  for (let line of hlines) {
437    let pre = findPreParent(line);
438    if (pre && !preParents.includes(pre)) preParents.push(pre);
439  }
440  preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
441})(document);</script>

441
442

443<script>
444slideshow._releaseMath = function(el) {
445  var i, text, code, codes = el.getElementsByTagName('code');
446  for (i = 0; i < codes.length;) {
447    code = codes[i];
448    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
449      text = code.textContent;
450      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
451          /^\$\$(.|\s)+\$\$$/.test(text) ||
452          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
453        code.outerHTML = code.innerHTML;  // remove <code></code>
454        continue;
455      }
456    }
457    i++;
458  }
459};
460slideshow._releaseMath(document);
461</script>

461
462<!-- dynamically load mathjax for compatibility with self-contained -->

463<script>
464(function () {
465  var script = document.createElement('script');
466  script.type = 'text/javascript';
467  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
468  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
469    script.src  = script.src.replace(/^https?:/, '');
470  document.getElementsByTagName('head')[0].appendChild(script);
471})();
472</script>

472
473  </body>
474</html>