1<!DOCTYPE html> 2<html lang="en"> 3 4<head> 5 <link href='//fonts.googleapis.com/css?family=Roboto:900,400' rel='stylesheet' type='text/css'> 6 <link href='//fonts.googleapis.com/css?family=Droid+Sans+Mono' rel='stylesheet' type='text/css'> 7 <link rel="stylesheet" type="text/css" href="./theme/css/style.css"> 8 <link rel="stylesheet" type="text/css" href="./theme/css/syntax.css"> 9 <meta charset="utf-8" /> 10 <meta name="viewport" content="width=device-width; initial-scale=1.0"> 11 <meta name="author" content="Mark Litwintschik"> 12 <meta name="description" content="Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More..."> 13 14 <link rel="shortcut icon" href="./theme/images/mark.jpg" type="image/x-icon" /> 15 16 <link href="https://tech.marksblogg.com/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Tech Blog Full Atom Feed" /> 17 18 19 <title> 20Tech Blog </title> 21 22</head> 23 24<body> 25 <aside> 26 <div id="user_meta"> 27 <center> 28 <a href="."> 29 <img src="./theme/images/mark.jpg" alt="Mark"> 30 </a> 31 <h2><a href=".">Mark Litwintschik</a></h2> 32 </center> 33 <p>I'm a Big Data, AI, GIS & Networking Consultant with clients in the UK, USA, Sweden, Ireland & Germany. Past clients include BAA plc, Bank of America Merrill Lynch, Blackberry, Bloomberg, British Telecom, Ford, Google, ITV, IMG, Nando's, News UK, Pizza Hut, Royal Mail, T-Mobile, Williams Formula 1, Wise & UBS. I hold both a Canadian and a British passport as well as permanent residence in Estonia. Find me on <a href="https://uk.linkedin.com/in/marklitwintschik/">LinkedIn</a> & <a href="https://x.com/marklit82">X</a>. 34 </p> 35 <ul> 36 </ul> 37 </div> 38 </aside> 39 40 <main> 41 <header> 42 <p> 43 <a href=".">Home</a> 44 | <a href="./benchmarks.html">Benchmarks</a> 45 46 | <a href="./categories.html">Categories</a> 47 48 | <a href="https://tech.marksblogg.com/feeds/all.atom.xml">Atom Feed</a> 49 50 51 </p> 52 </header> 53 54 <article> 55 <div id="article_title"> 56 <h3><a href="./scikit-decide-openap-optimal-flight-planning.html">Saving Jet Fuel</a></h3> 57 </div> 58 <div id="article_text"> 59 <p class="first last">I'll walk through Scikit-decide's optimal flight planning examples.</p> 60 61 </div> 62 </article> 63 <hr /> 64 <article> 65 <div id="article_title"> 66 <h3><a href="./planet-labs-open-satellite-feed.html">Planet Labs' Open Satellite Feed</a></h3> 67 </div> 68 <div id="article_text"> 69 <p class="first last">I'll walk through Planet Labs' new open satellite imagery feed.</p> 70 71 </div> 72 </article> 73 <hr /> 74 <article> 75 <div id="article_title"> 76 <h3><a href="./iceye-open-satellite-feed.html">ICEYE's Open Satellite Feed</a></h3> 77 </div> 78 <div id="article_text"> 79 <p class="first last">I'll walk through ICEYE's new open SAR imagery feed.</p> 80 81 </div> 82 </article> 83 <hr /> 84 <article> 85 <div id="article_title"> 86 <h3><a href="./canada-radarsat.html">Tasking RADARSAT</a></h3> 87 </div> 88 <div id="article_text"> 89 <p class="first last">I'll walk through Canada's RCM SAR satellite constellation tasking schedule.</p> 90 91 </div> 92 </article> 93 <hr /> 94 <article> 95 <div id="article_title"> 96 <h3><a href="./china-hydro.html">Hydropower Stations in China in 2026</a></h3> 97 </div> 98 <div id="article_text"> 99 <p class="first last">I'll walk through a database of China's Hydropower Stations.</p> 100 101 </div> 102 </article> 103 <hr /> 104 <article> 105 <div id="article_title"> 106 <h3><a href="./little-navmap-flight-planning.html">Flight Planning with Little Navmap</a></h3> 107 </div> 108 <div id="article_text"> 109 <p class="first last">I'll walk through two example flight plans with Little Navmap.</p> 110 111 </div> 112 </article> 113 <hr /> 114 <article> 115 <div id="article_title"> 116 <h3><a href="./vantor-satellite-imagery.html">Vantor's Open Satellite Feed</a></h3> 117 </div> 118 <div id="article_text"> 119 <p class="first last">I'll walk through Vantor's open S3 bucket of satellite imagery that they launched this year.</p> 120 121 </div> 122 </article> 123 <hr /> 124 <article> 125 <div id="article_title"> 126 <h3><a href="./a380-seating.html">Optimal Seating on the Airbus A380</a></h3> 127 </div> 128 <div id="article_text"> 129 <p class="first last">I'll examine a solver that attempts to maximise passenger revenue on A380 flights.</p> 130 131 </div> 132 </article> 133 <hr /> 134 <article> 135 <div id="article_title"> 136 <h3><a href="./aviation-maps.html">Open Source Aviation Maps</a></h3> 137 </div> 138 <div id="article_text"> 139 <p class="first last">I'll examine a QGIS-based Aviation Map template that can be refreshed with current data from the FAA.</p> 140 141 </div> 142 </article> 143 <hr /> 144 <article> 145 <div id="article_title"> 146 <h3><a href="./asos-weather-observations.html">1.75B Airport Weather Observations</a></h3> 147 </div> 148 <div id="article_text"> 149 <p class="first last">I'll walk through a Parquet-formatted, Global Airport Weather Observations Dataset.</p> 150 151 </div> 152 </article> 153 <hr /> 154 <article> 155 <div id="article_title"> 156 <h3><a href="./nasa-artemis-2-jpegs.html">12K+ JPEGs from NASA's Artemis II Mission</a></h3> 157 </div> 158 <div id="article_text"> 159 <p class="first last">I'll classify their imagery with an OpenAI model and examine their EXIF metadata.</p> 160 161 </div> 162 </article> 163 <hr /> 164 <article> 165 <div id="article_title"> 166 <h3><a href="./ebike-fleet-monitoring.html">e-Bike Fleet Monitoring</a></h3> 167 </div> 168 <div id="article_text"> 169 <p class="first last">I collect and analyse e-Bike fleet data for three vendors in Edmonton, Canada.</p> 170 171 </div> 172 </article> 173 <hr /> 174 <article> 175 <div id="article_title"> 176 <h3>
176<a href="./gcat-satellite-database.html">10K+ Satellites in Space</a></h3> 177 </div> 178 <div id="article_text"> 179 <p class="first last">I look at GCAT's space objects dataset.</p> 180 181 </div> 182 </article> 183 <hr /> 184 <article> 185 <div id="article_title"> 186 <h3><a href="./reverse-geocoding-overture-maps.html">Reverse Geocoding with Overture Maps</a></h3> 187 </div> 188 <div id="article_text"> 189 <p class="first last">I explore using Overture as a data source for reverse geocoding satellite imagery footprints.</p> 190 191 </div> 192 </article> 193 <hr /> 194 <article> 195 <div id="article_title"> 196 <h3><a href="./american-solar-farms-v2.html">3.4M Solar Panels</a></h3> 197 </div> 198 <div id="article_text"> 199 <p class="first last">I look at GM-SEUS' revised US Solar Farm dataset.</p> 200 201 </div> 202 </article> 203 <hr /> 204 <article> 205 <div id="article_title"> 206 <h3><a href="./france-open-mobile-network-data.html">French Mobile Network Datasets</a></h3> 207 </div> 208 <div id="article_text"> 209 <p class="first last">I look at France's open mobile network datasets.</p> 210 211 </div> 212 </article> 213 <hr /> 214 <article> 215 <div id="article_title"> 216 <h3><a href="./canadian-wind-farms.html">Canadian Wind Farms</a></h3> 217 </div> 218 <div id="article_text"> 219 <p class="first last">I look at Natural Resources Canada's 8K-record Canadian Wind Farm dataset.</p> 220 221 </div> 222 </article> 223 <hr /> 224 <article> 225 <div id="article_title"> 226 <h3><a href="./google-street-view-coverage.html">Google Street View in 2026</a></h3> 227 </div> 228 <div id="article_text"> 229 <p class="first last">I look at Google Street View's global coverage.</p> 230 231 </div> 232 </article> 233 <hr /> 234 <article> 235 <div id="article_title"> 236 <h3><a href="./data-crew-route-optimiser-solver-framework.html">Data Crew's Route Optimiser Framework</a></h3> 237 </div> 238 <div id="article_text"> 239 <p class="first last">I look at two example solvers from Data Crew's new Route Optimiser Framework.</p> 240 241 </div> 242 </article> 243 <hr /> 244 <article> 245 <div id="article_title"> 246 <h3><a href="./overture-places-pois.html">72M Points of Interest</a></h3> 247 </div> 248 <div id="article_text"> 249 <p class="first last">I look at Overture's revamped POIs dataset.</p> 250 251 </div> 252 </article> 253 <hr /> 254 <article> 255 <div id="article_title"> 256 <h3><a href="./ms-buildings-2026.html">
256Microsoft's 2026 Global ML Building Footprints</a></h3> 257 </div> 258 <div id="article_text"> 259 <p class="first last">I look at Microsoft's refreshed ML Buildings Dataset.</p> 260 261 </div> 262 </article> 263 <hr /> 264 <article> 265 <div id="article_title"> 266 <h3><a href="./alltheplaces.html">Millions of Locations for Thousands of Brands</a></h3> 267 </div> 268 <div id="article_text"> 269 <p class="first last">I look at the All The Places project.</p> 270 271 </div> 272 </article> 273 <hr /> 274 <article> 275 <div id="article_title"> 276 <h3><a href="./administrative-boundaries.html">Better Boundaries</a></h3> 277 </div> 278 <div id="article_text"> 279 <p class="first last">I look several administrative boundary datasets.</p> 280 281 </div> 282 </article> 283 <hr /> 284 <article> 285 <div id="article_title"> 286 <h3><a href="./american-data-centers.html">American Data Centers</a></h3> 287 </div> 288 <div id="article_text"> 289 <p class="first last">I look at the Business Insider's 1,240-site data center dataset.</p> 290 291 </div> 292 </article> 293 <hr /> 294 <article> 295 <div id="article_title"> 296 <h3><a href="./american-wind-farms.html">American Wind Farms</a></h3> 297 </div> 298 <div id="article_text"> 299 <p class="first last">I look at the USGS' 76K-record US Wind Farm dataset.</p> 300 301 </div> 302 </article> 303 <hr /> 304 <article> 305 <div id="article_title"> 306 <h3><a href="./open-building-map.html">2.7 Billion Buildings</a></h3> 307 </div> 308 <div id="article_text"> 309 <p class="first last">I look at OpenBuildingMap's global building dataset.</p> 310 311 </div> 312 </article> 313 <hr /> 314 <article> 315 <div id="article_title"> 316 <h3><a href="./australia-coastline-satellite-imagery.html">Australian Coastline Imagery</a></h3> 317 </div> 318 <div id="article_text"> 319 <p class="first last">I look at the DEA's 2.6 TB Australian Coastline satellite imagery dataset.</p> 320 321 </div> 322 </article> 323 <hr /> 324 <article> 325 <div id="article_title"> 326 <h3><a href="./american-solar-farms.html">American Solar Farms</a></h3> 327 </div> 328 <div id="article_text"> 329 <p class="first last">I look at GM-SEUS's 15K-record US Solar Farm dataset.</p> 330 331 </div> 332 </article> 333 <hr /> 334 <article> 335 <div id="article_title"> 336 <h3><a href="./icmm-mining-data.html">A Global Mining Dataset</a></h3> 337 </div> 338 <div id="article_text"> 339 <p class="first last">I look at ICMM's 8K-record mining dataset.</p> 340 341 </div> 342 </article> 343 <hr /> 344 <article> 345 <div id="article_title"> 346 <h3><a href="./canadas-odb-buildings.html">Canada's 14M Buildings</a></h3> 347 </div> 348 <div id="article_text"> 349 <p class="first last">I look at Statistics Canada's 14M-building Open Database of Buildings (ODB) dataset.</p> 350 351 </div> 352 </article> 353 <hr /> 354 <article> 355 <div id="article_title"> 356 <h3><a href="./layercake-openstreetmap.html">Analysis-Ready OpenStreetMap</a></h3> 357 </div> 358 <div id="article_text"> 359 <p class="first last">I look at the Layercake's weekly Parquet-formatted OpenStreetMap files.</p> 360 361 </div> 362 </article> 363 <hr /> 364 <article> 365 <div id="article_title"> 366 <h3><a href="./canadas-buildings.html">Canada's 13M Buildings</a></h3> 367 </div> 368 <div id="article_text"> 369 <p class="first last">I look at Public Safety Canada's 13M-building Canada Structures dataset.</p> 370 371 </div> 372 </article> 373 <hr /> 374 <article> 375 <div id="article_title"> 376 <h3><a href="./building-footprints-gba.html">
376The World's 2.75B Buildings</a></h3> 377 </div> 378 <div id="article_text"> 379 <p class="first last">I look at TUM's GlobalBuildingAtlas Dataset.</p> 380 381 </div> 382 </article> 383 <hr /> 384 <article> 385 <div id="article_title"> 386 <h3><a href="./alberta-pipelines.html">Alberta's Pipelines</a></h3> 387 </div> 388 <div id="article_text"> 389 <p class="first last">I look at the Alberta Energy Regular's open datasets.</p> 390 391 </div> 392 </article> 393 <hr /> 394 <article> 395 <div id="article_title"> 396 <h3><a href="./planet-labs-tanager-hyperspectral-satellite-images.html">Planet Labs' Hyperspectral Imagery</a></h3> 397 </div> 398 <div id="article_text"> 399 <p class="first last">I look at Planet Labs' 426-band, open hyperspectral satellite imagery feed.</p> 400 401 </div> 402 </article> 403 <hr /> 404 <article> 405 <div id="article_title"> 406 <h3><a href="./overture-maps-esri-arcgis-pro.html">Overture Maps in ArcGIS Pro</a></h3> 407 </div> 408 <div id="article_text"> 409 <p class="first last">I review an add-in that imports Overture Maps datasets into Esri's ArcGIS Pro.</p> 410 411 </div> 412 </article> 413 <hr /> 414 <article> 415 <div id="article_title"> 416 <h3><a href="./apple-iphone-15-pro-depth-map-heic.html">The iPhone 15 Pro's Depth Maps</a></h3> 417 </div> 418 <div id="article_text"> 419 <p class="first last">I look at Apple's depth maps it embeds in its iPhone 15 Pro's HEIC images.</p> 420 421 </div> 422 </article> 423 <hr /> 424 <article> 425 <div id="article_title"> 426 <h3><a href="./apple-depthpro-maxar-ai-detection.html">Apple's DepthPro on Maxar's Imagery</a></h3> 427 </div> 428 <div id="article_text"> 429 <p class="first last">I look at Apple's depth estimation model on Maxar's satellite imagery of Bangkok, Thailand.</p> 430 431 </div> 432 </article> 433 <hr /> 434 <article> 435 <div id="article_title"> 436 <h3><a href="./geodeep-better-building-footprints.html">Better Building Footprints</a></h3> 437 </div> 438 <div id="article_text"> 439 <p class="first last">I straighten AI-detected building footprints with a minimalist Python library.</p> 440 441 </div> 442 </article> 443 <hr /> 444 <article> 445 <div id="article_title"> 446 <h3><a href="./arcgis-pro-35.html">Esri's ArcGIS Pro 3.5</a></h3> 447 </div> 448 <div id="article_text"> 449 <p class="first last">I review Esri's latest release of their flagship desktop GIS offering.</p> 450 451 </div> 452 </article> 453 <hr /> 454 <article> 455 <div id="article_title"> 456 <h3><a href="./depth-anything-v2-maxar-ai-detection.html">Satellites Spotting Depth</a></h3> 457 </div> 458 <div id="article_text"> 459 <p class="first last">I look at DepthAnything's depth estimation model on Maxar's satellite imagery of Bangkok, Thailand.</p> 460 461 </div> 462 </article> 463 <hr /> 464 <article> 465 <div id="article_title"> 466 <h3><a href="./gaussian-splatting.html">Turning Videos into 3D Worlds</a></h3> 467 </div> 468 <div id="article_text"> 469 <p class="first last">I turn a minute's worth of phone camera footage into a 3D world.</p> 470 471 </div> 472 </article> 473 <hr /> 474 <article> 475 <div id="article_title"> 476 <h3><a href="./geodeep-maxar-ai-detection.html">GeoDeep's AI Detection on Maxar's Satellite Imagery</a></h3> 477 </div> 478 <div id="article_text"> 479 <p class="first last">I look at GeoDeep's object detection on some of Maxar's satellite imagery of Myanmar and Bangkok, Thailand.</p> 480 481 </div> 482 </article> 483 <hr /> 484 <article> 485 <div id="article_title"> 486 <h3><a href="./canada-addresses.html">Canada's 15.8M Addresses</a></h3> 487 </div> 488 <div id="article_text"> 489 <p class="first last">I use Statistics Canada's National Address Register to produce centroids of Canada's major settlements.</p> 490 491 </div> 492 </article> 493 <hr /> 494 <article> 495 <div id="article_title"> 496 <h3><a href="./wyvern-open-data-feed.html">Wyvern's Open Satellite Feed</a></h3> 497 </div> 498 <div id="article_text"> 499 <p class="first last">I review Wyvern's 130 GBs of open satellite imagery.</p> 500 501 </div> 502 </article> 503 <hr /> 504 <article> 505 <div id="article_title"> 506 <h3><a href="./satellogic-open-data-feed.html">Satellogic's Open Satellite Feed</a></h3> 507 </div> 508 <div id="article_text"> 509 <p class="first last">I review Satellogic's 10 TBs of open satellite imagery.</p> 510 511 </div> 512 </article> 513 <hr /> 514 <article> 515 <div id="article_title"> 516 <h3><a href="./overture-dec-2024-update.html">Overture Maps' Refreshed Global Geospatial Datasets</a></h3> 517 </div> 518 <div id="article_text"> 519 <p class="first last">I look at Overture's latest open and global geospatial dataset update.</p> 520 521 </div> 522 </article> 523 <hr /> 524 <article> 525 <div id="article_title"> 526 <h3><a href="./planet-labs-webp.html">Smaller Satellite Images</a></h3> 527 </div> 528 <div id="article_text"> 529 <p class="first last">I look at a newer compression technique for satellite imagery.</p> 530 531 </div> 532 </article> 533 <hr /> 534 <article> 535 <div id="article_title"> 536 <h3><a href="./language-translation-ai-python.html">Language Translation with Python</a></h3> 537 </div> 538 <div id="article_text"> 539 <p class="first last">I look at LibreTranslate, a self-hosted, machine translation tool.</p> 540 541 </div> 542 </article> 543 <hr /> 544 <article> 545 <div id="article_title"> 546 <h3><a href="./foursquare-open-global-poi-dataset.html">Foursquare's 104M Points of Interest</a></h3> 547 </div> 548 <div id="article_text"> 549 <p class="first last">I look at Foursqaure's Open and Global Points of Interest (POI) Dataset.</p> 550 551 </div> 552 </article> 553 <hr /> 554 <article> 555 <div id="article_title"> 556 <h3><a href="./osm-mvt-vector-tiles.html">OpenStreetMap's New Vector Tiles</a></h3> 557 </div> 558 <div id="article_text"> 559 <p class="first last">I look at OpenStreetMap's new sharp, clear and customisable vector tiles.</p> 560 561 </div> 562 </article> 563 <hr /> 564 <article> 565 <div id="article_title"> 566 <h3><a href="./ornl-fema-buildings.html">131M American Buildings</a></h3> 567 </div> 568 <div id="article_text"> 569 <p class="first last">I look at Oak Ridge National Laboratory's AI-generated US Building Dataset.</p> 570 571 </div> 572 </article> 573 <hr /> 574 <article> 575 <div id="article_title"> 576 <h3><a href="./overture-land-cover.html">Land Cover from Satellite Imagery</a></h3> 577 </div> 578 <div id="article_text"> 579 <p class="first last">I look over Overture's 84 GB Land Cover dataset.</p> 580 581 </div> 582 </article> 583 <hr /> 584 <article> 585 <div id="article_title"> 586 <h3><a href="./microsofts-global-ml-building-footprints.html">
586Microsoft's 1.4 Billion Global ML Building Footprints</a></h3> 587 </div> 588 <div id="article_text"> 589 <p class="first last">I walk through Microsoft's Global ML Building Footprints Dataset.</p> 590 591 </div> 592 </article> 593 <hr /> 594 <article> 595 <div id="article_title"> 596 <h3><a href="./ai-sar-satellites-iceye-aircraft-detection.html">ICEYE's Satellite Imagery</a></h3> 597 </div> 598 <div id="article_text"> 599 <p class="first last">I try to detect aircraft in ICEYE's SAR imagery of Doha International Airport using SARDet_100K and MSFA.</p> 600 601 </div> 602 </article> 603 <hr /> 604 <article> 605 <div id="article_title"> 606 <h3><a href="./building-footprints-japan.html">Japan's Building Footprints</a></h3> 607 </div> 608 <div id="article_text"> 609 <p class="first last">I look at "Flateau", a Parquet-based Japanese Building Footprint Dataset.</p> 610 611 </div> 612 </article> 613 <hr /> 614 <article> 615 <div id="article_title"> 616 <h3><a href="./ai-sar-satellites-umbra-aircraft-detection.html">Satellites Spotting Aircraft</a></h3> 617 </div> 618 <div id="article_text"> 619 <p class="first last">I try to detect aircraft in Umbra's SAR imagery of Bangkok Airport using SARDet_100K and MSFA.</p> 620 621 </div> 622 </article> 623 <hr /> 624 <article> 625 <div id="article_title"> 626 <h3><a href="./baltic-ais-maritime-shipping-traffic-open-data-feed.html">Baltic Maritime Traffic Feed</a></h3> 627 </div> 628 <div id="article_text"> 629 <p class="first last">I look at an open ship tracking feed.</p> 630 631 </div> 632 </article> 633 <hr /> 634 <article> 635 <div id="article_title"> 636 <h3><a href="./ai-street-view-streetscapes-mapillary-kartaview.html">AI on Street View</a></h3> 637 </div> 638 <div id="article_text"> 639 <p class="first last">I look at the results of several models running inference on 10M images taken in cities across the globe.</p> 640 641 </div> 642 </article> 643 <hr /> 644 <article> 645 <div id="article_title"> 646 <h3><a href="./capella-open-data-free-satellite-sar-imagery.html">Capella's Open Satellite Feed</a></h3> 647 </div> 648 <div id="article_text"> 649 <p class="first last">I explore Capella's freely available satellite imagery.</p> 650 651 </div> 652 </article> 653 <hr /> 654 <article> 655 <div id="article_title"> 656 <h3><a href="./overture-2024-revisit.html">Revisiting Overture's Global Geospatial Datasets</a></h3> 657 </div> 658 <div id="article_text"> 659 <p class="first last">I look over some of the changes made to Overture's datasets over the past six months.</p> 660 661 </div> 662 </article> 663 <hr /> 664 <article> 665 <div id="article_title"> 666 <h3><a href="./asian-building-footprints-from-google-maps.html">AI-Extracted Asian Building Footprints</a></h3> 667 </div> 668 <div id="article_text"> 669 <p class="first last">I explore the results of a new model that aims to extract building footprints from Google Maps' Satellite Imagery.</p> 670 671 </div> 672 </article> 673 <hr /> 674 <article> 675 <div id="article_title"> 676 <h3><a href="./yolo-umbra-sar-satellites-ship-detection.html">Satellites Spotting Ships</a></h3> 677 </div> 678 <div id="article_text"> 679 <p class="first last">I train a model using YOLOv5 to detect ships in Umbra's Open Satellite Feed.</p> 680 681 </div> 682 </article> 683 <hr /> 684 <article> 685 <div id="article_title"> 686 <h3><a href="./maxar-hd-global-imagery-basemap.html">Maxar's HD Global Imagery Basemap</a></h3> 687 </div> 688 <div id="article_text"> 689 <p class="first last">
689I explore satellite imagery of eleven cities on Maxar's 30cm Global Basemap.</p> 690 691 </div> 692 </article> 693 <hr /> 694 <article> 695 <div id="article_title"> 696 <h3><a href="./heavyiq-faa-ai-llm-gpu-database.html">HeavyIQ: Understanding 220M Flights with AI</a></h3> 697 </div> 698 <div id="article_text"> 699 <p class="first last">HEAVY.AI's GPU-powered database can now understand English, not just SQL.</p> 700 701 </div> 702 </article> 703 <hr /> 704 <article> 705 <div id="article_title"> 706 <h3><a href="./av1-video-encoding.html">Minimalist Guide to AV1 Video Encoding</a></h3> 707 </div> 708 <div id="article_text"> 709 <p class="first last">I explore AV1 encoding with FFMPEG.</p> 710 711 </div> 712 </article> 713 <hr /> 714 <article> 715 <div id="article_title"> 716 <h3><a href="./umbra-open-data-free-satellite-imagery.html">Umbra's Open Satellite Feed</a></h3> 717 </div> 718 <div id="article_text"> 719 <p class="first last">I explore Umbra's freely available satellite imagery.</p> 720 721 </div> 722 </article> 723 <hr /> 724 <article> 725 <div id="article_title"> 726 <h3><a href="./open-charge-map-global-ev-charging-point-dataset.html">Global EV Charging Points with Open Charge Map</a></h3> 727 </div> 728 <div id="article_text"> 729 <p class="first last">I walk through Open Charge Map's EV Charging Station dataset.</p> 730 731 </div> 732 </article> 733 <hr /> 734 <article> 735 <div id="article_title"> 736 <h3><a href="./aircraft-route-analysis-adsb.html">Aircraft Route Analysis</a></h3> 737 </div> 738 <div id="article_text"> 739 <p class="first last">I explore ways to track individual aircraft found in adsb.lol's flight tracking dataset.</p> 740 741 </div> 742 </article> 743 <hr /> 744 <article> 745 <div id="article_title"> 746 <h3><a href="./clickhouse-14900k-1b-taxi-rides.html">1.1 Billion Taxi Rides using ClickHouse on Intel's Core i9-14900K</a></h3> 747 </div> 748 <div id="article_text"> 749 <p class="first last">I examine the performance of ClickHouse on Intel's Core i9-14900K against my 1.1B taxi rides benchmark.</p> 750 751 </div> 752 </article> 753 <hr /> 754 <article> 755 <div id="article_title"> 756 <h3><a href="./duckdb-1b-taxi-rides.html">1.1 Billion Taxi Rides using DuckDB</a></h3> 757 </div> 758 <div id="article_text"> 759 <p class="first last">I examine the performance of DuckDB against my 1.1B taxi rides benchmark.</p> 760 761 </div> 762 </article> 763 <hr /> 764 <article> 765 <div id="article_title"> 766 <h3><a href="./tokyo-walking-tour-guide.html">Tokyo Walking Tour Guide</a></h3> 767 </div> 768 <div id="article_text"> 769 <p class="first last">I build a walking tour guide using DuckDB and QGIS.</p> 770 771 </div> 772 </article> 773 <hr /> 774 <article> 775 <div id="article_title"> 776 <h3><a href="./extracting-osm-features.html">Extracting OSM Features</a></h3> 777 </div> 778 <div id="article_text"> 779 <p class="first last">I break up an OSM file into 1,087 themed GeoPackage files.</p> 780 781 </div> 782 </article> 783 <hr /> 784 <article> 785 <div id="article_title"> 786 <h3><a href="./global-flight-tracking-adsb.html">Global Flight Tracking</a></h3> 787 </div> 788 <div id="article_text"> 789 <p class="first last">I explore adsb.lol's flight tracking dataset.</p> 790 791 </div> 792 </article> 793 <hr /> 794 <article> 795 <div id="article_title"> 796 <h3><a href="./lidar-estonia.html">Mapping Estonia with LiDAR</a></h3> 797 </div> 798 <div id="article_text"> 799 <p class="first last">I explore the Estonian Land Board's LiDAR scans dataset.</p> 800 801 </div> 802 </article> 803 <hr /> 804 <article> 805 <div id="article_title"> 806 <h3><a href="./natural-earth-free-gis-data.html">Natural Earth's Global Geospatial Datasets</a></h3> 807 </div> 808 <div id="article_text"> 809 <p class="first last">I explore Natural Earth's freely available global geospatial datasets.</p> 810 811 </div> 812 </article> 813 <hr /> 814 <article> 815 <div id="article_title"> 816 <h3><a href="./maxar-open-data-free-satellite-imagery.html">Maxar's Open Satellite Feed</a></h3> 817 </div> 818 <div id="article_text"> 819 <p class="first last">I explore 1 TB of Maxar's freely available satellite imagery.</p> 820 821 </div> 822 </article> 823 <hr /> 824 <article> 825 <div id="article_title"> 826 <h3><a href="./overture-gis-data.html">Overture's Global Geospatial Datasets</a></h3> 827 </div> 828 <div id="article_text"> 829 <p class="first last">I explore Overture's three global and free-to-use mapping dataset releases.</p> 830 831 </div> 832 </article> 833 <hr /> 834 <article> 835 <div id="article_title"> 836 <h3><a href="./esri-imagery-mooc-review.html">A Review of Esri's Imagery in Action MOOC</a></h3> 837 </div> 838 <div id="article_text"> 839 <p class="first last">A review of their six-week spatial imagery course.</p> 840 841 </div> 842 </article> 843 <hr /> 844 <article> 845 <div id="article_title"> 846 <h3><a href="./vvc-encoding.html">Versatile Video Coding</a></h3> 847 </div> 848 <div id="article_text"> 849 <p class="first last">I walk through setting up a research and development environment for H.266 / VVC encoding.</p> 850 851 </div> 852 </article> 853 <hr /> 854 <article> 855 <div id="article_title"> 856 <h3><a href="./meta-ai-segment-anything-maxar.html">Segmenting Satellite Images</a></h3> 857 </div> 858 <div id="article_text"> 859 <p class="first last">
859I identify objects in aerial and phone camera imagery using Meta AI's Segmentation Model.</p> 860 861 </div> 862 </article> 863 <hr /> 864 <article> 865 <div id="article_title"> 866 <h3><a href="./esri-data-science-mooc-review.html">A Review of Esri's Spatial Data Science MOOC</a></h3> 867 </div> 868 <div id="article_text"> 869 <p class="first last">A review of their six-week course which focuses on their ArcGIS Pro offering.</p> 870 871 </div> 872 </article> 873 <hr /> 874 <article> 875 <div id="article_title"> 876 <h3><a href="./clickhouse-gis-rust.html">Enhancing ClickHouse's Geospatial Support</a></h3> 877 </div> 878 <div id="article_text"> 879 <p class="first last">I review Clickgis, a Rust-based extension that adds WKB and GeoJSON support to ClickHouse.</p> 880 881 </div> 882 </article> 883 <hr /> 884 <article> 885 <div id="article_title"> 886 <h3><a href="./langchain-llama-cpp-pgvector-local-llm.html">Asking a Large Language Model How YouTube Works</a></h3> 887 </div> 888 <div id="article_text"> 889 <p class="first last">I ask Platypus2 13B questions about a PDF.</p> 890 891 </div> 892 </article> 893 <hr /> 894 <article> 895 <div id="article_title"> 896 <h3><a href="./h3-duckdb-qgis.html">Geospatial Clustering with Uber's H3 in DuckDB & QGIS</a></h3> 897 </div> 898 <div id="article_text"> 899 <p class="first last">I revisit Uber's H3 with a more concise method for producing geospatial clusters.</p> 900 901 </div> 902 </article> 903 <hr /> 904 <article> 905 <div id="article_title"> 906 <h3><a href="./popular-airline-passenger-routes-2023.html">Popular Airline Passenger Routes Refresh</a></h3> 907 </div> 908 <div id="article_text"> 909 <p class="first last">I've extracted the most popular commercial airline passenger routes from 21 GB of Wikipedia articles.</p> 910 911 </div> 912 </article> 913 <hr /> 914 <article> 915 <div id="article_title"> 916 <h3><a href="./streaming-video-hls.html">Streaming Video</a></h3> 917 </div> 918 <div id="article_text"> 919 <p class="first last">I walk through hosting streaming videos using FFmpeg, Bento4, Caddy Server and HLS.</p> 920 921 </div> 922 </article> 923 <hr /> 924 <article> 925 <div id="article_title"> 926 <h3><a href="./ipinfo-free-ip-address-location-database.html">IPinfo's Free IP Address Location Database</a></h3> 927 </div> 928 <div id="article_text"> 929 <p class="first last">I walk through IPinfo's free IPv4 and IPv6 location database.</p> 930 931 </div> 932 </article> 933 <hr /> 934 <article> 935 <div id="article_title"> 936 <h3><a href="./duckdb-gis-spatial-extension.html">DuckDB's Spatial Extension</a></h3> 937 </div> 938 <div id="article_text"> 939 <p class="first last">DuckDB can now open 50+ GIS file formats. I use it to help examine the Bing Maps team's AI road detection project.</p> 940 941 </div> 942 </article> 943 <hr /> 944 <article> 945 <div id="article_title"> 946 <h3><a href="./duckdb-geospatial-gis.html">Geospatial DuckDB</a></h3> 947 </div> 948 <div id="article_text"> 949 <p class="first last">I walk through basic geospatial workflows in DuckDB.</p> 950 951 </div> 952 </article> 953 <hr /> 954 <article> 955 <div id="article_title"> 956 <h3><a href="./route-planning-europe-postgresql-pgrouting.html">European Route Planning</a></h3> 957 </div> 958 <div id="article_text"> 959 <p class="first last">I build a pan-European Bus Route Planner.</p> 960 961 </div> 962 </article> 963 <hr /> 964 <article> 965 <div id="article_title"> 966 <h3><a href="./postgresql-to-bigquery.html">Faster PostgreSQL To BigQuery Transfers</a></h3> 967 </div> 968 <div id="article_text"> 969 <p class="first last">I compare shipping data via CSV and Parquet from PostgreSQL to BigQuery.</p> 970 971 </div> 972 </article> 973 <hr /> 974 <article> 975 <div id="article_title"> 976 <h3><a href="./billion-taxi-rides-doublecloud-clickhouse.html">1.1 Billion Taxi Rides in ClickHouse on DoubleCloud</a></h3> 977 </div> 978 <div id="article_text"> 979 <p class="first last">I investigate how fast DoubleCloud can query 1.1 billion taxi journeys using their managed ClickHouse solution.</p> 980 981 </div> 982 </article> 983 <hr /> 984 <article> 985 <div id="article_title"> 986 <h3><a href="./valhalla-isochrones.html">Awesome Isochrones</a></h3> 987 </div> 988 <div id="article_text"> 989 <p class="first last">I show how you can create beautiful isochrone maps using Valhalla and QGIS.</p> 990 991 </div> 992 </article> 993 <hr /> 994 <article> 995 <div id="article_title"> 996 <h3><a href="./python-data-visualisation-echarts-graphs-plots.html">ECharts for Python</a></h3> 997 </div> 998 <div id="article_text"> 999 <p class="first last">I explore a Python wrapper for Apache ECharts.</p> 1000 1001 </div> 1002 </article> 1003 <hr /> 1004 <article> 1005 <div id="article_title"> 1006 <h3><a href="./python-data-visualisation-charts-graphs-plots.html">Python Data Visualisation</a></h3> 1007 </div> 1008 <div id="article_text"> 1009 <p class="first last">I explore Altair, a concise API for charting in Python.</p> 1010 1011 </div> 1012 </article> 1013 <hr /> 1014 <article> 1015 <div id="article_title"> 1016 <h3><a href="./hardening-ssh.html">
1016Hardening SSH</a></h3> 1017 </div> 1018 <div id="article_text"> 1019 <p class="first last">I walk through setting up BastionZero on an AWS EC2 instance.</p> 1020 1021 </div> 1022 </article> 1023 <hr /> 1024 <article> 1025 <div id="article_title"> 1026 <h3><a href="./pretty-maps-in-python.html">Pretty Maps in Python</a></h3> 1027 </div> 1028 <div id="article_text"> 1029 <p class="first last">I show how you can create beautiful maps in Python.</p> 1030 1031 </div> 1032 </article> 1033 <hr /> 1034 <article> 1035 <div id="article_title"> 1036 <h3><a href="./making-heatmaps-python-qgis.html">Making Heatmaps</a></h3> 1037 </div> 1038 <div id="article_text"> 1039 <p class="first last">I walk through a GIS toolchain for creating heatmaps.</p> 1040 1041 </div> 1042 </article> 1043 <hr /> 1044 <article> 1045 <div id="article_title"> 1046 <h3><a href="./poem-rust-web-framework.html">Minimalist Guide to Poem</a></h3> 1047 </div> 1048 <div id="article_text"> 1049 <p class="first last">A review of the Rust-based Web Framework Poem.</p> 1050 1051 </div> 1052 </article> 1053 <hr /> 1054 <article> 1055 <div id="article_title"> 1056 <h3><a href="./axum-rust-web-framework.html">Minimalist Guide to Axum</a></h3> 1057 </div> 1058 <div id="article_text"> 1059 <p class="first last">I review the features and community benchmarks of the Rust-based Web Framework Axum.</p> 1060 1061 </div> 1062 </article> 1063 <hr /> 1064 <article> 1065 <div id="article_title"> 1066 <h3><a href="./caddy-https-minio.html">File Sharing with Caddy & MinIO</a></h3> 1067 </div> 1068 <div id="article_text"> 1069 <p class="first last">Cost-effective, mobile-friendly file sharing using two Go-based offerings.</p> 1070 1071 </div> 1072 </article> 1073 <hr /> 1074 <article> 1075 <div id="article_title"> 1076 <h3><a href="./tree-heights-open5g.html">Deploying 5G Around Trees</a></h3> 1077 </div> 1078 <div id="article_text"> 1079 <p class="first last">I explain how Open5G digs through 3.5 trillion records produced by a deep learning algorithm trained on a massive cluster in Switzerland that was fed imagery of the entire earth from two satellites to decide how to roll out 5G in California.</p> 1080 1081 </div> 1082 </article> 1083 <hr /> 1084 <article> 1085 <div id="article_title"> 1086 <h3><a href="./streets-of-monaco-openstreetmap-postgis-qgis.html">The Streets of Monaco</a></h3> 1087 </div> 1088 <div id="article_text"> 1089 <p class="first last">I walk through a GIS toolchain for visualising the streets of Monaco and its Formula 1 circuit.</p> 1090 1091 </div> 1092 </article> 1093 <hr /> 1094 <article> 1095 <div id="article_title"> 1096 <h3><a href="./install-clickhouse-faster.html">Install ClickHouse Faster</a></h3> 1097 </div> 1098 <div id="article_text"> 1099 <p class="first last">I look at the latest way to get ClickHouse running quickly.</p> 1100 1101 </div> 1102 </article> 1103 <hr /> 1104 <article> 1105 <div id="article_title"> 1106 <h3><a href="./faster-geospatial-enrichment.html">Faster Geospatial Enrichment</a></h3> 1107 </div> 1108 <div id="article_text"> 1109 <p class="first last">I compare latitude and longitude to h3 binning times between PostgreSQL, BigQuery and ClickHouse.</p> 1110 1111 </div> 1112 </article> 1113 <hr /> 1114 <article> 1115 <div id="article_title"> 1116 <h3><a href="./where-are-ip-addresses-ipinfo.html">Where is every IP Address?</a></h3> 1117 </div> 1118 <div id="article_text"> 1119 <p class="first last">I describe how IPinfo finds the location of almost every IP address on earth.</p> 1120 1121 </div> 1122 </article> 1123 <hr /> 1124 <article> 1125 <div id="article_title"> 1126 <h3><a href="./rdns-domain-name-tld-extract-golang.html">Faster Top Level Domain Name Extraction with Go</a></h3> 1127 </div> 1128 <div id="article_text"> 1129 <p class="first last">I port a Python-based TLD extraction script to Go.</p> 1130 1131 </div> 1132 </article> 1133 <hr /> 1134 <article> 1135 <div id="article_title"> 1136 <h3><a href="./fastest-fizz-buzz.html">The Fastest FizzBuzz Implementation</a></h3> 1137 </div> 1138 <div id="article_text"> 1139 <p class="first last">I look at an implementation of FizzBuzz that can generate output at a rate of 56 GB/s.</p> 1140 1141 </div> 1142 </article> 1143 <hr /> 1144 <article> 1145 <div id="article_title"> 1146 <h3><a href="./roapi-rust-data-api.html">ROAPI: An API Server for Static Datasets</a></h3> 1147 </div> 1148 <div id="article_text"> 1149 <p class="first last">I review the features and benchmark ROAPI.</p> 1150 1151 </div> 1152 </article> 1153 <hr /> 1154 <article> 1155 <div id="article_title"> 1156 <h3><a href="./actix-rust-web-framework.html">
1156Actix: A Web Framework for Rust</a></h3> 1157 </div> 1158 <div id="article_text"> 1159 <p class="first last">I review the features and community benchmarks of Actix.</p> 1160 1161 </div> 1162 </article> 1163 <hr /> 1164 <article> 1165 <div id="article_title"> 1166 <h3><a href="./rocket-rust-web-framework.html">Rocket: A Web Framework for Rust</a></h3> 1167 </div> 1168 <div id="article_text"> 1169 <p class="first last">I review the features and community benchmarks of Rocket.</p> 1170 1171 </div> 1172 </article> 1173 <hr /> 1174 <article> 1175 <div id="article_title"> 1176 <h3><a href="./postgresql-extension-rust.html">Building PostgreSQL Extensions with Rust</a></h3> 1177 </div> 1178 <div id="article_text"> 1179 <p class="first last">I build a PostgreSQL function in Rust and use it to try and transform 1.27B records.</p> 1180 1181 </div> 1182 </article> 1183 <hr /> 1184 <article> 1185 <div id="article_title"> 1186 <h3><a href="./rdns-domain-name-tld-extract-rust.html">Faster Top Level Domain Name Extraction with Rust</a></h3> 1187 </div> 1188 <div id="article_text"> 1189 <p class="first last">I port a Python-based TLD extraction script to Rust.</p> 1190 1191 </div> 1192 </article> 1193 <hr /> 1194 <article> 1195 <div id="article_title"> 1196 <h3><a href="./git-track-changes-in-media-office-documents.html">Track changes in Excel, Word, PowerPoint, PDFs, Images & Videos with Git</a></h3> 1197 </div> 1198 <div id="article_text"> 1199 <p class="first last">I walk through tracking changes in rich documents using Git.</p> 1200 1201 </div> 1202 </article> 1203 <hr /> 1204 <article> 1205 <div id="article_title"> 1206 <h3><a href="./snappy-s2-compression-golang.html">Faster Compression with Snappy's S2 Extension</a></h3> 1207 </div> 1208 <div id="article_text"> 1209 <p class="first last">I walk through installing and running Snappy's S2 extension.</p> 1210 1211 </div> 1212 </article> 1213 <hr /> 1214 <article> 1215 <div id="article_title"> 1216 <h3><a href="./meilisearch-full-text-search.html">MeiliSearch: A Minimalist Full-Text Search Engine</a></h3> 1217 </div> 1218 <div id="article_text"> 1219 <p class="first last">I walk through installing and running MeiliSearch.</p> 1220 1221 </div> 1222 </article> 1223 <hr /> 1224 <article> 1225 <div id="article_title"> 1226 <h3><a href="./minio-aws-s3-hdfs.html">MinIO: A Bare Metal Drop-In for AWS S3</a></h3> 1227 </div> 1228 <div id="article_text"> 1229 <p class="first last">I walk through running an AWS S3-compatible storage service on HDFS.</p> 1230 1231 </div> 1232 </article> 1233 <hr /> 1234 <article> 1235 <div id="article_title"> 1236 <h3><a href="./clickhouse-prometheus-grafana.html">Monitor ClickHouse with Prometheus & Grafana</a></h3> 1237 </div> 1238 <div id="article_text"> 1239 <p class="first last">Keep an eye on ClickHouse with Prometheus and Grafana.</p> 1240 1241 </div> 1242 </article> 1243 <hr /> 1244 <article> 1245 <div id="article_title"> 1246 <h3><a href="./data-fluent-for-postgresql.html">Data Fluent for PostgreSQL</a></h3> 1247 </div> 1248 <div id="article_text"> 1249 <p class="first last">Build a better understanding of your data in PostgreSQL.</p> 1250 1251 </div> 1252 </article> 1253 <hr /> 1254 <article> 1255 <div id="article_title"> 1256 <h3><a href="./hydrolix-1b-taxi-rides-aws.html">1.1 Billion Taxi Rides using Hydrolix on AWS</a></h3> 1257 </div> 1258 <div id="article_text"> 1259 <p class="first last">I examine the performance of Hydrolix against my 1.1B taxi rides benchmark.</p> 1260 1261 </div> 1262 </article> 1263 <hr /> 1264 <article> 1265 <div id="article_title"> 1266 <h3><a href="./omnisci-macos-macbookpro-mbp.html">1.1 Billion Taxi Rides using OmniSciDB and a MacBook Pro</a></h3> 1267 </div> 1268 <div id="article_text"> 1269 <p class="first last">I investigate how fast OmniSciDB can query 1.1 billion taxi journeys using a 16" MacBook Pro.</p> 1270 1271 </div> 1272 </article> 1273 <hr /> 1274 <article> 1275 <div id="article_title"> 1276 <h3><a href="./python-scraper-wireguard-vpn-ssh-proxy.html">Python Web Scraping with Virtual Private Networks</a></h3> 1277 </div> 1278 <div id="article_text"> 1279 <p class="first last">Proxy Python and curl web requests through WireGuard and OpenSSH.</p> 1280 1281 </div> 1282 </article> 1283 <hr /> 1284 <article> 1285 <div id="article_title"> 1286 <h3><a href="./fast-ip-to-hostname-clickhouse-postgresql.html">Fast IPv4 to Host Lookups</a></h3> 1287 </div> 1288 <div id="article_text"> 1289 <p class="first last">I compare PostgreSQL and ClickHouse performance characteristics while performing IPv4 to hostname lookups.</p> 1290 1291 </div> 1292 </article> 1293 <hr /> 1294 <article> 1295 <div id="article_title"> 1296 <h3><a href="./faster-zip-decompression-unzip-deflate-zlib-crc32-adler32-7zip-archiver.html">Faster ZIP Decompression</a></h3> 1297 </div> 1298 <div id="article_text"> 1299 <p class="first last">I compare the decompression times of various DEFLATE implementations.</p> 1300 1301 </div> 1302 </article> 1303 <hr /> 1304 <article> 1305 <div id="article_title"> 1306 <h3><a href="./faster-clickhouse-imports-csv-parquet-mysql.html">Faster ClickHouse Imports</a></h3> 1307 </div> 1308 <div id="article_text"> 1309 <p class="first last">I compare import times of various formats into ClickHouse.</p> 1310 1311 </div> 1312 </article> 1313 <hr /> 1314 <article> 1315 <div id="article_title"> 1316 <h3><a href="./youtube-database-procella.html">YouTube's Database "Procella"</a></h3> 1317 </div> 1318 <div id="article_text"> 1319 <p class="first last">I analyse material recently published on Google's "Procella" query processing engine which powers YouTube.</p> 1320 1321 </div> 1322 </article> 1323 <hr /> 1324 <article> 1325 <div id="article_title"> 1326 <h3><a href="./is-hadoop-dead.html">Is Hadoop Dead?</a></h3> 1327 </div> 1328 <div id="article_text"> 1329 <p class="first last">I analyse and debate arguments surrounding the "demise" of Hadoop.</p> 1330 1331 </div> 1332 </article> 1333 <hr /> 1334 <article> 1335 <div id="article_title"> 1336 <h3><a href="./minimalist-guide-compression.html">Minimalist Guide to Lossless Compression</a></h3> 1337 </div> 1338 <div id="article_text"> 1339 <p class="first last">I look at various aspects of lossless compression.</p> 1340 1341 </div> 1342 </article> 1343 <hr /> 1344 <article> 1345 <div id="article_title"> 1346 <h3><a href="./faster-file-distribution-hadoop-hdfs-s3.html">Faster File Distribution with HDFS and S3</a></h3> 1347 </div> 1348 <div id="article_text"> 1349 <p class="first last">I look for faster ways of transferring files between HDFS and AWS S3.</p> 1350 1351 </div> 1352 </article> 1353 <hr /> 1354 <article> 1355 <div id="article_title"> 1356 <h3><a href="./minimalist-guide-tutorial-flume.html">A Minimalist Guide to Flume</a></h3> 1357 </div> 1358 <div id="article_text"> 1359 <p class="first last">I take a look at Apache Flume and walk through an example using it to connect Kafka to HDFS.</p> 1360 1361 </div> 1362 </article> 1363 <hr /> 1364 <article> 1365 <div id="article_title"> 1366 <h3><a href="./minimalist-guide-tutorial-foundationdb.html">A Minimalist Guide to FoundationDB</a></h3> 1367 </div> 1368 <div id="article_text"> 1369 <p class="first last">I take a short look at FoundationDB and walk through a leaderboard example using Python.</p> 1370 1371 </div> 1372 </article> 1373 <hr /> 1374 <article> 1375 <div id="article_title"> 1376 <h3><a href="./architecting-modern-data-platforms-book-review.html">"Architecting Modern Data Platforms" Book Review</a></h3> 1377 </div> 1378 <div id="article_text"> 1379 <p class="first last">I review the Hadoop-focused book "Architecting Modern Data Platforms".</p> 1380 1381 </div> 1382 </article> 1383 <hr /> 1384 <article> 1385 <div id="article_title"> 1386 <h3><a href="./billion-nyc-taxi-rides-clickhouse-cluster.html">1.1 Billion Taxi Rides: 108-core ClickHouse Cluster</a></h3> 1387 </div> 1388 <div id="article_text"> 1389 <p class="first last">I investigate how fast ClickHouse 18.16.1 can query 1.1 billion taxi journeys on a 3-node, 108-core AWS EC2 cluster.</p> 1390 1391 </div> 1392 </article> 1393 <hr /> 1394 <article> 1395 <div id="article_title"> 1396 <h3><a href="./faster-csv-to-orc-conversions.html">Convert CSVs to ORC Faster</a></h3> 1397 </div> 1398 <div id="article_text"> 1399 <p class="first last">I compare the ORC file construction times of Spark 2.4.0, Hive 2.3.4 and Presto 0.214.</p> 1400 1401 </div> 1402 </article> 1403 <hr /> 1404 <article> 1405 <div id="article_title"> 1406 <h3><a href="./billion-nyc-taxi-rides-spark-2-4-versus-presto-214.html">1.1 Billion Taxi Rides: Spark 2.4.0 versus Presto 0.214</a></h3> 1407 </div> 1408 <div id="article_text"> 1409 <p class="first last">I investigate how fast Spark and Presto can query 1.1 Billion Taxi Journeys using a 21-node EMR cluster.</p> 1410 1411 </div> 1412 </article> 1413 <hr /> 1414 <article> 1415 <div id="article_title"> 1416 <h3><a href="./working-with-hdfs.html">Working with the Hadoop Distributed File System</a></h3> 1417 </div> 1418 <div id="article_text"> 1419 <p class="first last">I explore several HDFS interfaces and compare them to the JVM-based Apache Hadoop HDFS CLI.</p> 1420 1421 </div> 1422 </article> 1423 <hr /> 1424 <article> 1425 <div id="article_title"> 1426 <h3><a href="./top-htop-glances.html">Systems Monitoring: top vs Htop vs Glances</a></h3> 1427 </div> 1428 <div id="article_text"> 1429 <p class="first last">An examination and comparison of top, Htop and Glances; three tools for performing ad-hoc monitoring of systems and application performance.</p> 1430 1431 </div> 1432 </article> 1433 <hr /> 1434 <article> 1435 <div id="article_title"> 1436 <h3><a href="./working-with-data-feeds.html">Working with Data Feeds</a></h3> 1437 </div> 1438 <div id="article_text"> 1439 <p class="first last">
1439This tutorial covers converting Wikipedia's XML dump of its English-language site into CSV, JSON, AVRO and ORC file formats as well as analysing the data using ClickHouse.</p> 1440 1441 </div> 1442 </article> 1443 <hr /> 1444 <article> 1445 <div id="article_title"> 1446 <h3><a href="./mssql-sql-server-linux-install-tutorial-and-guide.html">A Minimalist Guide to Microsoft SQL Server 2017 on Ubuntu Linux</a></h3> 1447 </div> 1448 <div id="article_text"> 1449 <p class="first last">This tutorial covers importing CSV data into SQL Server 2017, automating data pipeline tasks via Apache Airflow and visualising data using Pandas and Jupyter Notebooks.</p> 1450 1451 </div> 1452 </article> 1453 <hr /> 1454 <article> 1455 <div id="article_title"> 1456 <h3><a href="./billion-nyc-taxi-rides-sqlite-parquet-hdfs.html">1.1 Billion Taxi Rides with SQLite, Parquet & HDFS</a></h3> 1457 </div> 1458 <div id="article_text"> 1459 <p class="first last">I investigate how fast SQLite can query 1.1 billion taxi journeys from Parquet files off of HDFS.</p> 1460 1461 </div> 1462 </article> 1463 <hr /> 1464 <article> 1465 <div id="article_title"> 1466 <h3><a href="./install-and-configure-apache-airflow.html">Customising Airflow: Beyond Boilerplate Settings</a></h3> 1467 </div> 1468 <div id="article_text"> 1469 <p class="first last">I walk through setting up Apache Airflow to use Dask.distributed, PostgreSQL, logging to AWS S3 as well as create User accounts and Plugins.</p> 1470 1471 </div> 1472 </article> 1473 <hr /> 1474 <article> 1475 <div id="article_title"> 1476 <h3><a href="./presto-connectors-kafka-mongodb-mysql-postgresql-redis.html">Using SQL to query Kafka, MongoDB, MySQL, PostgreSQL and Redis with Presto</a></h3> 1477 </div> 1478 <div id="article_text"> 1479 <p class="first last">A guide to connecting to five different data stores using Presto.</p> 1480 1481 </div> 1482 </article> 1483 <hr /> 1484 <article> 1485 <div id="article_title"> 1486 <h3><a href="./python-big-data-airflow-jupyter-notebook-hadoop-3-hive-presto.html">Python & Big Data: Airflow & Jupyter Notebook with Hadoop 3, Spark & Presto</a></h3> 1487 </div> 1488 <div id="article_text"> 1489 <p class="first last">A guide to running Airflow and Jupyter Notebook with Hadoop 3, Spark & Presto.</p> 1490 1491 </div> 1492 </article> 1493 <hr /> 1494 <article> 1495 <div id="article_title"> 1496 <h3><a href="./billion-nyc-taxi-rides-ec2-versus-emr.html">1.1 Billion Taxi Rides: EC2 versus EMR</a></h3> 1497 </div> 1498 <div id="article_text"> 1499 <p class="first last">I investigate how fast Spark and Presto can query 1.1 Billion Taxi Journeys using an i3.8xlarge EC2 instance with 1.7 TB of NVMe storage versus a 21-node EMR cluster.</p> 1500 1501 </div> 1502 </article> 1503 <hr /> 1504 <article> 1505 <div id="article_title"> 1506 <h3><a href="./hadoop-3-single-node-install-guide.html">Hadoop 3 Single-Node Install Guide</a></h3> 1507 </div> 1508 <div id="article_text"> 1509 <p class="first last">A simple Hadoop 3 installation guide for Ubuntu 16 that includes Hive, Spark and Presto.</p> 1510 1511 </div> 1512 </article> 1513 <hr /> 1514 <article> 1515 <div id="article_title"> 1516 <h3><a href="./billion-nyc-taxi-rides-brytlytdb-ibm-minsky.html">1.1 Billion Taxi Rides with BrytlytDB 2.1 & a 5-node IBM Minsky Cluster</a></h3> 1517 </div> 1518 <div id="article_text"> 1519 <p class="first last">I investigate how fast BrytlytDB 2.1 can query 1.1 billion taxi journeys using five IBM Minsky servers with 20 Nvidia P100 GPUs.</p> 1520 1521 </div> 1522 </article> 1523 <hr /> 1524 <article> 1525 <div id="article_title"> 1526 <h3><a href="./billion-nyc-taxi-rides-p2-16xlarge-brytlytdb-2.html">1.1 Billion Taxi Rides with BrytlytDB 2.0 & 2 GPU-Powered p2.16xlarge EC2 Instances</a></h3> 1527 </div> 1528 <div id="article_text"> 1529 <p class="first last">I investigate how fast BrytlytDB 2.0 can query 1.1 billion taxi journeys using two p16.8xlarge AWS EC2 instances.</p> 1530 1531 </div> 1532 </article> 1533 <hr /> 1534 <article> 1535 <div id="article_title"> 1536 <h3><a href="./sqlite3-tutorial-and-guide.html">A Minimalist Guide to SQLite</a></h3> 1537 </div> 1538 <div id="article_text"> 1539 <p class="first last">
1539This tutorial covers importing CSV data into SQLite 3, manipulating data via Python and visualising data using Pandas and Jupyter Notebooks.</p> 1540 1541 </div> 1542 </article> 1543 <hr /> 1544 <article> 1545 <div id="article_title"> 1546 <h3><a href="./billion-nyc-taxi-rides-spark-raspberry-pi.html">1.1 Billion Taxi Rides with Spark 2.2 & 3 Raspberry Pi 3 Model Bs</a></h3> 1547 </div> 1548 <div id="article_text"> 1549 <p class="first last">I investigate how fast Spark 2.2 can query 1.1 billion taxi journeys using a cluster of three Raspberry Pis.</p> 1550 1551 </div> 1552 </article> 1553 <hr /> 1554 <article> 1555 <div id="article_title"> 1556 <h3><a href="./billion-nyc-taxi-rides-aws-ec2-p2-16xlarge-brytlytdb.html">1.1 Billion Taxi Rides with BrytlytDB & 2 GPU-Powered p2.16xlarge EC2 Instances</a></h3> 1557 </div> 1558 <div id="article_text"> 1559 <p class="first last">I investigate how fast BrytlytDB can query 1.1 billion taxi journeys using two p16.8xlarge AWS EC2 instances.</p> 1560 1561 </div> 1562 </article> 1563 <hr /> 1564 <article> 1565 <div id="article_title"> 1566 <h3><a href="./compiling-mapd-ubuntu-16.html">Compiling MapD's Source Code</a></h3> 1567 </div> 1568 <div id="article_text"> 1569 <p class="first last">In this tutorial I walk-through building MapD from source on an Ubuntu 16.04.2 machine.</p> 1570 1571 </div> 1572 </article> 1573 <hr /> 1574 <article> 1575 <div id="article_title"> 1576 <h3><a href="./billion-nyc-taxi-rides-aws-ec2-p2-8xlarge-mapd.html">1.1 Billion Taxi Rides with MapD 3.0 & 2 GPU-Powered p2.8xlarge EC2 Instances</a></h3> 1577 </div> 1578 <div id="article_text"> 1579 <p class="first last">I investigate how fast MapD 3.0 can query 1.1 billion taxi journeys using two p2.8xlarge AWS EC2 instances.</p> 1580 1581 </div> 1582 </article> 1583 <hr /> 1584 <article> 1585 <div id="article_title"> 1586 <h3><a href="./detect-bots-apache-nginx-logs.html">Detecting Bots in Apache & Nginx Logs</a></h3> 1587 </div> 1588 <div id="article_text"> 1589 <p class="first last">I explore the task of bot detection in web traffic logs.</p> 1590 1591 </div> 1592 </article> 1593 <hr /> 1594 <article> 1595 <div id="article_title"> 1596 <h3><a href="./tensorflow-vizdoom-bots.html">Doom Bots in TensorFlow</a></h3> 1597 </div> 1598 <div id="article_text"> 1599 <p class="first last">I walk through using TensorFlow to train AI Bots to play Doom, a classic first-person shooter.</p> 1600 1601 </div> 1602 </article> 1603 <hr /> 1604 <article> 1605 <div id="article_title"> 1606 <h3>
1606<a href="./petabytes-of-website-data-spark-emr.html">Analysing Petabytes of Websites</a></h3> 1607 </div> 1608 <div id="article_text"> 1609 <p class="first last">I demonstrate how to extract analytical data from petabytes worth of websites collected by Common Crawl.</p> 1610 1611 </div> 1612 </article> 1613 <hr /> 1614 <article> 1615 <div id="article_title"> 1616 <h3><a href="./designing-data-intensive-applications-review.html">A Review of "Designing Data-Intensive Applications"</a></h3> 1617 </div> 1618 <div id="article_text"> 1619 <p class="first last">I review an early release of Martin Kleppmann's book "Designing Data-Intensive Applications".</p> 1620 1621 </div> 1622 </article> 1623 <hr /> 1624 <article> 1625 <div id="article_title"> 1626 <h3><a href="./billion-nyc-taxi-clickhouse.html">1.1 Billion Taxi Rides on ClickHouse & an Intel Core i5</a></h3> 1627 </div> 1628 <div id="article_text"> 1629 <p class="first last">I investigate how fast ClickHouse can query 1.1 billion taxi journeys on an Intel Core i5 4670K.</p> 1630 1631 </div> 1632 </article> 1633 <hr /> 1634 <article> 1635 <div id="article_title"> 1636 <h3><a href="./billion-nyc-taxi-vertica.html">1.1 Billion Taxi Rides on Vertica & an Intel Core i5</a></h3> 1637 </div> 1638 <div id="article_text"> 1639 <p class="first last">I investigate how fast Vertica Community Edition 8.0.1 can query 1.1 billion taxi journeys on an Intel Core i5 4670K.</p> 1640 1641 </div> 1642 </article> 1643 <hr /> 1644 <article> 1645 <div id="article_title"> 1646 <h3><a href="./billion-nyc-taxi-rides-spark-2-1-0-emr.html">1.1 Billion Taxi Rides on AWS EMR 5.3.0 & Spark 2.1.0</a></h3> 1647 </div> 1648 <div id="article_text"> 1649 <p class="first last">I investigate how fast an 11-node Spark 2.1.0 cluster can query over a billion records.</p> 1650 1651 </div> 1652 </article> 1653 <hr /> 1654 <article> 1655 <div id="article_title"> 1656 <h3><a href="./billion-nyc-taxi-kdb.html">1.1 Billion Taxi Rides on kdb+/q & 4 Xeon Phi CPUs</a></h3> 1657 </div> 1658 <div id="article_text"> 1659 <p class="first last">I investigate how fast kdb+/q can query 1.1 billion taxi journeys on 4 Intel Xeon Phi 7210 CPUs.</p> 1660 1661 </div> 1662 </article> 1663 <hr /> 1664 <article> 1665 <div id="article_title"> 1666 <h3><a href="./billion-nyc-taxi-rides-aws-athena.html">1.1 Billion Taxi Rides on Amazon Athena</a></h3> 1667 </div> 1668 <div id="article_text"> 1669 <p class="first last">I investigate how fast Amazon Athena can query 1.1 billion taxi journeys.</p> 1670 1671 </div> 1672 </article> 1673 <hr /> 1674 <article> 1675 <div id="article_title"> 1676 <h3><a href="./alenka-open-source-gpu-database.html">Alenka: A GPU-Driven, Open Source Database</a></h3> 1677 </div> 1678 <div id="article_text"> 1679 <p class="first last">I walk through installing, loading in data and querying Alenka.</p> 1680 1681 </div> 1682 </article> 1683 <hr /> 1684 <article> 1685 <div id="article_title"> 1686 <h3><a href="./billion-nyc-taxi-rides-nvidia-pascal-titan-x-mapd.html">1.1 Billion Taxi Rides with MapD & 8 Nvidia Pascal Titan Xs</a></h3> 1687 </div> 1688 <div id="article_text"> 1689 <p class="first last">I investigate how fast MapD can query 1.1 billion taxi journeys using 8 Nvidia Pascal-based Titan X cards.</p> 1690 1691 </div> 1692 </article> 1693 <hr /> 1694 <article> 1695 <div id="article_title"> 1696 <h3><a href="./tensorflow-nvidia-gtx-1080.html">TensorFlow on a GTX 1080</a></h3> 1697 </div> 1698 <div id="article_text"> 1699 <p class="first last">I walk through setting up TensorFlow, a Deep Learning Framework, on Ubuntu 16 with an Nvidia GTX 1080 and use it to build "Deep Fizz buzz".</p> 1700 1701 </div> 1702 </article> 1703 <hr /> 1704 <article> 1705 <div id="article_title"> 1706 <h3><a href="./airflow-postgres-redis-forex.html">Building a Data Pipeline with Airflow</a></h3> 1707 </div> 1708 <div id="article_text"> 1709 <p class="first last">I walk through setting up a data pipeline for currency exchange rates using Airflow, PostgreSQL and Redis.</p> 1710 1711 </div> 1712 </article> 1713 <hr /> 1714 <article> 1715 <div id="article_title"> 1716 <h3><a href="./billion-nyc-taxi-rides-aws-ec2-mapd.html">1.1 Billion Taxi Rides with MapD & AWS EC2</a></h3> 1717 </div> 1718 <div id="article_text"> 1719 <p class="first last">I investigate how fast MapD can query 1.1 billion taxi journeys using 4 g2.8xlarge EC2 instances.</p> 1720 1721 </div> 1722 </article> 1723 <hr /> 1724 <article> 1725 <div id="article_title"> 1726 <h3><a href="./billion-nyc-taxi-rides-nvidia-titan-x-mapd.html">1.1 Billion Taxi Rides with MapD & 4 Nvidia Titan Xs</a></h3> 1727 </div> 1728 <div id="article_text"> 1729 <p class="first last">I investigate how fast MapD can query 1.1 billion taxi journeys using 4 Nvidia Titan X cards.</p> 1730 1731 </div> 1732 </article> 1733 <hr /> 1734 <article> 1735 <div id="article_title"> 1736 <h3><a href="./billion-nyc-taxi-rides-nvidia-tesla-mapd.html">1.1 Billion Taxi Rides with MapD & 8 Nvidia Tesla K80s</a></h3> 1737 </div> 1738 <div id="article_text"> 1739 <p class="first last">I investigate how fast MapD can query 1.1 billion taxi journeys using 8 Nvidia Telsa K80 GPU cards.</p> 1740 1741 </div> 1742 </article> 1743 <hr /> 1744 <article> 1745 <div id="article_title"> 1746 <h3><a href="./billion-nyc-taxi-rides-rds-postgres.html">1.2 Billion Taxi Rides on AWS RDS running PostgreSQL</a></h3> 1747 </div> 1748 <div id="article_text"> 1749 <p class="first last">I investigate how fast a series of graph generated using R can be created across 4 different types of AWS RDS instances.</p> 1750 1751 </div> 1752 </article> 1753 <hr /> 1754 <article> 1755 <div id="article_title"> 1756 <h3><a href="./billion-nyc-taxi-rides-redshift-large-cluster.html">1.1 Billion Taxi Rides on a Large Redshift Cluster</a></h3> 1757 </div> 1758 <div id="article_text"> 1759 <p class="first last">I investigate how fast a 6-node ds2.8xlarge Redshift Cluster can query over a billion records.<
1759/p> 1760 1761 </div> 1762 </article> 1763 <hr /> 1764 <article> 1765 <div id="article_title"> 1766 <h3><a href="./all-billion-nyc-taxi-rides-redshift.html">All 1.1 Billion Taxi Rides on Redshift</a></h3> 1767 </div> 1768 <div id="article_text"> 1769 <p class="first last">I investigate how fast a single Redshift ds2.xlarge instance can query over a billion records.</p> 1770 1771 </div> 1772 </article> 1773 <hr /> 1774 <article> 1775 <div id="article_title"> 1776 <h3><a href="./all-billion-nyc-taxi-rides-elasticsearch.html">All 1.1 Billion Taxi Rides in Elasticsearch</a></h3> 1777 </div> 1778 <div id="article_text"> 1779 <p class="first last">I look at ways of fitting every column of the 1.1 billion taxi rides into Elasticsearch on a single, 850 GB SSD.</p> 1780 1781 </div> 1782 </article> 1783 <hr /> 1784 <article> 1785 <div id="article_title"> 1786 <h3><a href="./50-node-presto-cluster-dataproc.html">50-node Presto Cluster on Google Cloud's Dataproc</a></h3> 1787 </div> 1788 <div id="article_text"> 1789 <p class="first last">I investigate how fast a 50-node Dataproc cluster queries the metadata of 1.1 billion taxi trips.</p> 1790 1791 </div> 1792 </article> 1793 <hr /> 1794 <article> 1795 <div id="article_title"> 1796 <h3><a href="./performance-impact-file-size-presto.html">Performance Impact of File Sizes on Presto Query Times</a></h3> 1797 </div> 1798 <div id="article_text"> 1799 <p class="first last">I investigate the performance impact of ORC file sizes on Presto query times using Google Cloud's Dataproc service.</p> 1800 1801 </div> 1802 </article> 1803 <hr /> 1804 <article> 1805 <div id="article_title"> 1806 <h3><a href="./faster-ipv4-whois-crawling.html">Faster IPv4 WHOIS Crawling</a></h3> 1807 </div> 1808 <div id="article_text"> 1809 <p class="first last">I examine the performance and reliably increases from using Redis across a 51-node IPv4 WHOIS crawling cluster.</p> 1810 1811 </div> 1812 </article> 1813 <hr /> 1814 <article> 1815 <div id="article_title"> 1816 <h3><a href="./faster-queries-google-cloud-dataproc.html">33x Faster Queries on Google Cloud's Dataproc</a></h3> 1817 </div> 1818 <div id="article_text"> 1819 <p class="first last">I look at speeding up Presto queries on 1.1 billion records run on a 10-node Dataproc cluster.</p> 1820 1821 </div> 1822 </article> 1823 <hr /> 1824 <article> 1825 <div id="article_title"> 1826 <h3><a href="./mass-ip-whois-django-kafka.html">Mass IP Address WHOIS Collection with Django & Kafka</a></h3> 1827 </div> 1828 <div id="article_text"> 1829 <p class="first last">I investigate how fast a cluster of EC2 instances can collect WHOIS records of IPv4 addresses.</p> 1830 1831 </div> 1832 </article> 1833 <hr /> 1834 <article> 1835 <div id="article_title"> 1836 <h3><a href="./billion-nyc-taxi-rides-s3-vs-hdfs.html">A Billion Taxi Rides: AWS S3 versus HDFS</a></h3> 1837 </div> 1838 <div id="article_text"> 1839 <p class="first last">I investigate the speed differences between S3 and HDFS when querying over a billion records using Presto on AWS EMR.</p> 1840 1841 </div> 1842 </article> 1843 <hr /> 1844 <article> 1845 <div id="article_title"> 1846 <h3><a href="./billion-nyc-taxi-rides-presto-dataproc.html">A Billion Taxi Rides on Google's Dataproc running Presto</a></h3> 1847 </div> 1848 <div id="article_text"> 1849 <p class="first last">I investigate how fast a small Dataproc cluster can query over a billion records using Presto.</p> 1850 1851 </div> 1852 </article> 1853 <hr /> 1854 <article> 1855 <div id="article_title"> 1856 <h3><a href="./50-node-emr-cluster-presto.html">50-node Presto Cluster on Amazon EMR</a></h3> 1857 </div> 1858 <div id="article_text"> 1859 <p class="first last">I investigate how fast a 50-node AWS EMR cluster can query over a billion records using Presto.</p> 1860 1861 </div> 1862 </article> 1863 <hr /> 1864 <article> 1865 <div id="article_title"> 1866 <h3><a href="./billion-nyc-taxi-rides-bigquery.html">A Billion Taxi Rides on Google's BigQuery</a></h3> 1867 </div> 1868 <div id="article_text"> 1869 <p class="first last">I investigate how fast BigQuery can query the metadata of 1.1 billion NYC taxi journeys.</p> 1870 1871 </div> 1872 </article> 1873 <hr /> 1874 <article> 1875 <div id="article_title"> 1876 <h3><a href="./bulk-ip-address-whois-python-hadoop.html">Bulk IP Address WHOIS Collection with Python and Hadoop</a></h3> 1877 </div> 1878 <div id="article_text"> 1879 <p class="first last">I investigate how fast a 40-node Hadoop cluster on AWS EMR can collect WHOIS records of IPv4 addresses.</p> 1880 1881 </div> 1882 </article> 1883 <hr /> 1884 <article> 1885 <div id="article_title"> 1886 <h3><a href="./billion-nyc-taxi-rides-postgresql.html">A Billion Taxi Rides in PostgreSQL</a></h3> 1887 </div> 1888 <div id="article_text"> 1889 <p class="first last">I look at query speeds on 1.1 billion records on a single PostgreSQL installation running on an SSD.</p> 1890 1891 </div> 1892 </article> 1893 <hr /> 1894 <article> 1895 <div id="article_title"> 1896 <h3><a href="./billion-nyc-taxi-rides-elasticsearch.html">
1896A Billion Taxi Rides in Elasticsearch</a></h3> 1897 </div> 1898 <div id="article_text"> 1899 <p class="first last">I investigate how fast a single instance of Elasticsearch can query over a billion records.</p> 1900 1901 </div> 1902 </article> 1903 <hr /> 1904 <article> 1905 <div id="article_title"> 1906 <h3><a href="./billion-nyc-taxi-rides-spark-emr.html">A Billion Taxi Rides on Amazon EMR running Spark</a></h3> 1907 </div> 1908 <div id="article_text"> 1909 <p class="first last">I investigate how fast a small AWS EMR cluster can query over a billion records using Spark.</p> 1910 1911 </div> 1912 </article> 1913 <hr /> 1914 <article> 1915 <div id="article_title"> 1916 <h3><a href="./billion-nyc-taxi-rides-presto-emr.html">A Billion Taxi Rides on Amazon EMR running Presto</a></h3> 1917 </div> 1918 <div id="article_text"> 1919 <p class="first last">I investigate how fast a small AWS EMR cluster can query over a billion records using Presto.</p> 1920 1921 </div> 1922 </article> 1923 <hr /> 1924 <article> 1925 <div id="article_title"> 1926 <h3><a href="./kafka-topic-latency.html">Kafka Producer Latency with Large Topic Counts</a></h3> 1927 </div> 1928 <div id="article_text"> 1929 <p class="first last">I look at the relationship between topic counts and producer latency with Kafka.</p> 1930 1931 </div> 1932 </article> 1933 <hr /> 1934 <article> 1935 <div id="article_title"> 1936 <h3><a href="./billion-nyc-taxi-rides-hive-presto.html">A Billion Taxi Rides in Hive & Presto</a></h3> 1937 </div> 1938 <div id="article_text"> 1939 <p class="first last">Import the metadata of over a billion Yellow and Green Taxi and Uber rides in New York City into ORC-formatted, columnar-based files on HDFS and query them using Hive & Presto.</p> 1940 1941 </div> 1942 </article> 1943 <hr /> 1944 <article> 1945 <div id="article_title"> 1946 <h3><a href="./billion-nyc-taxi-rides-redshift.html">A Billion Taxi Rides in Redshift</a></h3> 1947 </div> 1948 <div id="article_text"> 1949 <p class="first last">Import the metadata of over a billion Yellow and Green Taxi and Uber rides in New York City into a columnar-based Data Warehouse.</p> 1950 1951 </div> 1952 </article> 1953 <hr /> 1954 <article> 1955 <div id="article_title"> 1956 <h3><a href="./presto-parquet-airpal.html">Presto, Parquet & Airpal</a></h3> 1957 </div> 1958 <div id="article_text"> 1959 <p class="first last">Using Airpal to execute queries on Parquet-fomatted data via Presto.</p> 1960 1961 </div> 1962 </article> 1963 <hr /> 1964 <article> 1965 <div id="article_title"> 1966 <h3><a href="./importing-data-from-s3-into-redshift.html">A Million Songs on AWS Redshift</a></h3> 1967 </div> 1968 <div id="article_text"> 1969 <p class="first last">Parallel imports of CSV data from AWS S3 into Redshift.</p> 1970 1971 </div> 1972 </article> 1973 <hr /> 1974 <article> 1975 <div id="article_title"> 1976 <h3><a href="./hadoop-up-and-running.html">Hadoop Up and Running</a></h3> 1977 </div> 1978 <div id="article_text"> 1979 <p class="first last">I explore three ways to get Hadoop installed and running.</p> 1980 1981 </div> 1982 </article> 1983 <hr /> 1984 <article> 1985 <div id="article_title"> 1986 <h3><a href="./test-django-on-ram-drive.html">Faster Testing with RAM Drives</a></h3> 1987 </div> 1988 <div id="article_text"> 1989 <p class="first last">Reduce the I/O overhead of running tests in Django.</p> 1990 1991 </div> 1992 </article> 1993 <hr /> 1994 <article> 1995 <div id="article_title"> 1996 <h3><a href="./popular-airline-passenger-routes.html">Popular Airline Passenger Routes</a></h3> 1997 </div> 1998 <div id="article_text"> 1999 <p class="first last">
1999Scraping 29K Wikipedia pages to find the most popular commercial airline passenger routes.</p> 2000 2001 </div> 2002 </article> 2003 <hr /> 2004 <article> 2005 <div id="article_title"> 2006 <h3><a href="./recommendation-engine-spark-python.html">Recommendation Engine built using Spark and Python</a></h3> 2007 </div> 2008 <div id="article_text"> 2009 <p class="first last">An end-to-end guide to building a film recommendation engine.</p> 2010 2011 </div> 2012 </article> 2013 <hr /> 2014 <article> 2015 <div id="article_title"> 2016 <h3><a href="./django-admin-logins.html">Tightening Django Admin Logins</a></h3> 2017 </div> 2018 <div id="article_text"> 2019 <p class="first last">A strategy for blocking dictionary attacks and restricting access to a white list of IP addresses.</p> 2020 2021 </div> 2022 </article> 2023 <hr /> 2024 <article> 2025 <div id="article_title"> 2026 <h3><a href="./uk-postcodes.html">Linting UK Postcodes</a></h3> 2027 </div> 2028 <div id="article_text"> 2029 <p class="first last">
2029Parsing and linting UK postcodes is ripe with edge cases.</p> 2030 2031 </div> 2032 </article> 2033 <hr /> 2034 <article> 2035 <div id="article_title"> 2036 <h3><a href="./passwords-in-django.html">Passwords in Django</a></h3> 2037 </div> 2038 <div id="article_text"> 2039 <p class="first last">A review of Django auth's password storage format and password storage upgrading capabilities.</p> 2040 2041 </div> 2042 </article> 2043 <hr /> 2044 <article> 2045 <div id="article_title"> 2046 <h3><a href="./faster-python.html">Faster Python</a></h3> 2047 </div> 2048 <div id="article_text"> 2049 <p class="first last">Six tips for speeding up Python code.</p> 2050 2051 </div> 2052 </article> 2053 <hr /> 2054 <article> 2055 <div id="article_title"> 2056 <h3><a href="./crushing-caching-cdn-django.html">Crushing, caching and CDN deployment in Django</a></h3> 2057 </div> 2058 <div id="article_text"> 2059 <p class="first last">A strategy for crushing, caching and deploying front-end-optimised Django sites.</p> 2060 2061 </div> 2062 </article> 2063 <hr /> 2064 <article> 2065 <div id="article_title"> 2066 <h3><a href="./better-python-package-management.html">Better Python Package Management</a></h3> 2067 </div> 2068 <div id="article_text"> 2069 <p class="first last">Python's most popular package management tool is pip. I explore some tools to increase its functionality.</p> 2070 2071 </div> 2072 </article> 2073 <hr /> 2074 <article> 2075 <div id="article_title"> 2076 <h3><a href="./load-balancing-django.html">Load balancing Django</a></h3> 2077 </div> 2078 <div id="article_text"> 2079 <p class="first last">Setup a load-balanced, two-node Django cluster with a minimal Ansible footprint.</p> 2080 2081 </div> 2082 </article> 2083 <hr /> 2084 <article> 2085 <div id="article_title"> 2086 <h3><a href="./faster-django-testing.html">Faster Django Testing</a></h3> 2087 </div> 2088 <div id="article_text"> 2089 <p class="first last">Run Django tests concurrently with pytest-xdist.</p> 2090 2091 </div> 2092 </article> 2093 <hr /> 2094 <article> 2095 <div id="article_title"> 2096 <h3><a href="./django-exception-archaeology.html">Django exception archaeology</a></h3> 2097 </div> 2098 <div id="article_text"> 2099 <p class="first last">How to capture, monitor and analyse exceptions raised from a Django project.</p> 2100 2101 </div> 2102 </article> 2103 <hr /> 2104 <article> 2105 <div id="article_title"> 2106 <h3><a href="./website-cdn-with-pelican-and-s3cmd.html">Python's killer apps for blogging: Pelican and S3cmd</a></h3> 2107 </div> 2108 <div id="article_text"> 2109 <p class="first last">I look into the steps of creating a blog using Pelican and hosting it with low-cost CDN services from Amazon with the help of S3cmd.</p> 2110 2111 </div> 2112 </article> 2113 <hr /> 2114 <article> 2115 <div id="article_title"> 2116 <h3><a href="./all-ipv4-whois-records.html">Collecting all IPv4 WHOIS records in Python</a></h3> 2117 </div> 2118 <div id="article_text"> 2119 <p class="first last">An exploratory effort to see how hard it is to collect all IPv4's WHOIS records.</p> 2120 2121 </div> 2122 </article> 2123 <hr /> 2124 <article> 2125 <div id="article_title"> 2126 <h3><a href="./former-php-developer.html">Former PHP developer</a></h3> 2127 </div> 2128 <div id="article_text"> 2129 <p class="first last">I stopped coding in PHP in 2011, here are the thoughts that led me to that decision.</p> 2130 2131 </div> 2132 </article> 2133 <hr /> 2134 <article> 2135 <div id="article_title"> 2136 <h3><a href="./file-uploads-amazon-s3-django.html">File uploads to Amazon S3 in Django</a></h3> 2137 </div> 2138 <div id="article_text"> 2139 <p class="first last">How to upload files to Amazon S3 from a form in Django as well as (very important) how to test the upload process.</p> 2140 2141 </div> 2142 </article> 2143 <hr /> 2144 <article> 2145 <div id="article_title"> 2146 <h3><a href="./ip-address-lookups-in-python.html">IP Address lookups using Python</a></h3> 2147 </div> 2148 <div id="article_text"> 2149 <p class="first last">A comparison of four methods used to find the country of an IP address.</p> 2150 2151 </div> 2152 </article> 2153 <hr /> 2154 <article> 2155 <div id="article_title"> 2156 <h3><a href="./django-speaking-json.html">Django speaking JSON</a></h3> 2157 </div> 2158 <div id="article_text"> 2159 <p class="first last">django-jsonview offers a method decorator which will cause all responses (including exceptions) to return in API-friend, JSON format.</p> 2160 2161 </div> 2162 </article> 2163 <hr /> 2164 <article> 2165 <div id="article_title"> 2166 <h3><a href="./query-elasticsearch-from-google-app-engine-gae.html">Querying Elasticsearch from Google App Engine</a></h3> 2167 </div> 2168 <div id="article_text"> 2169 <p class="first last">GAE strips HTTP body payloads if sent via HTTP GET. Elasticsearch excepts post bodies sent via HTTP GET. Re-writing the HTTP verb fixes the communications problem.</p> 2170 2171 </div> 2172 </article> 2173 2174 <footer> 2175 </footer> 2176 2177 <div id="ending_message"> 2178 <p>
2178Copyright © 2014 - 2026 Mark Litwintschik. This site's template is based off a <a href="https://github.com/giulivo/pelican-svbhack" target="_blank">template</a> by Giulio Fidente.</p> 2179 </div> 2180 </main> 2181</body> 2182</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.