1 2<!DOCTYPE html> 3<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]--> 4<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]--> 5<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]--> 6<head> 7 <meta charset="utf-8"> 8 <title>Paul’s Blog</title> 9 <meta name="author" content="Paul Norman"> 10 11 12 <meta name="description" content="I’ve been looking at how many tiles are changed when updating OSM data in order to better guide resource estimations, and have completed some …"> 13 14 15 <!-- http://t.co/dKP3o1e --> 16 <meta name="HandheldFriendly" content="True"> 17 <meta name="MobileOptimized" content="320"> 18 <meta name="viewport" content="width=device-width, initial-scale=1"> 19 20 21 <link rel="canonical" href="http://www.paulnorman.ca"> 22 <link href="/favicon.png" rel="icon"> 23 <link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css"> 24 <link href="/atom.xml" rel="alternate" title="Paul's Blog" type="application/atom+xml"> 25
25<script src="/javascripts/modernizr-2.0.js"></script>
25 26
26<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
26 27
27<script>!window.jQuery && document.write(unescape('%3Cscript src="./javascripts/libs/jquery.min.js"%3E%3C/script%3E'))</script>
27 28
28<script src="/javascripts/octopress.js" type="text/javascript"></script>
28 29 <!--Fonts from Google"s Web font directory at http://google.com/webfonts --> 30<link href="http://fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css"> 31<link href="http://fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css"> 32 33 34 35</head> 36 37<body > 38 <header role="banner"><hgroup> 39 <h1><a href="/">Paul’s Blog</a></h1> 40 41 <h2>A blog without a good name</h2> 42 43</hgroup> 44 45</header> 46 <nav role="navigation"><ul class="subscription" data-subscription="rss"> 47 <li><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li> 48 49</ul> 50 51<form action="http://google.com/search" method="get"> 52 <fieldset role="search"> 53 <input type="hidden" name="q" value="site:www.paulnorman.ca" /> 54 <input class="search" type="text" name="q" results="0" placeholder="Search"/> 55 </fieldset> 56</form> 57 58<ul class="main-navigation"> 59 <li><a href="/">Blog</a></li> 60 <li><a href="/blog/archives">Archives</a></li> 61</ul> 62 63</nav> 64 <div id="main"> 65 <div id="content"> 66 <div class="blog-index"> 67 68 69 70 <article> 71 72 <header> 73 74 <h1 class="entry-title"><a href="/blog/2024/01/minutely-updated-tiles/">Minutely Updated Tile Volume: Technical Details</a></h1> 75 76 77 <p class="meta"> 78 79 80 81 82 83 84 85 86 87 88 89 90<time datetime="2024-01-15T14:00:00-08:00" pubdate data-updated="true"></time> 91 92 </p> 93 94 </header> 95 96 97 <div class="entry-content"><p>I’ve been looking at how many tiles are changed when updating OSM data in order to better guide resource estimations, and have completed some benchmarks. This is the technical post with details, I’ll be doing a high-level post later.</p> 98 99<p>Software like Tilemaker and Planetiler is great for generating a complete set of tiles, updated about once a day, but they can’t handle minutely updates. Most users are fine with daily or slower updates, but OSM.org users are different, and minutely updates are critical for them. All the current minutely ways to generate map tiles involve loading the changes and regenerating tiles when data in them may have changed. I used osm2pgsql, the standard way to load OSM data for rendering, but the results should be applicable to other ways including different schemas.</p> 100 101</div> 102 103 104 <footer> 105 <a rel="full-article" href="/blog/2024/01/minutely-updated-tiles/">Read on →</a> 106 </footer> 107 108 109 110 </article> 111 112 113 <article> 114 115 <header> 116 117 <h1 class="entry-title"><a href="/blog/2023/05/tilelog-country-data/">Tilelog Country Data</a></h1> 118 119 120 <p class="meta"> 121 122 123 124 125 126 127 128 129 130 131 132 133<time datetime="2023-05-21T20:00:00-08:00" pubdate data-updated="true"></time> 134 135 </p> 136 137 </header> 138 139 140 <div class="entry-content"><p>I added functionality to <a href="https://github.com/openstreetmap/tilelog">tilelog</a> to generate per-country usage information for the OSMF Standard Map Layer. The output of this is a CSV file, generated every day, which contains country code, number of unique IPs that day, tiles per second, and tiles per second that were a cache miss, all for each country code.</p> 141 142<p>With a bit of work, I manipulated the files to give me the usage from the 10 countries with the most usage, for the first four months of 2023.</p> 143 144<p><img src="/blog/2023/05/tilelog-country-data/date.png" alt="Tile usage per country by date" /></p> 145 146<p>Perhaps more interesting is looking at the usage for each country by the day of week.</p> 147 148<p><img src="/blog/2023/05/tilelog-country-data/weekly.png" alt="Tile usage per country by date" /></p> 149</div> 150 151 152 153 154 </article> 155 156 157 <article> 158 159 <header> 160 161 <h1 class="entry-title"><a href="/blog/2022/10/tilekiln-storage/">Tilekiln Tile Storage</a></h1> 162 163 164 <p class="meta"> 165 166 167 168 169 170 171 172 173 174 175 176 177<time datetime="2022-10-09T20:00:00-08:00" pubdate data-updated="true"></time> 178 179 </p> 180 181 </header> 182 183 184 <div class="entry-content"><p>I’m rewriting <a href="https://github.com/pnorman/tilekiln">
184Tilekiln</a>, tile generation software which leverages PostGIS to allow using established toolchains like <a href="https://osm2pgsql.org/">osm2pgsql</a>.</p> 185 186<p>Tile storage is a difficult problem. For a tileset going to zoom 14, there are 358 million tiles, and for one going to zoom 15, there are 1.4 billion. Most tiles are smalled, with 80% being about 100 bytes typically, and the largest tiles might be about 1 megabyte.</p> 187 188<p>Tilekiln’s storage must be able to handle these numbers, but also handle incremental minutely updates, and maintenance work like deleting tilesets. A nice to have would be the ability to distribute tilesets easily, but this is not essential.</p> 189 190<h2>Options</h2> 191 192<h3><a href="https://github.com/protomaps/PMTiles">PMTiles</a></h3> 193 194<p>PMTiles is a file format designed to store an entire tileset in one file. It consists of a directory, which lists offsets for where tiles are within the larger file. Using range requests, any tile can be retrieved in 3 requests in the worst case, while any caching at all will bring this to 2 requests, and typical caching can bring it close to one.</p> 195 196<p>It features de-duplication, both for tiles that are bytewise-indentical, as well as for adjacent offset listings pointing at the same tile.</p> 197 198<p>There is client-side support for some map browser-based display libraries, but most applications will require a server returning conventional that handles conventional z/x/y URLs serving from the PMTiles file. As a fairly new format, support from other applications is limited.</p> 199 200<p>Updating the PMTiles archive in place is possible, because the clients use etags to identify when the archive has changed, invalidating the client-side cache. This means with minutely updates, every one minute, one request from each client will be the worst case, requiring 3 requests. In practice, this doesn’t matter, because for a large tileset, it is impossible to rewrite the entire archive that frequently, as it will take longer than that to write out the complete file.</p> 201 202<h4>Pros</h4> 203 204<ul> 205<li>Generally most space efficient single-file tileset archive format</li> 206<li>Easy to distribute</li> 207<li>Can directly serve to some clients</li> 208</ul> 209 210 211<h4>Cons</h4> 212 213<ul> 214<li>Impossible to minutely update</li> 215<li>Poor support for the archive format outside of specialized software and browser-based libraries</li> 216</ul> 217 218 219<h3><a href="https://github.com/mapbox/mbtiles-spec">MBTiles</a></h3> 220 221<p>Like PMTiles, MBTiles is a single-file archive format. It was developed by Mapbox for users to generate tiles and upload them to Mapbox’s servers. It’s format is a SQLite database with tables consisting of tile indexes and tile data data as binary blobs. Because it’s based on SQLite, and has been around for longer, support is wide-spread, with several generation. Browser-based support is limited, and it wasn’t designed with that in mind.</p> 222 223<p>Minutely updates are theoretically possible, but in practice, not a good idea. SQLite databases do not work well with high volumes of concurrent reads and writes, generally requiring all work to go through one process. This requires coupling the generation and serving systems.</p> 224 225<h4>Pros</h4> 226 227<ul> 228<li>Easy to distribute</li> 229<li>Good support for non-browser clients</li> 230</ul> 231 232 233<h4>Cons</h4> 234 235<ul> 236<li>Poor minutely support</li> 237<li>Not suitable for directly serving to browsers</li> 238</ul> 239 240 241<h3>PostgreSQL</h3> 242 243<p>Because Tilekiln already requires PostgreSQL, it would be possible to store tiles in it, the same way that MBTiles does.</p> 244 245<h4>Pros</h4> 246 247<ul> 248<li>Supports minutely updates</li> 249<li>Uses software already required</li> 250</ul> 251 252 253<h4>Cons</h4> 254 255<ul> 256<li>Custom format</li> 257<li>Impossible to distribute the archive</li> 258</ul> 259 260 261<h3>Tiles on disk</h3> 262 263<p>Instead of an archive format, it’s possible to store tiles on disk as files. This is the most well-established method, and simplest. Tiles can be updated atomically, and serving tiles is just serving files from disk. The downside comes to managing millions or billions of tiny files. File systems are not designed for this, and can have problems with</p> 264 265<ul> 266<li>minimum file sizes,</li> 267<li>inode usage,</li> 268<li>inodes per directory, and</li> 269<li>cleaning up tilesets.</li> 270</ul> 271 272 273<p>
273In particular, it can take a day or longer to delete a tileset.</p> 274 275<h4>Pros</h4> 276 277<ul> 278<li>Supports minutely updates</li> 279<li>Simple serving</li> 280</ul> 281 282 283<h4>Cons</h4> 284 285<ul> 286<li>Does not scale to planet-wide tilesets</li> 287<li>No archive to distribute</li> 288</ul> 289 290 291<h3>Object stores</h3> 292 293<p>A popular approach to store tiles in some form of object store, like S3. All commercial object stores I’ve looked perform badly with large numbers of small objects. While there are sometimes work-arounds for this, their pricing structure generally makes it very expensive to store tiles this way.</p> 294 295<h4>Pros</h4> 296 297<ul> 298<li>Easy to serve out of</li> 299<li>Supports minutely updates</li> 300</ul> 301 302 303<h4>Cons</h4> 304 305<ul> 306<li>Very expensive, or requires running your own object store</li> 307<li>Slow</li> 308</ul> 309 310 311<h3><a href="https://github.com/tapalcatl/tapalcatl-2-spec">Tapalcatl 2</a></h3> 312 313<p>Tapalcatl 2 is a system of using zip files to combine tiles, reducing the number of tiles that need to be stored. It is similar to how raster tiles are combined into metatiles, except that the vector tiles are pre-sliced within the zipfile and can contain multiple zooms.</p> 314 315<p>In a typical configuration, there are zip files generated for tiles on zooms 0, 4, 8, and 12. Each zip file contains the “root” tile and then tiles from the next three zooms that lie within it. This means that a zip archive contains 85 tiles, all tiles within a small area. By combining tiles into one zip archive, this reduces the number of files on disk to 16.8 million files, a small enough number to be reasonably managed on disk.</p> 316 317<p>The format hasn’t had a great deal of usage since it was developed, so support is limited to some server-side programs that take tapalcatl archives and present tiles to the user. These server-side programs are known to have some issues, like not supporting updates to remote tapalcatl tilesets.</p> 318 319<p>Updates are possible in two ways. The first is by taking an existing zip file, replacing the changed tiles within it, and generating a new zip file. The second is to completely regenerate all the tiles in the zip file, which is simpler, but involves more tile generation.</p> 320 321<h4>Pros</h4> 322 323<ul> 324<li>Supports minutely updates</li> 325<li>Allows good decoupling of serving and generation</li> 326</ul> 327 328 329<h4>Cons</h4> 330 331<ul> 332<li>Limited client support</li> 333<li>Minutely updates are more complicated</li> 334</ul> 335 336 337<h2>Recommendations</h2> 338 339<p>The two options which requires further investigation are PostgreSQL and Tapalcatl 2. Both support updates, but come with downsides.</p> 340</div> 341 342 343 344 345 </article> 346 347 348 <article> 349 350 <header> 351 352 <h1 class="entry-title"><a href="/blog/2021/07/standard-layer-2/">OpenStreetMap Standard Layer: Requests</a></h1> 353 354 355 <p class="meta"> 356 357 358 359 360 361 362 363 364 365 366 367 368<time datetime="2021-07-26T20:00:00-08:00" pubdate data-updated="true"></time> 369 370 </p> 371 372 </header> 373 374 375 <div class="entry-content"><p>This blog post is a version of my recent SOTM 2021 presentation on the OpenStreetMap Standard Layer and who’s using it.</p> 376 377<p>With the switch to a commercial CDN, weâve improved our logging significantly and now have the tools to log and analyze logs. We log information on both the incoming request and our response to it.</p> 378 379</div> 380 381 382 <footer> 383 <a rel="full-article" href="/blog/2021/07/standard-layer-2/">Read on →</a> 384 </footer> 385 386 387 388 </article> 389 390 391 <article> 392 393 <header> 394 395 <h1 class="entry-title"><a href="/blog/2021/07/standard-layer-1/">OpenStreetMap Standard Layer: Introduction</a></h1> 396 397 398 <p class="meta"> 399 400 401 402 403 404 405 406 407 408 409 410 411<time datetime="2021-07-10T20:00:00-08:00" pubdate data-updated="true"></time> 412 413 </p> 414 415 </header> 416 417 418 <div class="entry-content"><p>This blog post is a version of my recent SOTM 2021 presentation on the OpenStreetMap Standard Layer and who’s using it.</p> 419 420<p>The OpenStreetMap Standard Layer is the default layer on <a href="https://www.openstreetmap.org/#layers=M">
420openstreetmap.org</a>, using most of the front page. It’s run by the OpenStreetMap Foundation, and the Operations Working Group is responsible for the planning, organisation and budgeting of OSMF-run services like this one and servers running it. There are other map layers on the front page like <a href="https://www.openstreetmap.org/#layers=C">Cycle Map</a> and <a href="https://www.openstreetmap.org/#layers=T">Transport Map</a>, and I encourage you to try them, but they’re not hosted or planned by us.</p> 421 422</div> 423 424 425 <footer> 426 <a rel="full-article" href="/blog/2021/07/standard-layer-1/">Read on →</a> 427 </footer> 428 429 430 431 </article> 432 433 434 <article> 435 436 <header> 437 438 <h1 class="entry-title"><a href="/blog/2021/02/openstreetmap-visits/">OpenStreetMap Survey by Visits</a></h1> 439 440 441 <p class="meta"> 442 443 444 445 446 447 448 449 450 451 452 453 454<time datetime="2021-02-17T11:00:00-08:00" pubdate data-updated="true"></time> 455 456 </p> 457 458 </header> 459 460 461 <div class="entry-content"><p>In my <a href="/blog/2021/02/openstreetmap-survey/">last post</a> I looked at <a href="https://wiki.osmfoundation.org/wiki/2021_Survey_Results">survey responses</a> by country and their correlation with mappers eligible for a fee waver as an active contributor.</p> 462 463<p>I wanted to look at the correlation with OSM.org views. I already had a full day’s worth of logs on tile.openstreetmap.org accesses, so I filtered them for requests from www.openstreetmap.org and got a <a href="https://gist.github.com/pnorman/d6e7f5c82f5efd80a4d6a0c6f37cdb7f">per-country count</a>. This is from December 29th, 2020. Ideally it would be from a complete week, and not a holiday, but this is the data I had downloaded.</p> 464 465<p><img src="/blog/2021/02/openstreetmap-visits/tiles.png" alt="Preview image" /></p> 466 467<p>The big outlier is Italy. It has more visits than I would expect, so I wonder if the holiday had an influence. Like before, the US is overrepresented in the results, Russia and Poland are underrepresented, and Germany is about average.</p> 468 469<p>Like before, I made a graph of the smaller countries.</p> 470 471<p><img src="/blog/2021/02/openstreetmap-visits/tilessmall.png" alt="Preview image" /></p> 472 473<p>More small countries are above the average line - probably an influence of Italy being so low.</p> 474</div> 475 476 477 478 479 </article> 480 481 482 <article> 483 484 <header> 485 486 <h1 class="entry-title"><a href="/blog/2021/02/openstreetmap-survey/">OpenStreetMap Survey</a></h1> 487 488 489 <p class="meta"> 490 491 492 493 494 495 496 497 498 499 500 501 502<time datetime="2021-02-17T11:00:00-08:00" pubdate data-updated="true"></time> 503 504 </p> 505 506 </header> 507 508 509 <div class="entry-content"><p>The board has <a href="https://wiki.osmfoundation.org/wiki/2021_Survey_Results">started releasing</a> results from their 2021 survey. I’ve done some analysis on the response rates by country.</p> 510 511<p>There’s lots of data for activity on OSM by country, but for this I took the numbers from joost for how many “<a href="https://www.openstreetmap.org/user/joost%20schouppe/diary/395012">active contributors” there are according to the contributor fee waver criteria</a>.</p> 512 513<p><img src="/blog/2021/02/openstreetmap-survey/countries.png" alt="Preview image" /></p> 514 515<p>For the larger countries, Russia is the most underrepresented country. This is not surprising, as they are underrepresented in other venues like the OSMF membership.</p> 516 517<p>The US and UK are both slightly overrepresented in the survey, but less so than I would have expected based on other surveys and OSMF membership.</p> 518 519<p>The smaller countries are all crowded, so I did a graph of just them.</p> 520 521<p><img src="/blog/2021/02/openstreetmap-survey/countriessmall.png" alt="Preview image" /></p> 522 523<p>As with other surveys, Japan is underrepresented. Indonesia, although underrepresented is less underrepresented than I would have expected.</p> 524</div> 525 526 527 528 529 </article> 530 531 532 <article> 533 534 <header> 535 536 <h1 class="entry-title"><a href="/blog/2020/05/openstreetmap-cartographic/">OpenStreetMap Cartographic: A Client-side Rendered OpenStreetMap Carto</a></h1> 537 538 539 <p class="meta"> 540 541 542 543 544 545 546 547 548 549 550 551 552<time datetime="2020-05-24T18:00:00-08:00" pubdate data-updated="true"></time> 553 554 </p> 555 556 </header> 557 558 559 <div class="entry-content"><p>I’ve been working on a new project, OpenStreetMap Cartographic. This is a client-side rendering based on OpenStreetMap Carto. This is an ambitious project, as OpenStreetMap Carto is an extremely complex style which shows a large number of features. The technical choices I’m making are designed so the style is capable of handling the load of osm.org with minutely updates.</p> 560 561<p>I’ve put up a world-wide demo at <a href="https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html">https://pnorman.dev.openstreetmap.org/cartographic/mapbox-gl.html</a>
561, using data from 2020-03-16, and you can view the code at <a href="https://github.com/pnorman/openstreetmap-cartographic">https://github.com/pnorman/openstreetmap-cartographic</a>.</p> 562 563</div> 564 565 566 <footer> 567 <a rel="full-article" href="/blog/2020/05/openstreetmap-cartographic/">Read on →</a> 568 </footer> 569 570 571 572 </article> 573 574 575 <article> 576 577 <header> 578 579 <h1 class="entry-title"><a href="/blog/2019/01/creating-tarballs/">Creating Tarballs</a></h1> 580 581 582 <p class="meta"> 583 584 585 586 587 588 589 590 591 592 593 594 595<time datetime="2019-01-14T13:21:06-08:00" pubdate data-updated="true"></time> 596 597 </p> 598 599 </header> 600 601 602 <div class="entry-content"><p>With all the tiles <a href="/blog/2018/12/seeding/">generated</a> and <a href="/blog/2018/12/optimizing">optimized</a>, they just need to be packaged in a tarball. Before creating them, we want to create some files with metadata about what was used to generate the tiles. The commit of the stylesheet and the timestamp of the planet file can be extracted with a couple of commands.</p> 603 604</div> 605 606 607 <footer> 608 <a rel="full-article" href="/blog/2019/01/creating-tarballs/">Read on →</a> 609 </footer> 610 611 612 613 </article> 614 615 616 <article> 617 618 <header> 619 620 <h1 class="entry-title"><a href="/blog/2018/12/optimizing-pngs/">Optimizing PNGs</a></h1> 621 622 623 <p class="meta"> 624 625 626 627 628 629 630 631 632 633 634 635 636<time datetime="2018-12-27T23:40:23-08:00" pubdate data-updated="true"></time> 637 638 </p> 639 640 </header> 641 642 643 <div class="entry-content"><p>With the <a href="/blog/2018/12/seeding/">tiles generated</a> normally the next step would be to serve them, but because I’m planning to distribute them to others, I’m going to the unusual step of optimizing the PNGs. Optimizing PNGs can cut the file size in half, helping downstream users of the tiles I’m generating.</p> 644 645</div> 646 647 648 <footer> 649 <a rel="full-article" href="/blog/2018/12/optimizing-pngs/">Read on →</a> 650 </footer> 651 652 653 654 </article> 655 656 <div class="pagination"> 657 658 <a class="prev" href="2">← Older</a> 659 660 <a href="/blog/archives">Blog Archives</a> 661 662 </div> 663</div> 664<aside class="sidebar"> 665 666 <section> 667 <h1>Recent Posts</h1> 668 <ul id="recent_posts"> 669 670 <li class="post"> 671 <a href="/blog/2024/01/minutely-updated-tiles/">Minutely Updated Tile Volume: Technical Details</a> 672 </li> 673 674 <li class="post"> 675 <a href="/blog/2023/05/tilelog-country-data/">Tilelog Country Data</a> 676 </li> 677 678 <li class="post"> 679 <a href="/blog/2022/10/tilekiln-storage/">Tilekiln Tile Storage</a> 680 </li> 681 682 <li class="post"> 683 <a href="/blog/2021/07/standard-layer-2/">OpenStreetMap Standard Layer: Requests</a> 684 </li> 685 686 <li class="post"> 687 <a href="/blog/2021/07/standard-layer-1/">OpenStreetMap Standard Layer: Introduction</a> 688 </li> 689 690 </ul> 691</section> 692 693 694 695 696 697 698</aside> 699 700 </div> 701 </div> 702 <footer role="contentinfo"><p> 703 Copyright © 2025 - Paul Norman - 704 <span class="credit">Powered by <a href="http://octopress.org">Octopress</a></span> 705</p> 706 707</footer> 708 709 710 711 712 713 714 715 716
716<script type="text/javascript"> 717 (function(){ 718 var twitterWidgets = document.createElement('script'); 719 twitterWidgets.type = 'text/javascript'; 720 twitterWidgets.async = true; 721 twitterWidgets.src = '//platform.twitter.com/widgets.js'; 722 document.getElementsByTagName('head')[0].appendChild(twitterWidgets); 723 })(); 724 </script>
724 725 726 727 728 729 730</body> 731</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.