Wikimedia Maps/Tile generation report
|
Full planet tile regeneration
Tile generation report for the work on the OS upgrade.
|
Introduction
[edit]The map infrastructure store vector-tiles in a Cassandra DB and can store data from zoom levels 0 to 15 where each zoom level has 2zoom tiles. Tilerator generates 22 tiles per second in average for each CPU available. Generating the whole planet can take more than a month for 24 CPU's.
| Zoom | Total tiles | Estimated Time to finish | |
|---|---|---|---|
| Hours | Days | ||
| 0 | 1 | 0.00 | 0.00 |
| 1 | 4 | 0.00 | 0.00 |
| 2 | 16 | 0.00 | 0.00 |
| 3 | 64 | 0.00 | 0.00 |
| 4 | 256 | 0.00 | 0.00 |
| 5 | 1024 | 0.00 | 0.00 |
| 6 | 4096 | 0.00 | 0.00 |
| 7 | 16384 | 0.01 | 0.00 |
| 8 | 65536 | 0.03 | 0.00 |
| 9 | 262144 | 0.14 | 0.01 |
| 10 | 1048576 | 0.55 | 0.02 |
| 11 | 4194304 | 2.21 | 0.09 |
| 12 | 16777216 | 8.83 | 0.37 |
| 13 | 67108864 | 35.31 | 1.47 |
| 14 | 268435456 | 141.22 | 5.88 |
| 15 | 1073741824 | 564.89 | 23.54 |
| Total | 1431655765 | 753.19 | 31.38 |
Debian upgrade to stretch
[edit]Because of the Debian upgrade to stretch, OSM data needed to be reloaded and all tiles needed to be generated from scratch. The tile generation started at 10/22/2018 and it's an ongoing process until now (last update was 12/19/2018). After some delay on tile generation we started keeping manual log through time regarding the tile generation health using a node tool that collect metrics generated by Tilerator, which can be seem in the following table:
| processed | tilegenok | tilenodata | processedRate | missing | okTilesRate | totalTiles | timestamp |
|---|---|---|---|---|---|---|---|
| 329605850 | 18235862 | 311369988 | 23.02% | 1102049915 | 5.53% | 1431655765 | 1543273715182 |
| 329631323 | 18236329 | 311394994 | 23.02% | 1102024442 | 5.53% | 1431655765 | 1543273762771 |
| 329656677 | 18236516 | 311420161 | 23.03% | 1101999088 | 5.53% | 1431655765 | 1543273810907 |
| 329678926 | 18236752 | 311442174 | 23.03% | 1101976839 | 5.53% | 1431655765 | 1543273851202 |
| 329839665 | 18238151 | 311601514 | 23.04% | 1101816100 | 5.53% | 1431655765 | 1543274169998 |
| 330184897 | 18244086 | 311940811 | 23.06% | 1101470868 | 5.53% | 1431655765 | 1543274814486 |
| 330242080 | 18244458 | 311997622 | 23.07% | 1101413685 | 5.52% | 1431655765 | 1543274924304 |
| 357986666 | 18878807 | 339107859 | 25.01% | 1073669099 | 5.27% | 1431655765 | 1543327057836 |
| 358478689 | 18890753 | 339587936 | 25.04% | 1073177076 | 5.27% | 1431655765 | 1543327999390 |
| 358481072 | 18890909 | 339590163 | 25.04% | 1073174693 | 5.27% | 1431655765 | 1543328003357 |
| 359466359 | 18915835 | 340550524 | 25.11% | 1072189406 | 5.26% | 1431655765 | 1543329887495 |
| 359468695 | 18915835 | 340552860 | 25.11% | 1072187070 | 5.26% | 1431655765 | 1543329890136 |
| 359501841 | 18916801 | 340585040 | 25.11% | 1072153924 | 5.26% | 1431655765 | 1543329957355 |
| 360883733 | 18960382 | 341923351 | 25.21% | 1070772032 | 5.25% | 1431655765 | 1543332566057 |
| 361105061 | 18968522 | 342136539 | 25.22% | 1070550704 | 5.25% | 1431655765 | 1543332988692 |
| 369736755 | 19204832 | 350531923 | 25.83% | 1061919010 | 5.19% | 1431655765 | 1543349355705 |
| 369754943 | 19205421 | 350549522 | 25.83% | 1061900822 | 5.19% | 1431655765 | 1543349393845 |
| 371169554 | 19214191 | 351955363 | 25.93% | 1060486211 | 5.18% | 1431655765 | 1543352016372 |
| 404768668 | 19856922 | 384911746 | 28.27% | 1026887097 | 4.91% | 1431655765 | 1543414812441 |
| 445075588 | 21248972 | 423826616 | 31.09% | 986580177 | 4.77% | 1431655765 | 1543490292663 |
| 448271613 | 21337248 | 426934365 | 31.31% | 983384152 | 4.76% | 1431655765 | 1543496265671 |
| 448305075 | 21338033 | 426967042 | 31.31% | 983350690 | 4.76% | 1431655765 | 1543496330227 |
| 448361068 | 21339359 | 427021709 | 31.32% | 983294697 | 4.76% | 1431655765 | 1543496440993 |
| 448376008 | 21339996 | 427036012 | 31.32% | 983279757 | 4.76% | 1431655765 | 1543496464526 |
| 448376958 | 21339996 | 427036962 | 31.32% | 983278807 | 4.76% | 1431655765 | 1543496466399 |
| 448444000 | 21342452 | 427101548 | 31.32% | 983211765 | 4.76% | 1431655765 | 1543496592815 |
| 448454022 | 21342935 | 427111087 | 31.32% | 983201743 | 4.76% | 1431655765 | 1543496611603 |
| 448455063 | 21342935 | 427112128 | 31.32% | 983200702 | 4.76% | 1431655765 | 1543496613191 |
| 448456352 | 21342935 | 427113417 | 31.32% | 983199413 | 4.76% | 1431655765 | 1543496616785 |
| 448463340 | 21343345 | 427119995 | 31.32% | 983192425 | 4.76% | 1431655765 | 1543496632324 |
| 448488739 | 21344399 | 427144340 | 31.33% | 983167026 | 4.76% | 1431655765 | 1543496680542 |
| 448493339 | 21344827 | 427148512 | 31.33% | 983162426 | 4.76% | 1431655765 | 1543496686367 |
| 454149805 | 21521204 | 432628601 | 31.72% | 977505960 | 4.74% | 1431655765 | 1543507258034 |
| 454419148 | 21531201 | 432887947 | 31.74% | 977236617 | 4.74% | 1431655765 | 1543507751739 |
| 459882332 | 21665626 | 438216706 | 32.12% | 971773433 | 4.71% | 1431655765 | 1543517792995 |
| 495010058 | 22871478 | 472138580 | 34.58% | 936645707 | 4.62% | 1431655765 | 1543582997617 |
| 495015752 | 22871535 | 472144217 | 34.58% | 936640013 | 4.62% | 1431655765 | 1543583005146 |
| 500146062 | 23023677 | 477122385 | 34.93% | 931509703 | 4.60% | 1431655765 | 1543592543127 |
| 501753227 | 23090414 | 478662813 | 35.05% | 929902538 | 4.60% | 1431655765 | 1543595574348 |
| 643485413 | 27850894 | 615634519 | 44.95% | 788170352 | 4.33% | 1431655765 | 1543858114418 |
| 643557080 | 27851243 | 615705837 | 44.95% | 788098685 | 4.33% | 1431655765 | 1543858251078 |
| 643575347 | 27851292 | 615724055 | 44.95% | 788080418 | 4.33% | 1431655765 | 1543858283434 |
| 643588132 | 27851365 | 615736767 | 44.95% | 788067633 | 4.33% | 1431655765 | 1543858306499 |
| 647378317 | 27923230 | 619455087 | 45.22% | 784277448 | 4.31% | 1431655765 | 1543865399600 |
| 647390779 | 27923944 | 619466835 | 45.22% | 784264986 | 4.31% | 1431655765 | 1543865417667 |
| 554720434 | 23601894 | 531118540 | 38.75% | 876935331 | 4.25% | 1431655765 | 1543935910238 |
| 554724936 | 23601894 | 531123042 | 38.75% | 876930829 | 4.25% | 1431655765 | 1543935916982 |
| 571188609 | 24069424 | 547119185 | 39.90% | 860467156 | 4.21% | 1431655765 | 1544026116760 |
| 586166907 | 24103862 | 562063045 | 40.94% | 845488858 | 4.11% | 1431655765 | 1544053956011 |
| 568689760 | 22641417 | 546048343 | 39.72% | 862966005 | 3.98% | 1431655765 | 1544156936945 |
| 589400156 | 22071213 | 567328943 | 41.17% | 842255609 | 3.74% | 1431655765 | 1544221066989 |
| 589418129 | 22071231 | 567346898 | 41.17% | 842237636 | 3.74% | 1431655765 | 1544221094608 |
| 637406144 | 21566271 | 615839873 | 44.52% | 794249621 | 3.38% | 1431655765 | 1544394201838 |
| 637408830 | 21566273 | 615842557 | 44.52% | 794246935 | 3.38% | 1431655765 | 1544394205758 |
| 637411640 | 21566273 | 615845367 | 44.52% | 794244125 | 3.38% | 1431655765 | 1544394209308 |
| 680100749 | 22261714 | 657839035 | 47.50% | 751555016 | 3.27% | 1431655765 | 1544472254313 |
| 717843703 | 22810362 | 695033341 | 50.14% | 713812062 | 3.18% | 1431655765 | 1544541762797 |
| 718145412 | 22816663 | 695328749 | 50.16% | 713510353 | 3.18% | 1431655765 | 1544542312249 |
| 756847246 | 23204744 | 733642502 | 52.87% | 674808519 | 3.07% | 1431655765 | 1544641669818 |
| 773737425 | 23618129 | 750119296 | 54.04% | 657918340 | 3.05% | 1431655765 | 1544710360518 |
| 819218612 | 23969150 | 795249462 | 57.22% | 612437153 | 2.93% | 1431655765 | 1544794097109 |
| 820425720 | 23984577 | 796441143 | 57.31% | 611230045 | 2.92% | 1431655765 | 1544796310087 |
| 826459063 | 24036728 | 802422335 | 57.73% | 605196702 | 2.91% | 1431655765 | 1544807325635 |
| 863417905 | 24334279 | 839083626 | 60.31% | 568237860 | 2.82% | 1431655765 | 1544875191371 |
| 962442186 | 26073481 | 936368705 | 67.23% | 469213579 | 2.71% | 1431655765 | 1545057598335 |
| 1056702638 | 27869655 | 1028832983 | 73.81% | 374953127 | 2.64% | 1431655765 | 1545228533232 |
| 1057790348 | 27879600 | 1029910748 | 73.89% | 373865417 | 2.64% | 1431655765 | 1545230495655 |
| 1078537988 | 28093576 | 1050444412 | 75.34% | 353117777 | 2.60% | 1431655765 | 1545268190817 |
| 1078577797 | 28094882 | 1050482915 | 75.34% | 353077968 | 2.60% | 1431655765 | 1545268269267 |
| 1078605334 | 28095272 | 1050510062 | 75.34% | 353050431 | 2.60% | 1431655765 | 1545268313809 |
| 1078621315 | 28095300 | 1050526015 | 75.34% | 353034450 | 2.60% | 1431655765 | 1545268340157 |
| 1078662686 | 28095300 | 1050567386 | 75.34% | 352993079 | 2.60% | 1431655765 | 1545268411926 |
| 1078675899 | 28095300 | 1050580599 | 75.34% | 352979866 | 2.60% | 1431655765 | 1545268439748 |
| 1078681693 | 28095300 | 1050586393 | 75.35% | 352974072 | 2.60% | 1431655765 | 1545268451792 |
| 1114218777 | 28670702 | 1085548075 | 77.83% | 317436988 | 2.57% | 1431655765 | 1545332461909 |
| 1117927315 | 28776224 | 1089151091 | 78.09% | 313728450 | 2.57% | 1431655765 | 1545339308889 |
| 1146342225 | 29491813 | 1116850412 | 80.07% | 285313540 | 2.57% | 1431655765 | 1545391152361 |
| 1150046974 | 29624996 | 1120421978 | 80.33% | 281608791 | 2.58% | 1431655765 | 1545397953049 |
| 1150162021 | 29625299 | 1120536722 | 80.34% | 281493744 | 2.58% | 1431655765 | 1545398162508 |
| 1150170358 | 29625299 | 1120545059 | 80.34% | 281485407 | 2.58% | 1431655765 | 1545398177374 |
| 1212590800 | 30434496 | 1182156304 | 84.70% | 219064965 | 2.51% | 1431655765 | 1545512936226 |
| 1212608207 | 30434600 | 1182173607 | 84.70% | 219047558 | 2.51% | 1431655765 | 1545512969903 |
| 1714812975 | 37884031 | 1676928944 | 119.78% | 0 | 2.21% | 1431655765 | 1546432643215 |
| 1715522034 | 37884184 | 1677637850 | 119.83% | 0 | 2.21% | 1431655765 | 1546433933383 |
| 1957220075 | 38965688 | 1918254387 | 136.71% | 0 | 1.99% | 1431655765 | 1546869993652 |
| 1374582898 | 34267596 | 1340315302 | 96.01% | 57072867 | 2.49% | 1431655765 | 1546961480404 |
| 1377981578 | 34267596 | 1343713982 | 96.25% | 53674187 | 2.49% | 1431655765 | 1546967794888 |
| 1416730878 | 34267675 | 1382463203 | 98.96% | 14924887 | 2.42% | 1431655765 | 1547040742682 |
| 1419676228 | 34267736 | 1385408492 | 99.16% | 11979537 | 2.41% | 1431655765 | 1547046408039 |
| 1340948139 | 37151623 | 1303796516 | 93.66% | 90707626 | 2.77% | 1431655765 | 1547133254764 |
| 1342254006 | 38076370 | 1304177636 | 93.76% | 89401759 | 2.84% | 1431655765 | 1547138839505 |
It is possible to see that Tilerator processed more tiles than expected and also, after some jobs failing with message "TTL exceeded" the number of tiles processed dropped and Tilerator rerun the jobs. It's not known if the work is committed and tiles were stored in Cassandra, even though maps1004 increased its size of generated tiles compared with maps1001 (old master). The remaining jobs were stopped at 2019-09-01 15:23 freeing CPU to enable the OSM sync replication script.
General Updates
[edit]Tile processing reached near 100% (01-08-2019)
[edit]After a long run of tile generation jobs, and some difficulties to reach an end, the team decided that would be reasonable to continue with the migration plan without waiting for the full tile processing. Analyzing the collected data we figured out that only a small part of the tiles are stored and we might have enough tiles stored.
Also, it was found a failure in the strategy to regenerate the full planet tiles. It was documented a different process that is faster if you have a set of tiles pre-generated before to use as a basis to generate only the specific indexes. The next time that full planet tile regeneration is needed, we should consider this approach which would generate tiles according to the following benchmark:
| Zoom | Total tiles | Estimated Time to finish | |
|---|---|---|---|
| Hours | Days | ||
| 0 | 0 | 0.00 | 0.00 |
| 1 | 4 | 0.00 | 0.00 |
| 2 | 16 | 0.00 | 0.00 |
| 3 | 57 | 0.00 | 0.00 |
| 4 | 218 | 0.00 | 0.00 |
| 5 | 846 | 0.00 | 0.00 |
| 6 | 3301 | 0.00 | 0.00 |
| 7 | 13024 | 0.01 | 0.00 |
| 8 | 51509 | 0.03 | 0.00 |
| 9 | 202145 | 0.11 | 0.00 |
| 10 | 159340 | 0.08 | 0.00 |
| 11 | 526282 | 0.28 | 0.01 |
| 12 | 1822626 | 0.96 | 0.04 |
| 13 | 6099772 | 3.21 | 0.13 |
| 14 | 19000727 | 10.00 | 0.42 |
| 15 | 61208604 | 32.20 | 1.34 |
| Total | 36.87 | 1.54 | |
Is important to notice that after the tile regeneration we had an increase of tiles on maps1004. For example, z15 increased the most with 61,208,604 tiles, that is 12,749,948 more tiles than our current production (~26% increase).
Useful Commands
[edit]The following Tilerator command was used to schedule and queue tile generation jobs:
sudo -u tileratorui node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js --config /etc/tileratorui/config.yaml \
-j.zoom 0 \
-j.fromZoom 0 \
-j.beforeZoom 16 \
-j.generatorId gen \
-j.storageId v4 \
-j.deleteEmpty true \
-j.keepJob true
Problems during tile generation
[edit]- Tile generation for zoom levels 14 and 15 failed. Tilerator tries to run jobs with too much tiles and got the error
TTL exceeded.- Solution: restart jobs using Tilerator UI
What we have learned
[edit]A few things that was learned during this process are worth to mention:
- Generating an entire zoom level is painful and slow and we should avoid at all costs for zoom levels greater than 13
- This work is an incentive for a refactor of the available documentation
- There isn't yet a reliable tool for Cassandra data copy, which would save us a lot of time too