Parsing/Replacing Tidy/Linter/Stats/Pixel Diff Testing Stats
We are stopping these diffing runs as of May 29th.
Every week, we run a mass pixel-diff testing on about 72K pages from 60 different wikis from 4 different projects. We generate a rendering of the (latest revision of the) test page that uses Tidy and another rendering of the test page that uses Remex and then compare the two images via the uprightdiff package. If the two images are not completely identical, the diffing algorithm checks if the differences can be accounted for by purely vertical whitespace shifts.
The table below shows how the results changed since August 2017 when we first started these tests. At that time, our internal target for complete Tidy replacement was 95% pages render identically (after accounting for vertical whitespace shifts). Because of shortcomings of our tooling, it is not possible to get to 100% because of false positives and noisiness in tests.
As the table shows, as editors fix linter-identified issues on pages, this percentage has been slowly increasing.
| Date | % vertical
whitespace shifts only |
% pixel-perfect
rendering |
Notes |
|---|---|---|---|
| 2017-08-03 | 91.90 | 63.23 | |
| 2017-08-07 | 91.94 | 62.89 | |
| 2017-08-10 | 91.77 | 63.52 | |
| 2017-08-16 | 91.82 | 62.96 | |
| 2017-08-21 | 91.87 | 62.98 | |
| 2017-08-30 | 91.77 | 63.07 | |
| 2017-09-04 | 91.81 | 63.07 | |
| 2017-09-11 | 91.78 | 62.99 | |
| 2017-09-18 | 91.72 | 62.95 | |
| 2017-09-25 | 91.94 | 62.84 | |
| 2017-10-02 | 92.11 | 62.97 | |
| 2017-10-09 | 92.27 | 62.73 | |
| 2017-10-16 | 92.35 | 63.23 | |
| 2017-10-23 | 92.50 | 63.50 | |
| 2017-10-30 | 92.23 | 63.84 | |
| 2017-11-02 | 93.53 | 67.73 | This big bump is because of fixes to the visual diff testing setup.
Redirects weren't being following for both Tidy and Remex and was causing false diffs to be reported. |
| 2017-11-06 | 93.56 | 67.71 | |
| 2017-11-13 | 93.68 | 67.87 | |
| 2017-11-20 | 93.72 | 68.00 | |
| 2017-11-27 | 93.91 | 68.05 | |
| 2017-12-04 | 94.39 | 68.06 | |
| 2017-12-11 | 94.47 | 68.10 | |
| 2017-12-18 | 94.53 | 68.25 | |
| 2017-12-25 | 94.62 | 68.16 | |
| 2018-01-01 | 94.61 | 68.32 | |
| 2018-01-08 | 94.64 | 68.27 | |
| 2018-01-15 | 94.65 | 68.27 | |
| 2018-01-22 | 94.68 | 68.30 | |
| 2018-01-29 | 94.74 | 68.32 | |
| 2018-02-05 | 94.83 | 68.42 | |
| 2018-02-12 | 94.89 | 68.33 | |
| 2018-02-19 | 94.87 | 68.32 | |
| 2018-02-26 | 94.91 | 68.44 | |
| 2018-03-05 | 94.94 | 68.46 | |
| 2018-03-12 | 94.91 | 68.43 | |
| 2018-03-19 | 94.92 | 68.43 | |
| 2018-03-26 | 95.59 | 70.82 | This is after the whitespace trimming patch was deployed. Surprised to see the big change here. |
| 2018-04-02 | 95.64 | 70.95 | |
| 2018-04-09 | 95.63 | 70.94 | |
| 2018-04-16 | 95.63 | 71.00 | |
| 2018-04-23 | 95.66 | 70.90 | |
| 2018-04-30 | 95.66 | 70.93 | |
| 2018-05-07 | 95.72 | 70.78 | |
| 2018-05-14 | 95.66 | 70.68 | |
| 2018-05-21 | 95.63 | 70.75 | |
| 2018-05-29 | 95.68 | 70.86 |