Data³/metric-dictionary
< Data³
| Name | Submitted By | Description | Status | Purpose | Category(s) | Source(s) | Used In | Impacts/Contributes to | Calculation | UOM | Granularity | Aggregations | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Code Coverage | jbranaa | Measures the amount of code that is covered by unit tests. | Current Version
Tasks: |
To help drive the adoption of unit testing practices. | Code Health | PHPUnit, QUnit | Code Health dash, Code Stewardship dash | Code Health score | Lines | Code unit - class, function, etc | File, Component, Extension, Service | ||
| Code Review Responsiveness | jbranaa | Measure how quickly code reviews are performed. | Current Version
Tasks: |
To help drive improved responsiveness to overall developer community. | Code Stewardship | Gerrit | Code Stewardship dash | Cycle Time | Days | commit or patchset | Repo, Component/Extension/Service, Patch Submitter - Team, Staff, non-Staff | ||
| Bug Triage responsiveness | jbranaa | Measures the amount of time it takes to triage a new bug. | Tasks: | Drive improved responsiveness when new prospective bugs are reported. Will also potentially help manage backlog growth. | Code Stewardship | Phabricator | Code Stewardship dash | Days | task | Phab project, team, organizationally. Patch Submitter - Team, Staff, non-Staff | |||
| Bug resolution responsiveness | jbranaa | Measures the amount of time it takes to address/resolve a bug. | Current Version
Tasks: |
To help improve overall software maintenance and manage bug backlog. | Code Stewardship | Phabricator | Code Stewardship dash | Days | task | Phab project, team, organizationally. Patch Submitter - Team, Staff, non-Staff | |||
| Test Execution Duration - Test metrics (eg: quibble) | Greg | Measure how long our tests take and identifying problematic (long running) tests. | Current Version
Tasks: |
To reduce the time to feedback for our developers using CI. | Code Health? | Zuul, quibble | CI health dashboard? | Cylce Time | seconds | test run | Repo, Component/Extension/Service | ||
| Page load time | Gilles | Real world measurement of how long it takes to fully load a page | Current Version
Tasks: |
To help improve reader experience, equity of access, search engine ranking and catch regressions | Performance | Coal/Grafana | Site performance | Median navigationTiming loadEventEnd - fetchStart for 1 in every 1000 requests | ms | Labeled as "loadEventEnd" on our performance dashboards | |||
| Edit save time | Gilles | Real world measurement of how long it takes to save an edit | Current Version
Tasks: |
To help improve key editor experience and catch regressions | Performance | Coal/Grafana | Site performance | ms | |||||
| Cycle time (mediawiki/core + extensions) | thcipriani | Time it takes for a patch to make it from gerrit to prod | Tasks: | Reduce time to production by streamlining delivery/review/feedback/etc | code review | scap, gerrit | developer satisfaction | hours | Repo, team | ||||
| Bugs per train | thcipriani | Tasks: | deployment | phab | Site stability | count | Component/extension/service | ||||||
| Test wait time | thcipriani | time spent waiting on hardware availibility | Tasks: | Tighten feedback loop | testing | zuul | developer satisfaction | ms | test/extension | ||||
| Patches per train | thcipriani | How many patches are we deploying | Current Version
Tasks: |
gerrit | site stability | count | |||||||
| Team/Technology bugs reported/fixed | thcipriani | count bugs fixed and created overtime | Current Version
Tasks: |
phab | |||||||||
| Team/Tech patches created over time | thcipriani | Count of patches created/merged over time by team | Current Version
Tasks: |
gerrit | |||||||||
| Lead time | Naïké | Time needed to move a ticket from the first stage to the last stage of a process. This metric calculates the number of work days that passed since the ticket was created. | Tasks: | 1) See how the team reduces time to finish
2) Identify what is stalling the process |
Performance | Phabricator | Code stewardship dashboard | Code stewardship | Count of days | Days | Task | Phabricator board | Only reportable by looking at each task manually |
| Cycle time | Naïké | Cycle time is the total elapsed time you need to move a task from the moment it enters the “in progress” (working) stage to the moment it is considered as finished. Cycle Time = End Date – Start Date | Tasks: | 1) Optimize the process by identifying bottlenecks
2) Create value for our users more often |
Performance | Phabricator | Code stewardship dashboard | Code stewardship | Cycle Time = End Date – Start Date | Days | Task | Phabricator board | Only reportable by looking at each task manually |
| Throughput | Naïké | The number of tickets completed per period (i.e moved to “Done” and not moved backward). | Current Version
Tasks: |
1) Forecast how much work the team is able to complete at a given period
2) Measure team performance over time |
Performance | Phabricator | Code stewardship dashboard | Code stewardship | Count | Days, weeks, months, and quarters | Task | Phabricator board | Only reportable by looking at each task manually |
| Total work in progress (WIP) | Naïké | The number of tasks that can be in progress at any one time. | Tasks: | 1) Prevent tasks from getting stuck
2) Reduce loss of productivity from multitasking 3) Get more work done |
Performance | Phabricator | Code stewardship dashboard | Code stewardship | Count | Count | Task | Phabricator board | Only reportable by looking at each task manually |
| Blockers | Naïké | Anything that stops or slows down progress on a ticket | Tasks: | Remove impediments as soon as possible | Performance | Phabricator | code stewardship dashboard | Code stewardship | Count | Count | Task | Phabricator board | Can be reported by manually counting |
| Deployment frequency | Naïké | Measure of software deployment to production | Tasks: | Performance | Gerrit? | Code stewardship dashboard | Code stewardship | Count | Hours, days, weeks, months, and quarters | Task | |||
| Mean time to restore | Naïké | Measures the availability of systems. Time it takes to restore a system. | Tasks: | Know whether or not we are meeting our SLAs/SLOs | Performance | Code stewardship dashboard | Code stewardship | Mean | hours, days | Task | calculated by dividing the total unplanned maintenance time spent on an asset by the total number of failures that asset experienced over a specific period. | ||
| Change fail percentage | Naïké | Change fail percentage is a measure of the percentage of changes that result in a failure. A failure is a change that
“result[s] in degraded service or subsequently require remediation (e.g., leads to service impairment or outage, require a hotfix, a rollback, a fix-forward, or a patch.) |
Tasks: | Measure delivery performance to reduce rework | Performance | Code stewardship dashboard | Code stewardship | Percentage | Task | ||||
| Time to first response | Naïké | First Response Time is the number of minutes, hours, or days between when a customer submits a support ticket and when a customer support representative provides an initial response. | Tasks: | Reduce the time stakeholders have to wait before we respond to an issue | Performance | Code stewardship dashboard | Code stewardship | Minutes, hours, days | Hours | Task | |||
| Average age of issues | Naïké | Average age of unresolved issues | Tasks: | Making sur the team meets its SLAs/SLOs by preventing too many issues from aging | Performance | Code stewardship dashboard | Code stewardship | average | Days | Task | |||
| Workload distribution | Naïké | Number of tickets assigned per individual contributor | Tasks: | Prevent too many tickets from being assigned to the same person | Performance | Code stewardship dashboard | Code stewardship | count | Count | Task | |||
| Defect Escapes | jbranaa | Number of unknown defects that escape to production. Sometimes we find defects prior to deployment, but decide to deploy without fixing them. This is NOT them. | Tasks: | To help measure testing effectiveness. | Testing | Phabricator | Code stewardship dashboard | Site Reliability, Test Effectiveness | count | Quantity | Repo, Patch (?) | ||
| Test Case Failure Rate | jbranaa | Percentage of time that a testcase catches a problem. | Tasks: | To help identify tests are no longer relavant. | Testing | Gerrit + TBD | Test Effectiveness | # of failures/total # of executions. | Percent | Testcase | This one requires the historical tracking of testcase exectuion results. Automated tests can be pulled from Gerrit, but we don't currently have a common way to track manual testcase execution (this may not be a big deal as much of our manual testing is exploritory/non-repetitive). | ||
| nDAR | jbranaa | The rate of defects being found normalized by testing effort. | Tasks: | To help assess release readiness by the rate of defects being found over time normalized by testing effort. The testing effort can be measured in hours and/or test cases being executed. | Testing | Phabricator + TBD | Release readiness dashboard | # of pre-deployment defects reported by WMF divided by effort measured in testing hours or test cases executed. | Count | Repo, Patch (?) | |||
| Pre-production defect backlog | jbranaa | The number of defects that where found pre-deployment that are still open | Tasks: | To help assess release readiness by identifying the size of the defect backlog that will be deployed to production. | Code Health | Phabricator | Release readiness dashboard, Code Health Dashboard | Code Health, Site Reliability | # of defect found during development cycle that are open. | Count | Repo, Patch (?) | ||
| Pre-production UBNs reported per train | Rummana | The Number of UBNs found during pre-deployment testing/ code-review period | Tasks: | To measue test effectiveness | Code Health, Testing | Gerrit, Phabricator | Site Reliability | # of pre-deployment UBNs reported on Phabricator per deployment cycle | Count | Task | |||
| Pre-production UBNs fixed per train | Rummana | Pre-deployment count of UBNs fixed | Tasks: | To measure reposiveness and release readiness | Code Health | Gerrit,
Phabricator |
Test Effectiveness | # of UBNs marked as resolved on Phabricator before deployment per release cycle | Count | Task and/or Patch | |||
| Rate of re-surfaced bugs | Rummana | Number of defects that were fixed but re-surfaced | Tasks: | To identify unstable or less stable area that can be improved and/or identify test completeness in those area. | Code Health, Testing | Gerrit, Phabricator | Site Reliability + Test effectiveness | # of issues that got re-opened on Phabricator per deployment cycle per repo. | Count | Task and/or Patch |