Quality Assurance/User priorities
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
From an email of Erik Moeller Oct 12 2012
I wanted to make sure you're aware of some of the non-Bugzilla channels through which we hear about bugs. I've CC'd a few folks who can chip in on some stuff that I may be missing.
The main point is this: Most of our users aren't devs and aren't active on Bugzilla. Nonetheless, they're the ones using the software continuously, and are usually the first ones noticing when stuff breaks. For this reason we have lots of wiki pages that are used as collection points for initial bug reports.
The TLDR version of my note: I recommend that we
1) (near-term) grow the community of volunteers who monitor these pages, 2) (mid-term) consistently use standard templates for tagging bugs that have been entered into Bugzilla, for headers for feedback pages, etc. 3) (mid-term) improve and more widely adopt MediaWiki's built-in feedback feature to collect feedback that has useful debugging information. 4) (long-term) improve the interface between MediaWiki and Bugzilla to provide better real-time information about bug statuses to our users.
Here are a few examples.
Visual Editor: Uses feedback feature built into the editor (see below), posted on the Visual Editor feedback page. Monitored by Product Manager James Forrester, who adds bugs to Bugzilla. Does not use the
template currently (see below). Visual Editor is not in production use yet (first real-world release is scheduled for December) and therefore even critical bugs in VE don't yet need to be addressed urgently. Upload Wizard: Uses feedback feature built into the wizard, posted on the Upload Wizard feedback page. No regular monitoring. Upload Wizard is in production use on Commons and has been used to upload >1M files. Breakage has serious consequences. Uses the
template. Page Curation: Feedback link (but no feedback feature) built into New Pages Feed, posted on Page Curation talk page. Monitored by Community Liaison Oliver Keyes, who adds bugs to Bugzilla. Uses the
template. In production use on English Wikipedia, probably soon on other wikis. Breakage has serious consequences. Article Feedback Version 5 (used for feedback about articles, but here we're interested in feedback about the tool itself): No consistent feedback link built into the various parts of the feature as far as I can tell, but most links seem to lead to the Article Feedback talk page. Monitored by Oliver. Uses the
template. Version 5 is on ~5% of English Wikipedia articles and will soon be scaled to the rest. Breakage has moderate to serious consequences (due to the high visibility of the feature in the UI). Feedback Dashboard / MoodBar (used for feedback from new users, but here we're interested in feedback about the tool itself): No direct feedback link on the Feedback Dashboard, but a link to the New Editor Feedback wiki page, so the talk page is used for bug reports and feature requests. Lightly monitored by Steven and Oliver. In production use on English and Dutch Wikipedia. Breakage has moderate consequences.
template is not used.
PDF/collections export feature. Used on many Wikimedia wikis. Feedback links exist in different language versions, at least English Wikipedia, German Wikipedia, French Wikipedia, Russian Wikipedia. Not actively monitored by anyone at WMF. Community tends to curate a bit more actively. Breakage has moderate to serious consequences (basic PDF download feature is used very frequently). Mobile sites/apps: There are many feedback channels here, including a "Contact us" form built into the mobile site. Not sure this list is up-to-date. Breakage has serious consequences. Internationalization features: This would be stuff like Narayam and WebFonts. I'm not seeing any wiki feedback pages for those; most pointers are directly to Bugzilla. However, there is extensive documentation and the help pages are translated, which is not the case for any of the above. Breakage has moderate to serious consequences.
The
template I created the
template to have a standard mechanism by which a Bugzilla bug ID# can be associated with a report on a wiki page. The template supports resolution parameters to make it obvious that a bug has been fixed.
In other words, when used consistently, it gives any viewer a quick glance of the status of the bugs on a page.
It would be ideal to have the bug status get pulled automatically from Bugzilla using the BZ API, and to asynchronously update the templates with "FIXED", "INVALID", etc. Rob Moen took a crack at this here, but it would still need some love to be usable and deployable without killing Bugzilla with too many requests.
The built-in feedback feature
MediaWiki has a built-in JavaScript library for providing feature-level feedback through a pop-up, and posting that feedback to a wiki page. As noted above, I think it's currently only used by Upload Wizard and Visual Editor (the latter brokenly).
This built-in feedback tool, until recently, automatically included user-agent/OS information with each feedback post. That capability was removed by Trevor, apparently after being advised to do so by legal for privacy reasons. I would like to get this capability restored in a privacy-sensitive manner as it has been very valuable for debugging purposes in the past.
Are feedback wiki pages useful?
In my experience, yes. They offer a user-friendly mechanism to get reports about severe breakage or browser issues from the "front-line", when it happens, as well as enhancement requests. I don't believe that the right approach, however, is for us to create many more of those pages and to attempt to make a commitment to monitor them and feed bugs into Bugzilla.
This does not scale, especially given the requirement to support comments in multiple languages. Moreover, some of the above examples show that users are perfectly willing to help on their own. The beautiful thing about wiki pages is that it's people's normal working environment, so they know how to respond, how to watch for updates, etc.
I think it's appropriate and necessary for a feature in active development to have a PM or liaison monitoring a feedback page, but once a feature is not actively developed anymore, this approach stops being viable.
I believe this could be a good use case for community engagement, whether we call it "bug squad", "wikitech ambassadors", "language teams", or whatever. Ideally we'll have a single role that describes people who do the legwork of monitoring pages, responding to users, and entering important reports into BZ.
I would also recommend more consistent use of the
template, and more consistent adoption of the built-in feedback feature. The advantage of built-in feedback capabilities is that we can collect debugging information that otherwise has to be obtained by asking the user. That information is absolutely essential, as so many features trigger obscure cross-browser issues.
Example: The difference between a report "UPLOAD WIZARD IS BROKEN IT DOES NOT WORK AT ALL" is between a useless report and a potentially useful report if we have the information that the user used, say, an iPad (where we know uploads don't work) or IE8 (where we expect them to work). Not having this information creates needless human work and reduces the perceived value of feedback.
All best, Erik