Beta Features/Roadmap

Here are some possible new functionalities and changes to Beta Features itself that may be added in the future. Note that programs using Beta Features should be added to the proposal page instead.

Pop-up guide to using Beta Features

Status: On hold – may not be needed as users seem to be able to find their way around OK.

As a user, I want to know what the 'Beta' label in my personal menu means. (See Mingle ticket #49)

Show a pop-up guide that explains what Beta Features is, the first time you load a page that contains the 'Beta' label in your personal menu.

It would show a couple lines explaining what this 'Beta' label does and point to it (between 'Preferences' and 'Watchlist'), as so:

Introducing $Sitename Beta Features!
Beta Features lets you test and give feedback about new features before everyone else gets to see them.
Actions: Maybe later Try Now'

'Try Now' would link to the Beta Features page
'Maybe later' would simply close the popup guide

See mockup in Mingle ticket #49.

Beta Link Preference

Status: On hold – adding yet another preference isn't a great plan either. Possibly solved by the forthcoming personal tools Beta Feature.

Some users may object to seeing the 'Beta' link in the personal menu (between 'Preferences' and 'Watchlist'). To prepare for this potential issue, we may want to consider a preference to let users remove that link, if they don't want it.

This could be a simple checkbox at the bottom of the 'Beta Features' preferences page, that would say something like:

[ x ] Show 'Beta' link in my personal menu.

There may be better ways to do this than preferences, since we are trying to update and improve the overall number of preferences to simplify the user experience. However, this is a relatively easy task that could be implemented quickly, without too much development; this could be the most practical approach for now, if we have to move fast to respond to a large number of community requests. We will update this section if we decide to implement this proposed feature.

Only show Beta link when there are new features

Status: On hold – complicated logic with "mystery meat" navigation issues ("Why is it here? Why did it disappear? Help!?")

Another more sophisticated way to address complaints about the 'Beta' link in the personal menu would be to only show that link if there are new features you haven't seen yet (perhaps with a little number next to the 'Beta' label. Once you have seen the new features, the Beta link would disappear until there are more new features -- or feature updates.

This may be a bit more elegant than implementing the 'Beta Link Preference' described above. However, this would require some new designs and more development than our current limited resources can support. So it may not be as practical or expedient as the Beta Link Preference at this time. We will update this section if we decide to implement this proposed feature.

Have the "enable all new Beta Features" checkbox also enable all current ones too

Status: Ready for a developer to implement.

A number of users have told us they are confused by the current implementation of the first check box, "Automatically enable all new beta features". They are puzzled as to why it works that way and believe that it should enable all beta features right away, not just future ones.

As a result, we recommend changing both the functionality and the wording to say something like 'Automatically enable all current and new beta features'. This would match user expectations more closely than the current solution, which is disappointing.

Sorting options

Status: Ready for a developer to implement. Low priority as there aren't that many features.

As more beta features are added, we may need to provide affordances for sorting the list by importance, by name, or by date. For example, we will want the option to place an important feature like Visual Editor at the top of the page (because of its significance for the movement's strategic goals), but also give users the option to change that default sorting to list features instead by date (e.g. most recent first) or by name (e.g. alphabetically). It may also become necessary to make the page design more compact, if we are beta testing dozens of different features at the same time.

Echo notifications

Status: In progress – see Gerrit change 84625.

Beta Features will trigger Echo notifications at different times, as soon as practical. This is to increase visibility and get additional feedback, so the product teams can react, update and improve features before making a decision to integrate into core, iterate the idea further, or abandon the experiment.

Here's a preliminary list of notifications that seem desirable, subject to feasibility and team discussion:

New Feature Enabled for you: Sent to logged-in users who have enabled automatic feature enrolment and enabled this notification category. Setting default: on for site (and not disable-able), off for e-mail.
- Later: Want link to that feature (anchor) in notification, highlight on that feature (yellow box?).

New Feature Available: Sent to all logged-in users who have signed up for Beta features and enabled this notification category, when a new feature is available to test. Setting default: off for both.
Feature Graduation: When features have achieved a high level of stability, acceptance, and polish, they may be chosen to be integrated by default as an extension, on a project-by-project basis. Setting default: off for both.
Feature Updated (Future): When major changes are made to a feature, a notification may be sent to enrolled users who have signed up for Beta features and enabled this notification category. Setting default: off for both.
Back to the Drawing Board (Future): Experiments that don't pass muster may be discontinued, and removed from the Beta Experiments program to be re-evaluated. Setting default: off for both.

Popup Guider

Status: Not done

Beta Features would trigger a Popup Guider when a new feature is released (as an alternative to the more complex Echo notifications) .

Goals

From a user's viewpoint, I want to know when a beta feature is about to be released, so I can try it out and learn more before it launches. From the foundation's viewpoint, we want to increase visibility and get additional feedback, so product teams can improve features before they are released, as well as let users know when a feature has been released.

This feature is also described in this onwiki specification (and is tracked in this Mingle ticket: Popup Guider for Beta Features (card #447)).

Here are the popup notices we propose to implement first:

New Feature Available: Shown to all logged-in users when a new feature is available for testing.
New Feature Release: This feature has now been released to everyone.

To make this work, this task will also be needed:

Beta Features Update: We want to add anchor links and section highlights to the BetaFeatures extension (card #432)

Notes: The task force that created this proposal views this popup solution as more practical than the Echo notification proposal above, for a number of reasons:

Echo notifications are intended for activity that is particularly relevant to you as a user — not for random announcements that may not interest you, which this new feature will inevitably encourage.
Many sites like Facebook support a range of notification styles for different types of activities — without any serious adverse effects.
From a development standpoint, the Echo implementation will take longer to implement, due to the more complex nature of this tool (e.g. email notifications, default settings, bundling, cross-wiki issues, etc.).
Our communities will need to define a sound policy about who can use Echo to contact millions of users for announcements like these. Who would have the right to do fire off a notification? What’s the review process? What rights does the foundation have vs. other stakeholders? The complex issues raised by these questions would require many conversations on all our projects, and would take a long time to resolve.

Also note that James Forrester is helping coordinate BetaFeatures with the design team, and should be contacted before implementing any of the proposed solutions.

Unsaved changes

Status: Ready for a developer to implement.

Right now, if I select a feature and leave the page without clicking 'Save', the preference isn't saved (understandably).

This feature would showing a message letting users know they have unsaved changes, something like:

"Your changes have not been saved. Discard changes? [ OK ] [ Cancel ]"

This feature has been partly implemented across all preferences, which all share the same basic issue, since the save button is often below the fold and hard to see. However, the current implementation uses confusing language that is redundant and a bit unclear, compared to the simpler copy proposed above. But it may not be technically feasible to implement the simpler copy, due to some browser limitations.

Guided Tour

Status: On hold – may not be needed as users seem to be able to find their way around OK.

As a user, I want to know how 'Beta' features work.

Show a guided tour that explains how Beta Features work the first time you load a page that contains the 'Beta' label in your personal menu.

Here is a proposed sequence of popups that would point to different parts of the Beta Features UI:

1) Introducing Beta Features!
Beta Features lets you test and give feedback about new features before everyone else gets to see them.
Actions: Maybe later [Try Now]
2) You can try out each feature separately
From here you can see what each feature does, learn more, or join the discussion about what you like about it, what you'd change, and if you encounter any issues.
Actions: Leave Tour [Next]
3) To turn on a feature check this box
You can always come back here to turn it off, once you try a feature be sure to go to its discussion page to let us know your thoughts on it.
Actions: Leave Tour [Next]
3) If you want new features to be enabled by default check here
New features will be turned on for you as they are released, you'll always be looking at the most up-to-date version. Have fun, and don't forget to let us know what you think.
Actions: [Done!]

Note that some team members think this guided tour seems like overkill for such a simple tool as Beta Feature, which is pretty self-explanatory. If we implement this feature, we would disable the Popup Guide described above.

Micro-survey when opting-out of a Beta Feature

Status: Ready for a developer to implement. Will need some Design & Product input on copy, look & feel, etc.

A micro-survey would be created to capture reasons why people decide to opt out of specific beta features, using a similar way to how we collect gender data for account registrations. See Trello ticket for more info about this proposal. This seems nice to have, but implementing micro-surveys could be a pretty labor-intensive task, which could be outside the scope of this project.

Use Flow for Beta feature discussion

Status: Done

We intend that the discussion page for all new Beta features will use the Flow discussion software. Migrating the Talk pages of existing Beta features discussions to Flow is possible; the existing discussions would be moved to an Archive subpage.

Research

Here is a preliminary list of research questions we hope to answer about the Beta Features tool, for discussion purposes.

Beta Features Adoption

Here are our goals for researching Beta Features adoption. (See also this Trello ticket, as well as Mingle ticket #55).

How many users are opting into beta features?
What's the breakdown of adoption by user tenure?
What are the most popular features by number of enrolled users?
How many users are selecting the auto-auto-enroll option for new features?
What proportion of users is disabling beta features after trying them out?

No ad-hoc instrumentation seems needed to measure adoption, as all data required to obtain descriptive usage stats are already available from these sources:

the PrefUpdate log
the user_properties table (core)
the betafeatures_user_counts table (extension)

For this product, we would like to build a set of LIMN dashboards to track basic usage patterns, so that research can inform our next steps. This development may need to be phased over time, based on available resources. Key questions are listed below in order of priority.

Beta Features Usage

Here are some other metrics we might want to track at a future stage.

Overall Stats

How many times did users click on the 'Beta' link in the personal bar?
How many times did users view the 'Beta features' preference page? (impressions)
What is the click-through rate for enabling feature x based on total impressions for the Beta preferences page?

Auto-enrol New Features

How many users have enabled all new beta features?
How many users have disabled new beta features after trying it out?

Individual Features

How many unique users enabled feature x?
How many unique users disabled feature x after trying it out?

(repeat for each feature x, z, z)

Info & Discussion Pages

How many times did users click on the info page icon for feature x?
How many times did users click on the discussion page icon for feature x?

These metrics could be collected through a combination of EventLogging and related technologies, and visualized with LIMN dashboards (see example). We would track this data on a daily basis, as well as on a cumulative basis.