Jump to content

Growth/Personalized first day/Newcomer homepage/Measurement and experiment plan/ja

From mediawiki.org
This page is a translated version of the page Growth/Personalized first day/Newcomer homepage/Measurement and experiment plan and the translation is 10% complete.
Languages:

利用者ホームページの目的は、利用者のうち特にアカウント登録したばかりの人に向けて、ヘルプの情報源や初学者でも取り組める作業のおすすめを伝え、さらに明確な情報として、ウィキペディアのコミュニティがどんな活発な活動をしていて、作業の取り組みがどんな影響があるか知らせる点にあります。 これらの情報は百科事典に建設的な投稿をしたくても壁を感じているならそれを低減し、結果として編集者の活動率を上げていきます（新規参加者のうち実際に編集作業を始める人の割合。） その情報がわかるとさらに、編集者の定着率を増やすと見込まれ（新規参加者がまた次の編集をする割合）、それこそ当 Growth チームの大きな目標です。

このページでは利用者ホームページの対照実験につき、担当チームの実施計画、利用者の皆さんとどのように影響を与え合うか、評価手法をご紹介します。

調査の質問内容

調査では高次の水準の質問にお答えいただきます。「ホームページ措置」とは、利用者ホームページを既定で有効に設定し、なおかつ利用者にホームページ確認を補佐する通知の受信を有効にしてあることで、通知とはバナーやトークページへのメッセージもしくはメインページに表示されるリンクを含みます。初期版では補助機能は1件だけで、個人向けツールにある利用者名のリンクを拾い、利用者ホームページに表示します。

  1. 編集者の活動はホームページ措置を受けると増えるか？
  2. 編集者の定着率はホームページ措置を受けると増えるか？
  3. 利用者登録直後の2週間で見る編集回数の平均は、ホームページ措置が利用できると増えるか？
  4. 建設的な編集の割合は、ホームページ措置を利用できると増えるか？
    • この文脈では「建設的な編集」とは差し戻しをされない編集のことです。Growthチームは新人編集者の定着率向上を図るため、この要素を総体と利用者の経験レベルごとの両側面から計測する計画です（例：登録後の日数、編集回数もしくは両方の組み合わせ）。
  5. ホームページが利用可能な利用者は、実際に別のページからホームページへ行ったり戻ったりするか？ ホームページ訪問に一定のパターンはあるか？
  6. Which modules engage newcomers and to what extent are they engaged? Do modules that ask users to take action lead to users taking those actions?
  7. Are we able to effectively personalize the homepage?
    • First step: do users behave in correspondence with their responses to the Welcome Survey? For example, do users who say they are interested in being contacted to get help with edits use the mentor module to reach out to a mentor; and do users who say they are interested in creating an article use the modules that help them learn more about how to do that?
    • Second step: once we can personalize the homepage, do we see increased engagement for users whose homepages are personalized?
  8. When the homepage is customizable, to what extent to do users avail themselves of that capability?
  9. If/when the effort exists to survey newcomers about the homepage, what can we learn qualitatively?  Such a qualitative study could be done through Quick Surveys, or through a homepage module that simply asks about the rest of the homepage.

Controlled experiment

In order to understand the Homepage’s impact on editor activation and retention, we propose an experiment that will be running for at least six months. During that time, 50% of new registrations on target wikis (currently Czech, Korean, Vietnamese, and Arabic) will have the homepage enabled by default, and 50% will have it disabled. We will most likely be running other experiments on the target wikis at the same time (e.g. variants of our Help Panel). If those experiments require stratified sampling, we will make sure to adjust our sampling strategies as necessary.

While the experiment is running, we expect to also be testing variants of the Homepage to understand how specific interface elements affect user behavior. We will remember that the stronger our hypothesis that an altered interface will affect activation or retention, the more a test on that interface confounds our longer term experiment on activation and retention.

Measurements

These measurements are driven by our research questions and focus on specific actions taken by users who have the ability to access the homepage and interact with its modules. The list below contains measurements for the four modules present on the Homepage at time of launch: Start, Help, Impact, and Mentor. When we develop new modules for the Homepage, we will also develop similar measurement plans for those.

  1. General
    1. What percent of users access the homepage?
      1. What percent of those users access the homepage multiple times?
      2. When users access the homepage multiple times, over what timeline does that happen?
      3. Are there differences between mobile and desktop users?
    2. How soon after creating their account does a user first visit the homepage?
    3. How do users get to the homepage?
      1. By clicking on their username in the top navigation?
      2. By clicking on the tab from their User or User talk page?
      3. Via another driver, such as a banner on wiki or link in an email?
      4. If the user clicked on a link to the Homepage from the top navigation or a tab, we want to capture the context they were in. Specifically, we want to know what namespace they were in when they clicked it, and whether they were reading or editing.
    4. How much time do users spend interacting with homepage modules per visit?
      1. During their first visit?
      2. During subsequent visits?
    5. What percent of users follow links from at least one module?
    6. Are users who have access to the homepage more or less likely to create a user page?
    7. Are users who have access to the homepage more or less likely to interact with other users through user talk pages?
    8. Are users who have access to the homepage more or less likely to interact with the community through article and project talk pages?
  2. Modules
    1. General
      1. How often is each module clicked on? What is the proportion of users who click on each one?
      2. How many modules do users tend to interact with (where “interact” means any click)?
      3. Which modules tend to be interacted with repeatedly?
      4. Which modules are interacted with most by those users who are activated and retained (acknowledging that the causality may be going in either direction)?
      5. How long do desktop users hover over each module? (analogous measurement for mobile depends on pending mobile design)
      6. Does the placement of the module on the page appear to influence interactions?
      7. What is the impact on retention from doing the call-to-action on each module?
        1. For example: what percentage of users who contact their mentor are retained?
    2. Specific modules
      1. Help module
        1. What percent of users click one or more of the help links?
          1. If so, which link(s) do they click?
        2. What percent of users use the search functionality?
          1. 1: Focus on the search box
          2. 2: Submit a search, number of characters in search
          3. 3: Click a result, number of results, ranking of clicked result
        3. How far do users go in the workflow of asking a question on the Help Desk?
          1. 1: Click to ask a question
          2. 2: Type a question
          3. 3: Submit
          4. 4: Click a link in the confirmation dialog
        4. What is the average length of questions asked?
        5. What percent of users who post a question appear to return to view the answer, by doing any of the following?
          1. Clicking the Echo notification saying they have been pinged in the response.
          2. Clicking on the link in the “Your recent questions” list.
          3. Responding to the answer on the Help Desk page.
        6. What percent of newcomers that had no email address add one when asking a question?
        7. What percent of newcomers confirm their email address (within 48 hours) after asking a question?
        8. What percent of newcomers ask a question without an email address?
        9. What percent of users who asked at least one question see one or more archived questions when they view the homepage?
        10. What percent of users who click on one of the links in the “Your recent questions” list clicked on a link to a question that was archived?
      2. Mentorship module
        1. What percent click on a link to learn more about the mentor?
        2. How far do users go in the workflow of asking their mentor a question?
          1. 1: Click to ask a question
          2. 2: Type a question
          3. 3: Submit
          4. 4: Click a link in the confirmation dialog
        3. What is the average length of questions asked?
        4. (Measured by hand) What percent of users who post a question receive an answer from their mentor?
          1. Do they get an answer from someone that is not their mentor?
        5. What percent of users who post a question appear to return to view the answer, by doing any of the following?
          1. Clicking the Echo notification saying they have been pinged in the response.
          2. Clicking on the link in the “Your recent questions” list.
          3. Responding to the answer on the talk page.
        6. (Measured by hand) What percent of newcomers who ask a question post a second time on their mentor’s talk page?
          1. How often does the first question become a conversation?
          2. How often is a second question asked?
          3. How often do conversations move beyond transactional wiki talk?
        7. What percent click on one of the links to view their question after they’ve asked it?
        8. What percent of newcomers that had no email address add one when asking a question?
        9. What percent of newcomers confirm their email address (within 48 hours) after asking a question?
        10. 新規参加者で質問をする時に自分のメールアドレスを書かない人は何パーセント？
        11. 編集初学者は質問に答えてもらうメンターとして、編集回数が多く直近の投稿からあまり時間が経っていない人を選ぶのか、回数が少なめだし／または少し前に前回の投稿をした人を選ぶのか？
        12. 質問をすでに1回した初学者のうち、利用者ホームページを開いて質問の過去ログの質問を1、2点調べた人は何パーセント？
        13. 「あなたの最近の質問一覧」のリンクを1つでも押した利用者のうち、過去ログの質問へのリンクを押したのは何パーセント？
      3. 影響度モジュール
        1. What percent click on a link when the module is in its “unactivated state” (when the user has no edits to articles)?
        2. What percent click to view an article?
        3. What percent click to view the pageviews analysis tool?
        4. What percent click to view all their contributions?
        5. How often do users return to open the pageviews analysis tool multiple times?
      4. Start module
        1. What percent of users that had no email address add an email address through this module?
        2. What percent of users confirm their email address through this module?
        3. What percent of users click the button for the tutorial?
        4. What percent of users click the button to start their user page, and what percent of them actually save a user page?
        5. What percent of users click the link to learn more about creating a user page?

Leading indicators and plans of action

The duration of the A/B test is at least six months because it is impossible to detect changes to new editor retention on mid-size wikis in less time than that (unless we drastically impact retention, but we see that as somewhat unlikely). While we wait for our results we want to be able to take action if we suspect that something is amiss. Below, we sketch out a set of scenarios based on the data described in the instrumentation strategy above. For each scenario, we propose a plan of action to take to remedy the situation.

Indicator Plans of action
Not visiting the homepage If >85% of users do not access the homepage, we prioritize testing a variant with increased affordance or investing in features to drive traffic to the homepage, such as banners, links, messages, or guided tours.
Quickly leaving the homepage after first visiting it If, on their first visit to the homepage, >50% of users leave the page within five seconds (or a different threshold identified through analysis), then we consider whether design changes are needed to improve the first impression of the page.
Not interacting with any homepage modules within 24 hours after first visiting the homepage If >75% of users who access the homepage do not interact with any of the modules, then we consider either different modules, design changes to existing modules, or improved personalization.
Not returning to homepage If <10% of users who have visited the homepage once visit it again within two weeks, then we consider either different modules, design changes to existing modules, or improved personalization.
Turning the homepage feature on/off If >1% of users turn the homepage feature on/off, we examine these users’ responses to the Welcome Survey to see if they appear to be experienced users. If they are not, we re-evaluate the design.
Not using a specific module If one module gets <10% of interactions, we consider whether that module should have its design changed or be replaced with a different module.
Help Desk overflowing If users post questions to the Help Desk through the Help Module, and users wait for responses for over an hour because the volume is sustained and too high for the community to keep up, we consult with the community to understand what these questions are and if a variant is needed.
Not enough mentors If users contact their mentors at a sustained rate that mentors cannot keep up with, we reach out to our ambassadors and/or the community to learn why they are contacted and if a variant of the module is needed.
Volume of hover events too high If the volume of hover events generated by users visiting the homepage exceeds acceptable limits, we reduce the sampling rate for these events. See T219435#5093702 for information on what “acceptable limits” are.

Status of leading indicators

The newcomer homepage was deployed to the Czech and Korean Wikipedias on May 6, 2019. We monitored the leading indicators regularly during the first month of the experiment and did not find issues that required immediate attention. Instead, we find that the Homepage is seeing a lot of traffic and interactions, in particular we notice that a high proportion of users are returning to it multiple times.

Table 1 below shows the leading indicators from the Measurement experiment plan together with results from both Wikipedias. This analysis is based on data of users who registered between deployment on May 6, 2019 and the time of analysis on June 3, 2019. Users who either self-selected into or out of having the Homepage (turning the option on or off) are excluded, as is any known test account from Growth Team members.

Table 1: Leading indicators, thresholds, plans of action, and results
Indicator Plans of action Czech Wikipedia Korean Wikipedia
Not visiting the homepage If >85% of users do not access the homepage, we prioritize testing a variant with increased affordance or investing in features to drive traffic to the homepage, such as banners, links, messages, or guided tours. 69.6% 77.3%
Quickly leaving the homepage after first visiting it If, on their first visit to the homepage, >50% of users leave the page within five seconds (or a different threshold identified through analysis), then we consider whether design changes are needed to improve the first impression of the page. 43.8% 44.3%
Not interacting with any homepage modules within 24 hours after first visiting the homepage If >75% of users who access the homepage do not interact with any of the modules, then we consider either different modules, design changes to existing modules, or improved personalization. 43.2% 61.2%
Not returning to homepage If <10% of users who have visited the homepage once visit it again within two weeks, then we consider either different modules, design changes to existing modules, or improved personalization. 58.5% 58.0%
Turning the homepage feature on/off If >1% of users turn the homepage feature on/off, we examine these users’ responses to the Welcome Survey to see if they appear to be experienced users. If they are not, we re-evaluate the design. 0.3% 0.0%
Not using a specific module If one module gets <10% of interactions, we consider whether that module should have its design changed or be replaced with a different module. 1 module 1 module
Help Desk overflowing If users post questions to the Help Desk through the Help Module, and users wait for responses for over an hour because the volume is sustained and too high for the community to keep up, we consult with the community to understand what these questions are and if a variant is needed. No No
Not enough mentors If users contact their mentors at a sustained rate that mentors cannot keep up with, we reach out to our ambassadors and/or the community to learn why they are contacted and if a variant of the module is needed. No No
Volume of hover events too high If the volume of hover events generated by users visiting the homepage exceeds acceptable limits, we reduce the sampling rate for these events. See T219435#5093702 for information on what “acceptable limits” are. No (max of 21 events per second)

There's one threshold that has been met, modules not being extensively used, and two areas of concern. Below we address these in more detail:

  • Not using a specific module: The only module that is not being used extensively is the Help module, which provides users with links to help content and a dialogue for posting questions to the Help Desk. We are not particularly concerned about this, partly because some users are interacting with it (mainly following the help links), and partly because it's not a problem if our users do not have a strong need to post questions to the Help Desk from the Homepage. Instead, we are seeing Homepage users posting questions to their mentors.
  • Not visiting the homepage: The threshold is not met on either wiki, but it can be labelled as "not far off" on Korean Wikipedia. Since we are in the process of developing features that help new users discover the Homepage, we do not see this as an issue as we expect that feature to reduce this proportion.
  • Not enough mentors: It does not appear that the volume of questions is difficult to deal with on either wiki. That being said, allowing new users to post questions to their mentors is a system where we are relying on individuals to respond, which can result in longer response times than what we might see on a wiki's Help Desk. At this point we do not find cause for concern, but it is an area that we will keep an eye on.
Retrieved from "https://www.mediawiki.org/w/index.php?title=Growth/Personalized_first_day/Newcomer_homepage/Measurement_and_experiment_plan/ja&oldid=5870651"