Visual Diffs

Research Questions

How does the visual diff compare to the wikitext diff for new/non-editors?
Do users understand what they’re seeing when they arrive at the visual diff page?
1. Can users easily and effectively identify particular changes that have been made?
2. Does the diff page communicate the type of change with appropriate amount of context?
What feedback do users have about the feature functionality, layout, interaction, visual elements?

Findings

Task Success

Fix a typo - 10/10 participants succeeded
Adding text - 10/10 participants succeeded
Removing text - 10/10 participants succeeded
Text formatting - 10/10 participants succeeded. 2/10 tried right clicking first. 1/10 changed formatting with a keyboard shortcut.
Add link - 9/10 participants succeeded in adding a link of some kind. 1/10 added an external link, thinking it was an internal link.
Remove link - 8/10 participants succeeded.
Add citation - 7/10 participants succeeded. 2/10 added external link. 1/10 skipped.
Reuse citation - 3/10 participants succeeded. 4/10 added another citation. 2/10 added an external link. 1/10 skipped.

Wikitext/Visual Diff, Participant Reactions

Table View

	Wikitext Diff	Visual Diff
P1	“trying to figure out what it's showing me”	“oh, so much better. so much easier to understand.”
P2	“a lot of information here, i have to intuit what the changes are, look at what was highlighted.”	“this is what i thought it would look like. very simple display of what i changed. looks like they grayed out the section i didn't change. i can skip the stuff not relevant to what i'm reviewing. relief. lot easier to look at and review.”
P3	“easy to understand”	“different format. looks interesting.”
P4	“it's a bit much. it's coded. a bit techie. insults your eyes with lots of words.”	“ooh! much better. much easier to see what you've done, basically.”
P5	“scans through my input”	“here, the view is a lot easier. you can actually see the bold vs quotation marks (re text formatting change.”
P6	n/a	“crossing out original and adding my edits.”
P7	“looks like mumbo jumbo clutter. hurts my eyes to look at this. don't know where to begin”	“easier for me to see, know what i'm looking at”
P8	“this can be tricky, even impossible, to read. i think it is html code for edits.”	“extremely easier to realize what you changed.” noted text formatting change he didn’t see in wikitext diff
P9	“it's showing what i have changed. compares new with previous.”	“nice, showing what i have changed and added. easy for me to see, better”
P10	“like the parallel views”	n/a

Raw notes

Wikitext Diff
- P1 - “trying to figure out what it's showing me”
- P2 - “a lot of information here, i have to intuit what the changes are, look at what was highlighted.”
- P3 - “easy to understand”
- P4 - “it's a bit much. it's coded. a bit techie. insults your eyes with lots of words.”
- P5 - “scans through my input”
- P6 - no reaction, just reviews changes
- P7 - “looks like mumbo jumbo clutter. hurts my eyes to look at this. don't know where to begin”
- P8 - “this can be tricky, even impossible, to read. i think it is html code for edits.”
- P9 - “it's showing what i have changed. compares new with previous.”
- P10 - “like the parallel views”

Visual Diff
- P1 - “oh, so much better. so much easier to understand.”
- P2 - “this is what i thought it would look like. very simple display of what i changed. looks like they grayed out the section i didn't change. i can skip the stuff not relevant to what i'm reviewing. relief. lot easier to look at and review.”
- P3 - “different format. looks interesting.”
- P4 - “ooh! much better. much easier to see what you've done, basically.”
- P5 - “here, the view is a lot easier. you can actually see the bold vs quotation marks (re text formatting change.”
- P6 - “crossing out original and adding my edits.”
- P7 - “easier for me to see, know what i'm looking at”
- P8 - “extremely easier to realize what you changed.” noted text formatting change he didn’t see in wikitext diff
- P9 - “nice, showing what i have changed and added. easy for me to see, better”
- P10 - no reaction, just reviews changes

Questionnaire Responses

Table View

	Which option better represented changes more clearly?	Which would you prefer to see when editing on the wikis?	Other comments?
P1	visual	visual	n/a
P2	wikitext	wikitext	both might be nice
P3	visual	visual	it was easy to switch modes
P4	visual	visual	more hints and tips when hovering/using icons
P5	visual	visual	both viewers should be used together
P6	visual	visual
P7	visual	visual
P8	visual	visual
P9	visual	visual
P10	wikitext	wikitext

Raw Notes

P1 - visual, visual
P2 - wikitext, wikitext, both might be nice
P3 - visual, visual, it was easy to switch modes
P4 - visual, visual, more hints and tips when hovering/using icons
P5 - visual, visual, both viewers should be used together
P6 - visual, visual
P7 - visual, visual
P8 - visual, visual
P9 - visual, visual
P10 - wikitext, wikitext

Conclusion

8/10 participants prefer visual over wikitext diffs.
1 of the 2 participants who preferred wikitext mentioned that visual diffs also has its advantages, and would like to have both modes available.

There is a possibility that the 2 participants who preferred wikitext may have conflated the mode with the '2-column' or 'side-by-side' view.

There were no red flags observed as to the participants' understanding of the edits they made while using the visual diff beta. Relatively more hesitation in identifying changes was observed when participants viewed the wikitext diff; a couple participants did not notice the text formatting edit they made.

Overall, the visual diff view is well-received. Recommendations for further testing:

Visual vs wikitext diffs with experienced editors cohort
Prototype views of more involved and complicated edits within visual diff to determine ideal representations

Researcher Notes

Searching for terms in the link inspector is not always consistent. ‘Dogs’ will find ‘dog’, ‘dog(s)’, and other pages including ‘dog’, but ‘dichromats’ will not find ‘dichromat’ or ‘dichromacy’. Can search find more matching/relevant terms including singular/plural versions, noun/verb forms, etc.?
Clicking somewhere on the word(s) to remove a link was not intuitive for many users. 8/10 highlighted the link to remove, and experienced some level of confusion. The delete button and its hover text (‘clear styling’) was also not intuitive for some.
Even though this cohort is comprised of new/non-editors, they all had difficulty with reusing a citation and over the course of testing, the tabs in the Cite menu did not clearly resonate. It is unclear if the issue is with the tabs' labels, visual styling/lack of contrast for the tabs on the hover window, or both.

Testing Methodology / Raw Notes

Set-up

User will log in with test account credentials to the 'Dog' article on wmflabs beta.
Users are directed to use Firefox, Chrome, or Safari for the test.
10 total participants
- Usertesting.com user panel
- New or non-editors
- 5 men, 5 women
  - 2 US/UK/CAN and 3 non-US/UK/CAN per group

Screener (Bold indicates accepted answer):

How many edits have you made on Wikipedia? (0-25, 26-50, 51-100, 100+)
Do you know the difference between the wikitext and the visual editors? (Y or N)

Protocol

Introduction

Keep in mind that though this test wiki is meant to replicate the live Wikipedia site as much as possible, there could be glitches that you might experience using this site. If you try something and it doesn’t work, try it once more before moving on to the next task. Please go to the next task.
Go to https://en.wikipedia.beta.wmflabs.org/wiki/Dog if not already there. Make sure to first log in with the credentials [redacted].

Tasks

After coming to this Wikipedia article, imagine you wanted to improve some things on the page. First thing you notice is that there’s a typo in the first sentence. Go ahead and fix that. (click ‘Edit’, may need to click ‘Start Editing’ on a pop-up, and ignore any other error messages)
You noticed that the ‘Biology’ section is missing some information you know, that dogs are highly variable in height and weight. Can you expand this section by adding this fact to the second paragraph?
You feel the last sentence in the Taxonomy section doesn’t add too much relevant information. Can you remove that sentence?
Currently, the latin name for dog (in the first paragraph) is not formatted in line with the Wikipedia Manual of Style (it’s in normal text). Change it so that it is in bold instead.
In the ‘Senses’ and ‘Vision’ section, you think it’d be helpful to add a link to the Wikipedia page on ‘dichromats’ in the first sentence. Go ahead and give that a try.
In that same paragraph, you notice that ‘dogs’ is linked multiple times, and you don’t think it’d be useful for a page to link to itself. Go ahead and remove those repetitive links.
You think the first sentence needs one more citation. Please cite that sentence to this article: http://www.sciencedirect.com/science/article/pii/S1055790398905139
In that same sentence, there is a citation to the http://onlinelibrary.wiley.com/doi/10.1002/zoo.20302/abstract source. Re-use this same citation on the last sentence of the first paragraph.
Click on ‘Save Changes’, and then on the next screen, the button ‘Review your changes’. What is your first impression of this page?
You made about 8 types of changes to the page. Based on what you’re seeing here, can you identify those changes without going back and reviewing your previous tasks? Take a look and talk through which edits you can identify here.
Now, click on “VISUAL (beta)” near the top right part of that same viewer window (you may need to scroll back up). What is your first impression of what you see here?
Again, which of the changes you made do you recall by reviewing this page? Talk us through them.

Post-task Questionnaire

Which of the two options do you feel represent the changes more clearly? Why?
Which would you prefer to see if you were editing on Wikipedia? Is this preferred option different than the one you indicated in the last question? If so, why?
If there is anything else you’d like to mention?
Thank you and please log out of the account.

Notes Spreadsheet

10 participants (5 male, 5 female) completed the editing tasks and answered post-test questions through UserTesting.com.

Raw notes on the 10 sessions can be found here.

Due to privacy concerns, the notes are shared internal to Wikimedia Foundation staff.