Jump to content

Büyüme/Kişiselleştirilmiş ilk gün/Yapısal görevler/Resim ekle

From mediawiki.org
This page is a translated version of the page Growth/Personalized first day/Structured tasks/Add an image and the translation is 17% complete.
Outdated translations are marked like this.
"Add an image" feature on mobile

Bu sayfada, Büyüme ekibinin yeni gelen ana sayfası aracılığıyla sunacağı bir yapılandırılmış görev türü olan "resim ekle" yapılandırılmış görev üzerindeki çalışmayı açıklar.

Bu sayfa önemli varlıkları, tasarımları, açık soruları ve kararları içerir.

İlerleme konusundaki artımlı güncellemelerin çoğu, burada yayınlanan bazı büyük veya ayrıntılı güncellemeler ile birlikte genel Büyüme ekibi güncellemeleri sayfasında yayımlanacaktır.

Mevcut durum

  • 2020-06-22: Resimleri önermek için basit bir algoritma oluşturmak için fikirler hakkında ilk düşünme
  • 2020-09-08: İngilizce, Fransızca, Arapça, Korece, Çekçe ve Vietnamca dillerinde bir eşleştirme algoritmasında yapılan ilk denemeyi değerlendirdi
  • 2020-09-30: İngilizce, Fransızca, Arapça, Korece, Çekçe, Vietnamca dillerinde ikinci bir eşleştirme algoritması denemesini değerlendirdi
  • 2020-10-26: resim öneri hizmeti için olası fizibilite hakkında dahili mühendislik tartışması
  • 2020-12-15: yeni gelenlerin bu görevde başarılı olup olmayacağını anlamaya başlamak için ilk kullanıcı testleri turunu çalıştırmak
  • 2021-01-20: Platform Mühendisliği ekibi, resim önerileri için kavram kanıtı API oluşturmaya başladı
  • 2021-01-21: Android ekibi, öğrenme amaçları için minimum uygun sürüm üzerinde çalışmaya başlar
  • 2021-01-28: kullanıcı test sonuçlarını gönderildi
  • 2021-02-04: topluluk tartışmalarının ve kapsam istatistiklerinin gönderilmiş özeti
  • 2021-05-07: Android MVP kullanıcılara yayımlandı
  • 2021-08-06: posted results from Android and mockups for Iteration 1
  • 2021-08-17: backend work begins on Iteration 1
  • 2021-08-23: posted interactive prototypes and began user tests in English and Spanish
  • 2021-10-07: posted findings from user tests and final designs based on the findings
  • 2021-11-19: ambassador begin testing the feature in their production Wikipedias
  • 2021-11-22: image suggestion dataset is refreshed in advance of releasing Iteration 1 to users
  • 2021-11-29: Iteration 1 deployed to 40% of mobile accounts on Arabic, Czech, and Bengali Wikipedias.
  • 2021-12-22: posted leading indicators
  • 2022-01-28: desktop version deployed for 40% of new accounts on Arabic, Czech, and Bengali Wikipedias.
  • 2022-02-16: Spanish Wikipedia newcomers start getting "add an image"
  • 2022-03-22: Portuguese, Farsi, French and Turkish Wikipedia newcomers start getting "add an image"
  • 2023-02-07: Complete evaluation of section-level image suggestions (T316151)
  • 2023-10-16: Image Recommendations added to the Android Wikipedia app
  • 2024-04-11: Publish "Add an image" Experiment Analysis
  • Sonraki: Release "Add an image" to more Wikipedias


Özet

Yapısal görevler, düzenleme görevlerini yeni başlayanlar için anlamlı olan ve mobil cihazlarda anlam ifade eden adım adım iş akışlarına bölmek içindir. Büyüme ekibi, bu yeni tür düzenleme iş akışlarının tanıtılmasının, daha fazla yeni insanın Vikipedi'ye katılmaya başlamasına izin vereceğine inanıyor; bunlardan bazıları daha önemli düzenlemeler yapmayı ve topluluklarına dahil olmayı öğrenecek. Yapılandırılmış görevler fikrini topluluklarla tartıştıktan sonra, ilk yapılandırılmış görevi oluşturmaya karar verdik: "bağlantı ekle". The Growth team believes that introducing these new kinds of editing workflows will allow more new people to begin participating on Wikipedia, some of whom will learn to do more substantial edits and get involved with their communities. After discussing the idea of structured tasks with communities, we decided to build the first structured task: "add a link".

Mayıs 2021'de "bağlantı ekle"yi dağıttıktan sonra, görevin yeni gelenler için ilgi çekici olduğunu ve düşük geri dönüş oranlarıyla düzenlemeler yaptıklarını gösteren ilk veri topladık. Bu da yapılandırılmış görevlerin yeni gelen için deneyim ve viki değerli göründüğünü gösteriyor.

İlk görevi oluştururken bile, bir sonraki yapılandırılmış görevin ne olabileceğini düşünüyorduk ve görüntü eklemenin yeni başlayanlar için iyi bir uygun olabileceğini düşünüyoruz. Buradaki fikir, basit bir algoritmanın, Commons'tan görüntüleri olmayan maddelerin üzerine yerleştirilmesini tavsiye etmesidir. Başlangıç ​​olarak, yalnızca Vikiveri'de bulunabilen mevcut bağlantıları kullanır ve yeni gelenler, resmi maddeye yerleştirmek veya yerleştirmek için kendi yargılarını kullanırdı.

Bunun nasıl çalışacağına dair pek çok açık soru olduğunu biliyoruz, doğru gitmemesi için birçok potansiyel neden var. Bu nedenle, birçok topluluk üyesinden haber almayı ve nasıl ilerleyeceğimize karar verirken devam eden bir tartışma yapmayı umuyoruz.

Android ekibi, aynı temel bileşenleri kullanan Vikipedi Android uygulaması için benzer bir görevin minimum sürümü üzerinde çalıştı.

Ek olarak, Yapısal Veri ekibi daha deneyimli kullanıcıları hedefleyen ve Commons'da Yapısal Veri'den yararlanarak benzer bir şeyi keşfetmenin ilk aşamalarındadır.

Neden resimler?

Expand to read the "Why images?" section

Önemli katkılar arıyoruz

Topluluk üyeleriyle yapılandırılmış görevleri ilk tartıştığımızda, birçok kişi vikibağlantı eklemenin özellikle yüksek değerli bir düzenleme türü olmadığına dikkat çekti. Topluluk üyeleri, yeni gelenlerin nasıl daha önemli katkılar sağlayabileceğine dair fikirler ortaya attı. Bir fikir görüntüler. Wikimedia Commons 65 milyon resim içerir, ancak çoğu Vikipedi'lerde maddelerin %50'sinden fazlasında resim yoktur. Commons'tan birçok görselin Vikipedi'yi önemli ölçüde daha resimli hâle getirebileceğine inanıyoruz. Community members brought up ideas for how newcomers could make more substantial contributions. One idea is images. Wikimedia Commons contains 65 million images, but in many Wikipedias, over 50% of articles have no images. We believe that many images from Commons can make Wikipedia substantially more illustrated.

Yeni gelenlerin ilgisi

Birçok yeni kişinin Vikipedi'ye resim eklemekle ilgilendiğini biliyoruz. "Resim eklemek", yeni gelenlerin neden hesaplarını oluşturduklarına ilişkin karşılama anketinde verdikleri yaygın bir yanıttır. Ayrıca en sık sorulan yardım paneli sorularından birinin, birlikte çalıştığımız tüm vikilerde geçerli olan resimlerin nasıl ekleneceği ile ilgili olduğunu görüyoruz. Bu yeni gelenlerin çoğu muhtemelen eklemek istedikleri kendi imajlarını getiriyor olsalar da, bu resimlerin nasıl ilgi çekici ve heyecan verici olabileceğine dair ipuçları veriyor. Yeni gelenlerin katıldığı diğer platformların (Instagram ve Facebook gibi) resim ağırlıklı unsurları göz önüne alındığında bu mantıklıdır.

Resimlerle çalışmanın zorluğu

Resimler hakkındaki birçok yardım paneli sorusu, onları makalelere ekleme işleminin çok zor olduğunu yansıtır. Yeni gelenler Vikipedi ve Commons arasındaki farkı, telif hakkıyla ilgili kuralları ve görselin doğru yere yerleştirilmesi ve başlıklandırılmasının teknik kısımlarını anlamalıdır. Resimsiz bir madde için Commons'ta bir resim bulmak, Vikiveri ve kategoriler bilgisi gibi daha da fazla beceri gerektirir.

"Fotoğraf İsteyen Vikipedi Sayfaları" kampanyasının başarısı

Fotoğraf İsteyen Vikipedi Sayfaları kampanyasında 600 kullanıcı 85.000 sayfaya görsel ekledi.

Fotoğraf İsteyen Vikipedi Sayfaları (WPWP) şaşırtıcı bir başarıydı: 600 kullanıcı 85.000 sayfaya görseller ekledi. Bunu, resimleri olmayan sayfaları belirleyen ve Vikiveri aracılığıyla olası resimler öneren topluluk aracı adlı bir çiftin yardımıyla yaptılar. Yeni gelenlerin resim ekleyerek başarılı olmalarına nasıl yardımcı olacağına dair öğrenilecek önemli dersler olsa da, bu bize kullanıcıların resim ekleme konusunda hevesli olabileceği ve araçlarla desteklenebileceği konusunda güven veriyor.

Hepsini bir araya getirmek

Tüm bu bilgileri birlikte düşündüğümüzde, hem yeni gelenler için eğlenceli hem de Vikipedi'ker için üretken bir "resim ekleme" yapılandırılmış görev oluşturmanın mümkün olabileceğini düşünüyoruz.

Fikir doğrulama

Büyüme ekibi, Haziran 2020'den Temmuz 2021'e kadar topluluk tartışmaları, arka plan araştırması, değerlendirmeler ve "resim ekle" göreviyle ilgili kavram kanıtları üzerinde çalıştı. Bu, ilk yinelememizi Ağustos 2021'de oluşturmaya başlama kararına yol açtı (Yineleme 1'e bakın). Bu bölüm, yineleme 1'e kadar olan tüm arka plan çalışmalarını içerir.

"Fikir doğrulama" bölümünü okumak için genişletin

Algoritma

Resim eklemek için yapılandırılmış bir görev oluşturma becerimiz, yeterince iyi öneriler üreten bir algoritma oluşturup oluşturamayacağımıza bağlıdır. Kesinlikle yeni gelenleri maddelere yanlış görseller eklemeye teşvik etmek istemiyoruz, bu da devriyelerin arkalarını temizlemesine neden olacak. Bu nedenle, üzerinde çalıştığımız ilk şeylerden biri iyi bir algoritma yapıp yapamayacağımızı görmektir.

Mantık

Wikimedia Araştırma ekibi ile çalışıyoruz ve şu ana kadar doğruluğa ve insan yargısına öncelik veren bir algoritmayı test ediyoruz. Beklenmedik sonuçlar üretebilecek herhangi bir bilgisayar görüşü kullanmak yerine, yalnızca Vikiveri'deki mevcut bilgileri bir araya getirir ve deneyimli katılımcılar tarafından yapılan bağlantılardan yararlanılır. Bunlar, resimsiz maddelere eşleştirme önermesinin üç ana yoludur:

  • Madde için Vikiveri öğesine bakın. Bir resmi varsa (P18), o resmi seçin.
  • Madde için Vikiveri öğesine bakın. İlişkili bir Commons kategorisi varsa (P373), kategoriden bir resim seçin.
  • Aynı konuyla ilgili başka bir dildeki Vikipedi maddelerine bakın. Bu maddelerden bir ana resim seçin.

Algoritma ayrıca, olası simge olan veya bir maddede bir gezinti kutusunun parçası olarak bulunan resimleri dışlamak gibi şeyler yapmak için mantık içerir.

Doğruluk

Ağustos 2021 itibariyle, algoritmayı üç tur test ettik ve her seferinde altı dildeki maddelerle eşleşmeleri inceledik: İngilizce, Fransızca, Arapça, Vietnamca, Çekçe ve Korece. Değerlendirmeler, ekibimizin elçileri ve test edilen dilleri anadili olarak konuşan diğer uzman Wikimedialılar tarafından yapıldı.

İlk iki değerlendirme

Her dilde önerilen 50 eşleşmeyi inceledik ve bunları şu gruplara ayırdık:

Sınıflandırma Açıklama Örnek
2 Maddenin başlığı olan şeyi gösteren madde için harika bir eşleşme. Madde "Kelebek" ve bir kelebeğin resimi.
1 İyi bir eşleşme, ancak kullanıcının bir içeriği olmadığı ve iyi bir başlığa ihtiyacı olmadığı sürece madde için onaylanması zor. Madde "Kelebek" ve kelebekleri inceleyen önemli bir bilim adamının görüntüsü.
0 Maddeye hiç uymuyor. Madde, "Kelebek" ve resim, bir otomobil.
-1 Resim konu için doğru, ancak yerel kültürle uyuşmuyor. Maddenin adı "Kelebek" ve resim, dünyanın bir bölgesinden, yerel türden farklı kelebeklere sahip özel bir kelebektir.
-2 Yeni gelen birinin yanlışlıkla düşünebileceği yanıltıcı imaj doğrudur. Madde, "Kelebek" ve resim bir güve.
-3 Sayfanın bir resmi olmamalıdır. Anlam ayrımı sayfaları, listeleri veya "adı verilen" maddeler.

Çalışma boyunca böyle bir algoritma üzerinde çalışan bir soru şudur: ne kadar doğru olması gerekir? Eşleşmelerin %75'i iyiyse bu yeterli mi? %90 doğru olması gerekiyor mu? Ya da %50 kadar düşük doğrulukta olabilir mi? Bu, yeni gelenlerin yargılarının ne kadar iyi olduğuna ve zayıf eşleşmeleri için ne kadar sabırlı olduklarına bağlıdır. Algoritmayı gerçek yeni gelenlerle kullanıcı olarak test ettiğimizde bunun hakkında daha fazla şey öğreneceğiz.

İlk değerlendirmede, en önemli şey, algoritmada yapılacak çok sayıda kolay iyileştirme bulmamızdır. Bu iyileştirmeler olmasa bile, eşleşmelerin yaklaşık %20-40'ı madde için harika eşleşmeler anlamına gelen "2" idi (vikiye bağlı olarak). İlk değerlendirmenin tüm sonuçlarını ve notlarını burada görebilirsiniz.

İkinci değerlendirme için birçok iyileştirme dahil edildi ve doğruluk artırıldı. Eşleşmelerin %50-70'i "2" idi (vikiye bağlı olarak). Ancak doğruluğu artırmak, kapsamı, yani eşleşme yapabileceğimiz madde sayısını azaltabilir. İhtiyatlı ölçütleri kullanan algoritma, o vikide yüz binlerce veya milyonlarca madde olsa bile, belirli bir vikide yalnızca on binlerce eşleşme önerebilir. Bu tür bir hacmin, bu özelliğin ilk sürümünü oluşturmak için yeterli olacağına inanıyoruz. İkinci değerlendirmenin tüm sonuçlarını ve notlarını burada görebilirsiniz.

Üçüncü değerlendirme

Mayıs 2021'de Yapılandırılmış Veri ekibi, Arapça, Cebuano, İngilizce, Vietnamca, Bengalce ve Çekçe Vikipedi'de resim eşleştirme algoritması (ve MediaSearch algoritması) üzerinde çok daha büyük ölçekli bir test gerçekleştirdi. Bu testte, hem resim eşleştirme algoritmasından hem de MediaSearch'ten yaklaşık 500 eşleşme, her bir dildeki uzmanlar tarafından değerlendirildi ve bu eşleşmeleri "İyi", "Tamam" veya "Kötü" eşleşmeler olarak sınıflandırabilir. Aşağıda ayrıntıları verilen sonuçlar şunları göstermektedir:

  • Resim eşleştirme algoritması, “İyi” veya “İyi+Tamam” olarak saymanıza ve viki/değerlendiriciye bağlı olarak %65-80 doğruluk aralığında değişir. İlginç bir şekilde, resim eşleşmelerini değerlendirme deneyimimizde, uzman Wikimedialılar sıklıkla birbirleriyle aynı fikirde değiller, çünkü herkesin görüntülerin maddelere ait olup olmadığı konusunda kendi standartları vardır.
  • Wikidata P18 ("Wikidata") is the strongest source of matches, ranging from 85%-95% accurate. Images from other Wikipedias ("Cross-wiki") and from Commons categories attached to Wikidata items ("Commons category") are less accurate to a similar degree.
  • Images from other Wikipedias ("Cross-wiki") is the most common source of matches. In other words, more of those are available to the algorithm than the other two sources.
Source Accuracy (good) Accuracy (good+okay) Share of coverage
Vikiveri 85% 93% 7%
Cross-wiki 56% 76% 80%
Commons category 51% 76% 13%
All 63% 80% 100%

The full dataset of results can be found here.

Coverage

The accuracy of the algorithm is clearly a very important component. Equally important is its "coverage" -- this refers to how many image matches it can make. Accuracy and coverage tend to be inversely related: the more accurate an algorithm, the fewer suggestions it will make (because it is only making suggestions when it is confident). We need to answer these questions: is the algorithm able to provide enough matches that it is worthwhile to build a feature with it? Would it be able to make a substantial impact on wikis? We looked at 22 Wikipedias to get a sense of the answers. The table is below these summary points:

  • The coverage numbers reflected in the table seem to be sufficient for a first version of an "add an image" feature. There are enough candidate matches in each wiki such that (a) users won't run out, and (b) a feature could make a substantial impact on how illustrated a wiki is.
  • Wikis range from 20% unillustrated (Serbian) to 69% unillustrated (Vietnamese).
  • We can find between 7.000 (Bengali) and 155.000 (English) unillustrated articles with match candidates. In general, this is a sufficient volume for a first version of the task, so that users have plenty of matches to do. In some of the sparser wikis, like Bengali, it might get into small numbers once users narrow to topics of interest. That said, Bengali only has about 100.000 total articles, so we would be proposing matches for 7% of them, which is substantial.
  • In terms of how big of an improvement in illustrations we could make to the wikis with this algorithm, the ceiling ranges from 1% (cebwiki) to 9% (trwiki). That is the overall percentage of additional articles that would wind up with illustrations if every match is good and is added to the wiki.
  • The wikis with the lowest percentage of unillustrated articles for which we can find matches are arzwiki and cebwiki, which both have a high volume of bot-created articles. This makes sense because many of those articles are of specific towns or species that wouldn't have images in Commons. But because those wikis have so many articles, there are still tens of thousands for which the algorithm has matches.
  • In the farther future, we hope that improvements to the image matching algorithm, or to MediaSearch, or to workflows for uploading/captioning/tagging images yield more candidate matches.
Wiki Total articles Unillustrated articles Unillustrated percent Have image match Percent of unillustrated with match
enwiki 6.199.587 2.932.613 47% 154.508 5%
trwiki 382.825 151.620 40% 35.561 23%
bnwiki 99.172 33.642 34% 6.921 21%
frwiki 2.273.610 952.994 42% 94.594 10%
ruwiki 1.680.385 584.290 35% 60.415 10%
fawiki 755.709 304.253 40% 55.382 18%
arwiki 1.080.564 581.710 54% 59.551 10%
dewiki 2.506.229 1.190.517 48% 110.771 9%
ptwiki 1.048.255 388.605 37% 79.483 20%
hewiki 282.232 73.261 26% 14.453 20%
cswiki 467.573 182.177 39% 37.300 20%
kowiki 526.990 274.338 52% 48.417 18%
plwiki 1.441.429 560.334 39% 71.456 13%
ukwiki 1.058.563 365.209 35% 51.154 14%
svwiki 3.514.965 1.686.664 48% 91.337 5%
huwiki 479.215 170.936 36% 26.559 16%
euwiki 364.458 105.412 29% 21.481 20%
hywiki 278.487 96.729 35% 13.531 14%
arzwiki 1.171.440 759.418 65% 32.956 4%
srwiki 640.678 126.102 20% 27.326 22%
viwiki 1.259.538 867.672 69% 83.785 10%
cebwiki 5.377.763 1.357.405 25% 61.839 5%

MediaSearch

As mentioned above, the Structured Data team is exploring using the MediaSearch algorithm to increase coverage and yield more candidate matches.

MediaSearch works by combining traditional text-based search and structured data to provide relevant results for searches in a language-agnostic way. By using the Wikidata statements added to images as part of Structured Data on Commons as a search ranking input, MediaSearch is able to take advantage of aliases, related concepts, and labels in multiple languages to increase the relevance of image matches. You can find more information about how MediaSearch works.

As of February 2021, team is currently experimenting with how to provide a confidence score for MediaSearch matches that the image recommendations algorithm can consume and use to determine whether a match from MediaSearch is of sufficient quality to use in image matching tasks. We want to be sure that users are confident in the recommendations that MediaSearch provides before incorporating them into the feature.

The Structured Data team is also exploring and prototyping a way for user generated bots to use the results generated by both the image recommendations algorithm and MediaSearch to automatically add images to articles. This will be an experiment in bot-heavy wikis, in partnership with community bot writers. You can learn more about that effort or express interest in participating in the Phabricator task.

In May 2021, in the same evaluation cited in the "Accuracy" section above, MediaSearch was found to be far less accurate than the image matching algorithm. Where the image matching algorithm was about 78% accurate, matches from MediaSearch were about 38% accurate. Therefore, the Growth team is not planning to use MediaSearch in its first iteration of the "add an image" task.

Questions and discussion

Açık sorular

Resimler, Vikipedi deneyiminin çok önemli ve görünür bir parçası. Resimlerin kolayca eklenmesini sağlayan bir özelliğin nasıl çalışacağı, olası tuzakların neler olabileceği ve topluluk üyeleri için çıkarımlarının ne olacağı konusunda çok düşünmemiz çok önemlidir. Bu amaçla, pek çok açık sorumuz var ve topluluk üyelerinin gündeme getirebileceği daha fazlasını duymak istiyoruz.

  • Algoritmamız, bol miktarda iyi eşleşme sağlanacak şekilde yeterince doğru olacak mı?
  • Resminin eklenip eklenmeyeceğine karar vermek için yeni gelenler Commons'tan hangi meta verilere ve resimsiz maddeye ihtiyaç duyar?
  • Yeni gelenler tavsiyelere bakarken yeterince iyi muhakeme sahip olacak mı?
  • Commons meta verilerinin çoğu İngilizce olduğu düşünüldüğünde, İngilizce okumayan yeni gelenler eşit derecede iyi kararlar alabilecekler mi?
  • Yeni gelenler, maddelere yerleştirdikleri resimlerle birlikte iyi başlıklar yazabilecekler mi?
  • Yeni gelenler görselleri ne kadar "alaka düzeylerine" karşı "kalitelerine" göre değerlendirmelidir?
  • Yeni gelenler bu görevi ilginç bulacak mı? Eğlence? Zor? Kolay? Sıkıcı?
  • Hangi maddelerde görselin olmadığını tam olarak nasıl belirlemeliyiz?
  • Resim, resimsiz maddenin neresine yerleştirilmelidir? Yazının başına koymak yeterli mi?
  • Önerilerdeki olası önyargıları nasıl dikkate alabiliriz, yani belki algoritma Avrupa ve Kuzey Amerika'daki konular için çok daha fazla eşleşme yapacak.
  • Böyle bir iş akışı vandalizm için bir vektör olacak mı? Bu nasıl önlenebilir?

Notes from community discussions 2021-02-04

Starting in December 2020, we invited community members to talk about the "add an image" idea in five languages (English, Bengali, Arabic, Vietnamese, Czech). The English discussion mostly took place on the discussion page here, with local language conversations on the other four Wikipedias. We heard from 28 community members, and this section summarizes some of the most common and interesting thoughts. These discussions are heavily influencing our next set of designs.

  • Overall: community members are generally cautiously optimistic about this idea. In other words, people seem to agree that it would be valuable to use algorithms to add images to Wikipedia, but that there are many potential pitfalls and ways this can go wrong, especially with newcomers.
  • Algorithm
    • Community members seemed to have confidence in the algorithm because it is only drawing on associations coded into Wikidata by experienced users, rather than some sort of unpredictable artificial intelligence.
    • Of the three sources for the algorithm (Wikidata P18, interwiki links, and Commons categories), people agreed that Commons categories are the weakest (and that Wikidata is the strongest). This has borne out in our testing, and we may exclude Commons categories from future iterations.
    • We got good advice on excluding certain kinds of pages from the feature: disambiguations, lists, years, good, and featured articles. We may also want to exclude biographies of living persons.
    • We should also exclude images that have a deletion template on Commons and that have been previously removed from the Wikipedia page.
  • Newcomer judgment
    • Community members were generally concerned that newcomers would apply poor judgment and give the algorithm the benefit of the doubt. We know from our user tests that newcomers are capable of using good judgment, and we believe that the right design will encourage it.
    • In discussing the Wikipedia Pages Wanting Photos campaign (WPWP), we learned that while many newcomers were able to exhibit good judgment, some overzealous users can make many bad matches quickly, causing lots of work for patrollers. We may want to add some sort of validation to prevent users from adding images too fast, or from continuing to add images after being repeatedly reverted.
    • Most community members affirmed that "relevance" is more important than "quality" when it comes to whether an image belongs. In other words, if the only photo of a person is blurry, that is usually still better than having no image at all. Newcomers need to be taught this norm as they do the task.
    • Our interface should convey that users should move slowly and take care, as opposed to trying to get as many matches done as they can.
    • We should teach users that images should be educational, not merely decorative.
  • User interface
    • Several people proposed that we show users several image candidates to choose from, instead of just one. This would make it more likely that good images are attached to articles.
    • Many community members recommended that we allow newcomers to choose topic areas of interest (especially geographies) for articles to work with. If newcomers choose areas where they have some knowledge, they may be able to make stronger choices. Fortunately, this would automatically be part of any feature the Growth team builds, as we already allow users to choose between 64 topic areas when choosing suggested edit tasks.
    • Community members recommend that newcomers should see as much of the article context as possible, instead of just a preview. This will help them understand the gravity of the task and have plenty of information to use in making their judgments.
  • Placement in the article
    • We learned about Wikidata infoboxes. We learned that for wikis that use them, the preference is for images to be added to Wikidata, instead of to the article, so that they can show up via the Wikidata infobox. In this vein, we will be researching how common these infoboxes are on various wikis.
    • In general, it sounds like a rule of "place an image under the templates and above the content" in an article will work most of the time.
    • Some community members advised us that even if placement in an article isn't perfect, other users will happily correct the placement, since the hard work of finding the right image will already be done.
  • Non-English users
    • Community members reminded us that some Commons metadata elements can be language agnostic, like captions and depicts statements. We looked at exactly how common that was in this section.
    • We heard the suggestion that even if users aren't fluent with English, they may still be able to use the metadata if they can read Latin characters. This is because to make many of the matches, the user is essentially just looking for the title of the article somewhere in the image metadata.
    • Someone also proposed the idea of using machine translation (e.g. Google Translate) to translate metadata to the local language for the purposes of this feature.
  • Captions
    • Community members (and Growth team members) are skeptical about the ability of newcomers to write appropriate captions.
    • We received advice to show users example captions, and guidelines tailored to the type of article being captioned.

Kullanıcı testi için plan

Kullanıcı testinde kullanılan, potansiyel bir resim eşleştirme iş akışının prototipinden ekran görüntüsü. Kullanıcı, Commons'tan resim hakkında daha fazla meta veri görmek için aşağı kaydırabilir.

Yukarıdaki açık soruları düşünerek, topluluk girdisine ek olarak, bir "resim ekleme" özelliği oluşturmanın fizibilitesini değerlendirmemize yardımcı olacak bazı nicel ve nitel bilgiler üretmek istiyoruz. Algoritmayı personel ve Wikimedialılar arasında değerlendiriyor olsak da, yeni gelenlerin buna nasıl tepki verdiğini görmek ve bir resminin bir maddeye ait olup olmadığına karar verirken yargılarını nasıl kullandıklarını görmek önemlidir.

Bu amaçla, Vikipedi düzenleme yeni insanlar bir prototip ve yanıt "Evet", "Hayır" veya "Emin değil" potansiyel resim maçları geçmesi hangi usertesting.com ile testler, çalıştırmak için gidiyoruz. Biz şimdiki algoritma gerçek maçlarla destekli test için hızlı bir prototip oluşturuyoruz. Prototip, yalnızca hepsi bir beslemede olmak üzere birbiri ardına eşleşmeyi gösterir. Resimler, Commons'taki tüm ilgili meta verilerle birlikte gösterilir:

  • Dosya adı
  • Boyut
  • Tarih
  • Kullanıcı
  • Açıklama
  • Altyazı
  • Kategoriler
  • Etiketler

Gelecekte gerçek kullanıcılar için iş akışının nasıl olacağı bu olmasa da, prototip testçilerin çok sayıda potansiyel eşleşmeyi hızlı bir şekilde geçip pek çok bilgi üretebilmesi için yapıldı.

Etkileşimli prototipi denemek için bu bağlantıyı kullanın. Bu prototipin öncelikle algoritmadan eşleşmeleri görüntülemek için olduğunu unutmayın. Gerçek kullanıcı deneyimi hakkında henüz çok düşünmedik. Aslında herhangi bir düzenleme oluşturmaz. Algoritma tarafından önerilen 60 gerçek eşleşme içerir.

İşte testte arayacağımız şey:

  1. Katılımcılar, sağlanan önerilere ve verilere dayanarak eşleşmeleri güvenle onaylayabiliyor mu?
  2. Katılımcılar önerileri değerlendirmede ne kadar doğrudur? Gerçekte yaptıklarından daha iyi mi yoksa daha kötü bir iş mi yaptıklarını düşünüyorlar?
  3. Katılımcılar maddelere bu şekilde resim ekleme görevi hakkında ne düşünüyor? Kolay/zor, ilginç/sıkıcı, ödüllendirici/alakasız mı buluyorlar?
  4. Katılımcılar, resim ve madde eşleşmelerini değerlendirmelerine yardımcı olurken en değerli bulduğu bilgiler nelerdir?
  5. Katılımcılar, sağlanan verileri kullanarak eşleşme olarak gördükleri görseller için iyi başlıklar yazabiliyor mu?

Design

Concept A vs. B

In thinking about design for this task, we have a similar question as we faced for "add a link" with respect to Concept A and Concept B. In Concept A, users would complete the edit at the article, while in Concept B, they would do many edits in a row all from a feed. Concept A gives the user more context for the article and editing, while Concept B prioritizes efficiency.

In the interactive prototype above, we used Concept B, in which the users proceed through a feed of suggestions. We did that because in our user tests we wanted to see many examples of users interacting with suggestions. That's the sort of design that might work best for a platform like the Wikipedia Android app. For the Growth team's context, we're thinking more along the lines of Concept A, in which the user does the edit at the article. That's the direction we chose for "add a link", and we think that it could be appropriate for "add an image" for the same reasons.

Single vs. Multiple

Another important design question is whether to show the user a single proposed image match, or give them multiple images matches to choose from. When giving multiple matches, there's a greater chance that one of the matches is a good one. But it also may make users think they should choose one of them, even if none of them are good. It will also be a more complicated experience to design and build, especially for mobile devices. We have mocked up three potential workflows:

  • Single: in this design, the user is given only one proposed image match for the article, and they only have to accept or reject it. It is simple for the user.
  • Multiple: this design shows the user multiple potential matches, and they could compare them and choose the best one, or reject all of them. A concern would be if the user feels like they should add the best one to the article, even if it doesn't really belong.
  • Serial: this design offers multiple image matches, but the user looks at them one at a time, records a judgment, and then chooses a best one at the end if they indicated that more than one might match. This might help the user focus on one image at a time, but adds an extra step at the end.
Single: in this design, the user is given only one proposed image match for the article, and they only have to accept or reject it.
Multiple: this design shows the user multiple potential matches, and they could compare them and choose the best one, or reject all of them.
Serial: this design offers multiple image matches, but the user looks at them one at a time, records a judgment, and then chooses a best one at the end if they indicated that more than one might match.

User tests December 2020

Background

During December 2020, we used usertesting.com to conduct 15 tests of the mobile interactive prototype. The prototype contained only a rudimentary design, little context or onboarding, and was tested only in English with users who had little or no previous Wikipedia editing experience. We deliberately tested a rudimentary design earlier in the process so that we could gather lots of learnings. The primary questions we wanted to address with this test were around feasibility of the feature as a whole, not around the finer points of design:

  1. Are participants able to confidently confirm matches based on the suggestions and data provided?
  2. How accurate are participants at evaluating suggestions? And how does the actual aptitude compare to their perceived ability in evaluating suggestions?
  3. How do participants feel about the task of adding images to articles this way? Do they find it easy/hard, interesting/boring, rewarding/irrelevant?
  4. What metadata do participants find most valuable in helping them evaluate image and article matches?
  5. Are participants able to write good captions for images they deem a match using the data provided?

In the test, we asked participants to annotate at least 20 article-image matches while talking out loud. When they tapped yes, the prototype asked them to write a caption to go along with the image in the article. Overall, we gathered 399 annotations.

Summary

We think that these user tests confirm that we could successfully build an "add an image" feature, but it will only work if we design it right. Many of the testers understood the task well, took it seriously, and made good decisions -- this gives us confidence that this is an idea worth pursuing. On the other hand, many other users were confused about the point of the task, did not evaluate as critically, and made weak decisions -- but for those confused users, it was easy for us to see ways to improve the design to give them the appropriate context and convey the seriousness of the task.

Observations

To see the full set of findings, feel free to browse the slides. The most important points are written below the slides.

Slides showing the full user test findings
  • General understanding of the task matching images to Wikipedia articles was reasonably good, given the minimal context provided for the tool and limited knowledge of Commons and Wikipedia editing. There are opportunities to boost understanding once the tool is redesigned in a Wikipedia UX.
  • The general pattern we noticed was: a user would look at an article's title and first couple sentences, then look at the image to see if it could plausibly match (e.g. this is an article about a church and this is an image of a church). Then they would look for the article's title somewhere in the image metadata, either in the filename, description, caption, or categories. If they found it, they would confirm the match.
  • Each image matching task could be done quickly by someone unfamiliar with editing. On average, it took 34 seconds to review an image.
  • All said they would be interested in doing such a task, with a majority rating it as easy or very easy.
  • Perceived quality of the images and suggestions was mixed. Many participants focused on the image composition and other aesthetic factors, which affected their perception of the suggestion accuracy.
  • Only a few pieces of image metadata from Commons were critical for image matching: filename, description, caption, categories.
  • Many participants would, at times, incorrectly try to match an images to its own data, rather than to the article (e.g. "Does this filename seem right for the image?"). Layout and visual hierarchy changes to better focus on the article context for the image suggested should be explored.
  • “Streaks” of good matches made some participants more complacent with accepting more images -- if many in a row were "Yes", they stopped evaluating as critically.
  • Users did a poor job of adding captions. They frequently would write their explanation for why they matched the image, e.g. "This is a high quality photo of the guy in the article." This is something we believe can be improved with design and explanation for the user.

Metrics

  • Members of our team annotated all the image matches that were shown to users in the test, and we recorded the answers the users gave. In this way, we developed some statistics on how good of a job the users did.
  • Of the 399 suggestions users encountered, they tapped "Yes" 192 times (48%).
  • Of those, 33 were not good matches, and might be reverted were they to be added to articles in reality. This is 17%, and we call this the "likely revert rate".

Takeaways

  • The "likely revert rate" of 17% is a really important number, and we want this to be as low as possible. On the one hand, this number is close to or lower than the average revert rate for newcomer edits in Wikipedia (English is 36%, Arabic is 26%, French is 22%, Vietnamese is 11%). On the other hand, images are higher impact and higher visibility than small changes or words in an article. Taking into account the kinds of changes we would make to the workflow we tested (which was optimized for volume, not quality), we think that this revert rate would come down significantly.
  • We think that this task would work much better in a workflow that takes the user to the full article, as opposed to quickly shows them one suggestion after another in the feed. By taking them to the full article, the user would see much more context to decide if the image matches and see where it would go in the article. We think they would absorb the importance of the task: that they will actually be adding an image to a Wikipedia article. Rather than going for speed, we think the user would be more careful when adding images. This is the same decision we came to for "add a link" when we decided to build the "Concept A" workflow.
  • We also think outcomes will be improved with onboarding, explanation, and examples. This is especially true for captions. We think if we show users some examples of good captions, they'll realize how to write them appropriately. We could also prompt them to use the Commons description or caption as a starting point.
  • Our team has lately been discussing whether it would be better to adopt a "collaborative decision" framework, in which an image would not be added to an article until two users confirm it, rather than just one. This would increase the accuracy, but raises questions around whether such a workflow aligns with Wikipedia values, and which user gets credit for the edit.

Metadata

The user tests showed us that image metadata from Commons (e.g. filename, description, caption, etc.) is critical for a user to confidently make a match. For instance, though the user can see that the article is about a church, and that the photo is of a church, the metadata allowed them to tell if it is the church discussed in the article. In the user tests, we saw that these items of metadata were most important: filename, description, caption, categories. Items that were not useful included size, upload date, and uploading username.

Given that metadata is a critical part of making a strong decision, we have been thinking about whether users will need to be have metadata in their own language in order to do this task, especially in light of the fact that the majority of Commons metadata is in English. For 22 wikis, we looked at the percentage of the image matches from the algorithm that have metadata elements in the local language. In other words, for the images that can be matched to unillustrated articles in Arabic Wikipedia, how many of them have Arabic descriptions, captions, and depicts? The table is below these summary points:

  • In general, local language metadata coverage is very low. English is the exception.
  • For all wikis except English, fewer than 7% of image matches have local language descriptions (English is at 52%).
  • For all wikis except English, fewer than 0.5% of image matches have local language captions (English is at 3.6%).
  • For depicts statements, the wikis range between 3% (Serbian) and 10% (Swedish) coverage for their image matches.
  • The low coverage of local language descriptions and captions means that in most wikis, there are very few images we could suggest to users with local language metadata. Some of the larger wikis have a few thousand candidates with local language descriptions. But no non-English wikis have over 1.000 candidates with local language captions.
  • Though depicts coverage is higher, we expect that depicts statements don’t usually contain sufficient detail to positively make a match. For instance, a depicts statement applied to a photo of St. Paul’s Church in Chicago is much more likely to be “church”, than “St. Paul’s Church in Chicago”.
  • We may want to prioritize image suggestions with local language metadata in our user interfaces, but until other features are built to increase the coverage, relying on local languages is not a viable option for these features in non-English wikis.
Wiki Local language descripton Local language caption Depicts
enwiki 51.71% 3.65% 6.20%
trwiki 1.91% 1.32% 4.33%
bnwiki 0.51% 1.08% 5.74%
frwiki 5.95% 0.66% 8.52%
ruwiki 4.05% 0.61% 6.73%
fawiki 0.58% 0.59% 4.06%
arwiki 0.97% 0.59% 7.00%
dewiki 6.11% 0.49% 5.16%
ptwiki 1.38% 0.34% 4.27%
hewiki 1.20% 0.30% 6.18%
cswiki 1.82% 0.23% 5.71%
kowiki 0.97% 0.19% 4.80%
plwiki 1.82% 0.17% 5.93%
ukwiki 1.04% 0.12% 5.95%
svwiki 0.90% 0.07% 10.10%
huwiki 2.28% 0.03% 5.96%
euwiki 0.27% 0.03% 6.20%
hywiki 0.69% 0.03% 5.39%
arzwiki 0.02% 0.01% 6.84%
srwiki 0.36% 0.01% 3.46%
viwiki 0.08% 0.00% 6.63%
cebwiki 0.00% 0.00% 9.93%

Given that local-language metadata has low coverage, our current idea is to offer the image matching task to just those users who can read English, which we could ask the user as a quick question before beginning the task. This unfortunately limits how many users could participate. It's a similar situation to the Content Translation tool, in that users need to know the language of the source wiki and the destination wiki in order to move content from one wiki to another. We also believe there will be sufficient numbers of these users based on results from the Growth team's welcome survey, which asks newcomers which languages they know. Depending on the wiki, between 20% and 50% of newcomers select English.

Android MVP

See this page for the details on the Android MVP.

Background

After lots of community discussion, many internal discussions, and the user test results from above, we believe that this "add an image" idea has enough potential to continue to pursue. Community members have been generally positive, but also cautionary -- we also know that there are still many concerns and reasons the idea might not work as expected. The next step we want to in order to learn more is to build a "minimum viable product" (MVP) for the Wikipedia Android app. The most important thing about this MVP is that it will not save any edits to Wikipedia. Rather, it will only be used to gather data, improve our algorithm, and improve our design.

The Android app is where "suggested edits" originated, and that team has a framework to build new task types easily. These are the main pieces:

  • The app will have a new task type that users know is only for helping us improve our algorithms and designs.
  • It will show users image matches, and they will select "Yes", "No", or "Skip".
  • We'll record the data on their selections to improve the algorithm, determine how to improve the interface, and think about what might be appropriate for the Growth team to build for the web platform later on.
  • No edits will happen to Wikipedia, making this a very low-risk project.

Results

The Android team released the app in May 2021, and over several weeks, thousands of users evaluated tens of thousands of image matches from the image matching algorithm. The resulting data allowed the Growth team to decide to proceed with Iteration 1 of the "add an image" task. In looking at the data, we were trying to answer two important questions around "Engagement" and "Efficacy".

Engagement: do users of all languages like this task and want to do it?

  • On average, users in the Android MVP did about 11 annotations each. While this is less than image descriptions and description translations, it is greater than the other four kinds of Android tasks.
  • Image matching edits showed a substantially lower retention rate than other kinds of Android suggested edits, but there are concerns that it’s not possible to calculate an apples-to-apples comparison. Further, we think that the fact that the edits from this MVP do not actually change the wikis would lead to lower retention, because users would be less motivated to return and do more.
  • With respect to language, data was collected for users in English Wikipedia as well as from users who exclusively use non-English Wikipedia, including large numbers of evaluations from German, Turkish, French, Portuguese, and Spanish Wikipedias. We expected English and non-English users to have quite different experiences, because the majority of metadata on images in Commons is in English. But metrics were remarkably similar across the two groups, including number of tasks completed, time spent on task, retention, and judgment. This bodes well for this task being usable across wikis, although it's likely that many of the non-English Android users are actually bilingual.

Efficacy: will resulting edits be of sufficient quality?

  • 80% of the matches for which newcomers said "yes" are actually good matches according to experts. This is an improvement of about 5 percentage points over the algorithm alone.
  • This number goes up to 82-83% when we remove newcomers who have very low median time for evaluations.
  • Experts tend to agree with each other only about 85% of the time.
  • Because newcomer accuracy goes up when certain kinds of newcomers are removed (those who evaluate too quickly or who accept too many suggestions), we think that automated “quality gates” could boost newcomer performance to levels acceptable by communities.

See the full results are here.

Engineering

This section contains links on how to follow along with technical aspects of this project:

Iteration 1

In July 2021, the Growth team decided to move forward with building a first iteration of an "add an image" task for the web. This was a difficult decision, because of the many open questions and risks around encouraging newcomers to add images to Wikipedia articles. But after going through a year of idea validation, and looking through the resulting community discussions, evaluations, tests, and proofs-of-concepts around this idea, we decided to build a first iteration so that we could continue learning. These are the main findings from the idea validation phase that led us to move forward:

  • Cautious community support: community members are cautiously optimistic about this task, agreeing that it would be valuable, but pointing out many risks and pitfalls that we think we can address with good design.
  • Accurate algorithm: the image matching algorithm has shown to be 65-80% accurate through multiple different tests, and we have been able to refine it over time.
  • User tests: many newcomers who experienced prototypes found the task fun and engaging.
  • Android MVP: the results from the Android MVP showed that newcomers generally applied good judgment to the suggestions, but more importantly, gave us clues about how to improve their results in our designs. The results also hinted that the task could work well across languages.
  • Overall learnings: having bumped into many pitfalls through our various validation steps, we'll be able to guard against them in our upcoming designs. This background work has given us lots of ideas on how to lead newcomers to good judgment, and how to avoid damaging edits.

Hypotheses

We're not certain that this task will work well -- that's why we plan to build it in small iterations, learning along the way. We do think that we can make a good attempt using our learnings so far to build a lightweight first iteration. One way to think about what we're doing with our iterations is hypothesis testing. Below are five optimistic hypotheses we have about the "add an image" task. Our aim in Iteration 1 will be to see if these hypotheses are correct.

  1. Captions: users can write satisfactory captions. This is our biggest open question, since images that get placed into Wikipedia articles generally require captions, but the Android MVP did not test the ability of newcomers to write them well.
  2. Efficacy: newcomers will have strong enough judgment that their edits will be accepted by the communities.
  3. Engagement: users like to do this task on mobile, do many, and return to do more.
  4. Languages: users who don’t know English will be able to do this task. This is an important question, since the majority of metadata on Commons is in English, and it is critical for users to read the filename, description, and caption from Commons in order to confidently confirm a match.
  5. Paradigm: the design paradigm we built for "the add a link structured task" will extend to images.

Scope

Because our main objective with Iteration 1 is learning, we want to get an experience in front of users as soon as we can. This means we want to limit the scope of what we build so that we can release it quickly. Below are the most important scope limitations we think we should impose on Iteration 1.

  • Mobile only: while many experienced Wikimedians do most of the wiki work from their desktop/laptop, the newcomers who are struggling to contribute to Wikipedia are largely using mobile devices, and they are the more important audience for the Growth team's work. If we build Iteration 1 only for mobile, we'll concentrate on that audience while saving the time it would take to additionally design and build the same workflow for desktop/laptop.
  • Static suggestions: rather than building a backend service to continuously run and update the available image matches using the image matching algorithm, we'll run the algorithm once and use the static set of suggestions for Iteration 1. While this won't make the newest images and freshest data available, we think it will be sufficient for our learning.
  • Add a link paradigm: our design will generally follow the same patterns as the design for our previous structured task, "add a link".
  • Unillustrated articles: we'll limit our suggestions only to articles that have no illustrations in them at all, as opposed to including articles that have some already, but could use more. This will mean that our workflow will not need to include steps for the newcomer to choose where in the article to place the image. Since it will be the only image, it can be assumed to be the lead image at the top of the article.
  • No infoboxes: we'll limit our suggestions only to articles that have no infoboxes. That's because if an unillustrated article has an infobox, its first image should usually be placed in the infobox. But it is a major technical challenge to make sure we can identify the correct image and image caption fields in all infoboxes in many languages. This also avoids articles that have Wikidata infoboxes.
  • Single image: although the image matching algorithm can propose multiple image candidates for a single unillustrated article, we'll limit Iteration 1 to only proposing the highest-confidence candidate. This will make for a simpler experience for the newcomer, and for a simpler design and engineering effort for the team.
  • Quality gates: we think we should include some sort of automatic mechanism to stop a user from making a large number of bad edits in a short time. Ideas around this include (a) limiting users to a certain number of "add an image" edits per day, (b) giving users additional instructions if they spend too little time on each suggestions, (c) giving users additional instructions if they seem are accepting too many images. This idea was inspired by English Wikipedia's 2021 experience with the Wikipedia Pages Wanting Photos campaign.
  • Pilot wikis: as with all new Growth developments, we will deploy first only to our four pilot wikis, which are Arabic, Vietnamese, Bengali, and Czech Wikipedias. These are communities who follow along with the Growth work closely and are aware that they are part of experiments. The Growth team employs community ambassadors to help us correspond quickly with those communities. We may add Spanish and Portuguese Wikipedias to the list in the coming year.

We're interested to hear community members' opinions on if these scoping choices sound good, or if any sound like they would greatly limit our learnings in Iteration 1.

Design

Mockups and prototypes

Building on designs from our previous user tests and on the Android MVP, we are considering multiple design concepts for Iteration 1. For each of five parts of the user flow, we have two alternatives. We'll user test both to gain information from newcomers. Our user tests will take place in English and Spanish -- our team's first time testing in a non-English language. We also hope community members can consider the designs and provide their thoughts on the talk page.

Prototypes for user testing

The easiest way to experience what we're considering to build is through the interactive prototypes. We've built prototypes for both the "Concept A" and "Concept B" designs, and they are available in both English and Spanish. These are not actual wiki software, but rather a simulation of it. That means that no edits are actually saved, and not all the buttons and interactions work -- but the most important ones relevant to the "add an image" project do work.

Mockups for user testing

Below are static images of the mockups that we're using for user testing in August 2021. Community members are welcome to explore the Growth team designer's Figma file, which contains the mockups below in the lower right of the canvas, as well as the various pieces of inspiration and notes that led to them.

Feed

These designs refer to the very first part of the workflow, in which the user chooses an article to work on from the suggested edits feed. We want the card to be attractive, but also not confuse the user.

Onboarding

These designs refer to what the user sees after opening up their first task, meant to explain what the task is and how to do a good job. We want the user to understand that adding an image is a consequential edit that needs to be considered seriously. Note that this exact text has not been designed carefully yet -- rather, we are thinking now about the experience through which we deliver this content.

Adding the image

These designs refer to the part of the workflow in which the user sees the suggested image, views its metadata from Commons, and decides whether to add it to the article. We know from user tests that it is important for the user to read the image title, Commons description, and Commons caption in order to make this decision correctly. This is a challenging part of the design: making all that information available on the mobile screen.

Caption and publish

These designs refer to the part of the workflow after the user has decided to add an image to the article, and is now writing a caption to go with it. This may be the most challenging part for the newcomer, and we are still thinking about how to help them understand what sort of caption is appropriate.

Rejection

When a user rejects a suggestion, we want to collect data on why the match was wrong, so that we can improve the algorithm. This is also an opportunity to continually remind the user about the evaluation criteria they should be using as they evaluate images.

User test results September 2021

In August 2021, we ran 32 user tests amongst people who were new to Wikipedia editing, using respondents from usertesting.com. Half the respondents walked through Concept A and half through Concept B. In order to represent more diverse perspectives, this was the first time that the Growth team ran user tests outside of English: 11 respondents took the test in Spanish, all of whom were located outside the United States. This will help us make sure we're building a feature that is valuable and understandable for populations around the world.

Our goals for the testing were to identify which parts of the design concepts worked best, and to surface any other potential improvements. These are our main findings and changes to the designs we plan to make.

  • Findings
    • Concept B clearly performed better for participants in both English and Spanish tests, particularly:
      • Better understanding of the task. In Concept A, users sometimes thought the image was already in the article, because of its preview on the suggested edit card and the preview in the article.
      • More careful engagement and consideration of article contents and image metadata when evaluating image suitability to an article. We suspect this is because the article and metadata areas were clearly separated.
      • Greater use of image details and article contents during caption composition. The Concept B caption experience shows the full article text.
    • Other notes
      • Most people misunderstood the task initially as uploading images when they opened the Suggested edits module, regardless of design. But expectation about self-sourcing images was immediately corrected for almost all participants as soon as they opened the task, and overall, Design B evoked better task comprehension and successful image evaluation than Design A.
      • Newcomers would benefit from more user education around Commons and use of images on Wikipedia articles in their understanding of the broader editing ecosystem of Wikipedia and its sister projects.
      • Users understood the purpose of the caption, and understood that it would be displayed with the image in the Wikipedia article.
      • Spanish participants were far more interwiki-attuned than English participants. Potentially explore ways to better cater to multilingual/cross-wiki users.
      • Spanish participants needed to translate Commons metadata to themselves in order to write good captions in Spanish.
      • The current task requires several different skills, such as image evaluation, caption writing, and translation (for reading Commons metadata from a non-English Wikipedia). There may be benefits and opportunities for separating out this task into multiple tasks in future so that users don't have to have all the skills in order to complete the task.
  • Changes
    • Do not show a preview of the suggested image on the card in the suggested edits feed.
    • The onboarding tooltips worked well to help users understand the task. But they could be overwhelming or cluttering for smaller screens. Though we prefer to implement tooltips, we have decided to implement fullscreen overlays for onboarding in Iteration 1, because tooltips will take a substantially longer time to engineer well. We may implement tooltips in a future iteration.
    • Image and image metadata need to be next to each other -- when they are in separate parts of the screen, users become confused.
    • Because it is very important that users consider image metadata when making their decision and writing the caption, we need to increase the visibility of the metadata with clearer calls to read it.
    • Include simple validation on the free-text caption, such as enforcing a minimum length for captions, or not allowing the filename to be part of the caption.
    • Provide samples of good and bad captions in the explanation for the caption step.
    • When users reject a suggestion and give the reason for the rejection, some of the reasons should not remove the suggestion from the queue, e.g. "I do not know this topic". Perhaps another user will be able to confidently make the match.
  • Example captions: below are three image/article pairings used in the test and the actual captions written by user testers. This gives us a sense of the sorts of captions we can expect from newcomers. They all seem to be generally on the right track, though they range from more like "alt text" to more like captions. There are also a couple that miss the mark.

Final designs for Iteration 1

Based on the user test findings above, we created the set of designs that we are implementing for Iteration 1. The best way to explore those designs is here in the Figma file, which always contains the latest version.

Measurement

Leading indicators

Whenever we deploy new features, we define a set of "leading indicators" that we will keep track of during the early stages of the experiment. These help us quickly identify if the feature is generally behaving as expected and allow us to notice if it is causing any damage to the wikis. Each leading indicator comes with a plan of action in case the defined threshold is reached, so that the team knows what to do.

Indicator Plan of Action
Revert rate This suggests that the community finds the "add an image" edits to be unconstructive. If the revert rate for "add an image" is substantially higher than that of unstructured tasks, we will analyze the reverts in order to understand what causes this increase, then adjust the task in order to reduce the likelihood of edits being reverted.
User rejection rate This can indicate that we are suggesting a lot of images that are not good matches. If the rejection rate is above 40%, we will QA the image suggestion algorithm and adjust thresholds or make changes to improve the quality of the recommendations.
Over-acceptance rate This might indicate that some users aren't actually applying judgment to their tasks, meaning we might want to implement different quality gates. (What percentage of users who have a complete session have never rejected or skipped an image? What percentage of users who have five or more complete sessions have never rejected or skipped an image? How many sessions across all users contained only acceptances?)
Task completion rate This might indicate that there’s an issue with the editing workflow. If the proportion of users who start the "add an image" task and complete it is lower than 55% (completion rate for "add a link"), we investigate where in the workflow users leave and deploy design changes to enable them to continue.

We collected data on usage of "add an image" from deployment on November 29, 2021 until December 14, 2021. "Add an image" has only been made available on the mobile website, and is given to a random 50% of registrations on that platform (excluding our 20% overall control group). We therefore focus on mobile users registered after deployment. This dataset excluded known test accounts, and does not contain data from users who block event logging (e.g. through their ad blocker).

Overall: The most notable thing about the leading indicator data is how few edits have been completed so far: only 89 edits over the first two weeks. Over the first two weeks of "add a link", almost 300 edits were made. That feature was deployed to both desktop and mobile users, but that alone is not enough to make up the difference. The leading indicators below give some clues. For instance, task completion rate is notably low. We also notice that people do not do many of these tasks in a row, whereas with "add a link", users do dozens in a row. This is a prime area for future investigation.

Revert rate: We use edit tags to identify edits and reverts, and reverts have to be done within 48 hours of the edit. The latter is in line with common practices for reverts.

Task type N edits N reverts Revert rate
Add an image 69 13 18,8%
Add a link 209 4 1,9%
Copyedit 93 19 20,4%

The "add an image" revert rate is comparable to the copyedit revert rate, and it’s significantly higher than "add a link" (using a test of proportions). Because "add an image" has a comparable revert rate to unstructured tasks, the threshold described in the leading indicator table is not met, and we do not have cause for alarm. That said, we are still looking into why reverts are occurring in order to make improvements. One issue we've noticed so far is a large number of users saving edits from outside the "add an image" workflow. They can do this by toggling to the visual editor, but it is happening so much more often than for "add a link" that we think there is something confusing about the "caption" step that is causing users to wander outside of it.

Rejection rate: We define an edit “session” as reaching the edit summary dialogue or the skip dialogue, at which point we count whether the recommended image was accepted, rejected, or skipped. Users can reach this dialogue multiple times, because we think that choosing to go back and review an image or edit the caption is a reasonable choice.

N accepted % N rejected % N skipped % N total
53 41,7 38 29,9 36 28,3 127

The threshold in the leading indicator table was a rejection rate of 40%, and this threshold has not been met. This means that users are rejecting suggestions at about the same rate as we expected, and we don't have reason to believe the algorithm is underperforming.

Over-acceptance rate: We reuse the concept of an "edit session" from the rejection rate analysis, and count the number of users who only have sessions where they accepted the image. In order to understand whether these users make many edits, we measure this for all users as well as for those with multiple edit sessions and five or more edit sessions. In the table below, the "N total" column shows the total number of users with that number of edit sessions, and "N accepted all" the number of users who only have edit sessions where they accepted all suggested images.

Edits N total N accepted all %
≥1 edit 97 34 35,1
≥2 edits 21 8 38,1
≥5 edits 0 0 0,0

It is clear that over-acceptance is not an issue in this dataset, because there are no users who have 5 or more completed image edits, and for those who have more than one, 38% of the users accepted all their suggestions. This is in the expected range, given that the algorithm is expected usually to make good suggestions.

Task completion rate: We define "starting a task" as having an impression of "machine suggestions mode". In other words, the user is loading the editor with an "add an image" task. Completing a task is defined as clicking to save the edit, or confirming that you skipped the suggested image.

N Started a Task N Completed 1+ Tasks %
313 96 30,7

The threshold defined in the leading indicator table is "lower than 55%", and this threshold has been met. This means we are concerned about why users do not make their way through the whole workflow, and we want to understand where they get stuck or drop out.

Add an Image Experiment Analysis

Review the full report: "Add an Image" Experiment Analysis, March 2024.

"Add an image" to a Section

Section-level "add an image" suggestion

On wikis where it is deployed, newcomers have access to the “add an image” structured task from their Newcomer homepage. The existing "add an image" task suggests article-level image suggestions for entirely unillustrated articles. The image is then added to the article's lead section to help illustrate the article's concept as a whole.

Caption onboarding for the section-level "add an image" task

There will be onboarding for the task, followed by a specific suggestion (that includes the reason why the image is suggested). If the newcomer decides the image is a good fit for the article's section, then they receive guidance on caption writing. The structured task provides image details, further caption writing help if needed, and then prompts the newcomer to review and publish the edit.

A partnership with the Structured Data team

The Growth team is partnering with the Structured Data team to work on a section-level variation of the “add an image” task.

This is one aspect of the Structured Data Across Wikipedia project. This new task will provide image suggestions that are relevant to a particular section within an article. This section-level image suggestion task will be considered a more difficult task that will only be suggested to newcomers who are successful at the current article-level “add an image” task.

Read more about the Structured Data Across Wikimedia team’s work here: Section-level image suggestions .

Hypotheses

  • Structured editing experience will lower the barrier to entry and thereby engage more newcomers, and more kinds of newcomers than an unstructured experience.  
  • Newcomers with the workflow will complete more edits in their first session, and be more likely to return to complete more.
  • Adding a new type of “add an image” task will increase the number of image suggestions available for each language.

Scope

Key deliverable: completion of the Section-Level Images (newcomer structured task) Epic (T321754).

Design

Screenshots from two mobile designs can be seen on the right. Section-level "add an image" designs are visible in this Figma file.

User testing

Initial user testing of designs was completed in April 2023 in English. Six testers were given instructions, asked to experiment with this section-level design prototype, and evaluate the easiness and enjoyableness of the task. Testers ranged in age from 18 to 55, were from 5 different countries, and most had not previously edited Wikipedia. Three of the testers were male, and three were female. They were asked to review two image suggestions, one was a "good" image suggestion and one was a "bad" image suggestion.

Some key take-aways from the user testing:

  • The onboarding was understood by all participants: “Clear, easy to understand, straightforward.
  • Participants seemed to understand the task and that they needed to focus on the section when making their decision. One participant accepted a "bad" image suggestion:
    • 2/6 participants accepted the "good" image suggestion (3 rejected the image, 1 participant skipped it).
    • 5/6 participants rejected the "bad" image suggestion.
      • Note: the algorithm that powers image suggestions should provide more "good" suggestions than "bad" suggestions, but the algorithm isn't perfect, which is why this task needs human review and is suitable for new editors.
  • Some participants mentioned wanting more than one image to review per section:One suggestion is not enough, maybe you can present more images to choose from so I can select the most appropriate image.

Algorithm evaluation

The Growth team aims to ensure structured tasks for newcomers provide suggestions to newcomers that are accurate at least 70% of the time. We have conducted several rounds of evaluation to review the accuracy of the image suggestion algorithm.

In the initial evaluation, suggestions were still fairly inaccurate (T316151). Many images were suggested in sections that shouldn't have images, or the image related to one topic in the section but didn't represent the section as a whole. Based on feedback from this evaluation, the Structured Data team continued to work on logic and filtering improvements to ensure suggestions were more accurate (T311814).

In the second evaluation, on average, suggestions were much better (T330784). Of course results varied widely by language, but the average accuracy was fairly high for many wikis. However, there are some wikis in which the suggestions are still not good enough to present to newcomers, unless we only utilized the "good intersection" suggestions. That would severely limit the number of image suggestions available, so we are looking instead at increasing the confidence score of the suggestions we provide.

wiki % good alignment % good intersection % good p18/p373/lead image total rated suggestions
arwiki 71 91 54 511
bnwiki 28 86 26 204
cswiki 41 77 23 128
enwiki 76 96 75 75
eswiki 60 67 48 549
frwiki N.A. N.A. 100 3
idwiki 66 81 37 315
ptwiki 92 100 84 85
ruwiki 73 89 69 250
overall 64 86 57 2,120

It's good to note that this task will be Community configurable via Special:EditGrowthConfig. We hope to improve the task to the point that it can work well on all wikis, but communities will ultimately decide if this task is a good fit and should remain enabled.

Community consultation

A discussion with Growth pilot wikis is planned for May 2023 (T332530). We will post designs, plans, and questions at Arabic Wikipedia, Bengali Wikipedia, Czech Wikipedia, Spanish Wikipedia, as well as share further details here on MediaWiki.

Measurement

We decided to not deploy this feature in an A/B test and instead allow users to opt in to using it through the task selection dialogue on the Newcomer Homepage, or through the "Try a new task" post-edit dialogue that's part of the Levelling Up features. This meant that we focused on measuring a set of leading indicators to understand if the task was performing well. More details about this can be found in T332344.

We pulled data from Growth's KPI Grafana board from 2023-07-31 to 2023-08-28 (available here) for Section-Level and Article-Level suggestions. This timeframe was chosen because it should not be as much affected by the June/July slump in activity that we often see on the wikis. The end date is limited by the team shutting off image suggestions in late August (see T345188 for more information). This data range covers four whole weeks of data. While this dataset does not allow us to separate it by platform (desktop and mobile web), nor does it allow us more fine-grained user filtering, it was easily available and provides us with a reasonably good picture that's sufficient for this kind of analysis at this time. Using this dataset we get the overview of task activity shown in Table 1.

Table 1: Task clicks, completions, and reverts by task type.
Task type Task clicks Saved edits Reverts Task completion rate Revert rate
Section-level 1.149 688 60 58,1% 9,0%
Article-level 6.800 2.414 105 35,5% 4,3%

We see from the table that the task completion rate for section-level image suggestions is high, on par with Add a Link (ref) when that was released. This is likely because the section-level task is something users either choose themselves in the task selection dialogue, or choose to try out after being asked through the "Try a new task" dialogue that's part of Levelling Up. Those users are therefore likely already experienced editors and don't have too many issues with completing this task.

The revert rate for the section-level task is higher than the article-level task. We don't think this difference is cause for concern for two reasons. First, it might be harder to agree that an article is clearly improved by adding a section-level image compared to adding an article-level image. Secondly, articles suggested for section-level images already have a lead image, which might mean that they're also longer and have more contributors scrutinizing the edit.