Wikimedia Technology/Annual Plans/FY2019/TEC5: Scoring Platform/Goals
Program Goals and Status for FY18/19
[edit]
- Goal Owner: Aaron Halfaker
- Program Goals for FY18/19: Improve the efficiency of wiki processes and mitigate the effects of algorithmic biases that are introduced.
- Annual Plan: TEC5 Scoring Platform
- Primary Goal is Knowledge Equity: Grow new contributors and content
- Tech Goal: Supporting our Community of contributors
Outcome 1 / Output 1
[edit]More wiki communities benefit from semi-automated curation support
- ORES supports the edit quality prediction models for more wikis/languages
Dependencies on: SRE team
Goal(s)
[edit]- Keep ORES online and improve robustness
Status
[edit]
Note: July 30, 2018
In progress
Note: August 9, 2018
- In progress
Note: September 13, 2018
- In progress...and will be forever and ever :)
Outcome 2 / Output 3
[edit]Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
- Published posts about ORES, AI, wiki processes, etc. in the ORES manual
Dependencies on: Cloud Services team
Goal(s)
[edit]- Documentation -- Threshold optimizations in The ORES Manual
Status
[edit] Note: July 30, 2018
- In progress
Note: August 9, 2018
- In progress
Note: September 13, 2018
- In progress and is in review with Sarah R
Outcome 2 / Output 4
[edit]Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
- Workshops run, papers published, datasets published, tutorials published, hackathons co-organized
Goal(s)
[edit]- Developing a focus group for JADE
Status
[edit] Note: July 30, 2018
- In progress
Note: August 9, 2018
- In progress
Note: September 13, 2018
- In progress
Outcome 3 / Output 5
[edit]Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
- JADE (our auditing system) accepts and stores human judgements
Dependencies on: SRE Team
Goal(s)
[edit]- JADE --> Production
Status
[edit] Note: July 30, 2018
Blocked The block on our JADE --> Production goal is a serious threat at this point, but there's nothing to do except to follow an RFC in the short term.
Note: August 9, 2018
- Blocked DBA's have confirmed a ceiling of growth for JADE in MediaWiki. An RFC discussion is scheduled for Aug. 22nd
Note: September 13, 2018
- Currently still Blocked because TechCom has requested iteration on our secondary tables for MediaWiki before they will consider closing the discussion
Outcome 1 / Output 1
[edit]More wiki communities benefit from semi-automated curation support
- ORES supports the edit quality prediction models for more wikis/languages
Dependencies on: SRE, Security teams
Goal(s)
[edit]- Upgrade Celery to version 4 
Done - Fix logging (redirect useful logs to logstash)  Done
- Implement edit quality models for translatewiki  In progress should be done by end of quarter
Status
[edit] Note: October 18, 2018
To do
Note: December 13, 2018
- These goals are
Partially done at this point and will be fully Done by end of quarter.
Outcome 2 / Output 3
[edit]Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)
- Published posts about ORES, AI, wiki processes, etc. on the Wikimedia blog in ORES manual
Dependencies on: Cloud Services Team
Goal(s)
[edit]- Documentation -- Feature Injection in The ORES Manual In progress
- Blog announcement of the Undisclosed Paid Editors dataset In progress
- Resubmit ORES paper to the Journal of Social Computing Cancelled
Status
[edit] Note: October 18, 2018
- To do
Note: December 13, 2018
- This is In progress and will be finished by end of this quarter; although we will not resubmit the paper.
Outcome 3 / Output 5
[edit]Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
- JADE (our auditing system) accepts and stores human judgements
Dependencies on: SRE Team
Goal(s)
[edit]- JADE --> Production (4 small wiki deployments pending local consensus)
Status
[edit] Note: October 18, 2018
- In progress
Note: December 13, 2018
- This is Partially done and will be finished in early Q3
Outcome 3 / Output 5
[edit]Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
- JADE (our auditing system) accepts and stores human judgements
Goal(s)
[edit]- Implement basic Jade integration (Phab:T183381) (carryover + additional commitment)
Status
[edit] Note: January 10, 2019
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is
Note: February 14, 2019
- Working with design on the production of wireframes/mocks In progress
- Working with design on the production of wireframes/mocks
Note: March 18, 2019
- Stalled due to a loss in Engineering Capacity Not done
- Stalled due to a loss in Engineering Capacity
Outcome 1 / Output 1
[edit]More wiki communities benefit from semi-automated curation support
- ORES supports the edit quality prediction models for more wikis/languages
Dependencies on: SRE, Security teams
Goal(s)
[edit]- Prepare ORES for kubernetes (ETA: 2 quarters)
- Implement high-availability strategy for redis
- Implement safe task serialization for celery (JSON)
- Implement new models in response to demand
- Perform research in response to community/WMF needs
Status
[edit] Note: January 10, 2019
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is
Note: February 14, 2019
- New models for Italian, Korean, and German Wikipedia as well as Spanish Wikibooks.
- Nearly finished with conversion from Pickle to JSON in Celery
- The Teahouse AI-Hostbot study has started
- Kubernetes work is In progress
Note: March 18, 2019
- Large scale deployments of February achievements (new/improved models, JSON for celery, etc.)
- Kubernetes work continues (as planned, this will take multiple quarters)
- Wikilabels CI is now running fully on docker
Strategy/Annual planning
[edit]Engage with 3-5 year strategy and annual planning process. Dependencies on: Audiences, Research, Search
Goals
[edit]- Develop proposal for building "topic spaces" infrastructure
- Develop annual plan program proposal for FY20
Status
[edit] Note: January 10, 2019
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress
- Discussed that as we've just gotten back from our vacations, this work is ramping up and is
Note: February 14, 2019
- Topic spaces proposal has been socialized and a document is In progress
- Topic spaces proposal has been socialized and a document is
Note: March 18, 2019
- Topic spaces proposal is complete and has been socialized. Talks with Audiences started Done
- Annual plan work in progress In progress
- Topic spaces proposal is complete and has been socialized. Talks with Audiences started
Outcome 1 / Output 1
[edit]More wiki communities benefit from semi-automated curation support
- ORES supports the edit quality prediction models for more wikis/languages
Dependencies on: SRE
Goal(s)
[edit]- Prepare ORES for kubernetes (ETA: 2 quarters)
- Implement high-availability strategy for redis
- Improve monitoring of ORES components in grafana
- Implement new models in response to demand
- Perform research in response to community/WMF needs
Status
[edit] Note: May 9, 2019
- ORES for kubernetes is Stalled for this quarter
- Implement high-availability strategy for redis is currently Blocked on figuring out the correct solution to take
- Improve monitoring of ORES components in grafana is Done
- Implement new models is In progress and we've completed svwiki article quality (model built -- pending deploy Partially done) already; srwiki goodfaith (model improved) is Done; eswikiversity (modeling In progress); dewiki model is Done but also Blocked waiting on Audiences / Growth team for RCFilters deployment.
- Perform research in response to community/WMF needs - srwiki, ruwiki, hewiki now have low quality "goodfaith" models Partially done for now. Additional work will still be In progress this quarter.
- Also In progress is the work written up here.
- ORES for kubernetes is
Note: June 20, 2019
- ORES for kubernetes is Stalled for this quarter
- Implement high-availability strategy for redis is currently Blocked on figuring out the correct solution to take
- Implement new models is In progress
- eswikiversity Done
- jawiki Done
- zhwiki Done
- nlwiki improvements Done
- eswikiversity
- Perform research in response to community/WMF needs
- m:Research:Exploring systematic bias in ORES is still Partially done (continue into next quarter)
- m:Research:Exploring systematic bias in ORES is still
- ORES for kubernetes is
Outcome 3 / Output 5
[edit]Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions
- JADE (our auditing system) accepts and stores human judgements
Dependencies on: Design/Audiences
Goal(s)
[edit]- Complete Jade design work. Prepare for implementation once Engineering capacity is restored.
Status
[edit] Note: May 9, 2019
- This design work is In progress but progressing very slowly with reviews of the wireframes.
- This design work is
Note: June 20, 2019
- This design work is Postponed
- This design work is
Misc Goals
[edit]- Hiring a Senior SWE and possibly an Engineering Manager
- Interviewing is In progress for the Sr SWE and Eng Mgr roles.
- Done Hired ACraze
- Interviewing is
- Annual planning work: Propose growth model for ORES and related AI technology teams
- This is now Done in regards to annual planning.
- This is now