Requests for comment/Job queue redesign
| Job queue redesign | |
|---|---|
| Component | General |
| Creation date | |
| Author(s) | Brion Vibber |
| Document status | accepted |
Current model
[edit]Each wiki carries a "job" table with a queue like this, used to record actions for future background processing:
-- Jobs performed by parallel apache threads or a command-line daemon
CREATE TABLE /*$wgDBprefix*/job (
job_id int unsigned NOT NULL auto_increment,
-- Command name
-- Limited to 60 to prevent key length overflow
job_cmd varbinary(60) NOT NULL default '',
-- Namespace and title to act on
-- Should be 0 and '' if the command does not operate on a title
job_namespace int NOT NULL,
job_title varchar(255) binary NOT NULL,
-- Any other parameters to the command
-- Presently unused, format undefined
job_params blob NOT NULL,
PRIMARY KEY job_id (job_id),
KEY (job_cmd, job_namespace, job_title)
) /*$wgDBTableOptions*/;
New jobs are pushed in on the end at runtime; multiple duplicates of a single job may be pushed.
A later queue-run operation (either from running runJobs.php on the command line/cron, or a random "background operation" during a web hit) pulls an item from the front, loads up the associated class, executes it, and when completed deletes it and any duplicates of it in the queue.
Example jobs
[edit]- RefreshLinksJob
- Re-renders pages to update their link tables after a template they used was edited
- RefreshLinksJob2
- Same thing, but optimized for high-use templates (pulls the list of pages to update at job-run time instead of job-insert time)
- HTMLCacheUpdateJob
- Updates page_touched timestamps and clears pages from the HTTP proxy or HTML file caches, if in use. Again, used to hit pages for indirect updates via templates.
- EnotifNotifyJob
- Sends e-mails of page change notifications, if configured. This is relatively time-critical.
- DoubleRedirectJob
- Automatically tidies up double-redirects that might have been created by a page rename to point to the final target.
- RenameUserJob
- For the RenameUser extension -- updates *_user_text fields in various tables with a users' new name. Delays in this are highly noticed.
Problems
[edit]- Lack of a timestamp in the queue entry means we can't easily get a sense of how lagged our operations are.
- Duplicate-entry model allows the queue to grow disproportionately large when things happen like multiple edits to a widely-used template; it becomes difficult to tell how many actual jobs there are to do.
- This model was used over avoiding duplicate inserts to maximize insert speed; a unique index is not needed, and we don't have to manually check for dupes.
- There's not a good way to get a summary of what's currently in the table; if there are a million entries, querying them to get a breakdown is pretty slow.
- There's no prioritization, making it wildly unsuitable for things like e-mail notifications which should happen within a couple minutes -- if we spend six hours updating link tables due to a template change, we shouldn't be pushing everything else back.
Execution model
[edit]On the Wikimedia setup, we have a number of app servers which additionally serve as "job queue runners", cycling through the various wikis running runJobs.php. There are some issues here as well:
- It can take a while to cycle through 700+ separate sites. While you're running through empty queues you're not processing something with a giant queue.
- Wikia, with a much larger number of smaller wikis, have been experimenting with a single shared job queue table to make this easier to handle.
- runJobs.php has been known to hang sometimes when database servers get rearranged. Needs better self-healing?
Monitoring model
[edit]Job queue runners log activity to files in /home/wikipedia/logs/jobqueue/. While sometimes useful, it'd be nice to have more of a "top"-style view that's more openly accessible:
- Web-accessible list of which jobs are being run on which wikis on which queue runners, with count, rate & lag info
- Processing rate per machine in Ganglia