User talk:Peter17/Reasonably efficient interwiki transclusion

Comments

Hi Peter, I'm not so deep into wiki software, so ignore my comments if fool.

1. When you mention templates, I presume, you'll address to any wiki page, since any wiki page can be seen as a template, and works as a template.
2. Please consider deeply and carefully the software and the philosopy of Extension:Labeled Section Transclusion. It's an extension mainly used into wikisource projects, but it would be extremely useful to any wiki project for very sound reasons, I only wait that it is discovered.... it is simply unknown. Just an example: I usually write multi-axial arrays into a single wiki page with a very simple code... sometimes with hundreds, or thousands of elements. I'd like to post here an example of its features, but I can't since the extension isn't installed here.... ;-) --Alex brollo 09:27, 25 May 2010 (UTC)

Hi Alex,

# Yes, I noticed that my text was somehow unclear... The aim is to be able to transclude any page. I will correct that.

# I have read that page, but I didn't know this extension before. Transcluding sections by heading (like #lsth) would be great for my project. I don't know if I can use the API for that but I will check. I don't know if #lst or #lstx are in the field of my project, but I will keep it in mind! I don't really understand what you want to to with multi-axial arrays... Do you have an example on another wiki?

Best wishes. Peter17 09:49, 25 May 2010 (UTC)

What about a more comprehensive approach, allowing a unified page table, and certain pages therein, to be shared across two or more wikis

Hi Peter, I am working on an extension to fix a number of bugs that are within the purview of Project:WikiProject Interwiki Integration. I have proposed that we add a field to the page table that could be set to one of three values: (1) The wildcard "*" for all wikis; (2) The name/index of the one wiki/database to which the page pertains; or (3) A foreign key leading to a table with wikis/databases as its fields and boolean values as its contents. There would be a function, e.g. Title::getDatabases(), that would access that table and return an array of all the wikis/databases that a page pertains to. (This need not be set up as just one field in the page table; in fact, it might be better to set up as 2 or 3 fields.)

The idea is that wikis on a wiki farm could use $wgSharedTables to share the page table, and then the wikis could have "global" or multi-wiki pages that the software would recognize as being present and identical on all the wikis in that array. Thus, an edit to the page on one wiki would change it for all the wikis that page is shared on. I suppose pages could also be set up as having a "home wiki" that would be the only wiki the page could be edited from (creating something functionally equivalent to mere interwiki transclusion), or restrictions could be set up on a per-wiki basis. Maybe users blocked from editing on the page's "home wiki" would be unable to edit the article, but otherwise users could edit it from any wiki that the page is shared on. There are a lot of possibilities.

Probably setting up a page to be shared could be done in a similar manner as moving an article; that is, the page would have to be non-existent on the local wiki, or the only item in the page history would have to be a redirect; otherwise, admin assistance would be required. Tisane 14:16, 1 June 2010 (UTC)

You can't share the page table; it would break database integrity everywhere unless you also share revision (and hence text), templatelinks, pagelinks, recentchanges, page_props, basically the entire database. Then you have your entire wiki farm running off one database, which doesn't scale: WMF's wikis are split into six clusters for precisely the reason that one database can't handle the load. Happy‑melon 18:37, 1 June 2010 (UTC)

Also, this project is specifically about interwiki transclusion, which is difficult in and of itself. That alone is a reason not to do this; Happy-melon's arguments for why this is a bad idea are also right on the money. --Catrope 19:50, 1 June 2010 (UTC)

@Happy Melon: I pretty much do think that the whole database should be shared, much as different namespaces share the same tables. Or rather, I think they should be shared if a way can be found to do it without causing the scaling problems you mention. An alternative is to set up global page tables kinda like the global user tables that CentralAuth sets up; that would probably be the superior method if an extension is going to be used. @Catrope: I'm not trying to steal anyone's thunder or step on anyone's toes if you guys just want to do interwiki transclusion, so I'll leave off further commentary on my proposal here. Tisane 21:45, 1 June 2010 (UTC)

@Tisane: Hi! Your approach seems interesting. I think it could replace interwiki transclusion (inside WMF's farm), as the requested foreign pages would not be distant anymore. As Catrope said, it is a little different from my project, but that could be a more comprehensive way of implementing it. However, Happy-melon's remarks are to be taken into account and my knowledge of the system is not complete enough to answer on this point. I think it should be discussed on wikitech-l before, so, we can get ideas from more people.

Another goal of my project is transclusion to foreign wikis, so, the two projects are actually different even if they have a common part. Peter17 00:17, 2 June 2010 (UTC)

As the saying goes, There's more than one way to do it. As you've pointed out, local caching of transcluded foreign pages could help mitigate server load. I've done a bit of work on foreign interwiki integration, specifically Extension:RPED, which maintains a table of data on foreign wiki page titles. There is also a bot to read the IRC recent changes feed and update the table as page creations/deletions/moves occur on the foreign wiki. Perhaps your local wiki could maintain a cache of transcluded pages on the foreign wiki, and whenever an edit to one of those pages appears on the IRC feed, it could grab the latest version of that page via the API. Maybe the bot could run on a centralized server and push the data to each of its client wikis, so that there wouldn't have to be a bunch of local wikis running the bots and making redundant API requests. Tisane 00:48, 2 June 2010 (UTC)

Core or extension?

Is this functionality envisaged as being implemented as part of the core or as an extension? If the latter, I would think that rather than adding fields to the interwiki table, one would want to put them in a new table, to be added by $wgExtNewTables, unless it is planned that the field will find other uses as well (which is probably the case). Then again, there are the performance issues mentioned at Extension talk:InterwikiIntegration that occur when one reads from the database rather than from variables set in, e.g., InitialiseSettings.php. It's a subject I've been mulling over a lot lately.

Are you envisaging that the interwiki table will be shared among the wikis on a wiki farm? I figure that way, if a change needs to be made (e.g. a new wiki is added), one SQL query can change it for all the wikis. Tisane 00:04, 15 June 2010 (UTC)

Hello, Tisane. Sorry for this late answer. I am currently programming the functionality in the core. This aspect (core or extension) has not been discussed yet, but as my code will replace "scary transclusions", which is in the core, I assume that I have to program this in the core. When discussing on the list and with my mentor, it has never been mentioned that it would be an extension...

About the other implications of my change, I just know that adding the URL of api.php in the interwiki table has already been discussed and will be useful for other purposes (I don't know which ones yet).

About sharing the interwiki table, I really have no idea. This table already exists and I don't need it to be shared. Sharing it should be discussed more globally... Peter17 16:00, 16 June 2010 (UTC)

Sounds cool

This project sounds really cool! I've been trying to use Wikipedia templates on my private wiki for a couple of years now. I've configured the interwiki table. I've set scaryTranscluding. I've tried everything. But nothing works properly :( If you can make this project work, it'll be a major boost to MW functionality. I can't believe it's not already implemented! I want to be able to use templates like Infobox_Economist in my own private work. Please keep us updated on progress. Anything we can do to help? Adrian

One thing you could do is help test things, but you can only start doing that in a later stage; right now, Peter is still busy writing the code. Once he gets to the testing stage, I'll see about setting up a public test wiki for this. --Catrope 17:22, 22 July 2010 (UTC)

You would still need to copy Infobox to your local wiki to do that. Rich en:User talk:Rich FarmbroughFarmbrough

About multilingual namespacename and fallback-calling

Hi, I read this page and looks pretty cool. Two points came to me though.

Hi! Peter17 16:42, 13 August 2010 (UTC)

Namespacename

I see the namespacename is stored as text. I didn't quite get why this is needed, but I'm guessing this is to be used in a dropdown-menu on the special-page for the usage (as is with Special:WhatLinksHere/Template:Test). However, what about matching namespaces from different wikis. Commons would have for example "(Gallery)" and "Template". But nlwikipedia "Hoofdnaamruimte" and "Sjablonen" (Main namespace and templates respectively). Perhaps not filter on namespace at all, or a way is needed to merge the matching ones (atleast the standard core-ones. Maybe a generic name (not a number or the wiki-text, because the former doesn't always match and the latter is different per language) like NS_PROJECT, NS_USERTALK etc.

Actually, I store the namespace id (number) and work with it. The namespace text is just stored for display purpose. Peter17 16:42, 13 August 2010 (UTC)

Also what comes to mind is where the usage would be tracked. For files, the "Special:WhatLinksHere" is only the local one. And the file-page shows both local usage and global usage summarized in two sections. For templates the toolbox'd have a seperate Special:GlobalTemplateUsage ? Or merge it into the existing GlobalUsage for files.

I am working on this. Currently, it is tracked in Special:GlobalTemplateUsage. I would like to merge it with the existing GlobalUsage, but that one is an extension, so, I would like to have it built-in, so, I can merge both. Peter17 16:42, 13 August 2010 (UTC)

Calling-method / fallback

I see that the call-method used is {{interwikiid:ToIncludePagename}}. The advantage here is that this allows several inclusion from several different wikis and point exactly the one that is wanted. However, for files this is done by a fallback principle. Local has preference, and if not existant the shared repo(s) are checked and included accordingly. That may be more intuitive as an end-user. Though, that might also bring the expectation that going to the template-page locally (ie. a table from "wikix" doesn't exist on "wikiy" but can be included like {{Foobar}} form wikix, but what would Template:Foobar show when visited on wikiy ?

Hum... I think it is more intuitive to copy the way interwiki links work : just give the prefix and it will point the wanted page. My project will finish on August 16th, so, I will have no time to program a fallback feature. This could be discussed on the mailing-list and developed later if judged useful... Peter17 16:42, 13 August 2010 (UTC)

First off, I realise your end-date has reached, but wanted to reply here just for the record.

I agree that from a technical point of view it's like the interwiki-system. However I believe that most users, especially on WMF-wikis (but ofcourse by all means, that's just one of the many sites that use MediaWiki), users would compare it to the way FileRepo works. Asuming, on WMF-wikis atleast, interwiki transclusion will only allowed from 1 wiki (either Commons or a new one), it would makes sense to not need to give an interwiki prefix and let a local equally named template take precedence of a remote one. That would also allow a more convenient way of centralizing and replacing commonly used templates (since these often have the same name, they can just be moved to Commons and deleted locally, no need to do any funky replacement stuff). Krinkle 22:15, 29 August 2010 (UTC)

Just a few thoughts, Krinkle 15:27, 13 August 2010 (UTC)

Thanks for your feedback. You can test the system if you want, just look the « Running prototype » section in this page. Sincerely. Peter17 16:42, 13 August 2010 (UTC)

Database error

I get a db error on the prototype w8ki. Rich en:User talk:Rich FarmbroughFarmbrough

Solved, thanks for reporting it. Peter17 16:09, 13 November 2010 (UTC)

Prototype Down/Project Status

I'm getting "Fatal error: Unsupported operand types in /srv/org/wikimedia/prototype/extensions/Vector/Vector.php on line 109" on all three wikis.

Also, is there somewhere I can look to for updates on the project? I see some of the svn commits, but was wondering if there had been a decision to add it to MW 1.17 or some later version? Really interested in having this functionality, thanks for tackling this issue! 75.144.133.76 18:28, 3 December 2010 (UTC)

Hi!

Sorry for this error. I don't know what's happening... I don't even have a extensions/Vector/Vector.php file on my local repository... I will try to solve that when I have some time, which might not be soon...

As far as I know, there is currently no decision about when this will be merged to trunk. Maybe in 1.18...

You can view all the commits made on my branch here: [1].

Regards. Peter17 19:00, 3 December 2010 (UTC)

Also (on prototype) Fatal error: Call to undefined method MediaWiki::getAction() in /srv/org/wikimedia/prototype/extensions/InputBox/InputBox.hooks.php on line 55. Presumably this is due to mixing versions of extensions that expect revisions of mediawiki that havn't been merged into your branch, or something along those lines. Bawolff 15:03, 19 January 2011 (UTC)

Indeed, MediaWiki::getAction() was added in r78512. The InputBox extension was updated to use it, but the iwtransclusion branch doesn't have that method yet. Reach Out to the Truth 16:20, 19 January 2011 (UTC)

---

Hello all, I'm also geting the same error. I just installed wiki and tried to add vector extension. "PHP Fatal error: Unsupported operand types in /Applications/MAMP/htdocs/lang/extensions/Vector/Vector.php on line 109"

interWiki authentication

Hi Peter, This may or may not be relevent, and may even be against Wiki ethos (i.e. share information :-) but I'm looking to use the transclude functionality to share internal information between two Wikis, but would like to retain user authentication to not display pages to the world.

I'd like to be able to still transclude those pages though on another Wiki - to users authenticated on that Wiki.

What would be useful would be a mechanism for one Wiki to authenticate against another in order to serve up/or pull that transcluded data...

Best wishes Brett

Hi Brett,

Well, this project is almost finished and it is not really the good page to post this.

If I can suggest, you should post a bug report on Wediawiki's bugzilla to ask for this functionality.

Regards

Peter17 19:53, 24 March 2011 (UTC)

Thank you

This is good stuff. Keep up the great work :)

--Craig131 (talk) 05:36, 11 January 2013 (UTC)

Current Status?

While checking the possibilities for using interwiki and transclusions together I found your great project here. But what I haven't found is a message that it is completed/implemented into a certain version or such. Maybe I have just read over it, but it would be great to get some informations about your progress 195.88.117.25 16:29, 3 September 2015 (UTC)Dang, forgot to log in last time
Ciannicay (talk) 12:22, 4 September 2015 (UTC)