So... there are two possible different problems: first one was that wysiwyg converted character strings [[ and ]] in wrong way. I am not sure anymore if my modification with these characters were right. Second problem is your relative links.
On my installation of MW1.26.2 (no extra plugins than wysiwyg) result of relative link is like in picture Relative_link.png (page has been edited with wikieditor). As soon as link text begins with two dots, my installation of mediawiki does not display it as link, it is displayed as text. After my latest modification with wysiwyg I can also edit page using wysiwyg and that line of text containing two dots can be saved without problems.
Is it so that you have enabled subpage feature in your wiki in case relative links work in your system? ... this is enabled only with some namespaces by default, see Subpage feature and Relative links. It seems like subpages are somehow stored in different way than "normal" pages in MW. If I create pages startting with "/", then I can get relative links to these pages work with MW. It looks like wysiwyg can not handle this kind of links properly and treats it as text.
I do not quite understand what you mean... are you saying that you have modified some files of mediawiki in order to make relative link work in your system or have you made some changes with files of extension wysiwyg? If you have made changes with files of wysiwyg, could you please post your changes so that I can try to learn from them and merge possible valid fix into wysiwyg extension.
To understand you better, please use shorthands "MW" if you are referring to mediawiki, "wysiwyg" if you are referring to extension wysiwyg, "wikieditor" if you referring to extension wikieditor (text based editor of mediawiki), visual editor if you are referring to mediawiki extension visual editor, which is completely different way to edit wiki pages and has nothing to do with extension "wysiwyg".