Wikimedia projects down due to power problem in primary data center
Posted by Mark Bergsma in outage, power on July 5th, 2010
Starting at 0:10 UTC on July 5th, the Wikimedia Foundation suffered from
intermittent, partial power failures in the internal power network of
one of its main data centers in Tampa, Florida. Due to the temporary
unavailability of several critical systems and the large impact on the
available systems capacity, all Wikimedia projects went down. The power
situation stabilized at 1:12 UTC, and systems and services recovery has
been taking place since. We expect all projects to be back online and
editable around 4:00 UTC.
Usability: Why Did We Move The Search Box?
On May 13th, we changed the default appearance of the English Wikipedia to use the new look developed as part of the Wikimedia Usability Initiative. On June 9th, we unveiled the new look in the remaining top 9 languages (by access volume). Other languages will follow in the coming weeks.
The key elements of the new design had been in public beta testing for many months, and hundreds of thousands of users had already adopted the new look. But, nothing compares to the real thing, and we tried to make the switch as painless as possible — by offering a quick way back to the old layout, by explaining our reasoning, observing and listening to comments carefully, fixing bugs and implementing changes quickly.
The single most frequently expressed concern about the changes we’ve made is the relocation of the search box from the left sidebar to the top right corner. This blog post will give an extended explanation of why we made the change, the other changes we made to the search, and what we’re planning to do next.
The old search box location
The default location of the search box in MediaWiki, the software used by Wikipedia, is below the “navigation” box in the top left corner. This was also the location in the English language Wikipedia, as well as many other language editions. Some language editions, including the German one, had customized the location of the search box, and moved it directly below the logo.
What do we know about search usability?
There are essentially three factors that influenced our decision to relocate the search box:
- common user expectations regarding the placement of the search box on web pages, as determined by the preexisting body of usability research;
- usability research regarding ideal search box width, and implications for the search box placement in our layout;
- ability of our test subjects to locate and use the Wikipedia search box, as determined by Wikimedia usability tests in a research lab.
There are several scientific studies that have examined the ideal placement of common objects on web pages. One early study by Michael Bernard conducted in 2001 by surveying participants regarding the expected placement of web objects such as internal links, external links, and search found that both new and experienced web users “generally expected internal search engines to be located in the upper and bottom-center of a web page. A smaller number expected it to be located at the top right of the page.”
This study was followed up five years later by A. Dawn Shaikh and Keisi Lenz (”Where’s the Search? Re-examining User Expectations of Web Objects”) in a survey of 142 participants. The study found that expectations had changed significantly, especially regarding the placement of the site search engine. The figure below illustrates the areas where participants expected the search to be found:
Expected location of site search engine
As the authors speculate and as seems intuitively plausible, early expectations of the placement of the search box were likely driven by the fact that search was commonly associated only with search engines of the time like AltaVista, not with site-specific searches. As more and more sites developed internal search functions, those were increasingly placed in slightly less exclusive screen real estate than the top center, shifting users’ expectations to look for search features in the top right corner.
Another factor that may have influenced user expectations is the common placement of search engine features in the top right corner of the web browser window.
There are practical advantages of positioning the search in the top right. As summarized in this research paper, several usability studies have pointed out a key advantage of navigational elements being placed on the right: it gives immediate access to the browser scrollbar. This is particularly valuable when a) scrolling up and down a list of search results, b) scrolling up and down an article you’ve just called up for information.
Search box width, and placement implications
The old search box is approximately 20 characters wide, the new search box accommodates 24 characters. More importantly, due to the placement of the old search box in the sidebar of the layout, widening the search was impossible without either relocating it or widening the sidebar.
The search box placement in the top right allows us to maintain a fixed standard width from one page to the next, while giving us maximum flexibility as to what that width should be. To make it even easier for users, we are experimenting with an expandable search, which is currently deployed in our sandbox 3. When you click the box, it will expand significantly to the left. We may or may not end up deploying this feature as we continue to look at ways to make search more accessible and user-friendly.
Our own research
In the course of the usability and user experience work since last year, we have so far completed a total of three usability studies, all of which are documented on the usability wiki:
- Usability and Experience Study (March 2009)
- Usability, Experience, and Progress Study (October 2009)
- Usability, Experience, and Evaluation Study (March 2010)
These studies included both remote and San Francisco based participants. While the primary focus of our studies were obstacles people encountered when editing, finding search in the navigation was clearly one of them, and our test subjects tended to resort to common web search engines to navigate Wikipedia instead of using the site’s own search. With the new search box placement, users’ ability to find and use the site search was markedly improved. One user intuitively used the search box in its new location and then consciously realized that it had been moved. To see videos of the other subjects finding and using the search box with ease, please see here.
For those unfamiliar with usability testing, it’s important to note that small samples and agile, iterative tests are commonly understood to be an effective method for discovering most key user interface issues. Our sample sizes were actually larger than strictly necessary, and more diverse than typical due to our use of remote testing methods.
With that said, we didn’t test the English Wikipedia against other languages which had placed the search box directly below the logo, and we recognize that this alternative placement is already an improvement to match user expectations. However, based on the cited research above, as well as the design reasons for moving the search box to the top right, we still believe that the overall case for moving the search is compelling even for those languages, if slightly less so.
So .. why did you move the search box? I liked it where it was!
In sum, we moved the search box to a) match web practices and user expectations, b) make it possible to widen it consistent with common usability recommendations, c) in response to actual observed problems of test subjects when using the old search.
We also recognize that millions of Wikipedia users had adjusted to the old placement, and will now have to re-adjust to the new placement. However, Wikipedia’s global audience grows by tens of millions of users every year (it is currently at 375 million unique visitors/month world-wide), and we hope to grow it by hundreds of millions in this decade. That will require that we adapt to common user expectations, rather than expecting every new user to adapt to us.
This will unfortunately inconvenience those who have adapted to the old placement. Do we absolutely know that to be the correct decision? No, but the fact that existing users are temporarily inconvenienced by it is not at all indicative that it is not.
Other search changes we made
It’s worth noting that the search box placement isn’t the only thing we changed about the search function. Perhaps most notably, the old search had two buttons (”Go” and “Search” in English). If you asked even an experienced user what the difference between those buttons was, you would get wildly different answers, and bug 577 had been open since 2004 because of this.
To answer the mystery: the “Go” button attempts to find an article with the same title as the entered search term and, if it fails, runs a full-text search of all articles. “Search” will always run the full-text search. “Search” is necessary where you want to search for a word instead of displaying the article of that title (say, you want to search for instances of “George W. Bush” all across Wikipedia).
In the new design, the less common case (search all across Wikipedia for a phrase, regardless of exact match) can be accessed using the “containing …” option in the drop-down menu. We believe this is both a more discoverable implementation, and it reduces overall clutter and complexity of the search.
Measures and coming changes
We are monitoring overall search volume. In the first week since the deployment, we have observed neither a statistically significant increase nor a decrease in search volume, but it’s too early to draw conclusions. There are also confounding variables. As noted above, the search box has changed not just in placement, but also in appearance and behavior. Finally, search volume isn’t the only interesting metric: search convenience (how long does it take users to, on average, find the search) is another one.
We’ll try to get our hands on solid metrics, but we’ll also continue to look for ways to make search more user-friendly (such as the auto-expansion), fix bugs, and so forth. In continuing our efforts to improve the user experience of all our projects, both for new and experienced users, we’ll try to share our thoughts with you frequently, and work with you to figure out the right answer. And, if you just can’t get used to the new search — you can always switch back to the old layout, which will continue to be there for you.
Warmly,
the User Experience Team
Wikimedia Foundation
Upcoming code changes for Flagged Revisions
Posted by William Pietri in wikimedia on June 14th, 2010
As mentioned in this post in January, the English Wikipedia will be trying out the Flagged Revisions extension, using a configuration we’re calling Pending Changes.
This new configuration requires new features, which in turn required substantial code changes to Flagged Revisions. For technical reasons, we can’t release that code just to the English Wikipedia, so we will upgrade all copies of Flagged Revisions in use on Wikimedia Foundation projects. Happily, that will result in a number of minor improvements for all Flagged Revisions users.
We’ll be rolling this code out on Monday, June 14th, around 23:00 UTC. We believe this won’t cause any trouble, but if bugs do crop up, we’d like to hear about them. Our team will be standing by, prepared to fix any small issues immediately, or roll back if there are any big problems. The best way to contact us will be via the wikimedia-tech IRC channel or our issues page. (If you happen to read this blog and are a participant in a wiki that uses Flagged Revisions, we’d appreciate you checking your local Village Pump around then, so as to minimize language issues.)
For more information on the rollout of Pending Changes for the English Wikipedia, keep an eye on the main Wikimedia Foundation blog for an upcoming post, and check out the Pending Changes help page and the details of the trial.
XML dumps resumed
Posted by ArielGlenn in mediawiki, software, wikimedia on May 29th, 2010
Folks that use XML dumps of our projects will know that the dumps process has been stalled while we investigated bug 23264. We have been running individual project dumps manually and asking people to inspect them carefully. We have just started the automated dumps up again, and various code fixes should be checked in shortly. Thanks to all for your assistance and your patience.
If you are working with the XML dumps of the English language Wikipedia containing all page revisions (pages-meta-history), please note the following issues with the two completed runs.
The January 30 run is missing the text for a large number of old revisions of articles, primarily revisions created between January 1 2005 and May 14 2005. This was due to bug 20757 which was subsequently fixed. If you are doing analysis using the text data, you can retrieve the missing text by extracting it from an earlier file; see the archives.
The March 12 run is incomplete; it is missing about the last third of the revisions, due to early termination during the compression step.
The stubs files and the current page dumps appear to be fine, so statistical or other analyses that only use these files should not be impacted. The mysql table dumps are also unaffected.
We apologize for the inconvenience and are working on getting out a set of complete full history dumps with all revision text intact.
Commons Gets Collapsible Navigation
Posted by Trevor Parscal in wikimedia on May 5th, 2010
Collapsible navigation for Vector's side bar
Wikimedia Commons now has a new feature, collapsible navigation for the sidebar of the Vector skin. Now you will see an arrow icon next to the title of each section. Each section title can be clicked to hide and show the links within the section. We refer to this hiding and showing action as expanding and collapsing.
The motivation for this feature came from observing users being distracted by and lost within the sidebar while trying to find various links. This solution helps to tuck away less frequently accessed items, without making them entirely inaccessible or fatiguing to access frequently for users who want do make use of them.
Part of our research also included tracking the frequency of clicks on sidebar links to discover which items were used the most. The results of our click-tracking experiment guided our choice to make the first navigation section always visible, the second section initially visible but collapsible, and all other sections after that initially collapsed but expandable. As you expand and collapse each section, your preference is saved on your computer so that it’s remembered between pages and sessions.
Simplified Search for Vector
Posted by Trevor Parscal in mediawiki on May 4th, 2010
The Vector skin differs in several distinct ways from its predecessor Monobook. One of the most prominent distinctions is the location of search controls, now in the top-right of the screen. User-testing has shown that this change has improved users’ ability to find and access the search controls. But there still remained some obvious areas for improvement. Our work to ensure that projects running on MediaWiki can be accessed from a wide range of devices brought us to conclude that the search controls were too space-consuming, specifically taking up too much horizontal space that is otherwise used for displaying top-level menu items. Meanwhile in several of our user experience studies we found that the search controls are generally confusing to users, specifically they did not understand the distinction between go and search.
Simplified search interface for the Vector skin
To solve these problems, we’ve developed a simplified search interface for Vector. The “Search” and “Go” buttons are gone, but their functionality live on. As you type, search suggestions are offered and accessible via the mouse or keyboard using the up and down arrow keys. “Go” is still the default action, executed by pressing the enter key on the keyboard. To perform a full-text search, users can click on the “containing” option within the search suggestions or press the up arrow key on the keyboard. Also, the new search uses less horizontal screen real-estate, making more room for top-level menu items.
You can experience the new search interface by visiting our prototype site.
The change in interface is coming – Part 2
Following-up the announcement by the user experience team on March 25, the default user interface of Wikimedia Commons was switched to the new user interface on April 5th. The feedback from Commons users is found here. The majority of logged-in users of Commons continue using the new user interface, while about 9% of logged-in users switched back to the previous interface. After the switch was made to Commons, Ops team has determined that extra system resources is recommended for precautions as transitioning to the new interface while supporting the existing interface can tax the cache system severely. The new hardware is being installed, and we have a scheduled date for switching the user interface of English Wikipedia.
The default interface switch of English Wikipedia will take place at 5:00am UTC on May 13. The deployment process is expected to last for two to three hours.
Maintainer of user scripts, style customization and gadgets are recommended to review the previous post that describes the impact of these changes on bots, scripts, and gadgets interfacing with Wikimedia projects
We are building a frequently-asked-questions page. Will you help to compile the topics you find missing by posting topic ideas to the discussion page? We want to make sure that we provide answers to anticipated questions.
We plan to conduct wider outreach about the upcoming changes to our global user base in the coming week, which would likely include press outreach, blogging, and developing more public-friendly questions and answers. We’ll share more info soon on the Wikimedia blog.
Thank you for your interest,
Naoko Komura
User Experience Programs
WMF announces our Google Summer of Code 2010 projects
Posted by danese in summer of code on April 26th, 2010
Once again in 2010, Wikimedia Foundation is participating in Google Summer of Code. I’m happy to announce that we’ve selected six students to participate this summer:
- Extension management platform
Student: Jeroen De Dauw
Mentor: Brion Vibber
Goal: Creating an awesome extension management platform for MediaWiki, facilitating the installation, updating, removal and configuration of extensions. (student application) - Improve metadata support
Student: Brian Wolff
Mentor: Chad Horohoe
Goal: Improve metadata support for uploaded media in mediawiki by displaying embedded IPTC and XMP metadata (student application) - General RDF export/import in Semantic MediaWiki
Student: Samuel Lampa
Mentor: Denny Vrandecic
Goal: Extend the import/export functionality of Semantic MediaWiki (SMW) to allow also full, general RDF import. (student application) - Javascript overhaul of Semantic MediaWiki
Student: Sanyam Goyal
Mentor: Yaron Koren
Goal: Improve and extend the Javascript for Semantic MediaWiki and some of its spinoff extensions, most notably Semantic Forms – this would include transferring over much of the Javascript to use the jQuery library, which is now becoming a MediaWiki standard. (student application) - Wikisource Legal Tool
Student: Stephen LaPorte
Mentor: Ariel Glenn
Goal: Creating a tool to format judicial decisions, legal scholarship, and statutes for Wikisource. (student application) - Reasonably efficient interwiki template transclusion
Student: Peter Potrowl
Mentor: Roan Kattouw
Goal: The aim is to allow MediaWiki users to insert (transclude) templates from a wiki to another on WikiMedia Foundation (WMF) wikis (Wikipedia, Wikimedia Commons, etc.). (student application)
We had an exceptional set of really great proposals this year, and an engaged mentor group helping with the selection process. It was both wonderful to have so many choices, and really sad that we couldn’t pick them all, but in the end, we had to narrow the list down. Our six slots represent 100% growth from previous years’ Summer of Code engagements and that’s a pretty exciting stretch.
To the students that weren’t selected: do know that we were inspired by the quality level of all of the proposals, and we had to turn down some really exceptional proposals. Please don’t be discouraged, and do consider us next year!
To the students selected: congratulations! Welcome aboard! We really look forward to working with you to make sure you are successful and have a great time in the process.
To everyone volunteering as a mentor who helped with the selection process: thank you for your effort and dedication! There was a lot to sort through, but I think we can all feel great that we have a group of very capable students on the case this year thanks to your work.
More detailed info available here.
Towards the Fun!
Template folding
Based on several usability studies, the usability user experience team has identified that template text and syntax is a major hindrance to new users, making them feel less comfortable editing pages.
As such, one approach that we’ve been experimenting with is collapsing templates into expandable “capsules”. This improves the readability of the wikitext.[1]
The full wikitext of the template is available with the expansion arrow. Additionally, a more user-friendly template editing form is available by clicking on the template name or the ‘pop-out’ symbol to the right of the name.
Since this is an experimental feature that is largely proof-of-concept, it does have a few limitations:
- Currently only works on Firefox with the editing iframe enabled
- Pasting content into the expanded template (or inserting a newline in Linux) can break the template, depending on the source of the content.
- The implementation is relatively slow, so slower and older computers can appear to hang, especially on pages with large templates
- Templates are not converted into capsules as you type; only templates that were there on initial page load are wrapped
We’re still working on these, but in the meantime, test it out on our sandbox[2] and let us know what you think!
[1]We’re working on making the displayed name more customizable on a per-template basis so the collapsed version more accurately summarizes what it’s collapsing, ie displaying the title of an infobox rather than the word “infobox”.
[2]This is currently prepopulated with some articles about large US cities. For some good examples, check out:
New York City, Boston, or Chicago
The change in interface is coming
Starting the week of April 5th, the Wikimedia Foundation will begin rolling out changes to the default settings on all projects. Wikimedia Commons is planned to be switched over first, and English and other language Wikipedias, and our sister projects will follow as our development and operations teams are ready.
What will change?
The default MediaWiki skin in all Wikimedia Foundation projects will change from Monobook to Vector, and the editing toolbar will be replaced with the new enhanced editing toolbar with dialogs for inserting links, tables and more. These features will be familiar to users who have chosen to opt-in to the beta that has been available since August 6th, 2009.
Who will be affected by this change?
Anyone using the site anonymously, or logged in users whose settings are set to the current site defaults. Users who are affected by these changes may also have user scripts and styles which may not perform similarly to how they did before the swtich-over, or may not be loaded at all. Also, some gadgets, community-developed features available through the user preferences, may not be compatible with these changes. Users who have enabled those gadgets may experience issues related to these incompatibilities. Fortunately, many of the most popular gadgets have already been adapted to Vector during the beta testing period.
Finally, external tools which use screen-scraping to access content rather than the API may be affected by this change, as the HTML structure of Vector is different in some ways from that of Monobook. If, on the other hand, you have built an application that relies on the MediaWiki API to process content, your application will not be affected by this deployment.
How can users make sure their user scripts, user styles, gadgets and external tools will still work?
By testing them with the new settings, you will be able to verify compatibility and resolve any issues. By clicking the “Try Beta” link at the top of any page, and then opting into the beta, you will have turned on the same settings which will be made default.
Another way to test skin compatability is to append “?useskin=vector” to the end of the URL of a page.
For example, the URL http://en.wikipedia.org/wiki/Wikimedia_Foundation would become http://en.wikipedia.org/wiki/Wikimedia_Foundation?useskin=vector to see how the page would look using the Vector skin.
What if users decide they want to return to the previous settings?
When a site is switched over to the new settings, the “Try Beta” link at the top of each page will be replaced with a set of links which will provide more information about what changes have been made and a convenient way to change users settings back to how they were before the switch over.
Naoko Komura and the Wikimedia Foundation User Experience team
The power of translators
Posted by Naoko in mediawiki, open-source, software, wikimedia on March 25th, 2010
Wikimedia projects support over 270 languages. This amazing global reach is powered by volunteers who translate not only the contents but also text used in MediaWiki so that localized wikis can be easily navigated and operated by users in their local language. Translatewiki.net is the amazing translation engine which not only supports Wikimedia projects but other open source projects. Siebrand and Nike are leading this translation platforms.
The user experience programs at Wikimedia Foundation is also benefited from translatewiki.net and translation volunteers. The usability beta has been completely translated into thirteen languages and twelve languages are 99% complete. These stats can be found at the translation completion status page for the usability extension by courtesy of GeardM.
The usability beta is planned to be switched to be the default interface in April. Additional translation boost for languages which are not fully translated will improve the usability of the new interface greatly.
GerardM had a great example of the interface in Nepali, whose localization is not complete, in his blog.
Translation help for such as Indonesian, Greek, Thai, Arabic, Hebrew, Italian, Sinhala, Korean and much more, are greatly appreciated.
Global Outage (cooling failure and DNS)
Posted by Mark Bergsma in wikimedia on March 24th, 2010
Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries.
However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.
We apologize for the inconvenience this has caused.
Update: Unfortunately, for many, this outage seems to have lasted longer than an hour. It appears that many ISPs’ DNS resolvers do not honor the so-called Negative Cache TTL that we send (1 hour), and instead use a longer value. We have circumvented this problem by renaming the affected DNS record to something else.
Update 21:32 UTC: Our SSL gateway, secure.wikimedia.org, was disabled due to overload issues, but is now back up.
Registration open for the Developer Workshop in Berlin!
Registration for the Developers’ Workshop in Berlin on April 14.-16 is now open: please use the registration form. Registration is required and will be open until March 21., but there are only 50 places available. So, sign up soon!
Wikimedia Germany invites all MediaWiki developers, Toolserver users, Gadget hackers, and other people interested in the technical side of Wikimedia projects to come to the Workshop. We have a very nice venue and a cool option for accommodation, details to be announced soon.
For updates and more information, watch meta:Wikimedia_Conference_2010/Developers’ Workshop. You can also get updates via twitter or identi.ca. If you have questions, please contact us at conference@wikimedia.de.
Wikimedia Bugzilla upgraded to version 3.4.5 with REST APIs
Posted by Priyanka Dhanda in wikimedia on February 26th, 2010
This morning we upgraded the Wikimedia Bug Tracker to Bugzilla version 3.4.5. The release notes are available here.
For those of you who have been eagerly awaiting the REST APIs, wait no more. You can now find our Bugzilla server’s APIs at https://bugzilla.wikimedia.org/bzapi. Documentation about the APIs is available at https://wiki.mozilla.org/Bugzilla:REST_API.
If you find any issues with the bugzilla instance or the APIs, submit a bug. Make sure you set the Component to Bugzilla so that it gets my attention.
Server Decommissioning Donations
We have been upgrading and adding new servers for capacity increases as we always do, unfortunately, thanks to the more updated tech, some of our older systems just are not worth the space in our racks. These are servers that Wikimedia has used on the projects for 3+ years, so they are out of warranty. These are good systems though, and while we may overload them and want replacements, they are fine for many, many non-profits to use.
Most systems (but possibly not all) have the following specifications:
- Dual CPU 2.5 GHz AMD (some intel, but most AMD in this batch.)
- 3-4GB RAM Each
- Most have 80 GB or larger HDD (Some have two hard drives, some drives are 160GB or possibly even 250GB)
Disclaimers: The Wikimedia Foundation does not guarantee the operation or use of these servers in any shape or form. They are old, some may have dying fans, bad hdd sectors, and the like. Servers have been wiped of information, and they ran through that, but no promises on function! Also, most servers have rails, but occasionally one may not, and we do not sort through them for these things. However, they are standard supermicro servers in most cases, and getting replacement rails is fairly simple. Some servers are well over 3 years old, we do not just turn off servers when they hit the 3 year mark, we turn them off when they are no longer worth using in any role or function on our cluster in a reliable manner. In most cases, it is simply the hardware technology has updated to the point that a new server is much faster, and since we demand high performance of our servers, it is worth upgrading for our needs.
Now, last time we opened this up, we got a lot of individuals asking for servers for various uses. Some requests were not very clear, and overall we ended up with a lot of stuff to sort through to find those eligible for these servers.
We try to only donate servers to other non-profits whose core values are similar or in support of our own. This means we do not donate them for individual use. Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent. This means that we cannot, in good conscience, donate these servers for profit or personal use to individuals or corporations.
If you would like to receive some of these servers for your NON-PROFIT use, please email servers@wikimedia.org. TO BE ELIGIBLE YOUR EMAIL MUST INCLUDE THE FOLLOWING:
- Subject: Server Decommissioning Donations for <NONPROFIT NAME HERE>
- Your name, contact information, relationship with non-profit requesting the servers
- Registered non-profit name and information.
- Information on the non-profit, who they are, what their mission statement and goals are.
- Shipping address information for a FedEx Ground delivery to where the servers need to go.
- How the servers will be used. (We like to know and share with folks!)
Please keep in mind that deciding where these go is pretty tough, so the more detailed you can be in your email is best. (IE: ‘Wikimedia uses these for our sites.’ is pretty vague where ‘Wikimedia is the non-profit foundation that runs Wikipedia. Server donations to us would be used to run our websites that allow access to Wikipedia and its sister projects.’ is a lot nicer. ;) Also, by submitting and possibly accepting servers from us, you are giving us permission to post about it here on our technical blog.
The submission period will remain open no less than two weeks from this posting.
The last time we did this was in September 2009, which took a long while to get sorted out! We try to do these in multiple, smaller batches, it just easier on operations that way.
If there are any questions or concerns, anyone can feel free to email me @ the servers@wikimedia.org address, or you can also feel free just to post the questions in the comments. (Which is even better, avoids duplicate questions!)
Rob Halsell
Systems Administrator
Iframe bugs
The usability initiative recently deployed some changes to the beta features, most notably the replacement of the textarea on the edit page with a content-editable iframe (mistakenly referred to as a “rich text editor” by some users). Users at various wikis have reported some issues resulting from this change. We’re fixing these bugs as fast as we can and deploying fixes as soon as possible (once properly tested of course). Below is a list of bugs associated with the rich text editor and their status; we’ll keep this page up to date as we push out fixes. If you’re bothered by these bugs and don’t want to wait for fixes, you can turn off the content-editable iframe by leaving beta (click the “Leave Beta” link at the top of any wiki page).
UPDATE Feb 12 21:01 UTC: It’s been reported that the loading cover doesn’t go away in Firefox when certain gadgets are enabled. For this reason we’ve removed the loading cover for now.
UPDATE Feb 12 22:31 UTC: Because there’s still some issues with the iframe and most users are just using the toolbar, we’ve disabled the iframe for users that only use the toolbar. This means that if you disable the navigable table of contents in your preferences, you shouldn’t experience any of the iframe-related issues (such as those with pasting text).
UPDATE: Around midnight UTC in the night between Feb 12 and Feb 13, we’ve disabled the navigable table of contents and the enhanced dialogs until we work out the remaining bugs.
Roan Kattouw (Catrope)
- bug 22311 (Pressing tab in edit box should go to summary box): fix deployed
- bug 22393 (Charinsert tools at the bottom of the edit page and old toolbar don’t work): fix deployed. Charinsert may have to be fixed in site JS on some wikis, depending on the local Charinsert implementation; this has been done on the English and Portuguese Wikipedias as well as Commons; I checked the other top 10 Wikipedias but they didn’t need a local fix. You may heve to Shift-Refresh for this to work on Commons.
- bug 22394 (Pasting formatted text preserves formatting): fix deployed.
- bug 22398 (Extra line breaks inserted when pasting text): fix ready, will be deployed soon
- bug 22402 (Leaving beta doesn’t turn off experimental features like TOC and dialogs): fix deployed.
- bug 22413 (Old toolbar disappears when TOC enabled): inadvertently fixed when bug 22393 was fixed (love it when that happens)
- bug 22428 (Line breaks in pasted text not saved in Safari, Chrome): fix ready, will be deployed soon
- bug 22435 (Literal and <p> in article text gets removed on save): fix deployed.
- bug 22440 (Random cursor jumps in Firefox 3.5 and above): fix ready, will be deployed soon
Tech folk will again meet in Berlin
Developer Meet-Up (by Raymond, CC-BY-SA)
Wikimedia Germany invites all MediaWiki developers, Toolserver users, Gadget hackers, and other people interested in the technical side of Wikimedia projects to come to Berlin for a Developer Meet-Up on April 14.-16. Last year’s meet-up in Berlin was a great success, and we hope to make it even better this time! This year we want to focus on structured (meta) data, search, and community building. The future of the Toolserver will also be a subject.
The dates are set, but it’s not clear yet if we start full throttle on Wednesday the 14th or if we have just an arrival event on that date and a full day on Friday the 16th instead – this depends on venue arrangements that are not sorted out yet. Note that registration in advance will be required – a website will be set up for this soon, we will announce it on blogs and mailing lists.
On that Friday, April 16., the Wikimedia Chapters and Board start their convention in Berlin. This will be a great opportunity to meet, to discuss interesting topics, to network and to exchange ideas and thoughts! Wikimedia Germany will host the event, so we will organize the venue, the hotel(s), some fun things to do in Berlin, food & drinks and lots of other things – and there might even be a party at the c-base again…
See you in Berlin!
Update: Registration is now open!
Wikimedia donates servers to deserving non-profits.
Posted by RobH in hardware, open-source, wikimedia on February 3rd, 2010
Every year, Wikipedia usage goes upward, and every year the technical folks working and volunteering with Wikimedia have to plan, purchase, and implement new servers to keep up to the growing popularity of Wikipedia and its sister projects. With the advances in computing, running 9 new application servers this year took the load of 36 application servers from 3 years ago.
So when we upgrade, what happens to the old equipment that is too slow for Wikipedia, but not too slow for MANY other non-profits? We donate them! These systems were 1U rackmount servers, dual cpu 2.5-3, single core, 2-4GB of RAM, and 2-4 HDD Bays with 1-2 80-250GB HDDs. This year, we have three non-profits who received our older systems (in alphabetical order): Drupal.org, OpenStreetMap Foundation, and Sugar Labs.
Drupal is a free software package that allows an individual or a community of users to easily publish, manage and organize a wide variety of content on a website. Tens of thousands of people and organizations are using Drupal to power scores of different web sites.
The OpenStreetMap Foundation is an international non-profit organisation supporting but not controlling the project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anybody to use and share.
OpenStreetMap is an open initiative to create and provide free geographic data such as street maps to anyone who wants them.
The mission of Sugar Labs® is to produce, distribute, and support the use of the Sugar learning platform; it is a support base and gathering place for the community of educators and developers to create, extend, teach, and learn with the Sugar learning platform.
We hope the recipients of our servers will be able to put them to good use!
Below are some common questions involving Wikimedia and the server donation process:
Q. How can I get some of the decommissioned donation servers?
A. The best place to follow the goings on of our technical team is here, on the Wikimedia Technical Blog. When we have a batch of servers up for decommissioning and donation, we will announce it on the tech blog, and instructions on how to apply to receive some servers.
Q. Who is eligible to apply for servers?
A. We try to only donate servers to other non-profits whose core values are similar or in support of our own. This means we do not donate them for individual use. Since these servers were purchased with donations to support Wikimedia, we feel we need to further donate them to other like-minded organizations, since that is how the money for the servers was meant to be spent.
Q. How often does this happen?
A. Most servers are kept in use by Wikimedia beyond three years. Many of our servers that we have turned off in this batch are anywhere from 3 to 5 years old. We only replace them when it makes sense from the technical standpoint to do so. This means we cannot just say ‘we will do this every X months.’ We try to get the most use out of every server, as they were donated or purchased with donations. So there is no set date, just keep checking the Wikimedia Technical Blog, when we have more to donate, we will say so there!
Q. I am a student/person/so and so, and I want to learn to develop and do such and such. Can you send me a server?
A. Sorry, unfortunately it is just not realistic or fair of us to try to sort out which personal use requests for servers are legitimate and which are folks wanting computers for any other reason. We choose to limit our donations to other like minded non-profit organizations.
Rob Halsell
Systems Administrator
Deployment of Babaco Enhancments
Posted by Trevor Parscal in mediawiki on January 27th, 2010
The Usability Team is preparing a supplemental release which will bring more stability and functionality to the features of their Babaco release, which has been available to logged in users of Wikimedia sites since October 2009. Among the changes which have been made are many improvements in interactivity and aesthetics, but the most critical change is using an HTML iframe element together with a special design mode that modern browsers support, in favor of the previous HTML textarea. This paves the way for developing a rich editing experience with collapsible templates and syntax highlighting, as well as provides a foundation on which a WYSIWYG editor may eventually be built upon.
The table of contents, which now features controls for expanding, collapsing and resizing, is also much more accurate than before. The link dialog which once had a tabbed interface for making internal or external links now intelligently detects the type of link you are making, an improvement we designed and prototyped while literally watching users struggle with the software during usability testing. Finally, there is now support for language specific icons in the toolbar, with a several languages ready to go such as German, French, Spanish, Dutch, Portuguese, and Polish. This feature allows us to provide a more native experience by using language specific mnemonics. So far we’ve applied this feature to the bold and italic buttons, which are now B and I for English, F and K for German and G and C for Spanish. Languages without customized icons will continue using A and A.
The team will be deploying these upgrades in the coming days. To learn more about the Wikipedia Usability Initiative, check out their website at http://usability.wikimedia.org.
Bugzilla upgrade.
Posted by aZaFred in linux, open-source, wikimedia on January 18th, 2010
Thanks to Priyanka’s wonderful work in theming Bugzilla and ironing out the last couple bugs and extensions, we are finally ready to move on with the upgrade.
As a side effect, Bugzilla will be down for a couple of hours (let’s say 2 to be on the safe side) around lunch-time. (Edit Addition: 2010-01-19 between 20:00 GMT and 22:00 GMT)
Flagged Revisions: Your questions answered!
Posted by William Pietri in mediawiki, wikimedia on January 6th, 2010
There has been a lot of interest lately in Flagged Revisions, a quality control mechanism for MediaWiki. In particular, people want to know when and how that’s getting used on the English language Wikipedia.
I’m William Pietri, a San Francisco software consultant who recently came on part time to do project management for this. In addition to my long experience building web software for on-line communities, I’m also a Wikipedian. Although I haven’t done much more than small corrections lately, I started editing in 2004, and became an admin in 2007.
My job on this has two main parts. The first is to make sure that everybody working on this gets everything they need to make progress. The second is to communicate progress to the wider world. In the spirit of open communication, this is an update in question-and-answer form. Mostly from real questions people have actually asked me, but I’m going to sneak in a couple that I expect somebody will ask shortly.
What is Flagged Revisions?
You can find more detail here, but Flagged Revisions is basically a way to insert a quality review step between someone editing an article and that article version being published for the general public to see. It has been in use on the German Wikipedia since May 2008, and implemented in other languages and projects since then (see this page on Meta for a full list). Typically, in those use cases, every single article is treated in this way, and every change by a new user has to be reviewed. There are a number of ways Flagged Revs can be used, and the proposed implementation for English Wikipedia (described below) is quite different.
Fundamentally, the objective of this technology is to reduce the exposure of readers both to subtle and not-so-subtle malicious changes in articles (whether it’s the insertion of blatant nonsense, or claiming the death of a celebrity), and to reduce the workload of people patrolling these changes by reducing duplication of effort.
What about the English language Wikipedia?
The use of Flagged Revisions on the English Wikipedia has been under discussion for a long time. Ultimately, the English Wikipedia volunteer community developed a proposal titled “Flagged protection and patrolled revisions“, which garnered strong support. It is fundamentally different from the way the technology has been used so far. Instead of requiring every change by a new or untrusted user to be reviewed, the mechanism would be activated on a per-page basis only, as an alternative to existing tools to restrict editing.
Notably, thousands of articles in the English Wikipedia, typically pages with a very high risk of malicious editing (e.g. major political figures), are currently “semi-protected”, meaning that new or unregistered users cannot make any changes at all. This new tool would make it possible to open up these pages for editing, provided that potentially problematic changes receive positive review. As a result of the more open approach, more high-risk pages could be made subject to this level of community moderation.
Initially, the English Wikipedia volunteer community wants to trial the system for two months. In addition to this alternative to page protection, the proposal calls for implementation of a new feature called “patrolled revisions”, which doesn’t impact what readers see, but is designed to make it easier for change patrollers to organize their work.
How is the Wikimedia Foundation responding to this proposal?
The Wikimedia Foundation (WMF), together with Wikimedia Germany, has driven and funded the development of the FlaggedRevisions technology since 2007 to the point where it has been able to scale to more than a year of production use in our second-largest project, the German language Wikipedia. WMF has carefully reviewed the English Wikipedia proposal, and allocated resources to assess its impact and support implementation along the principles outlined in the proposal.
The technology as proposed markedly differs from the way it’s been used before, so it’s a substantial development and design effort to get it right. For example, the notion of a per-page setting necessitates an entire set of user interface changes that allow changing the setting of a page, and that make it clear to a reader what the state of a particular page is. See this mock-up by Howie Fung as an example of what revised per-page controls could look like. WMF will post further mock-ups for feedback and prototypes for testing as they are built.
Who is currently working on the project?
- Aaron Schulz, a contract developer with Wikimedia, is the lead developer of FlaggedRevisions.
- Howie Fung, a contract product manager who also works with Wikimedia’s Usability Initiative, is supporting usability and product review of the software.
- I, William Pietri, support the project management of the English Wikipedia roll-out as described above.
- Erik Zachte, Wikimedia’s Data Analyst, will develop metrics specifically assessing the impact of the English Wikipedia rollout.
When will it be done?
This question has been asked a lot lately. Because this isn’t the flipping of a switch but a software development project, answering it requires me to let you in on a secret about software development projects. There are basically four ways to deal with dates, but only three of them are sane:
- It’s done when it’s done. Nobody mentions dates. The developers code until they’re finished. Then you release, get feedback, and code some more.
- Measure progress and project dates. You lay out all the work, estimate relative size, and then measure how much you get done over time. That data is used to figure out release dates.
- Pick a date and release whatever you finish. If you’re building, say, annual tax return software, it’s better to ship on time and drop features than it is to finish late with everything.
- Make up dates to please people. This is very popular, and has the advantage of making people happy at first, but it rarely works out well.
Until recently, we were using the first approach. That’s how most Mediawiki development (and most other open source development) works, and it has many advantages. But because a lot of people are eager for this project to launch, we’re shifting to the second approach.
The developer, Aaron Schulz, has estimated all of the items on the work list and already started in on them. The holidays complicate things some, but I expect we’ll have enough data to make a first guess at the estimated release date by the middle of January.
Wait, there’s only one developer on this? Is the Wikimedia Foundation taking this seriously?
Yes, absolutely. Aaron has been working on the Flagged Revisions extension for years, and nobody knows it better. We talked about adding developers, but unfortunately adding more people now wouldn’t help. I haven’t dug into the history much, but it looks like the real slowdown lately wasn’t in the coding; it was in turning the many-voiced community response into a clear set of things to do.
Having spent time with all the people involved, it’s clear to me that the Foundation takes this project very seriously. It’s one of small number of high-priority projects, which include things like keeping the site running, organizing the annual fundraiser, and Wikimedia’s usability initiative.
How can I keep track?
There are a few ways. First, we’ll mention big updates (and the eventual release) here on this blog. Second, keep an eye on the labs site. That will be updated regularly with the latest code and configs. You can judge for yourself how we’re doing, and make sure we do it right. And third, I’ve put the work queue into a public web-based tool called Pivotal Tracker. It’s one of the few software project management tools made for the measure-and-project approach we’re using; if you’re the sort of person who likes way too much detail, you can find real-time updates there.
How can I get involved?
Go to the labs site, play with the current implementation, give feedback, post your own user interface design suggestions, report bugs, and so forth. Further community discussion in the English Wikipedia about the proposed roll-out is happening on the page about flagged protection and patrolled revisions. I’m always glad to hear feedback, either on my talk page or via email. And of course, you can comment on this very blog post.
William Pietri
Contractor, Wikimedia Foundation
Priyanka Dhanda joins Wikimedia tech team
I’m very pleased to welcome Priyanka Dhanda to the Wikimedia Foundation as Code Maintenance Engineer. Priyanka joins us from SourceForge Inc., where she worked since 2002 as a software developer and also was involved in operations, working on most pieces of the infrastructure, and integrating third party software with the SourceForge platform (including MediaWiki). Priyanka holds a Master’s Degree in Computer Science from the University of Toledo, Ohio, and a Bachelor of Technology in Computer Science and Engineering from the Pondicherry Engineering College in India.
She is starting today and will work in the San Francisco office.
Priyanka will be a key interface between software developers and the operations team, helping us to catch up with our code and bug review backlog, to mentor new developers, to push projects to completion, and to improve testing and automation. Please don’t swamp her immediately with requests as she’ll need some time to get more deeply oriented in the MediaWiki codebase. :-) You’ll be seeing her in the IRC channels, on SVN, Code Review, BugZilla, wikitech-l, and so forth.
Please join me in welcoming Priyanka to the Wikimedia team! :-)
– Erik Moeller
Deputy Director, Wikimedia Foundation
LiquidThreads almost ready to deploy
Hi all,
With the Foundation’s support, I’ve spent the last few months churning away at LiquidThreads, a new discussion system that is proposed for use on Wikimedia projects.
Essentially, it’s an attempt to marry the radical openness of the wiki paradigm with the usability and practicality of a forum-like system. As the name implies, LiquidThreads is designed to allow any user to easily refactor discussions while maintaining edit history, to edit other users’ comments, and to collaborate on a summary of an ongoing discussion. LiquidThreads also brings many standard communication features lacking from wiki discussion pages, such as watching and protecting individual discussion threads, RSS feeds of comments in a discussion or on a discussion page. In the world of online communication, its approach is entirely unique.
LiquidThreads has been in alpha testing on Wikimedia Labs for several months, and, more recently, it’s been used in a production context on the strategy wiki, where it has been quite well-received. It’s been easy to run these smaller trials, as the extension allows the activation and deactivation of LiquidThreads discussions on individual pages with a simple parser function.
While there are still some issues remaining before wider trials, I believe I can resolve most of them quite quickly (within a few weeks when my vacation finishes at the end of next month), and I’d like to get the ball rolling in proposing small-scale trials on some of the larger wikis, so that a full discussion can be had, and so that adjustments can be made on the basis of ongoing feedback. I’d especially like to see LiquidThreads used on some of the higher-traffic discussion pages on English Wikipedia (such as the technical village pump), and progressive rollout on some of our mid to large sized wikis.
So, I’d like to encourage you to have a play with LiquidThreads, either on the strategy wiki or on the test site (which generally runs a newer version). Tell me what you like about it, and (far more importantly) what improvements you think it needs before we can expand our trials to wider parts of the Wikimedia Universe, and perhaps move towards a full rollout of this very exciting technology.
I should give the following caveats about LiquidThreads as it stands. These are all issues that I intend to address before any trial expansion occurs.
- Presently the system is somewhat vulnerable to abuse. I intend to make changes to the way signatures work, and improve tracking and listing of thread actions by specific users.
- While LiquidThreads allows for thread summaries and discussion headers, the system does not currently have support for collaboratively-edited posts which are unsigned or signed by a group of people. These are a key piece of any decision-making framework, and I intend to make adjustments to make this possible.
- There is no support for embedding LiquidThreads discussion pages on other pages.
- There are plenty of minor interface issues which I intend to clean up.
Feedback is best directed to the dedicated feedback page, or, alternatively, to bugzilla (although before filing a bug, you should check the list of existing LiquidThreads bugs).
Thanks,
Andrew Garrett
Software Development Contractor
Mobile Homepage in your Language!
Posted by Hampton Catlin in mobile, software on December 9th, 2009
Setting up mobile home pages for different languages is a very important part of my job here at Wikimedia. The English mobile home page has been setup for a while and it is based on CSS selectors. A couple other languages, (like Spanish) were easy to implement CSS solutions for and therefore I had gone ahead and created mobile home pages with the help of those communities. However, I am only one man and manually contacting each Wikipedia admin structure individually was taking far too long. Besides, different languages have different items on their home page!
With the help of Petter Strandmark at the Swedish Wikipedia, we have come up with another method that should hopefully work better for lots of different languages: A customized mobile home page. If you want a mobile home page in your language, just send us the name of the page and I’ll wire it up. You can see this is the Swedish mobile main page and here is the corresponding specialized mobile home page on the main site.
It’s one of those obvious solutions that takes way too long to come up with… but at least we have it now.
Now, each community can build the mobile homepage that they are looking for and maintain it themselves with whatever content they want.
If your language wants to produce a mobile home page, then open a ticket in Bugzilla that includes the URL of an already setup MainPage version and I’ll sort it out!
Cheers!
Fun with Subtitles
Posted by Michael Dale in wikimedia on November 9th, 2009
You can select your subtitle language
We have just finished up the Multimedia Usability Project Meeting here in France. I am sure there will be more general wrap up coverage of the meeting shortly… but I wanted to share a hack we worked here.
For a long time people have requested subtitle support for mediaWiki embed videos and at this meeting we were finally able to sit down and hack up an initial solution. The system works by putting the srt files into the wiki so people can collaborate on translation and contribution. The naming scheme is “TimedText” followed by the file name followed by the language code followed by .srt . We include a basic editor to upload srt files and switch between between languages. We presently display the subtitles on the side of the video but they should make there way below the video soon. There are lots of supporting projects to work on if anyone is interested ;)
How do I try it out
To try it out install the mwEmbed gadget and then visit either video linked to in this post. Hopefully we can produce some more documentation soon :)
Some quick ideas for enhancements ( I am sure you can think of some too) :
- A translation interface maybe borrowing from the techniques used on translatewiki.net
- A simple search of subtitles from the current video
- A more complex search system for subtitles across all videos
- Timed metadata a-la-metavid
subtitles on commons using the mwEmbed gadget
Localization Update: As mentioned in the comments we are still missing some of the localizations msgs. They should be making their way in there soon, along with some other updates.
