A brief update on Mix’n’match. When you are logged in, you can reach your “user page” in Mix’n’Match through the user icon in the top navbar. On that page, you will now find a link to “Settings”. The first setting (besides dark mode) there is a configuration of the search buttons on unmatched lists of entries. This doesn’t sound like much, but I found that looking for an entry in a bespoke search engine, or in the Mix’n’Match “creation candidates”, is very helpful.
My Mix’n’match tool has a little-known feature that I think deserves a little attention.
A catalog in Mix’n’match is a list of entries in other datasets that should be matched to Wikidata. Besides some required core data for each entry, the user importing the catalog can add key-value-pairs for Wikidata statements that can help to match the entry (e.g. VIAF ID), or can be nice-to-have for item creation (e.g. occupation).
Now, many datasets contain such additional information, but often in free-form text. That person here was born in Berlin – but is it the Berlin in Germany, or the one in New Hampshire? This often depends on the context of the dataset. This makes conversion from free-form text to proper Wikidata IDs difficult and time-consuming, and a hurdle before import into Mix’n’match. So often such data gets thrown away.
But there is another way: If you know the Wikidata property for a “factoid” in your dataset (e.g. “place of birth”), you can import it as a “statement text”. It is still a key-value-pair, but while the key is a Wikidata property ID, the value can be anything you like.
So that “statement text” is now sitting in the Mix’n’match database (~1.4 million rows, at the time of writing this). But what can you do with that? You can convert it to “normal” auxiliary statement data via a handy web interface! But wait, do you have to do this for every single “statement text”? Not necessarily! As it turns out, within a catalog and a property, there are often repeat values. As you can see in the video, a taxon rank called “form” is used over 300 times in this catalog. In this context, they all refer to the same thing. Search results for this value are presented to you, and in this case, it happens to be the first hit on the list. One click, and all 328 taxon ranks are converted into the appropriate Wikidata item value.
The link to “statement text” in the action drop-down will appear only if the catalog has any (unmatched) statement text for you to work on. So, feel free to import free (-form text) with your next catalog!
WikiFlix is one of my tools that got some press when it started, but had stalled a bit due to lack of development. Thanks to recent developments, I was able to pay some attention to it again.
First, there are a lot of invisible changes to the code itself, moving the front-end to refactored ES6 JavaScript, and making the hourly update job more efficient.
On the visible side of things, there is a new “Recently viewed” section that is stored in your browser only, so you can remember and resume recent movies. The “special sections” (e.g. “Highly ranked”, “Female directors”) are now clickable, leading to the full respective list. The lists themselves are now loading more entries on scrolling, rather than loading all of them at once, making for a faster, smoother experience. Several pages now show initial “placeholder boxes” that are filled when loaded, which makes a for less jarring experience than “blank page, stand by”.
The search box is now more prominent, and there is a “random movie”button, if you are bored. Also, playing a movie (Commons, YouTube, Internet archive) will now immediately launch a full-screen player, which is what I suspect most people would prefer over the page-with-iframe.
Finally, I am now including a more edge cases of movies (e.g. “Public domain in the United States” rather than just “PD”). This pushed the movie count to 5100.
Update: I now added free TV episodes as well, pushing the count to over 5200. Example series: Calvin and the Colonel
This is going to be controversial, so I would ask you to reserve judgement until the end of this post (with the average internet crowd, such an appeal would be pointless, but people who read this blog are usually familiar with the NPOV concept, so I have hope).
I have been writing code for over 40 years. I have worked in the usual (and some unusual) programming languages. I can work “full stack”. I wrote the original version of MediaWiki during a semester holiday 25 years ago, and I have been writing MediaWiki-related code ever since. This is not to brag, but to set the stage.
With over 350 tools juggled in the air as a hobby, I find that I can keep the big ones running, and occasionally I find time for a new one, or a rewrite. But all of that code is technical debt, and it’s slowly rotting away. And while my tools usually work, they are often written organically, growing into strange shapes as time passes and features are added.
That makes most of my tools difficult to maintain, even for me. Passing a tool to someone else to take care of is… difficult at best.
I had looked into LLMs (mostly ChatGPT at the time) for coding when it first came into fashion, so to speak, but the results were spurious and, frankly, unacceptable. Then, about half a year ago, I played around with Claude Opus again, and it was… better. Around the same time, I was notified that one of my oldest tools, “geohack”, was exhausting its assigned Toolforge resources. As many of you know, I like to experiment, so I told it to rewrite the PHP codebase in Rust (vibe-transpiling?). The result was astonishing. Even though I told it in no uncertain terms to reproduce all details of the original code, I would never have expected it to spit out an essentially feature-complete, fully working version.
Yes, I am quite capable of doing this work myself. For example, I have previously rewritten CatScan (PHP) into PetScan (C++) into PetScan (Rust). But to do that, I would have had to drop other things, for quite a while. And it would have been quite tedious. This way, I had it done in an afternoon. Was the code good? Kinda. But I was very specific about sacrificing style for faithful reproduction of functionality, so that’s OK. I could probably outsource part of it to an existing Rust crate. But it works well as it is, much better than the original.
Finally, there was hope for me to get my tools at least to a maintainable level. I started asking Claude to refactor, write tests, find bugs (and it did, including potentially security-relevant ones), move the codebase towards SOLID and DRY principles, reduce cognitive complexity, use best practices and language standards — all things you would want for your code. I did this with Rust codebases first, since the stringent but helpful compiler works very well at pushing back against LLM-induced issues, but eventually I expanded into PHP and JavaScript territory.
An incomplete list of things I had Claude help with:
Is that “vibe coding”? I don’t think so (that would be more like “go write a banking app in assembler, YOLO!!1!”), but call it whatever you like. Is the code “slop”? It does have a different “taste” from my own code. It can be a bit verbose. But given the upsides of using LLMs, especially with the directives I put in place (writing tests, SOLID/DRY, etc.), the codebase as a whole is more accessible. And with very few, small exceptions, I have never had Claude write a de novo codebase (transpiling from PHP is mostly mechanical, so it doesn’t count in my book), so I am iterating on my own code, and actually reducing lines of code (e.g. merging the Mix’n’match PHP code into the Rust one) that I need to maintain. Of course, new features require new code, but that is true of all coding, and I try not to go overboard.
So am I an “AI” fanboy now? Most certainly not. Likewise, I am not apologetic about using it. It is a technology that has helped me. Using LLMs has improved my tools, both at the code level and in the user experience. That is a win in my book. I would love to have a little box with a copy of Claude (or similar) on my desk as a permanent assistant (usage is not cheap!), but if LLMs get outlawed tomorrow, go bankrupt, or increase prices even more, I will shrug and move on. I’ll be fine without it. At least I got some use out of it while it lasted.
Some of my tools are JavaScript scripts on Wikidata. They are added by users on their commons.js page and run in their web browser when browsing Wikidata. I have about a dozen of them, of various ages. Some of them I had updated four years ago, and put a “shared library” sort-of-thing in place.
One thing that always happens with these scripts is that I start developing them in a nice editor on toolforge, but once they reach a certain maturity, I put them on a Wikidata “code page”, announce them, fix the inevitable bug or two, and then leave them be; it is too awkward to copy them into the Toolforge file again, switch the include to Toolforge, work on it, and then switch everything back, just to be able to edit in a decent editor.
Using my own scripts on a daily basis, I had accumulated a few things that bugged (see what I did there?) me, and I decided to do something about it. And then I paused and did something bigger. I took my “shared library”, and the (still working) scripts, and put them in a repo on codeberg. I also wrote a Python script that can sync between Wikidata and the local repo. I can now use my trusty editor (Zed) to edit files, push them to repo, and to Wikidata, with one line. I also put a decent README that briefly describes the tools. As a side effect, I now offer the issue tracker to file bug reports and feature requests; I hope this gets take up over “adding it to some talk page” where it is too easily forgotten.
Now, what did I do with this awesome power? Mostly things behind the scenes. One visible change is how statements get added to the Wikidata interface (DOM). Until now, I just added statement-lookalike boxes as placeholders, but they were just Potemkin’s <div>s; they could not be edited, had no references, no new references could be drag’n’dropped on them, etc. Now, I am using the (officially undocumented) Wikibase JS functions to add new statements, so they become not only visible immediately, but are working, usable members of the statement society, as if I had added them manually, with keystrokes, like some caveman. And if this undocumented hack ceases to function one day, it will just fall back on the previous, display-only version.
I hope this will future-proof the scripts, at least for a while, and maybe even tempt others to use the “shared library” in their own scripts.
I have a collection of tools around images (or these days files in general) and their use on the various Wikimedia projects, that run under a single Toolforge tool name. These are:
These tools have been around many years, with FIST predating Wikidata. They were in various states of decay, WDFIST holding up best, the original FIST tool a broken PHP script. So I thought to bring them up to date a bit.
FIST was completely rewritten into FIST2. File Candidates was absorbed into WDFIST as a way to find candidate images. All tools were upgraded to use Vue2 ES6 JavaScript modules. They now share common, reusable components, and a common visual language. And all of them should work better on mobile, especially WDFIST. Also, all tools are hardened against the recent Wikimedia API rate limits, with automatic retries, and keeping API queries to an absolute minimum. I hope that these refurbished tools will be of use to people, so happy (re-)discovery!
Mix’n’match is one of my more popular tools, and with over 262 million entries one of the largest. It also has been around for awhile (November 2013), almost as old as Wikidata. It started out as a PHP script, which then grew into a collection of classes, but eventually the size of the data, as well as changes on Toolforge, resulted in me writing a server in Rust, to process data in the background (scraping websites, matching entries, maintenance functions). This resulted in a massive duplication of code, as the Rust back-end needs to deal with the same database and data structures as the PHP front-end; plus, all the (now legacy) PHP back-end code.
Recent changes in a catalog. The user links show recent edits by the user.
I finally got around to implementing the missing part, which is a Rust rewrite of the PHP web API. This leverages the existing Rust data structures, database access etc., but also re-uses the HTML/JS for the front-end. One reason that kept me from doing this was me letting users add small PHP scripts to perform specific functions, e.g. extract birth and death dates from entry descriptions for a specific catalog. As you can see in my recent blog entry, I am now offering Lua instead, but keep the legacy PHP code around, for comparison mostly.
A migration of this magnitude (the web API alone was >3600 lines of PHP; over 15K lines of PHP across all PHP classes) can not be expected to go perfectly smooth, when you are a team of 1 (well, more 0.2, as I have a day job, and one of these pesky “life” things that are all the rage now), but it’s up and running now, with surprisingly little fallout.
Besides the consolidation of the two code bases into the Rust one, and the Rust web server (axum) being a bit more snappy than the lighttp/PHP one, I have also put some effort into improving the interface. For example, you can now see more details about a catalog and how it behaves, and (if you created the catalog, or are a “catalog admin”) you can also change it; that includes setting up specific auto-matchers to specify what kind of items you are looking to match the catalog against.
Adding and removing catalogs from a group or search filter is now much easier.
Many pages have an improved UI, e.g. Recent Changes and User Changes (which is actually a new page type). Importing files as a catalog is now a wizard-style component, and the scraper page has more information feedback about just why that regexp doesn’t work (and is now linked from every catalog that has a scraper in the “Action” drop-down).
As you might have already seen, the landing page has changed as well. Top groups has been revamped to be more usable. There are also small improvements all over, e.g. proper pagination for issues or jobs.
Please leave all bug reports, improvement suggestions etc. in the issue tracker of the Rust repo. Otherwise, happy matching!
A minor tech announcement for Mix’n’match:
Code fragments for Mix’n’match catalogs are switching from PHP to Lua. User-supplied PHP is not secure enough, and will not scale for future development. I have converted many code fragments from PHP to Lua already, and new code fragments have to be Lua.
Lua is not exactly my favorite language, but it has a sweet spot between the number of people who know it or can easily learn it, and safety/resource usage etc. for letting user-written code run within Mix’n’match/Toolforge.
PHP code that was not “translated” yet will be supported for a while, but not forever. Existing PHP code will be shown as a “hint” for Lua, for the time being.
Example: https://mix-n-match.toolforge.org/#/code/7098
Mix’n’match is one of my more popular tools. It has grown over the years in both functionality and data size. It currently contains >260M entries in over 6300 catalogs. The code base has been growing organically with the data size. I felt it was time to modernize some things.
First, the speed. Some pages, most notably the landing page, would load details on all active catalogs in the database. On every load. This was not sustainable. The updated version has some layout simplification, but shows the same information as before, and reduces the (uncached) loading time from ~35 to 1.8 seconds (YMMV).
The interface was updated to bootstrap5, and many interfaces have ever so slightly changed, for the better I think.
I have also taken pains to support mobile better. This includes moving the “mobile game” from its own, separate site to integrate into Mix’n’match.
Finally, under the hood, this is my first tool where I moved from my home-grown Vue module loading system to proper ES6 components. While this will make little difference to users, it is an important step towards maintainability. It also allowed me to write JavaScript unit tests.
Please give the new version a spin, and let me know about issues and suggestions in the issue tracker.
AutoDesc has been a longstanding tool to generate descriptions for Wikidata items. Originally a spin-off from Reasonator, it started as my only JavaScript-based server application, generating both short and (for biographical items, and two languages) long descriptions. When the usual bitrot set in, I decided to move it over to Python, hoping to attract more developers; that did, however, not pan out. Also, I only managed to port the short description code, dropping support for the long descriptions in the process.
Now, I have moved it to trusty Rust, and also ported the long description mode from the original JavaScript. I threw in some caching, so descriptions are now much faster – often <1sec for short descriptions.
In other news, my MediaWiki Rust API has now added all of the MediaWiki Action API functionality as proper Rust elements. Generators are also supported, as is continue. Results that are based on page_info results can be automatically parsed into an appropriate struct. I even threw in a prelude::* module to get everything you need in one line. Due to the massive addition of functionality, the version has also been bumped to 0.5.0.
