Blog on Taavi Väänänen

Wikimedia Hackathon Northwestern Europe 2026

taavi@majava.org (Taavi Väänänen) — Wed, 18 Mar 2026 00:00:00 +0000

Wikimedia Nederland organised a new type of event this year, the Wikimedia Hackathon Northwestern Europe 2026, which was held last weekend in Arnhem, the Netherlands. And I'm very happy they did, since unlike last years, I will unfortunately be missing from the "main" Wikimedia Hackathon (which is happening in Milan at the start of May).

I continue to believe the primary reason for these events existing is the ability to connect with old and new friends in person. That being said, I did get a bit of technical tinkering done during the weekend as well. These include a dark mode fix to MediaWiki's notification interface, fixes to some visual bugs in MediaWiki's two-factor authentication and OAuth functionality. I did also get an older patch of mine about disabling Composer's new auditing functionality merged. And, as usual, I spent a bunch of time helping various people use with the various infrastructure pieces I'm familiar with (or at least had to suddenly get familiar with) and approved a bunch of OAuth consumers and other requests.

We also managed to continue the tradition from the past two Wikimedia Hackathons of nominating more people to receive +2 access to mediawiki/*. That request is still open as of writing, as those have to run for at least a week, but looks very likely to pass at this point.

Overall, the event was very well-organized: the venue was great, except that the number of stairs was described in a rather misleading way, food was great, and the atmosphere was amazing. The pressure that you must Just Get Things Done to justify your attendance that the main hackathon seems to have recently gained was clearly missing here which was great. Also, I will clearly need to bring more Finnish chocolate next time.

The timing of Friday and Saturday works great for us with other things (like university for me) during the week, as it takes full advantage of the weekend but still only eats workdays from a single calendar week. My main gripe with the logistics was the focus on a single sketchy non-free messaging platform for all event-related communications with the IRC bridge used on the main hackathon channel notably missing.

ps. Like Lucas, I do have Opinions about so many proudly mentioning they've used "vibe coding" tools during the introduction and showcase. Those opinions are best left for an another time, but I do want to note that all of my work and mistakes have still been lovingly handcrafted.

How to import a new Wikipedia language edition (in hard mode)

taavi@majava.org (Taavi Väänänen) — Sat, 06 Dec 2025 00:00:00 +0000

I created the latest Wikipedia language edition, the Toki Pona Wikipedia, last month. Unlike most other wikis which start their lives in the Wikimedia Incubator before the full wiki is created, in this case the community had been using a completely external MediaWiki site to build the wiki before it was approved as a "proper" Wikipedia wiki,¹ and now that external wiki needed to be imported to the newly created Wikimedia-hosted wiki. (As far as I'm aware, the last and previously only time an external wiki has been imported to a Wikimedia project was in 2013 when Wikitravel was forked as Wikivoyage.)

Creating a Wikimedia wiki these days is actually pretty straightforward, at least when compared to what it used to be like a couple of years ago. Today the process mostly involves using a script to generate two configuration changes, one to add the basic configuration for a wiki to operate and an another to add the wiki to the list of all wikis that exist, and then running a script to create the wiki database in between of deploying those two configuration changes. And then you wait half an hour while the script to tell all Wikidata client wikis about the new wiki runs on one wiki at a time.

The primary technical challenge in importing a third-party wiki is that there's no SUL making sure that a single username maps to the same account on both wikis. This means that the usual strategy of using the functionality I wrote in CentralAuth to manually create local accounts can't be used as is, and so we needed to come up with a new way of matching everyone's contributions to their existing Wikimedia accounts.

(Side note: While the user-facing interface tries to present a single "global" user account that can be used on all public Wikimedia wikis, in reality the account management layer in CentralAuth is mostly just a glue layer to link together individual "local" accounts on each wiki that the user has ever visited. These local accounts have independent user ID numbers — for example I am user #35938993 on the English Wikipedia but #4 on the new Toki Pona Wikipedia — and are what most of MediaWiki code interacts with except for a few features specifically designed with cross-wiki usage in mind. This distinction is also still very much present and visible in the various administrative and anti-abuse workflows.)

The approach we ended up choosing was to re-write the dump file before importing, so that a hypothetical account called $Name would be turned $Name~wikipesija.org after the import.² We also created empty user accounts that would take ownership of the edits to be imported so that we could use the standard account management tools on them later on. MediaWiki supports importing contributions without a local account to attribute them to, but it doesn't seem to be possible to convert an imported actor³ to a regular user later on which we wanted to keep as a possibility, even with the minor downside of creating a few hundred users that'll likely never get touched again later.

We also made specific decisions to add the username suffix to everyone, not to just those names that'd conflicted with existing SUL accounts, and to deal with renaming users that wanted their contributions linked to an existing SUL account only after the import. This both reduced complexity and thus risk from the import phase, which already had much more unknowns compared to the rest of the process, but also were much better options ethically as well: suffixing all names meant we would not imply that those people chose to be Wikimedians with those specific usernames (when in reality it was us choosing to import those edits to the Wikimedia universe), and doing renames using the standard MediaWiki account management tooling meant that it produced the normal public log entries that all other MediaWiki administrative actions create.

With all of the edits imported, the only major thing remaining was doing those merges I mentioned earlier to attribute imported edits to people's existing SUL accounts. Thankfully, the local account -based system makes it actually pretty simple. Usually CentralAuth prevents renaming individual local accounts that are attached to a global account, but that check can be bypassed with a maintenance script or a privileged enough account. Renaming the user automatically detached it from the previous global account, after which an another maintenance script could be used to attach the user to the correct global account.

That external site was a fork of a fork of the original Toki Pona Wikipedia that was closed in 2005. And because cool URIs don't change, we made the the URLs that the old Wikipedia was using work again. Try it: https://art-tokipona.wikipedia.org. ↩︎
wikipesija.org was the domain where the old third-party wiki was hosted on, and ~ was used as a separator character in usernames during the SUL finalization in the early 2010s so using it here felt appropriate as well. ↩︎
An actor is a MediaWiki term and a database table referring to anything that can do edits or logged actions. Usually an actor is a user account or an IP address, but an imported user name in a specific format can also be represented as an actor. ↩︎

Tracking my train travel by parsing tickets in emails

taavi@majava.org (Taavi Väänänen) — Sat, 05 Jul 2025 00:00:00 +0000

Rumour has it that I might be a bit of a train nerd. At least I want to collect various nerdy data about my travels. Historically that data has lived in manual form in several places,¹ but over the past year and a half I've been working on a toy project to collect most of that information into a custom tool.

That toy project² uses various sources to get information about trains to fill up its database: for example, in Finland Fintraffic, the organization responsible for railway traffic management, publishes very comprehensive open data about almost everything that's moving on the Finnish railway network. Unfortunately, I cannot be on all of the trains.³ Thus I need to tell the system details about my journeys.

The obvious solution is to make a form that lets me save that data. Which I did, but I got very quickly bored of filling out that form, and as regular readers of this blog know, there is no reason to settle for a simple but boring solution when the alternative is to make something that is ridiculously overengineered.

Parsing data out of my train tickets

Finnish long-distance trains generally require train-specific seat reservations, which means VR (the train company) knows which trains I am on. We just need to find a way to extract that information in some machine-readable format. So my plan for the ridiculously overengineered solution was to parse the booking emails to get the details I need.

Now, VR ticket emails include the data I want in a couple of different formats: they're included as text in the HTML email body, they're in the embedded calendar invite, as text in the included PDF ticket, and encoded in the Aztec Code in the included PDF ticket. I chose to parse the last option with the hopes of building something that could be ported to parse other operators' tickets with relative ease.

Example Aztec code

After a bit of digging (thank you to the KDE Itinerary people for documenting this!) I stumbled upon an European Union Agency for Railways PDF titled ELECTRONIC SEAT/BERTH RESERVATION AND ELECTRONIC PRODUCTION OF TRANSPORT DOCUMENTS - TRANSPORT DOCUMENTS (RCT2 STANDARD) which, in its Appendix C.1, describes how the information is encoded in the code.⁴ (As a side note, various sources call these codes SSB version 1 codes, although that term isn't used in this specification. So maybe there are more specifications about the format that I haven't discovered yet!)

I then wrote a parser in Go for the binary data embedded in these codes. So far it works, although I wouldn't be surprised if there are some edge cases that it doesn't handle. In particular, the spec specifies a custom lookup table for converting between text and binary data, and that only has support for characters 0-9 and A-Z. But Finnish railway station codes can also use Ä and Ö.. maybe I need to buy a ticket to a station with one of those.

Extracting barcodes out of emails

A parser just for the binary format isn't enough here if the intended source input is the emails that VR sends upon making a booking. Time to write a single-purpose email server! In short, the logic in the server, again written in Go and with the help of go-smtp and go-message, is:

Accept any mail with a reasonable body size
Process through all body parts
For all PDF parts, extract all images
For all images, run them through ZXing
For all decoded barcodes, try to parse them with my new ticket parsing library I mentioned earlier
If any tickets are found, send the data from them and any metadata to the main backend, which will save them to a database

The custom mail server exposes an LMTP interface over TCP for my internet-facing mail servers to forward to. I chose LMTP for this because it seemed like a better fit in theory than normal (E)SMTP. I've since discovered that curl doesn't support LMTP which makes development much harder, and in practice there's no benefit of LMTP here as all mails are being sent to the backend in a single request regardless of the number of recipients, so maybe I'll migrate it to regular SMTP at some point.

Side quest time

The last missing part is automatically forwarding the ticket mails to the new service. I've routed a dedicated subdomain to the new service, and the backend is configured to allocate addresses like i2v44g2pygkcth64stjgyuqz@somedomain.example for each user. That's great if we wanted to manually forward mails to the service, but we can go one step above that. I created a dedicated email alias in my mail server config that routes both to my regular mailbox and the service address. That way I can update my VR account to use the alias and have mails automatically processed while still receiving backup copies of the tickets (and any other ~~important~~ mail that VR might send me).

Unfortunately that last part turns out something that's easier said than done. Logging in on the website, I'm greeted by this text stating I need to contact customer service by phone to change the address associated with my account.⁵ After a bit of digging, I noticed that the mobile app suggests filling out a feedback form in order to change the address. So I filled that, and after a day or two I got a "confirm you want to change your email" mail. Success!

Including (but not limited to): a page of this website, the notes app on my phone, and an uMap map. ↩︎
Which I'm not directly naming here because I still think it needs a lot more work before being presentable, but if you're really interested it's not that hard to find out. ↩︎
Someone should invent human cloning so that we can fix this. ↩︎
People who know much more about railway ticketing than I do were surprised when I told them this format is still in use somewhere. So, uh, sorry if you were expecting a nice universal worldwide standard! ↩︎
In case you have not guessed yet, I do not like making phone calls. ↩︎

lua entry thread aborted: runtime error: bad request

taavi@majava.org (Taavi Väänänen) — Mon, 12 May 2025 00:00:00 +0000

The Wikimedia Cloud VPS shared web proxy has an interesting architecture: the management API writes an entry for each proxy to a Redis database, and the web server in use (Nginx with Lua support from ngx_http_lua_module) looks up the backend server URL from Redis for each request. This is maybe not how I would design this today, but the basic design dates back to 2013 and has served us well ever since.

However, with a recent operating system upgrade to Debian 12 (we run Nginx from the packages in Debian's repositories), we started seeing mysterious errors that looked like this:

2025/04/30 07:24:25 [error] 82656#82656: *5612 lua entry thread aborted: runtime error: /etc/nginx/lua/domainproxy.lua:32: bad request
stack traceback:
coroutine 0:
[C]: in function 'set_keepalive'
/etc/nginx/lua/domainproxy.lua:32: in function 'redis_shutdown'
/etc/nginx/lua/domainproxy.lua:48: in main chunk, client: [redacted], server: *.wmcloud.org, request: "GET [redacted] HTTP/2.0", host: "codesearch.wmcloud.org", referrer: "https://codesearch.wmcloud.org/search/"

The code in question seems straightforward enough:

function redis_shutdown()
 -- Use a connection pool of 256 connections with a 32s idle timeout
 -- This also closes the current redis connection.
 red:set_keepalive(1000 * 32, 256) -- line 32
end

When searching for this error online, you'll end up finding advice like "the resty.redis object instance cannot be stored in a Lua variable at the Lua module level". However, our code already stores it as a local variable:

local redis = require 'nginx.redis'
local red = redis:new()
red:set_timeout(1000)
red:connect('127.0.0.1', 6379)

Turns out the issue was with the function definition: functions can also be defined as local. Without that, something somewhere in some situations seems to reference the variables from other requests, instead of using the Redis connection for the current request. (Don't ask me what changed between Debian 12 and 13 making this only break now.) So we needed to change our function definition to this instead:

local function redis_shutdown()
 -- Use a connection pool of 256 connections with a 32s idle timeout
 -- This also closes the current redis connection.
 red:set_keepalive(1000 * 32, 256)
end

I spent almost an entire workday looking for this, ultimately making a two-line patch to fix the issue. Hopefully by publishing this post I can save that time from everyone else stumbling upon the same problem after myself.

Wikimedia Hackathon Istanbul 2025

taavi@majava.org (Taavi Väänänen) — Sat, 10 May 2025 00:00:00 +0000

It's that time of the year again: the Wikimedia Hackathon 2025 happened last weekend in Istanbul. This year was my third time attending what has quickly become one of my favourite events of the year simply due to the concentration of friends and other like-minded nerds in a single location.¹

Image by Chlod Alejandro is licensed under CC BY-SA 4.0.

This year I did a short presentation about the MediaWiki packages in Debian (slides), which is something I do but I suspect is fairly obscure to most people in the MediaWiki community. I was hoping to do some work on reproducibility of MediaWiki releases, but other interests (plus lack of people involved in the release process at the hackathon) meant that I didn't end up getting any work done on that (assuming this does not count).

Other long-standing projects did end up getting some work done! MusikAnimal and I ended up fixing the Commons deletion notification bot, which had been broken for well over two years at that point (and was at some point in the hackathon plans for last year for both of us). Other projects that I made progress on include supporting multiple types of two-factor devices, and LibraryUpgrader which gained support for rebasing and updating existing patches².

In addition to hacking, the other highlight of these events is the hallway track. Some of the crowd is people who I've seen at previous events and/or interact very frequently with, but there are also significant parts of the community and the Foundation that I don't usually get to interact with outside of these events. (Although it still feels extremely weird to heard from various mostly-WMF people with whom I haven't spoken with before that they've heard various (usually positive) ~~rumours~~ stories about me.)

Unfortunately we did not end up having a Cuteness Association meetup this year, but we had an impromptu PGP key signing party which is basically almost as good, right?

However, I did continue a tradition from last year: I ended up nominating Chlod, a friend of mine, to receive +2 access to mediawiki/* during the hackathon. The request is due to be closed sometime tomorrow.

(Usual disclosure: My travel was funded by the Wikimedia Foundation. Thank you! This is my personal blog and these are my own opinions.)

Now that you've read this post, maybe check out posts from others?

Unfortunately you can never have absolutely everyone attending :( ↩︎
Amir, I still have not forgiven you about this. ↩︎

Writing a custom rsync server to automatically update a static site

taavi@majava.org (Taavi Väänänen) — Wed, 09 Apr 2025 00:00:00 +0000

Inspired by some friends,¹ I too wanted to make a tiny website telling which event I am at this exact moment. Thankfully I already had an another toy project with that information easily available, so generating the web page was a matter of just querying that project's API and feeding that data to a HTML template.

Now the obvious way to host that would be to hook up the HTML-generating code to a web server, maybe add some caching for the API calls, and then route external HTTPS traffic to it. However, that'd a) require that server to be constantly available to serve traffic, and b) be boring.

For context: I have an existing setup called staticweb, which is effectively a fancy name for a couple of Puppet-managed servers that run Apache httpd to serve static web pages and have a bunch of systemd timers running rsync to ensure they're serving the same content. It works really well and I use it for things ranging from my website or whyisbetabroken.com to things like my internal apt repository.

Now, there are two ways to get new content into that mechanism: it can be manually pushed in from e.g. a CI job, or the system can be configured to periodically pull it from a separate server. The latter mechanism was initially created so that I could pull the Debian packages from my separate reprepro server into the staticweb setup. It turns out that the latter makes a really neat method for handling other dynamically-generated static sites as well.

So, for my "where is Taavi at" site, I ended up writing the server part in Go, and included an rsync server using the gokrazy/rsync package. Initially I just implemented a static temporary directory with a timer to regularly update the HTML file in it, but then I got an even more cursed idea: what if the HTML was dynamically generated when an rsync client connected to the server? So I did just that.

For deployment, I slapped the entire server part in a container and deployed it to my Kubernetes cluster. The rsync server is exposed directly as a service to my internal network with no authentication or encryption - I think that's fine since that's a read-only service in a private LAN and the resulting HTML is going to be publicly exposed anyway. (Thanks to some DNS magic, just creating a LoadBalancer Service object with a special annotation is enough to have a DNS name provisioned for the assigned IP address, which is neat.)

Overall the setup works nice, at least for now. I need to add some sort of a cache to not fetch unchanged information from the API since for every update. And I guess I could write some cursed rsyncd reverse proxy with per-module rules if I end up creating more sites like this to avoid creating new LoadBalancer services for each of them.

Mostly from Sammy's where.fops.at. ↩︎

Automatically updating reverse DNS entries for my Hetzner servers

taavi@majava.org (Taavi Väänänen) — Fri, 03 Jan 2025 00:00:00 +0000

Some parts of my infrastructure run on Hetzner dedicated servers. Hetzner's management console has an interface to update reverse DNS entries, and I wanted to automate that. Unfortunately there's no option to just delegate the zones to my own authoritative DNS servers. So I did the next best thing, which is updating the Hetzner-managed records with data from my own authoritative DNS servers.

Generating DNS zones the hard way

The first step of automating DNS record provisioning is, well, figuring out which records need to be provisioned. I wanted to re-use my existing automation for generating the record data, instead of coming up with a new system for these records. The basic summary is that there's a Go program creatively named dnsgen that's in charge of generating zone file snippets from various sources (these include Netbox, Kubernetes, PuppetDB and my custom reverse web proxy setup).

Those snippets are combined with Jinja templates to generate full zone files to be loaded to a hidden primary running Bind9 (like all other DNS servers I run). The zone files are then transferred to a fleet of internal authoritative servers as well as my public authoritative DNS server, which in turn transfers them to various other authoritative DNS servers (like ns-global and Traficom anycast) for redundancy.

There's also a bunch of other smaller features, like using Bind views to server different data to internal and external clients, and resolving external records during record generation time to be used on apex records that would use CNAME records if they could. (The latter is a workaround for Masto.host, the hosting provider we use for Wikis World, not having a stable IPv6 address.) Overall it's a really nice system, and I've spent quite a bit of time on it.

Updating records on Hetzner-managed space

As mentioned above, Hetzner unfortunately does not support custom DNS servers for reverse records on IP space rented from them. But I wanted to use my existing, perfectly working DNS record generation setup since that works perfectly fine. So the obvious answer is to (ab)use DNS zone file transfers.

I quickly wrote a few hundred lines of Go to request the zone data and then use the Hetzner robot API to ensure the reverse entries are in sync. The main obstacle hit here was the Hetzner API somehow requiring an "update" call (instead of a "create" one) to create a new record, as the create endpoint was returning an HTTP 400 response no matter what. Once I sorted that out, the script started working fine and created the few dozen missing records. Finally I added a CronJob in my Kubernetes cluster to run the script once in a while.

Overall this is a big improvement over doing things by hand and didn't require that much effort. The obvious next step would be to expand the script to a tiny DNS server capable of receiving zone update NOTIFYs to make the updates happen real-time. Unfortunately there's now no hiding of the records revealing my ~~ugly hacks~~ clever networking solutions :(

Custom domains on the Wikimedia Cloud VPS web proxy

taavi@majava.org (Taavi Väänänen) — Fri, 01 Nov 2024 00:00:00 +0000

The shared web proxy used on Wikimedia Cloud VPS now has technical support for using arbitrary domains (and not just wmcloud.org subdomains) in proxy names. I think this is a good example of how software slowly evolves over time as new requirements emerge, with each new addition building on top of the previous ones.

According to the edit history on Wikitech, the web proxy service has its origins in 2012, although the current idea where you create a proxy and map it to a specific instance and port was only introduced a year later. (Before that, it just directly mapped the subdomain to the VPS instance with the same name).

There were some smaller changes in the coming years like the migration to acme-chief for TLS certificate management, but the overall logic stayed very similar until 2020 when the wmcloud.org domain was introduced. That was implemented by adding a config option listing all possible domains, so future domain additions would be as simple as adding the new domain to that list in the configuration.

Then the changes start becoming more frequent:

In 2022, for my Terraform support project, a bunch of logic, including the list of supported backend domains was moved from the frontend code to the backend. This also made it possible to dynamically change which projects can use which domains suffixes for their proxies.
Then, early this year, I added support for zones restricted to a single project, because we wanted to use the proxy for the *.svc.toolforge.org Toolforge infrastructure domains instead of coming up with a new system for that use case. This also added suport for using different TLS certificates for different domains so that we would not have to have a single giant certificate with all the names.
Finally, the last step was to add two new features to the proxy system: support for adding a proxy at the apex of a domain, as well as support for domains that are not managed in Designate (the Cloud VPS/OpenStack auth DNS service). In addition, we needed a bit of config to ensure http-01 challenges get routed to the acme-chief instance.

Bulk downloading Wikimedia Commons categories

taavi@majava.org (Taavi Väänänen) — Sun, 13 Oct 2024 00:00:00 +0000

Wikimedia Commons, the Wikimedia project for freely licensed media files, also contains a bunch of photos by me and photos of me at various events. While I don't think Commons is going away anytime soon, I would still like to have a local copy of those images available on my own storage hardware.

Obviously this requires some way to query for photos you want to download. I'm using Commons categories for this, since that's easy to implement and works for both use cases. The Commons community tends to come up with very specific categories that you can use, and if not, you can usually categorize the files yourself.

thankfully Commons has no such thing as a Conflict of interest (COI) policy

There is almost an existing tool for this: Sam Wilson's mwcli project has support for exporting images one has uploaded to Commons. However I couldn't use that to upload photos of me others have uploaded, plus it's written in PHP and I don't exactly want to deal with the problem of figuring out how to package it in a way I could neatly install it on my NAS.

So I wrote my own tool for it, called comload. It's written in Python because Python is easy to deploy (I can just throw it in a .deb and upload it to my internal repository), and because I did not find a Go library to handle Action API pagination for me. The basic usage is like this:

$ comload --subcats "Taavi Väänänen"

This will download any files in Category:Taavi Väänänen and its sub-categories to the current directory. Former image versions, as well as the image description and SDC data, if any, is also included. And it's smart enough to not download any files that are already there on future runs, so you can just throw it in a systemd timer to get any future files. I'd still like it to handle moved files without creating a duplicate copy, but otherwise I'm really happy with the current state.

comload is available from PyPI and from my Git server directly, and is licensed under the GPLv3.

Wikimedia Hackathon Tallinn 2024

taavi@majava.org (Taavi Väänänen) — Tue, 14 May 2024 00:00:00 +0000

This year's Wikimedia Hackathon was held in early May in Tallinn, Estonia. Like last year, it was a great opportunity to both see people I work with regularly, including people in my own team that I had not seen in person before, and to work with and help people that I have had very limited interactions with before.

Image by Olari Pilnik is licensed under CC BY-SA 4.0.

I presented a session about Puppet (slides), the configuration management tool used on Wikimedia infrastructure (and some other projects I've been involved on) which I think went quite well. I also organized (read: picked a spot for in the schedule) the cuteness meetup.

In addition to the sessions, the main focus of the event was, of course, hacking. As usual, I didn't make any major plans beforehand, and instead ended up working on several smaller projects as they popped up.

Here is a list of things I can remember working on:

I fixed several small issues in LibUp that makes it pass on more MediaWiki repositories (including core.git). James and I also migrated the LibUp configuration to GitLab.
I finished up an MR to grunt-banana-checker to add support for automatically fixing some common issues that were causing LibUp failures and to fix some minor bugs.
I worked with Piotr to get some of my patches to the OATHAuth and WebAuthn MediaWiki extensions merged. This is a part of my project to add support for more than one two-factor authentication device at a time that I was also working on during the Wikimania 2023 hackathon. Next up on this project is writing some UI code.
I fixed Wikimedia Gerrit twice after it had some issues that needed SRE intervention.
I sent a patch to Wikimedia's Phabricator/Phorge fork to add a new fox token. This ended up being deployed on Sunday and I got to showcase this during the hackathon showcase.
Reedy and I implemented support for foxes in WikiLove. I also wrote a bot to spam foxes to Sammy's talk pages on the beta cluster.¹ (This also involved a fun side quest to get a working thumbnail for the fox image we used to show up on Beta since the thumbnailing there is broken.)
I removed some deprecated code from core to earn the MediaWiki track T-shirt. I also reviewed a bunch of patches by others trying to earn that T-shirt.²
I found and reported some bugs relating to Parsoid read views on Commons.
I processed some Toolforge account approval requests and Cloud VPS project requests. I also helped some people debug some Cloud VPS issues.
I helped Bryan debug and fix an issue with HTTP/1.1 streams through the Toolforge front proxy.
I made some queries on the Wiki Replicas accidentally very slow and then fixed them to be fast again on the next day.
Got a 100% helpful, harmless, useful, etc. patch merged to something. I will provide no more details on this one.

Finally, a conversation I had at the hackathon resulted in me nominating Novem Linguae for mediawiki/* +2 access a few days after the hackathon.

I had a great time, and the ferry trip to Tallinn was much nicer than the very early flight I had last year. I can't wait to see you all again :-)

Disclosure: I am currently a Wikimedia Foundation contractor, and the Foundation did pay for my travel to Tallinn. This is my personal blog and these are my own opinions.

Since backporting this change felt too risky to do on the weekend, and also I have a feeling I'd get in troble if I ran an unapproved bot that edited on random wikis on our production wiki farm. ↩︎
Anyone who got 5 or more patches to core.git merged during the Hackathon got a cool MediaWiki T-shirt. ↩︎

Wikimania 2023

taavi@majava.org (Taavi Väänänen) — Mon, 28 Aug 2023 00:00:00 +0000

I've returned from (very hot and humid) Singapore where Wikimania was held this year. The primary reason I was there on-site was the Wikimedian of the Year stuff (I was awarded the tech award last year), but the rest of the conference was great too.

Jay and I on-stage for the awards ceremony. Image by Zack McCune, CC BY-SA 4.0.

Unlike the main Wikimedia Hackathon (which I attended too this year), Wikimania is primarly not a technical event. This resulted in several unexpected situations where I had to explain what Cloud Services is and I actually do,¹ and also gave me a much better understanding about all of the other background work going on to improve the wikis. We also had a panel about how we run WMCS collaboratively.

Wikimedian of the Year ceremony

The award ceremony was definitely the highlight of my trip. The stage was much larger than I expected, thankfully you couldn't really see the size of the audience from up there. Everything went mostly well, except that there was some confusion on which microphone(s) to use and I almost tripped on the stairs when walking up.

Congrats to this year's winners! The surprise was mostly ruined for me, thanks to a calendar invite that leaked the winners a few days earlier than planned. At least the rehearsal schedule kept changing so you still had a mystery (whether you were in the right place at the right time or not) to be excited about. And then there was the challenge of not being suspicious when leaving the hackathon table to go to a rehearsal without revealing the other, not-yet-announced winner at the same table.

It was quite funny to see people editing the Wikipedia article about the award live during the ceremony. (I believe this was shown on Depths of Wikipedia too, it was me who showed it to Annie. And on the same subject, the "Depths of Wikimania" spin-off at the closing ceremony was great!)

And yes, my name was mispronounced a few times. I'm kind of used to that by now.

Hackathon

As expected I spent a large portition of my time during the conference somewhere around the hackathon room. I didn't have any plans on what I wanted to do beforehand (as those never end up matching reality), and instead planned to just show up and find something to do after talking to people. This ended up working nicely, and my primary project during the hackathon was to pick up my work from some months ago on improving the OATHAuth extension by making it possible to have multiple different two-factor devices.

I did some fixes to make it possible to run the already-existing migration script² in Wikimedia production, and after those were reviewed and merged I changed the first wikis to WRITE_BOTH mode. I'm expecting to continue work on this even after I return back home. It even got included in the very cute Wikimania illustration as did all other projects shown during the hackathon showcase.

In addition I helped several people with several other projects and reviewed some patches. I'm also taking credit for the slide layout used on the hackathon showcase which ensured that each project had at least one slide (to avoid confusion) and that all projects used differently colored slides than projects before and after it.

Unfortunately my impression was that there weren't that many people at the hackathon who were completely new to the technical scene; I did meet a bunch of people that weren't in Athens but most of those still seemed like people who were already involved in one way or another. Although there were some folks who were mostly active on user scripts and other on-wiki activities, and I hope I managed to convince some of those folks to try working on MediaWiki itself too. In my view getting existing (power) users involved is one of the best ways to recruit new long-term contributors and maintainers.³

I also used the opportunity to have a chat with with Selena Deckelmann (the WMF's current CPTO). I'm not going to go into detail here, but I'm quite excited about some of the plans we talked about.

Other

Travelling there and back was very smooth:

I definitely was way too early at the airport when flying there. When you have a morning flight you can just sleep longer and not be that early, but I had an evening flight this time (the last departure of the evening, in fact) and I just couldn't wait at home doing nothing. I guess I could at least have taken the train the wrong way around the loop and wasted time that way.
Helsinki Airport was great as usual, although I wish there would have been more places to eat on the non-Schengen side. Singapore (Changi) was nice too, although the security-at-the-gate model was a bit unusual and the arrival security checks for the gate next to ours was causing quite a bit of confusion.
The ~13 hour flights were quite long, and about double the length of the previous longest flight for me. I did survive, though, partially thanks to the WiFi that was at the same time very slow by modern standards but still very impressive for a metal tube that high and fast.⁴
I still had a direct flight, which was definitely appreciated especially after seeing several people from Western Europe connect here in Helsinki.
Singapore border control was much simpler than I expected. (A benefit of being an EU citizen, it seems.) I'm pretty sure it took me more time to leave the Schengen zone than it took for me to enter Singapore as the gates here in Helsinki require you to take your hat and glasses off but the ones in Singapore do not.
The day I was flying back happened to be my birthday. So naturally we found some cake at the airport. :-P

I'm pretty sure this was the closest I've ever been to a Wikimedia data center. We didn't get a tour of it, but at least logged-out page views were incredibly fast.

I continued the trend of getting other people to do COI editing for me, this time with a Commons category.

Thank you to the Wikimedia Foundation for providing me a scholarship to attend this year, especially for the no-application-required process as a last year's Wikimedian of the Year winner.

Wikimania will be in Kraków next year. I believe I won't be able to attend due to IRL stuff going on at the same time, but there should be more Wikimanias after that :-P. And there are Wikimedia Hackathons and other conferences of course.

Now I need a next event/trip to be excited about as I currently have nothing planned. FOSDEM in February of next year maybe? It should even be possible to travel there by land, which would be nice considering how much I've flown over the last half a year or so.

A good reminder about living in a bubble if you only hang out in the technical channels, I guess. ↩︎
Which was already included included in a MediaWiki release before being run on Wikimedia production. Oops. ↩︎
Currently [[New Developers]] and related pages seem to mostly focus on completely new contributors that don't have an idea what they want to work on, so I might need to do some marketing for my view at some point. ↩︎
Although a speedtest became suspiciously slower several orders of magnitude slower after enabling a VPN. ↩︎

Finding files not managed by Puppet

taavi@majava.org (Taavi Väänänen) — Sat, 01 Jul 2023 00:00:00 +0000

With Puppet, it's fairly typical to have a directory where all files not managed by Puppet will be automatically purged, as that helps to ensure consistency between servers. For example, you could do something like this:

# @summary manages the ferm firewall frontend
class ferm () {
 # install package

 # manage main ferm.conf file

 file { '/etc/ferm/ferm.d':
 ensure => directory,
 owner => 'root',
 group => 'adm',
 mode => '0755',
 recurse => true,
 force => true,
 purge => true,
 notify => Service['ferm'],
 }

 # manage service, etc
}

However, sometimes you end up with a directory that's not managed this way and you want convert it to a fully managed directory. This is a bit risky: there's a possibility that some hosts have stale files in that directory and removing those could break things. And you couldn't use PuppetDB to search for unmanaged files, either, as the unmanaged files are by definition not in the Puppet state stored there.

To solve this, I wrote a tiny Python script that compares a directory on disk with the Puppet state file. On it's own it's not very useful, as it only works with the local system, but it really becomes useful when paired with a tool like Cumin to run it on many servers. Cumin can be integrated with PuppetDB too, so you can run the command on exact set of servers a potential change would affect.

The script

#!/usr/bin/python3
# SPDX-License-Identifier: Apache-2.0
# Copyright (c) 2023 Taavi Väänänen <hi@taavi.wtf>
"""puppet-unmanaged - detect files that are not managed by Puppet

This script determines which files in the given directory are not
managed by Puppet. It is intended to be used to detect stale unmanaged
files when converting directories to purge mode. The script only
interacts with the local system, for a full picture you should run it
on all affected hosts via Cumin.

Usage example:
 $ sudo cumin "C:ferm" "locate-unmanaged /etc/ferm/ferm.d/"
"""
import argparse
from pathlib import Path
from sys import exit

import yaml


def report_constructor(loader, node):
 return loader.construct_mapping(node)


def main() -> int:
 parser = argparse.ArgumentParser(description=__doc__)
 parser.add_argument("directory", type=Path)
 parser.add_argument(
 "--report",
 type=Path,
 default=Path("/var/lib/puppet/state/last_run_report.yaml"),
 help="Puppet report to work on",
 )
 args = parser.parse_args()

 # parse the special report tag as a plain object
 yaml.add_constructor(
 "!ruby/object:Puppet::Transaction::Report",
 report_constructor,
 Loader=yaml.SafeLoader,
 )

 report = yaml.safe_load(args.report.read_text())

 all_managed_paths = [
 Path(resource.get("path", resource["title"]))
 for resource in report["resource_statuses"].values()
 if resource["resource_type"] == "File"
 ]

 for file in sorted(args.directory.glob("**/*")):
 if file in all_managed_paths:
 continue
 print(str(file))

 return 0


if __name__ == "__main__":
 exit(main())

Wikimedia Hackathon Athens 2023

taavi@majava.org (Taavi Väänänen) — Thu, 01 Jun 2023 00:00:00 +0000

I attended my first in-person technical event last month: the 2023 Wikimedia Hackathon in Athens, Greece! After initially getting involved just before certain major world events it was really nice to finally get an opportunity to meet both people I'd consider friends and that I've never worked with before.¹

I'm the one in the middle here, with a green t-shirt and a hat. Image credit: CC BY 4.0 by Tiago Lubiana.

Back in February I predicted that I'd be working on "Likely something WMCS related? Or MW auth stuff (CA/OATHAuth)". In the end I ended up hopping between multiple projects. To summarize:

I worked with Dreamy Jazz on this CheckUser patch. I really enjoyed getting a perspective on how the CheckUser tool is actually used day-to-day and what kinds of features the CUs find useful.
Kunal and I deployed the RealMe extension to Wikimedia sites. I think I was the first person to verify their Wikimedia user page on a Mastodon profile, even though I was beaten by Raymond on announcing the new feature to the public. This was certainly one of my more interesting experiences deploying stuff to production, considering it was a Friday evening² and there was a very real chance of missing dinner if something went wrong.
I helped Arturo with using toolforge-weld, which is a new helper library project I started a few weeks before the event.
I helped a bunch of people with using Toolforge and Cloud VPS, and processed access and quota requests to unblock people.
I used the opportunity to get some signatures on my GPG key as those are still useful when working on Debian.

I got a "I broke Wikipedia and then I fixed it" sticker. Apparently this was one of only three given out this event so I've been really good at breaking (and then fixing) stuff. I think my go-to story for this is the time when I broke the ability to be logged in, including the possibility to log in or out for anyone with an existing session. Unfortunately they don't hand out t-shirts for breaking stuff anymore.

I ended up attending a couple of sessions that I found interesting. I wish I could've attended more, but as expected I couldn't consentrate on the presentations anymore after attending a couple, and I slept over one that I really wanted to attend :(. I still managed to have very interesting conversations with lots of different people.

Thank you to the WMF for providing me a scholarship to attend the event.

About the travel experience itself:

This was my first time travelling internationally alone. Everything went well, which was nice.
I liked the Copenhagen and Stockholm-Arlanda airports, everything worked as expected and there were not too many people there. I've been in Stockholm and Copenhagen before many times, but never flown via either of those.
I flew with two new aircraft types I have not flown on before: B738 and CRJ900. I already dislike the CRJ family as they are clearly designed for shorter people.

I can't wait to meet everyone again, whether that's at Wikimania in Singapore later this year, next year's hackathon or something else.

Although most of the attendees had at least heard of my name, which made this a really weird experience meeting new people. ↩︎
We did undeploy an another extension just before deploying this one, so technically we didn't change anything. ↩︎

Wikimedia needs to re-think MediaWiki staging environments

taavi@majava.org (Taavi Väänänen) — Fri, 18 Nov 2022 00:00:00 +0000

Wikimedia's Beta Cluster (aka deployment-prep) needs to be replaced with something competely different.

The Beta Cluster Wikitech page describes the projects' ambitions like this:

The Beta Cluster aims to provide a staging area that closely resembles the Wikimedia production environment. It runs MediaWiki and extensions from their master branch, allowing developers and power users to test new code before it goes live on Wikimedia websites.

This was written in early 2013, nearly a decade ago. Back then, the Wikimedia technical community and the WMF were much smaller. The Beta Cluster was one of the first projects on Wikimedia Labs (which is these days known as Wikimedia Cloud Services).

The Beta Cluster has from the very beginning attempted to re-use the same Puppet code used in production, with the intention being that Beta could be used by community members to test changes. This hasn't always been easy as a large part of the code was not designed to run outside production; there even was an attempt in 2015 to build a "stabler Beta Cluster" with an explicit goal of having Puppet do all of the provisioning.

To summarize: The original intention of the Beta Cluster was to allow testing changes to both MediaWiki and the underlying infrastructure.

The infrastructure is developed elsewhere

As far as I can tell, the Beta Cluster was never maintained by the same people taking care of the equivalent production infrastructure. The people maintaining production infrastructure (originally called TechOps, these days known as the SRE team) have different needs than what the MediaWiki developer and testers do.

The nature of the Beta Cluster made it very inflexible for the infrastructure people: for example it was hard to test multiple changes for the same component at the same time, and you needed to be very careful to not break the cluster entirely because that would be disruptive to the MediaWiki developers.

Over time, the SRE team developed other systems for testing the infrastructure. Today the main way used to test infrastructure changes in a production-like environment is Pontoon. Pontoon's primary aim is to simplify starting disposable 'stacks' that are largely independent of each other and are much closer to the actual production environment than what standard Cloud VPS are. Cloud VPS itself has also moved from its original use case of being a development environment for services that were either currently living or planned to live in production.

A staging cluster that's trying to emulate production as closely as possible should be maintained by the same people maintaining production. Otherwise it's going to be impossible to keep up with all the changes and code that for whatever reason can't be easily used outside the environment it was originally written for.

Rather unsuprisingly this kind of environment hasn't been very stable. Even worse, since the people responding to outages are usually not familiar with the system, most fixes end up being hacks that are decreasing the long-term reliability of the entire platform.

There is a demand for a better MediaWiki testing environment

Beta Cluster outages get noticed very quickly, which suggests that people rely on the Beta Cluster working at least somewhat. However, not everyone needs it for the same reason. Common reasons seem to include:

Demoing new features to other people
Testing features that need restricted rights in production
Testing changes in a more 'production-like environment', for example on
- a wiki running the same software versions as the production cluster does
- a multi-wiki setup (a wiki farm) using CentralAuth
- a wiki running a similar configuration compared to production
- a wiki that uses Swift for media storage
- a wiki with a working VisualEditor
- a wiki that's integrated with Wikidata
- a wiki with a statsd stack
- a wiki using CirrusSearch
- a wiki with a proper job queue setup
- ... and the list goes on. You get the point.
Testing with a wider range of real world data¹

Some features are hard to configure. Others need specialized dependencies. Either way, considering the Beta Cluster is (at least currently) only for code already merged to the master branch, we should instead focus on making it easier to run those features locally or make the relevant interfaces safer so we can be more confident in for example something storing files on the local disk also working properly with a Swift backend.

Going forward

The current model for Beta Cluster maintenance has been unsustainable for years, and it shows. The current model doesn't work unless it's maintained by the SRE team directly, which is not optimal for the SRE team. Therefore I think it's reasonable to make the conclusion that we need to replace the current Beta Cluster with a different solution (well, solutions) that are more sustainable to maintain and solve the same problems more efficiently.

What might that solution look like? Honestly, I'm not completely sure. What I do know is that we need to drop the requirement to be as close as possible to production, and instead need to focus on what we need to work on MediaWiki as efficiently as possible.

There are a couple of promising projects I'd like to showcase:

mwcli is a command-line tool which supports managing other services running in Docker.
Patch demo can be used to spin up a MediaWiki instance running a particular patch from Gerrit.

Acknowledgements

Thanks to Tyler Cipriani for providing me access to the 2018 Beta Cluster Survey, which provided helpful insights on how people use the Beta Cluster.

I'll admit that I didn't even think of that, thanks cscott for pointing it out. ↩︎

Introducing Terraform support on Wikimedia Cloud VPS

taavi@majava.org (Taavi Väänänen) — Mon, 24 Oct 2022 00:00:00 +0000

I've been working for a while to make it possible to use Terraform to manage Wikimedia Cloud VPS, and I'm finally happy to announce that for the most part, it's now possible*. Terraform is a popular open-source Infrastructure as Code tool that lets you manage your infrastructure configuration (such as Cloud VPS instances) with a special coding language/framework. You can then manage and review that code with familiar tools, such as Git.

Using Terraform to attach a volume to an existing instance.

If you want to just get started with Terraform on your own project, read the docs. Otherwise keep reading to learn about the technical challenges of making it all possible.

Opening the OpenStack APIs

The core Cloud VPS platform is powered by OpenStack, an open-source cloud computing platform. OpenStack consists of various separate services, some of which are used on our deployment. These services already expose HTTP APIs, and for example the web-based dashboard (Horizon) uses it internally. However, until now these APIs had always been firewalled off from the public internet, and only some specific accounts were allowed to log in from the internal Cloud VPS network without the 2-factor authentication code that we require from all users by default.

Since Cloud VPS uses Wikimedia developer accounts, the passwords used to log in to the dashboard can also be used to log in to other critical tools. For this reason, we don't want to encourage our users to store these passwords as plain text on their computers. Thankfully, OpenStack's Identity service, Keystone, contains a solution that works for this use case: Application Credentials. These are essentially API keys that are tied to a specific user and a specific project. As a part of this project, we've enabled the use of Application Credentials in our configuration and wrote some documentation on how to use them properly.

The second major change needed on our setup was to open up the firewall rules that previously restricted API access to Wikimedia networks. It's now possible to reach the APIs from anywhere from the internet. As a part of this, we've also updated our load balancer configuration to make it easier to limit or block misbehaving clients.

Integrating custom services with OpenStack authentication

Not everything on Cloud VPS uses an upstream OpenStack projects. Some components, most notably the current web proxy service and the Puppet integration (internally called the Puppet ENC API), are powered by custom code that's mostly been written using Python and the Flask framework. Historically they didn't have any proper access control, and instead we simply had configured our firewalls to block access to the APIs from everything except the Cloud VPS control plane servers.

Since this model doesn't let external users use the APIs directly, we had to come up with a new model. I ended up updating both of the affected services to use the Keystone API. After those changes, we've made the web proxy API publicly available like the vanilla OpenStack services, but the Puppet API is still private until it's fixed to work properly with external consumers.

Writing a custom Terraform provider

Just having the web proxy API accessible on the internet doesn't mean that you can directly use it with Terraform. Instead, you need something called a "Terraform provider". Providers are programs that interact with Terraform and the external service (the Cloud VPS web proxy API in this case). There's an existing provider for OpenStack, which works great for anything that uses the vanilla OpenStack APIs, but I ended up writing a custom provider to work with our custom features. Since Terraform and providers are written in Go, I also ended up writing a Go library to work with the web proxy API. Support for the Puppet ENC API is planned once it's been updated to support external clients.

Since the official Terraform module registry (where Terraform downloads the modules your code uses) is heavily built around GitHub, a propiertary platform, I ended up deploying a self-hosted registry on terraform.wmcloud.org to host the new provider. The registry is based on the rekisteri project by Hugo Martins and has been lightly customized to work for this use case.

What's next

It's now possible to do most things via Terraform that you can do via horizon.wikimedia.org. However, there are still a few major exceptions:

You can't manage the Puppet ENC data, as mentioned above.
You can't manage project membership due to some upstream limitations.

It'd be nice to get those fixed. In addition, I'm planning on working to make the entire system more streamlined with the Puppet setup we use to provision instances. Most WMCS managed projects use a standalone Puppetmaster to manage secrets. There are a few manual steps when provisioning or decomissioning instances to sign and revoke the TLS certificates Puppet uses internally, and I want to eventually make Terraform do that for you.

If this sounds interesting: get involved! The entire stack is licensed under free licenses and welcomes new contributors, and in my experience it's a great way to experiment with technology that might be otherwise hard be able toto play with.

This post was originally published on the Wikimedia Technical Blog.

Adding OpenStack Keystone authentication to a Flask application

taavi@majava.org (Taavi Väänänen) — Sat, 01 Jan 2022 00:00:00 +0000

The Wikimedia Cloud VPS platform is powered by the OpenStack platform. In addition to the standard OpenStack services, we've needed to build our own services and APIs that add features specific to our installation.

Historically our OpenStack APIs have not been open to direct use and we've required everyone to use the OpenStack dashboard (Horizon) to manage their VPS project. Our custom APIs have been "secured" by just firewalling access from everywhere else except the OpenStack control hosts, which has worked fine for us in the past. However, we are planning on letting our users directly access the APIs to build their own integrations, and as a part of this work we needed to replace the firewall-based security model with something more flexible. The natural solution is to integrate our custom APIs with OpenStack's Identity service (Keystone).

Our services use the Flask framework. Thankfully this means that we can just use the code that Rackspace has written: the flask_keystone package does exactly what we want. The documentation is a bit lacking though.

flask_keystone makes heavy use of OpenStack's shared library Oslo and uses Rackspace's flask_oslolog too, so you need to install them. At Wikimedia we built our own Debian packages for both Rackspace libraries (Oslo is already packaged for on the official Debian repositories).

A minimal code example looks something like this:

import flask

from flask_keystone import FlaskKeystone
from flask_oslolog import OsloLog
from oslo_config import cfg
from oslo_context import context

key = FlaskKeystone()
log = OsloLog()

cfg.CONF(default_config_files=['/etc/path-to-your/config.ini'])

app = flask.Flask(__name__)

log.init_app(app)
key.init_app(app)


def get_oslo_context(rule, project_id):
 # headers in a specific format that oslo.context wants
 headers = {
 f'HTTP_{name.upper().replace("-", "_")}': value
 for name, value in flask.request.headers.items()
 }

 return context.RequestContext.from_environ(headers)


@app.route("/")
def hello():
 ctx = get_oslo_context()
 return f'Hello, {ctx.user_id} on project {ctx.project_id}!'

if __name__ == '__main__':
 app.run()

You will also need a config file in the path you gave to oslo_config. A sample configuration looks something like this:

[DEFAULT]
auth_strategy=keystone

[keystone_authtoken]
identity_uri = https://openstack.example.org:25000
www_authenticate_uri = https://openstack.example.org:25000

http_connect_timeout = 5

interface = public
auth_version = 3.14
auth_type = password
auth_url = https://openstack.example.org:25000
user_domain_id = default
project_domain_id = default

username = some-username
password = some-password
project_name = some-project

service_type = some-keystone-service-type

delay_auth_decision = True

[flask_keystone]
roles = ""

In the config file, you need to add credentials for an OpenStack account that has access to the identity:validate_token rule. We added a new role for that purpose and granted our service account that role domain-wide.

Authenticating your requests

Now that your service is secured with Keystone authentication, you probably want to make some requests to it. This can be done using the keystoneclient python package:


from keystoneauth1 import session as keystone_session
from keystoneauth1.identity import v3
from keystoneclient.v3 import client

auth = v3.Password(
 auth_url='https://openstack.example.org:25000',
 username='some-username',
 password='some-password',
 project_id='some-project',
 user_domain_name='default',
 project_domain_name='default',
)

session = keystone_session.Session(
 auth=auth,
 user_agent='some-app',
)

client = client.Client(
 session=session(),
 interface='public',
 timeout=5,
)

service = client.services.list(type='some-keystone-service-type')[0]
endpoint = client.endpoints.list(service=service.id, interface='public', enabled=True)[0]
url = endpoint.url

print(session.get(f'{url}/test').json())

Policy-based access control

We can also use OpenStack oslo.policy for policy-based access control, similar to other OpenStack services.

First, create an Enforcer object for your policy rules:

from oslo_config import cfg
from oslo_policy import policy

# you should already have this
cfg.CONF(default_config_files=['/etc/path-to-your/config.ini'])

enforcer = policy.Enforcer(cfg.CONF)
enforcer.register_defaults([
 policy.RuleDefault('admin', 'role:admin'),
 policy.RuleDefault('proxy:index', ''),
 policy.RuleDefault('proxy:view', ''),
 policy.RuleDefault('proxy:create', 'rule:admin'),
 policy.RuleDefault('proxy:update', 'rule:admin'),
 policy.RuleDefault('proxy:delete', 'rule:admin'),
])

Then you can enforce the policies in your route handlers:

@app.route('/v1/proxy')
def all_proxies():
 ctx = get_oslo_context()
 result = enforcer.authorize('proxy:index', {}, ctx)

 if not result:
 return flask.json({'error': 'forbidden'}), 403

 return flask.json(get_proxies(ctx.project_id))

Google Code-In 2019

taavi@majava.org (Taavi Väänänen) — Fri, 24 Jan 2020 00:00:00 +0000

UPDATE: I was a Runner-Up for Wikimedia!

I participated in Google Code-In for the first time this winter. Looks like other people are starting to blog about so here's my blog post so I do not look like a weird antisocial hermit. To be honest, I had a lot of fun participating.

In the end, I completed 31 tasks and had done little under 4.5% of all completed Wikimedia tasks.

such a taboo talking about task counts

All of the tasks I did were in Wikimedia. I've contributed some small patches there before, and I have about 2,000 edits in the English Wikipedia, so I wasn't a newcomer to the environment.

My first task was "Convert two plain text files in MediaWiki core in /doc to Markdown format (III)". It was a fairly simple task, but it was my first patch to the MediaWiki core.

When I claimed my second task, "Add a namespace filter", I a message in IRC from Martin, who I had worked with before (they are a SWAT deployer) asking if I was really this young. Martin also mentored multiple tasks that I completed, so thanks for those.

For the first ~2 weeks, I had completed at least one task every day. Unfortunately then there were some school stuff that required my attention and my streak was cut short. After the streak, I did not do any tasks for about some time due to IRL stuff (school, etc) and lack of motivation. After sometime the new years I returned to work.

I claimed my last task ([Tracker] Create maintenance script to force re-queueing MediaWiki communication methods for specified list of tickets) about two hours before task claiming was closed.

I personally feel that there weren't enough tasks in MW Core or extensions as they are the most interesting projects on (excluding the jsonlint or jshint to eslint converting tasks and extension manifest schema upgrading tasks), most actual coding tasks were on Toolforge tools. I ended working with Python most as it's really common on Toolforge for some reason.

I want to thank Florian for making the notable expections and mentoring a couple of tasks for Core (Create tests for UploadFromUrl::isAllowedHost) and for the Translate extension (GettextFFS should implement isContentEqual).

GCI was definitely a great experience. The mentors knew what they were doing and I had a ton of fun. I am definitely looking for being a part of it in some form next year. Thanks to everyone who was involved this year!

Other people have also written about this years GCI. Check them out after reading my one.