DEV Community: Hoelee

Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About

Hoelee — Mon, 09 Nov 2026 04:25:45 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run authentik as the single sign-on gate in front of my self-hosted stack —
email, dashboard, Synology apps, a remote-access outpost. For a long while it
sat on 2025.8.3, and a year of releases piled up. This is the story of
bringing it all the way to 2026.8.1 in one sitting, and everything that
broke along the way — especially the one mistake that took Single Sign-On
completely offline and made every internal app ask for a password again.

What is authentik, and why run it?

authentik is an open-source identity provider (IdP) — the thing that answers
one question over and over: "who is this person, and are they allowed in?"
Think of it as the front door to a whole building of apps. Instead of every
app running its own username/password screen (and its own list of bugs, its own
"forgot password" flow, its own 2FA), you make all of them ask authentik instead.

It speaks the protocols that matter for real deployments:

OIDC / OAuth2 — the modern single sign-on standard (what "Sign in with Google" uses under the hood).
SAML — the enterprise standard (for tools like Grafana, Jira, or any legacy line-of-business app).
LDAP — so older clients and NAS boxes that only know LDAP can join too.
Proxy / forward-auth — it sits in front of an app via a reverse proxy and gatekeeps access before a request ever reaches the app.

The practical benefits, in order of how much they actually matter:

One login for everything. Log in once, move freely between every app. Users stop juggling a dozen passwords (and the support tickets that go with them).
One place to lock things down. MFA, password policy, session limits, and account recovery live in authentik once — not re-implemented per app.
One place to audit. Every login, on every app, for every user, in one log. That's the difference between "we think nothing happened" and "we can prove it" when a question comes up.
Self-contained and self-hosted. You own the data. No per-user SaaS fees that scale with your headcount, no lock-in — it's AGPL, runs in Docker.

For a small business or a solo operator running twenty-something services
(email, dashboards, NAS apps, internal tools), authentik is the difference
between "every app has its own flimsy password" and "one hardened front door."

Why upgrade at all

Version 2025.8.3 wasn't broken. But it had fallen far enough behind that a
stack of CVEs had landed in the releases after it, and I was getting ready to
do per-application branding. authentik's own policy is that you can't jump
major versions — it enforces a stepwise path. So the plan was:

2025.8.3 → 2025.10 → 2025.12 → 2026.2 → 2026.4 → 2026.6 → 2026.8

Six hops, one at a time, with a migration and a health-check between each.
Before touching anything, the one non-negotiable step: back up the database.
authentik doesn't support downgrades. If a migration half-runs, you're restoring
from dump, not rolling back an image tag.

sudo docker exec authentik-postgres pg_dump -U authentik authentik > authentik-backup.sql

Pitfall 1: Portainer is the source of truth, not the compose file

My first instinct was to edit the docker-compose.yml on disk and up it.
Wrong. The stack is managed by Portainer (stack 143), which keeps the real
compose and the real environment variables in its own store. The .env on disk
was stale — its PG_PASS didn't match what Portainer actually ran.

The correct update path is via the Portainer API, not the filesystem:

Update the image tags in the compose content.
docker pull the new images first (so the API call doesn't time out mid-pull).
docker stop + docker rm the running containers (fixed container_name will otherwise collide on redeploy).
PUT /api/stacks/143?endpointId=2 with the new StackFileContent + Env.

I hit the colliding-container error, the pull-timeout error, and a network-attach
problem where the rebuilt authentik-server only joined one of its two networks
and couldn't resolve postgres-server. Each one is a five-minute fix once you
know what you're looking at, but together they ate the better part of the evening.

Pitfall 2: the storage mount moved (2025.12)

Up to 2025.12, brand assets lived under /media/ and were served at /media/....
After 2025.12, the storage layout changed: files moved to a /data/media
structure served under a new /files/media/public/<name>?token=... URL with a
JWT signature. My containers were still mounting ./media:/media, so every
logo, favicon, and background image 404'd the moment I crossed that version.

The fix is the documented migration:

mkdir -p data && mv media data/media

…and change the mounts to ./data:/data. The new file backend also refuses to
work unless /data is an actual mount point — the earlier version of my fix
used a symlink, which the backend's is_mount() check rejected with
No file management backend configured.

Pitfall 3: RBAC migration leaves a poisoned session table

2025.12 removed the old authentik_core.User_groups model in favor of the RBAC
rework. The migration ran clean, but old sessions still held references to
the deleted model. Result: the login page kept throwing
LookupError: App 'authentik_core' doesn't have a 'User_groups' model.

Not django_session — that was empty. The real culprit was authentik's own
authentik_core_session table. Clearing it (and the other session tables)
forced everyone to log in again and cleared the error:

TRUNCATE authentik_core_session; -- named authentik_core_authenticatedsession in some versions

One side effect worth knowing: this also invalidates every OIDC refresh token
your apps were holding. They'll bounce the user to a fresh login once, then
recover. It's a one-time annoyance, not a bug.

Pitfall 4: trusted proxies are now opt-in (2026.8)

2026.8 tightened the default forwarded-header handling. Previously authentik
trusted all the private ranges; now it only trusts what you list explicitly.
Behind a Synology reverse proxy forwarding to localhost, that means:

environment:
  AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,::1/128

Skip this and the proxy headers get rejected, which surfaces as auth failures
that look like almost anything except what they actually are.

Pitfall 5: the one that broke SSO — authorization_flow vs authentication_flow

This was the expensive mistake, and it's the kind of thing that's easy to get
wrong if you're doing per-application branding.

A provider in authentik has two flow fields, and they mean different things:

# "Flow used for authentication when the associated application is
#  accessed by an un-authenticated user."      ← the LOGIN page
authentication_flow = models.ForeignKey(...)

# "Flow used when authorizing this provider."  ← the OAuth consent page
authorization_flow = models.ForeignKey(...)

authentication_flow is the login page. This is where you put a per-application flow to customise the title and background.
authorization_flow is the consent/authorize step for a user who is already logged in.

I wanted different background images per app, so I created one flow per
application and pointed the provider's authorization_flow at it. Instant
breakage: a user who was already authenticated still got walked through the
full identification → password → MFA flow every time they opened another app.
Single Sign-On was, effectively, gone.

The fix was a single UPDATE to put the two fields back where they belong:

UPDATE authentik_core_provider
SET authorization_flow_id = '1d85b1b1-...',   -- explicit-consent flow
    authentication_flow_id = 'aa##-per-app-flow'
WHERE ...;

And the branding itself — the per-app background and title — goes on the
flow's own background and title fields (writable since 2026.8), not on
the brand's domain match.

There was a second, related gotcha hiding behind this one. When I created the
18 per-app flows, they all landed with designation=authentication, and my
auth.hoelee.com brand had flow_authentication set to NULL. authentik's
fallback when a brand has no explicit auth flow is to pick the first
authentication flow by slug, alphabetically — which happened to be
auth-agent, not default-authentication-flow. So the root login page started
showing my agent's background image. Setting the brand's flow_authentication
to the real default flow fixed it.

And one more that compounds with the RBAC pitfall: creating a flow sets its
background but not its stages — stages is read-only on the flow object.
An empty flow with no stage bindings is exactly what produced an infinite
redirect loop on the login page earlier in the migration. Stage bindings are
created separately:

POST /api/v3/flows/bindings/   # { target: "<flow pk>", stage: "<stage pk>", order: N }

What I'd do differently

The whole ordeal came down to three preventable patterns:

Never guess at a field's semantics — I treated authorization_flow as "the login flow" when the source-of-truth is the model definition, which spells out the difference in the field's own docstring.
Keep the API surface at arm's length — API tokens kept expiring mid-run with every server restart, so I ended up doing the critical fixes directly against the database with psql. Reliable, but worth scripting before the cluster is on fire, not during.
One breaking change per restart — I tried to reason about storage, RBAC, and proxy changes all in one go. Each would have been trivial if isolated and verified independently.

The result

authentik now runs 2026.8.1 — current, patched, all containers healthy —
with 18 applications each showing their own background and title on the login
page, and SSO working across every subdomain. Seventeen of the applications I
use daily went from "asks me to log in again every time I switch apps" back to
"log in once, move freely."

The lesson worth carrying: a major-version upgrade on auth infrastructure is
roughly 10% "change the image tag" and 90% "the data model, storage layout,
and proxy rules all shifted underneath you." Back up, go one version at a time,
and when something behaves in a way that makes no sense, read the field name
again before you reach for another config.

Want single sign-on for your business?

If you're running several internal apps — a dashboard, an email server, a help
desk, a file server — and your team is still logging into each one separately
(or reusing the same password everywhere), I set up and maintain exactly this
kind of infrastructure. I'll deploy authentik, wire it to your existing apps,
add MFA, and make sure "log in once" actually works — then leave you with a
handover so you're never locked in.

Reach me at me@hoelee.com or WhatsApp
+60 12-797 2969, or see what I do at
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Upgrading n8n v1 to v2: Seven Deprecations in One Log File

Hoelee — Sun, 08 Nov 2026 04:25:21 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run n8n as the automation backbone for my self-hosted stack — it handles
file-delivery permissions, database backups, and a text-to-speech API that a
reading app depends on. It had been sitting on the 1.123.x line for the
better part of a year, quietly doing its job.

Then I pulled n8nio/n8n:2.40.1 and restarted the container. The upgrade
itself took about ninety seconds. Understanding what it broke took the rest
of the evening — and almost all of it was already written down in a single log
file that n8n prints on boot. I just hadn't read it carefully enough the first
time.

This is that log file, decoded, so you can plan your own v1 → v2 jump instead
of discovering these at 11 PM.

Why it matters

Major-version upgrades of an automation platform are different from upgrading
a leaf service. n8n is the thing that runs everything else: if it comes up
broken, your backups, your access-control syncs, and your internal APIs all
stop with it. Worse, most of what breaks in v2 doesn't throw an error — it
logs a deprecation notice once and then quietly does something different.

The seven items below are the ones that actually applied to a real, messy,
production-shaped install. Three of them changed behavior in my stack. One of
them silently turned a feature off.

Start here: n8n tells you what's wrong at boot

Before touching a single workflow, read the container log from the top. On a
fresh v2 boot, n8n prints an explicit deprecation block:

There are deprecations related to your n8n setup. Please take the recommended
actions to update your configuration:
 - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for
   both test and production webhooks.
 - N8N_UNVERIFIED_PACKAGES_ENABLED -> The default for this variable will
   change to `false` in a future version.
 - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be
   removed in a future version.
 - N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced
   from 300 (5 minutes) to 60 (1 minute) in a future version.
 - N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> The default will be
   reduced from 2 GiB to 256 MiB in a future version.
 - N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> The default will be reduced from
   5000 to 1000 in a future version.

That block is the core of your migration checklist. Five of my seven configuration and deprecation gotchas are right inside it, and the other two surfaced a few lines away in the same startup log.

1. Your env values can now fail schema validation

Here is the one that confused me most, because it looked like a nonsense error:

Telemetry event "Instance started" failed schema validation:
execution_variables.executions_data_save_on_error: Invalid option:
expected one of "all"|"none"

I had EXECUTIONS_DATA_SAVE_ON_ERROR=error set — a value that was perfectly
legal in v1, and that I'd chosen deliberately, because saving only failed
executions is the sane default for a busy instance. In v2 that value is no
longer in the allowed set, which is now all or none.

The failure mode is the interesting part. It didn't crash. It didn't even warn
in a way that reads like an error at a glance — it emitted a telemetry schema
validation message, which sounds like an n8n-internal problem, not a
configuration problem of mine. The setting was effectively ignored.

The fix is to move that intent somewhere coherent: pick a legal value and
control volume with pruning instead.

EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=none
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_MAX_AGE=336
EXECUTIONS_DATA_PRUNE_MAX_COUNT=10000

Lesson: in v2, treat your environment variables as a typed interface with
a schema. An invalid value may be dropped silently rather than rejected loudly.

2. `WEBHOOK_URL` is deprecated for `N8N_WEBHOOK_URL`

If you publish webhooks behind a reverse proxy — which you almost certainly do,
because that's how they become reachable — the base URL variable is load-bearing.
It's what makes n8n report the public webhook path instead of
http://localhost:5678/....

The old name still works today, so this one won't bite immediately. But note
the wording: the new variable sets the base URL for both test and production
webhooks. In my setup those had drifted apart in behavior, which is exactly
the class of bug this consolidation is meant to eliminate.

# before (still functional, deprecated)
WEBHOOK_URL=https://auto.example.com/

# after
N8N_WEBHOOK_URL=https://auto.example.com/

3. Internal task runner mode is going away — and mine was already broken

This one was sitting in my logs the whole time, several lines above the
deprecation block, and I'd been reading past it for months:

Failed to start Python task runner in internal mode. because Python 3 is
missing from this system. Launching a Python runner in internal mode is
intended only for debugging and is not recommended for production.

If any workflow of yours uses a Python Code node, it has not been running
in internal mode at all — there's no Python in the stock image. JavaScript
Code nodes are fine (a JS runner registers normally), which is why this can go
unnoticed indefinitely: everything looks healthy.

v2 makes the direction of travel explicit: switch to external mode and share
an auth token with a separate launcher process.

N8N_RUNNERS_MODE=external
N8N_RUNNERS_AUTH_TOKEN=<a long random string>

(Note: switching to external mode means pairing the main container with the n8nio/runners sidecar image in your compose stack sharing that same token, which actually ships both the JS and Python runtimes.)

Lesson: "internal mode is deprecated" is the headline, but the real
finding is that a runner type can be silently non-functional for months.
Check docker logs for the runner registration line, not just for up/down.

4. Task timeout drops from 300s to 60s

This is the one I'd flag hardest for anyone with slow workflows:

N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced
from 300 (5 minutes) to 60 (1 minute) in a future version.

Be honest about your own workloads. Do you have a Code node that loops over
thousands of records, or an HTTP call to a slow upstream? Mine do — a nightly
reconciliation walks every customer and calls an external API per record. On a
future upgrade, that stops at sixty seconds with no config change on my side.

Set it explicitly now, while you're already in the file:

N8N_RUNNERS_TASK_TIMEOUT=300

The general principle for every deprecation of the form "the default will
change": if you rely on the current default, pin it explicitly. Otherwise
the upgrade is a silent behavior change, and you'll debug it as a bug rather
than recognise it as a stale default.

5 & 6. Two compression-node limits shrink (2 GiB → 256 MiB, 5000 → 1000 entries)

These two travel together and matter only if you use compression/decompression
nodes on large payloads — which is easy to end up doing when you're shuttling
database dumps or archives through a workflow.

N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> reduced from 2 GiB to
256 MiB in a future version.
N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> reduced from 5000 to 1000 in a
future version.

An eighth of the memory ceiling and a fifth of the entry limit. Nothing errors;
the node just refuses at a threshold you didn't set. Pin both if you're near
either.

7. The storage path renames in v3 — and you have a volume mounted at the old one

Not a v2 breakage, but v2 warns about it, and it's the one with real data-planning
implications:

Deprecation warning: The storage directory "/home/node/.n8n/binaryData" will
be renamed to "/home/node/.n8n/storage" in n8n v3. To migrate now, set
N8N_MIGRATE_FS_STORAGE_PATH=true. If you have a volume mounted at the old
path, update your mount configuration after migration.

Read that last sentence again: if you have a volume mounted at the old path,
update your mount configuration after migration. If you set the migration flag
and keep your old bind mount, you now have two directories and your binary data
lives in whichever one the container is actually pointed at. Do the rename and
the mount change in the same maintenance window — not one now, one "later".

The one outside the deprecation block: my AI sandbox disabled itself

Here's the finding that had nothing to do with a deprecation notice, and that
I'd never have caught without reading the full boot sequence:

Sandbox: enabled=false provider=n8n-sandbox (DB override; env was enabled=true
provider=n8n-sandbox)

My environment said enabled. The database said otherwise. The database won.

The env var was N8N_INSTANCE_AI_SANDBOX_ENABLED=true, and it was still set
correctly on the container. But a value persisted in n8n's own configuration
store overrode it at startup — and the only place that conflict is reported is
inside a parenthetical in a log line.

This is a genuinely valuable debugging lesson beyond n8n: when a feature is off
despite the env var being obviously right, suspect a persisted settings layer
that outranks your environment. The container config is not always the last
word. Grep for the feature name in the logs, and don't stop at the env value.

What I'd do differently

Read the boot log before declaring the upgrade done. Every deprecation that mattered to me was printed on startup, in one block, on the first run. My v1 habit was to check "is it up, do the workflows run" — which is exactly the check that misses all seven of these.
Treat "the default will change" as a to-do, not a warning. Five of the seven items are future-default changes. Pinning them now costs one edit and converts a mystery outage later into a config diff.
Diff the container config against n8n's own stored config. The sandbox override taught me that env is one of two inputs, not the source of truth. When behavior and configuration disagree, believe the behavior and go find the higher-priority layer.
Pin the image tag and keep the previous one. I jumped v1 → 2.40.1 directly. Having the old image on disk is what makes a rollback a docker run instead of a rebuild.

The result

n8n is on 2.40.1 with the full deprecation block resolved: the schema-invalid
value corrected, N8N_WEBHOOK_URL in place, the JS runner registering cleanly,
and the task timeout and compression limits pinned so the next upgrade is a
no-op rather than a surprise.

Everything downstream kept working — the file-permission syncs, the nightly
backups, the text-to-speech endpoint. That's the outcome worth aiming for with
an upgrade like this: not "it came back up", but "it came back up and the
next three upgrades are already paid for".

The uncomfortable part is how much of it I could have known in advance. n8n
handed me the entire list, unprompted, at startup. The upgrade was never the
hard part — reading the output was.

Running automation you'd rather not babysit?

I build and maintain self-hosted automation — n8n workflows, Docker stacks,
and the glue between apps that were never designed to talk to each other. If
you're facing a major version upgrade, or you have automation that works until
it doesn't, I plan the migration, do it in a maintenance window, and document
every config decision so the next upgrade is boring.

Reach me at me@hoelee.com or
WhatsApp +60 12-797 2969, or see what I do at
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses

Hoelee — Sat, 07 Nov 2026 04:25:54 +0000

Originally published on my blog — read it there with the full context, code and updates.

I upgraded a CodeIgniter 4 app from 4.6.3 to 4.7.4. I read the upgrade guide, checked every documented breaking change against the codebase, and confirmed none of them applied. Then I ran the app and it fataled twice.

Both fatal errors were the same kind of problem, and that kind of problem is not mentioned anywhere in the upgrade guide. This is a short post about what actually breaks and why the guide can't warn you.

Why this matters

Composer has a scope boundary that is easy to forget. composer update writes to vendor/ — the system scope, the framework's own files, which it owns and can replace freely.

Your app/Config/*.php files are project scope. They are yours. Composer will never touch them, and the upgrade guide describes them as "you may want to merge these changes" rather than "this will break".

But when 4.7's framework code reads a property that your 4.6-era config class does not define, you get a fatal — not a deprecation, not a warning. The guide cannot enumerate this, because the set of properties is the intersection of what the new framework reads and what your config file happens to contain, and it has no visibility into the second half.

If you are upgrading a CI4 app more than a minor version behind, budget for this. It took me about an hour and 14 config files.

What I checked first

The documented breaking changes in 4.7.0, and whether each applied:

Documented change	Affects this app?
`Model::insertBatch()` / `updateBatch()` return values	No — no Models used
Entity casting changes	No — no Entities
Validation rule changes (`regex_match`, `differs`)	No
Encryption handler defaults	No
Uploaded-file / image validation behaviour	No — no uploads
`IncomingRequest` internal changes	No
Removed `Session` class properties	No — see below
`PageCache` constructor signature	No — see below

Two of those needed a closer look, and both turned out to be false alarms worth describing, because they look identical to a real hit if you only grep:

"Removed Session class properties" — grepping for session in my configs lit up immediately. But the properties in question (sessionDriver, sessionCookieName, and friends) on my Config\App are Config\App's own properties, which were not removed. The guide refers to properties on the Session class. Same word, different class.

"PageCache constructor signature changed" — I reference pagecache — but only as a filter alias in Config\Filters. That does not extend or instantiate the class, which is the only thing that breaks. A reference is not a subclass.

Both checks taught me the same thing: grep finds mentions, not usages. Verify the class in the match is the class the guide means, and that you actually extend or call the thing that changed.

The two fatals that were not in the guide

Both surfaced on a clean boot. Both are properties that 4.7 framework code reads and 4.6-era config files do not define.

1. `Config\App::$permittedURIChars`

Undefined property: Config\App::$permittedURIChars

Required by 4.7's Router, which validates incoming URI characters against this property. Absent from a 4.6-era App.php, the Router throws before it routes anything.

// app/Config/App.php — append to the class
/**
 * CI4 4.7 compatibility: allowed characters in a URI.
 * Required by the 4.7 Router.
 */
public string $permittedURIChars = 'a-z 0-9~%.:_\-';

2. `Config\Format::$jsonEncodeDepth`

Undefined property: Config\Format::$jsonEncodeDepth

Required by 4.7's JSONFormatter. This one broke the app's report engine, because the engine's JSON response is decoded with json_decode() and the failure surfaced much further downstream as a confusing parse error rather than the real "undefined property" message.

// app/Config/Format.php — append to the class
/**
 * CI4 4.7 compatibility: json_encode() depth limit.
 */
public int $jsonEncodeDepth = 512;

That second one is worth dwelling on. The visible error was:

Failed to parse JSON string. Malformed UTF-8 characters

Which points at encoding. The actual cause was a missing config property. If a framework error names a symptom rather than a cause, go looking for the real exception in the log before acting on the message you were given — I lost time chasing UTF-8 that was never broken.

How to find the rest before they bite you

Both fatal errors were found by running the app. Running is the reliable method, but you can get ahead of it by diffing property coverage.

The approach: parse the framework's shipped config defaults from vendor/, parse your project's config classes, and list properties defined in the former but missing from the latter. That gives you the whole class of problem at once instead of one fatal per restart.

import re, pathlib

sys_ = pathlib.Path('vendor/codeigniter4/framework/system/Config')
app_ = pathlib.Path('app/Config')

def props(path):
    try:
        src = path.read_text(encoding='utf-8', errors='replace')
    except FileNotFoundError:
        return set()
    # public/protected/private $name = ...
    return set(re.findall(r'(?:public|protected|private)\s+(?:[\w\\\[\]|?]+\s+)?\$(\w+)', src))

# names whose properties 4.7 requires but a stale config may omit
for name in ['App', 'Cache', 'ContentSecurityPolicy', 'CURLRequest', 'DocTypes',
             'Email', 'Encryption', 'Exceptions', 'Format', 'Honeypot',
             'Migrations', 'Paths', 'Routing', 'Toolbar', 'View']:
    missing = props(sys_ / f'{name}.php') - props(app_ / f'{name}.php')
    if missing:
        print(name, '->', sorted(missing))

Run it against the new vendor/ after composer update and before you boot the app. Anything it prints is a candidate fatal.

In this project it flagged 14 config files. I merged the missing properties into each one, appended in a marked block at the end of the class so existing settings and behaviour were untouched:

    // ----------------------------------------------------------------
    // CI4 4.7 compatibility — properties the 4.7 framework reads.
    // Appended as a block so existing settings above are unchanged.
    // ----------------------------------------------------------------

Two config files the framework now ships had no counterpart at all and needed creating rather than merging: Hostnames.php and WorkerMode.php. Copy those straight from vendor/codeigniter4/framework/app/Config/.

Also worth knowing before you upgrade

Validate app.baseURL properly. 4.7 throws a ConfigException on values 4.6 accepted. 'http://localhost/' — which had been sitting in my local .env for months, working fine — now throws. Production was unaffected because it used a real URL, but any dev environment with a shorthand baseURL will refuse to boot.

The security case is real. 4.7.4 alone patched an uploaded-file extension bypass (is_image / mime_in), a SQL injection in deleteBatch(), and a path traversal in UploadedFile::move(). None were exploitable in this app — it does no uploads and uses no database — but all three become live the moment those features are added. Upgrading before you need those features is the cheaper order.

PHP floor is now 8.2. Check your host before starting, not after.

What I'd do differently

Run the app against the new vendor/ before doing anything else. composer update, then boot it, then fix what breaks. Reading the guide tells you what might change; only running tells you what did.

Do the property diff as a matter of course. It converts a serial debugging session into one report.

Do the upgrade as its own commit, on its own. I did, and it made the two fatals trivially bisectable — one git show told me exactly which change introduced each. Mixing a framework bump into a feature branch turns a five-minute diagnosis into an archaeology exercise.

Read the changelog, not just the upgrade guide. The guide lists what the maintainers judged likely to break you. The changelog lists what changed. They are not the same set, and the gap is exactly where these two fatals lived.

The result

4.6.3 → 4.7.4 on a 17,000-line CodeIgniter app: 14 config files merged, two new config files added, two fatals found and fixed, and every route verified returning 200 — including a real report generation producing 19 A4 pages with byte-comparable output to the pre-upgrade baseline. Zero errors in the application log.

Running an older CodeIgniter app? I do framework upgrades like this one — CI4 minor bumps, PHP version migrations, and the config-merging work that the upgrade guide assumes you will figure out yourself.

WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

The JPA Field That Turns a Silent Lost Update Into a 409

Hoelee — Fri, 06 Nov 2026 04:25:50 +0000

Originally published on my blog — read it there with the full context, code and updates.

Two people open the same post. Both edit it. Both save. One of the saves
disappears without a trace — no error, no warning, and the API answered
HTTP 200 for both of them.

That is the lost update problem, and it is the quietest data-loss bug in
read-modify-write APIs. I hit it while building my Spring Boot reference
service — a Spring Boot 3.5.16 (Java 21) REST API where editors create and
update posts from a browser page. The fix turned out to be one @Version
field on the JPA entity, one field on the request contract, and one
exception that the API layer maps to HTTP 409 Conflict. This post traces
that path through the actual code.

The problem: an edit that vanishes with no one to blame

Why does one editor's change disappear when two people save the same
record?

Follow the timeline. Editor A and editor B both fetch the same post.
A saves first: the row is updated and the API answers 200. B saves a
moment later: the row is updated again, this time over A's text, and
the API answers 200 again. Every request succeeded; every response told
its caller that their save is the state of the record. A's edit is
simply gone, and neither client has any way to know.

The failure is silent because it is normal read-modify-write behaviour,
not an anomaly. That is exactly why the bug survives in real systems, and
why it is worth fixing properly in something meant to demonstrate
engineering judgement.

What I tried first, and why it failed

My first sketch of the update flow was the obvious one: GET the record,
edit it, PUT it back with the new title and body and nothing else.

GET  /api/posts/{id}    # read the record
PUT  /api/posts/{id}    # write it back with a new title and body

The server loads the row, applies the update, and returns 200. It has no
way to notice that the row moved underneath the client between the GET
and the PUT — the read and the write are two unrelated requests, and
nothing in the contract connects them. The last write wins, and both
writers are told they succeeded.

A 200 is worse than an error here. An error at least tells the losing
client that something happened and gives them a reason to look. A 200
tells both clients that their save is the current state of the record,
which is a lie for one of them. The losing edit now exists nowhere — not
on screen, not in the database — and because the client believes it
saved, nobody goes looking for the data.

The fix: a @version field, a contract, and a conflict

The fix is five small parts, all of them in the reference service's source.

1. The entity carries a version column

@Entity
@Table(name = "posts", indexes = @Index(name = "idx_posts_title", columnList = "title"))
@EntityListeners(AuditingEntityListener.class)
public class Post {

    @Id
    @GeneratedValue
    @UuidGenerator
    private UUID id;

    @Column(nullable = false, length = 160)
    private String title;

    @Column(nullable = false, length = 10_000)
    private String body;

    @Version
    private long version;
    // ...
}

@Version tells Hibernate to treat version as an optimistic lock. Every
UPDATE it generates becomes conditional:

UPDATE posts
SET title = ?, body = ?, version = version + 1, ...
WHERE id = ? AND version = ?

If the row's version no longer matches the value the statement was built
with, zero rows change and Hibernate rejects the flush instead of quietly
overwriting. The entity never bumps the counter by hand — the update
method only touches title and body; the version moves with the write
itself.

2. The version is part of the API contract

A lock is useless if the client cannot take part in it, so the version
travels through the API in both directions. The response record exposes
it:

public record PostResponse(UUID id, long authorId, String title, String body,
                           long version, Instant createdAt, Instant updatedAt) {
}

and the update request requires it back:

public record UpdatePostRequest(
        @NotBlank @Size(max = 160) String title,
        @NotBlank @Size(max = 10_000) String body,
        @NotNull @Min(0) Long version) {
}

The learning note next to UpdatePostRequest says it plainly: "the
expected version is part of the update contract, making optimistic
locking visible to clients."

3. The service checks it inside the transaction that owns the write

@CachePut(cacheNames = "posts", key = "#postId")
@Transactional
public PostResponse updatePost(UUID postId, UpdatePostRequest request) {
    Post post = postRepository.findById(postId)
            .orElseThrow(() -> new PostNotFoundException(postId));
    if (post.getVersion() != request.version()) {
        throw new PostVersionConflictException(postId);
    }
    post.update(request.title().trim(), request.body().trim());
    return PostResponse.from(postRepository.saveAndFlush(post));
}

The load, the comparison and the save all happen inside one
@Transactional method, and that is the part that makes the check
honest: the version is compared against the row as it stands at write
time, not against a snapshot from an earlier request. If the read and
the write lived in different transactions, the check would compare
against a version the row has already left behind, and the whole scheme
would be theatre. The learning note on PostService names the rule:
"service methods centralize transaction boundaries, cache coherence, and
optimistic-locking rules." The service also runs with
spring.jpa.open-in-view: false, so no lingering session serves a stale
entity across requests — the row is re-read inside the writing
transaction.

The explicit check catches stale requests deterministically. The
@Version column still matters underneath: if another commit slips in
between the check and the flush, the conditional UPDATE affects zero
rows and Hibernate refuses the write anyway.

4. The handler maps the exception to a 409

The exception itself carries the message a client can act on:

public class PostVersionConflictException extends RuntimeException {

    public PostVersionConflictException(UUID postId) {
        super("Post %s has changed; fetch it again before retrying".formatted(postId));
    }
}

and the error boundary turns it into the right HTTP answer:

@ExceptionHandler({PostVersionConflictException.class, ObjectOptimisticLockingFailureException.class})
ProblemDetail handleConflict(RuntimeException exception) {
    return problem(HttpStatus.CONFLICT, "POST_VERSION_CONFLICT", exception.getMessage());
}

HttpStatus.CONFLICT is HTTP 409, and the response is a problem document
— a type URI, a stable machine code (POST_VERSION_CONFLICT) and a
human-readable detail. Mapping both our explicit exception and Spring Data JPA's ObjectOptimisticLockingFailureException (thrown when Hibernate detects a race condition) guarantees that concurrent write collisions always return 409 rather than an unhandled 500. The service enables the framework's problem-details
support in its configuration (spring.mvc.problemdetails.enabled: true),
so validation and every other error ride the same shape.

5. The whole path, end to end

stale PUT /api/posts/{id}
  -> PostService.updatePost re-reads the row inside its transaction
  -> post.getVersion() != request.version()
  -> PostVersionConflictException: "fetch it again before retrying"
  -> ApiExceptionHandler.handleConflict
  -> HTTP 409, ProblemDetail with code POST_VERSION_CONFLICT
  -> the client knows its premise was stale

One @Version field, one version in the request contract, one
exception, one handler method. Nothing on the controller changed —
updatePost still looks like an ordinary PUT.

Why a conflict is the correct answer, not a retry

When two editors produce two different versions of the same record, the
server cannot know which intent is the one meant. Both saves are
legitimate from their authors' point of view, and replaying the stale
write just replays the same data loss.

The honest verdict is HTTP's own definition of 409: the request conflicts
with the current state of the resource, and the client is told exactly
how to resolve it — "fetch it again before retrying". The losing client
re-reads the record, sees the other writer's change, and a human decides
what to keep. The silent overwrite becomes a visible decision point,
which is the entire purpose of the exercise.

What the rest of the reference service proves

The version field is one layer of a small system, and the rest of it is
verifiable the same way — every row below is a class or a config file in
the repository, not a claim:

Layer	What it does	Where it lives
Caching	Bounded Caffeine cache on single-post reads (max 500, TTL 10 min, both from config); `@Cacheable` on read, `@CachePut` on update, `@CacheEvict` on delete; `GET /api/showcase/cache` exposes requests, hits, misses and hit rate	`CacheConfig`, `CacheProperties`, `ShowcaseMetricsController`
Security	Stateless HTTP Basic with BCrypt; pages and read APIs are public, every mutation needs the `EDITOR` role; local defaults are a throwaway `demo-editor`/`changeit` account overridable via env vars	`SecurityConfig`, `application.yml`
Errors	Request records validate at the boundary; one `@RestControllerAdvice` returns consistent problem documents with machine codes — `POST_NOT_FOUND` (404), `POST_VERSION_CONFLICT` (409), `VALIDATION_FAILED` (400) with a field-error map	`CreatePostRequest`, `UpdatePostRequest`, `ApiExceptionHandler`
Configuration	Local profile runs an in-memory H2 database in PostgreSQL mode with `ddl-auto: update` and seeded sample posts; `prod` selects PostgreSQL, takes every credential from `APP_*` env vars and sets `ddl-auto: validate`	`application.yml`, `application-prod.yml`
Tests	One `@SpringBootTest` suite asserts the rendered Thymeleaf pages, unauthenticated 401s versus editor 201s, the `VALIDATION_FAILED` problem response, public search, and cache hits after repeated reads	`DemoApplicationIntegrationTest`

Hardening This Pattern for Production Rollout

Two things change the moment this pattern moves to production. The project
README says both, and they are worth repeating as my own judgement.

First, schema management. I would add Flyway or Liquibase migrations
before ever setting ddl-auto to validate. The local development profile
uses update, which is convenient on a scratch database and dangerous as
a habit — production schema should be versioned code, not a side effect
of startup.

Second, secrets. The local editor account is intentionally a non-secret
demo account with default credentials. For anything real, I would keep
credentials in a managed secret store and refuse to boot without them,
instead of falling back to a default. The prod profile already takes
every credential from environment variables with no committed secrets —
that is the shape I would keep, minus the fallbacks.

The result

A stale update now returns HTTP 409 with code POST_VERSION_CONFLICT
instead of a 200 that silently discards the other writer's change. The
loss can no longer happen quietly: the failure is loud, actionable —
"fetch it again before retrying" — and the integration test suite asserts
the access and validation behaviour end to end.

The whole fix is one annotation on one field, plus plumbing that makes
the version part of the contract. If a record can be read by two people
at the same time, that field is the difference between a lost edit and a
conflict both clients can see.

I build Java/Spring back ends and full-stack web applications — REST APIs
with JPA, optimistic locking, caching, validation and integration tests,
from the browser page down to the database. If you have a form or a field
where edits keep quietly disappearing, tell me about it:
WhatsApp ·
me@hoelee.com ·
hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

When a Self-Hosted Model Gateway Tripped Trust & Safety

Hoelee — Fri, 06 Nov 2026 00:17:31 +0000

Originally published on my blog — read it there with the full context, code and updates.

A single self-hosted OpenAI-compatible gateway served every model tier: long-context reading, refactors, cheap background jobs. One key, one model picker, one dashboard where spend was visible in one place. Six independent workflow seats — repo indexing, CI triage, interactive refactors — sat behind it, each authenticating with its own subscription OAuth credential.

Then all six came back disabled inside one short detection window. None of them had been doing anything unusual. What tripped was a scheduled credential check, and what it found was a block already applied upstream.

The interesting part was not the block. It was the ten minutes I spent rotating credentials that were never wrong.

The error that sent me the wrong way

For the first ten minutes, every consumer of the gateway's error reported a credential failure — orchestrator, shell probe, CLI — because that is what the body said: 401 invalid_api_key. The upstream had actually returned a policy decision.

Three probes separated the layers, and they are worth keeping for any "the model stopped answering" ticket: confirm the gateway answers its own health endpoint; confirm a model from an unrelated provider succeeds on the same key; then loop the subscription models and compare the error codes. One model failing means a broken model. Every model on one provider failing identically, while other providers succeed on the same key, means the provider relationship is closed — and no key rotation moves that.

The real defect is the error contract, not the outage:

# Before: policy decisions disguised as auth failures
if upstream.status == 403:
    return jsonify(error="invalid_api_key"), 401

# After: distinct classes, distinct client behaviour
if "Terms of Service" in upstream.text or "disabled in this account" in upstream.text:
    return jsonify(error="upstream_policy_block",
                   upstream_status=403,
                   alert="credential-policy"), 502
return jsonify(error="invalid_api_key"), 401

A 401 that means "your key is wrong" and a 403 that means "the vendor closed this seat" must never collapse into the same response. Flattening them costs the responder the first ten minutes of every incident.

Why a shared token is not a shared key

An API key is a bearer credential: whoever holds it can use it. An OAuth token is bound to a client identity — client id, consent record, scopes. When a different process presents that token, the request keeps the claims and loses the behaviour: headers, TLS fingerprint, request cadence.

Two properties of the initial gateway topology made the pattern obvious to any detection system:

One egress. Every seat left through one container on a single public address.
One fingerprint. The gateway normalised headers and stripped client telemetry, so six workflow seats looked like one client at six times the volume.

Simultaneous failures across independent workflows are not six human reviews. They are one deterministic cluster rule firing. The vendor's terms had said it in advance: scripting the official CLI is acceptable, pooling seats or automating rotation across them is not.

Task dispatch instead of routing

The lesson is not "use fewer seats". It is that a subscription is a seat, not an API. The moment its token leaves the vendor's client, it becomes a shared credential.

So the gateway keeps serving models paid for by API key, and subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the official CLI bound to a dedicated workflow seat. No pooled credentials, no hijacked persistent session, no rotation. The token never leaves the vendor's binary.

orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
                       |
                       +- cwd, permission scope, timeout, tree-kill, no retries

Four dispatcher properties mattered more than any prompt:

Explicit working directory — each task starts inside one repository, never a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell.
Closed stdin — interactive prompts in a headless run become errors instead of silent hangs. One forgotten -y otherwise blocks the parent forever.
Process-group kill on timeout — killing the parent leaves grandchildren running; kill the group and return a timeout code, not a half-finished diff.
No automatic retries — an inner agent loop times an outer retry policy is hundreds of upstream calls in minutes, the exact cadence that got the previous design flagged.

proc = subprocess.Popen(
    argv, cwd=repo, stdin=subprocess.DEVNULL,      # no interactive hangs
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,
    start_new_session=True,                         # own process group
)
try:
    out, err = proc.communicate(timeout=task_timeout)
except subprocess.TimeoutExpired:
    os.killpg(proc.pid, signal.SIGKILL)             # kill the whole tree
    raise

Context stopped travelling in prompts as well: the delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged status: draft and final archiving stays with the orchestrator.

The economics survived, because the workload splits

Nine turns in ten are routine orchestration — tool loops, retrieval, cron, log triage, short edits. Those run on a curated open-weight plan: $10/month carrying a $100 monthly allowance on the fast model, with per-model pools beside it. At $0.15 per million input tokens and $0.60 per million output, a 4k-in/500-out turn costs about $0.0009 — under a tenth of a cent.

The remaining tenth is heavy engineering: multi-file refactors, legacy archaeology, anything with a real build-and-test contract. That dispatches to an authorised CLI seat, where the vendor's client owns the credential and the allowance belongs to the subscription rather than to a pooled bucket.

Pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.

💬 Let's Compare Notes

If a credential is bound to a client, where should a platform team draw the execution boundary when metered APIs and client-bound CLI tools have to coexist: standardise on pure APIs and pay the premium, or run a dispatcher that governs official binaries? I have moved to the second, and I would like to hear from anyone who has run the first at team scale and made it survive an audit.

Two related posts from the same infrastructure work:

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Embedding Fonts So a PDF Looks the Same on Every Machine

Hoelee — Thu, 05 Nov 2026 04:25:03 +0000

Originally published on my blog — read it there with the full context, code and updates.

The requirement was one sentence

A report generator has to produce a document that looks the same everywhere:
Chinese headings in a serif, Chinese body text in a sans, Latin headings and
cover numerals in a display serif, Latin body text and tables in Inter.
Weights matter — medium for emphasis, semibold for headings — across three
languages and two PDF renderers.

And the fonts must travel with the document. Not "the server has them
installed", not "the reader probably has them" — embedded, so a print
shop in another country sees the same pages.

The old pipeline failed on the last clause in a way that was invisible: it
depended on fonts installed on the server by a package manager, and it only
ever embedded a single CJK face. The second weight did not exist, so "bold"
was the renderer's synthetic approximation, and the only way to notice was
to open the PDF and look at what was actually inside it.

Rule one: read the fonts back out of the finished PDF

"Embedded" is a claim about the output, so I measured the output. The same
document, rendered before and after:

	before	after
zh page count	23	23
en page count	28	29
bm page count	30	30
fonts actually embedded	one CJK face + Roboto	Source Han Serif Medium + SemiBold · Source Han Sans Regular + Medium · Cormorant Garamond Medium + SemiBold · Inter Regular + SemiBold + Medium · Source Han Sans for the CJK runs inside Latin reports
render time / peak memory	baseline	same order (±10%, noise)
file size	baseline	roughly +3%

The glyph counts per face are the receipt: in the Chinese report, 15,988
glyphs land in the body sans, 2,247 in its medium weight, 634 in the heading
serif medium, 222 in its semibold. That is the difference between "the font
is on the server" and "the font is in the file" — a reader in another
country sees the same four faces, and a page count that moved by exactly one
page in one language (the Latin metrics are wider than what they replaced)
is the honest cost.

Rule two: static instances only, and the reason is in the parser

The obvious modern choice — ship one variable font per family — is wrong
here, for two measured reasons:

The PDF engine's font parser reads glyf/loca only. Feed it an OpenType file with PostScript/CFF outlines and it refuses: Fonts with postscript outlines are not supported. A variable font is not the problem; CFF outlines are.
It does not read the fvar table at all, so a variable font silently resolves to its default instance — and the default instances of the CJK families are the thinnest ones (Thin 100 and ExtraLight 200, visible in the name table). Every page would render in a hairline weight, with no error anywhere in the pipeline.

So the build step takes the upstream variable fonts and instantiates
static cuts: it runs the instancer, then rewrites name,
OS/2.usWeightClass and head.macStyle so the file identifies itself as
the weight it now is (the source's own name still says "Thin"). A manifest
records family, weight class, sha256, glyph count and code-point count for
every file, and the licence files ship next to them.

Rule three: the engine speaks two weights, so map onto two slots

The renderer's CSS handling honours exactly normal and bold — its source
says so in as many words ("Does not support: bolder, lighter, 100..900").
Numeric weights are therefore meaningless to it, while the headless-Chromium
renderer in the same pipeline honours them properly.

The mapping that works: each family declares an R slot and a B slot
(regular-ish and heavier), font-weight: 500 lands on the R slot of the
intended family, and 600 is rewritten to bold on the PDF side only,
which lands on the B slot. Chromium keeps 600 as written. Two tests pin
it: the B slot must be heavier than the R slot (read from usWeightClass,
not assumed), and font-weight: 600 must never appear in the stylesheet
handed to the PDF engine.

One subtlety worth the paragraph: a "medium" weight is not an emphasis
mechanism in this pipeline, it is a family member. Chinese headings use
the serif's medium as their regular and its semibold for emphasis; Latin
headings do the same with the display serif. Getting that wrong is invisible
on a screen and obvious in print.

Rule four: do not subset against a character set you do not control

The first attempt subset the CJK fonts to the union of two national
character sets and produced a ~29 MB bundle instead of ~50 MB. Then I ran
the name probe and found these were cut:

犇  骉  頔  祎  廸  玧  祾  赟  堃

Every one of them is a real given name character. The report prints a
customer's name on the cover, and the name is free input — so a subset-based
build will eventually put a tofu box on the most prominent page of the
product. Full coverage means keeping ~31k code points — roughly 50 MB of static font assets in the repository (adding only ~3% to each finished PDF after engine subsetting); that is a fine price for never showing a customer an empty rectangle.

Rule five: make missing glyphs a build failure, not a surprise

The gate runs three checks and exits non-zero on any of them:

every font file declared in the manifest is present, and its sha256 matches the file on disk;
every character in the content library (all three languages) plus the brand strings, plus probe classes — simplified, traditional, CJK punctuation, full-width forms, mixed-script, digits, Latin extended, symbols — plus the name characters, has a glyph in the face that will actually print it;
the lookup uses the PDF engine's own font parser rather than a third-party library, so the check cannot disagree with the renderer.

The known edges are written down instead of hidden. Dingbats and emoji (✅ ✔)
exist in none of these text fonts and the engine cannot draw colour emoji,
so they are kept out of the report body. The display serif has only 974
glyphs and does not draw symbols like ① ★ ℃, which is why the
content-library check covers all slots rather than just the body. And CJK
characters inside a Latin report must go through the dedicated CJK slot —
two tests keep the renderers from disagreeing about that.

What I would do differently

Choose faces and weights before styling anything. Retrofitting a second weight after the layout exists means re-checking every page count by hand.
Read the renderer's source before choosing a font format. The variable-font decision looked like a taste question and was actually a parser question — two files settled it in ten minutes.
Treat everything you ship inside a document as a licence artifact. Font files enter the repo with their licences, or they enter someone else's legal review.
Measure what got embedded. Page count, glyph-per-face counts, file size, render time: all four moved, and three of them moved in ways I would have guessed wrong.

The result

Before: one CJK face and a UI font, with synthetic bold, depending on what
the server had installed. After: four families and eight static faces
embedded in the file, weights that are real members of their families, page
counts within one page of the previous layout, and a lint gate that fails
the build when a character has no glyph — so the failure mode moved from "a
customer sees a box" to "CI refuses the release". Which is the same trade as
everywhere else in this pipeline: make the mistake impossible to ship rather
than easy to notice.

I build automated document generation and multi-lingual PDF publishing pipelines — from layout engine customisation and compliant font embedding to CI visual verification gates. If your business reports, invoices, or client deliverables need to look identical on every machine, get in touch: WhatsApp +60 12-797 2969 · me@hoelee.com · hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

How I Vet an Open-Source Dependency Before Betting On It

Hoelee — Wed, 04 Nov 2026 04:26:01 +0000

Originally published on my blog — read it there with the full context, code and updates.

I was handed a 1,306-line architecture spec for a partner program. It read like a
plan: consistent, cross-referenced, confident. It named a self-hosted
open-source engine as the attribution and commission layer, and it assumed three
things about that engine — that it spoke OIDC so my existing identity provider
could log partners in, that its commission rules covered the five shapes the
business needed, and that its white-label feature was shipped.

Ten minutes of curl and grep later, two of those were false and the third was
half-true. None of that is unusual. The README describes the product; the
repository describes the software. The gap between those two is exactly where
integration projects die — and it is cheap to measure before you write any code.

The problem, phrased as the question I'd actually search for

How do I know an open-source project does what its documentation claims before
I build on it?

A spec is a claim about other software. The more polished the spec, the more its
assumptions read like facts — this one cited sources, defined stable IDs, and had
a source-of-truth matrix. What it never did was check whether the engine down the
chain implemented the features being assigned to it. So before installing
anything, I went looking for those specific features.

What I tried first, and why it wasn't enough

1. Read the README end to end. It sells the pipeline well:
click → identity → event → attribution → commission → payout. It never once
says which authentication protocol the app speaks. For a feature you need,
silence isn't neutral — but an absent mention is easy to skim past when the rest
of the page is convincing.

2. Searched the web. Several of my queries returned nothing usable. The
project is a few months old and its SEO surface is thin. For software that's
fine — the source is the documentation. It just means the reading has to happen
in the repo, not in a blog post.

3. Found a 78 KB feature design doc and assumed the feature shipped. It was
labelled "FINAL, post-review", listed three reviewers, and carried a date. I took
that as proof. That was the worst inference of the three, and the easiest to check.

The six checks that actually settled it

1. Read the repo's vital signs before any feature claim

curl -s -H "User-Agent: hermes" https://api.github.com/repos/<owner>/<repo> \
  | grep -E '"created_at"|"pushed_at"|"stargazers_count"|"forks_count"|"open_issues_count"|"archived"|"spdx_id"'

What came back: MIT, created 2026-04-23, last push 2026-09-10, 8 stars, 3 forks,
15 open issues, not archived. Read the same as: real, still moving, and very
small — one vendor, which also sells a hosted tier. Now I know how much of my
business I want resting on it before I start grading its features.

2. Grep for the feature instead of reading for it

grep -in 'oidc\|sso\|saml' README.md ARCHITECTURE.md docs/*.md | wc -l

Zero hits. The app authenticates with its own magic-link sessions and
per-persona API keys; it does not speak OIDC at all. That one line of output
killed a phase of the plan — "put the portal behind our SSO provider" — which
would otherwise have surfaced weeks later, mid-integration, with a partner
already invited. Reading for a feature you can't find is how you lose an
afternoon; grepping makes the absence loud.

3. Read the target branch line at the top of a feature doc

head -5 docs/white-label-custom-domains.md

Target branch: multi-tenant. The doc is a design for a branch that is not the
default branch — it describes work in flight, so the feature is not in the code
you would install. A finished-looking document with reviewers and dates can
document something unshipped. That isn't dishonesty; it's what design docs are.

4. Read the data model, not the feature list

The commission section named exactly two rule types — percent (with a recurring
flag) and fixed. The business spec had enumerated five. That gap isn't a bug;
it's a modelling job I'd have to do in configuration instead of expecting from
the product:

Spec assumed	What the engine actually has
first-order bonus	a `fixed` rule on the first event
event-specific rules (`lead` vs `invoice_paid`)	one rule per campaign — so: two campaigns
recurring commission	`percent` + `recurring: true`
percentage	`percent` ✅
fixed	`fixed` ✅

Two rule primitives, five behaviours — workable, but only if you design for it up
front instead of discovering it during implementation.

5. Check which paid tier gates the feature you need

For the database tool in the same stack, the edition table was blunt:

Feature I needed	Community (free)	Enterprise (paid)
API tokens, so I can automate it	✅	✅
SSO / OIDC	❌	✅
Page designer, hide branding, 2FA	❌	✅

The thing I wanted for the partner-facing door (SSO) sits behind the commercial
gate, while the thing I wanted for automation (an API token) is free. Knowing
which side of the wall each requirement falls on changes the design — before
you've designed around it.

6. Check that the money rail works in your country

The payout model was stripe_connect | manual, and self-hosted mode was
documented as "operator manages out-of-band" — i.e. you move the money yourself.
Stripe Connect is not how Malaysian businesses pay partners, so that one line
decided the entire deployment shape: self-host mode, manual payouts, and event
ingestion pushed from my own order system rather than leaning on the provider's
payment integration. That line is never on a marketing page, and it mattered more
than any feature.

Then: ask the fallback question before you need it

"What if the feature I need isn't there?" For the missing SSO the answer was:
don't put login in the app, put it in front of the app. authentik's proxy
provider exists to protect "applications that do not support native
authentication protocols such as OIDC, SAML, or LDAP", and it injects
X-authentik-username and X-authentik-groups headers upstream, so the reverse
proxy can do the gating. That turned "no SSO support" from a disqualification
into an afternoon of config.

What I'd do differently

Write the requirement list first, then go look for it. Five lines — the features I need — and search for those. Starting from the README's feature list means grading the software on the vendor's exam.
Treat "a doc exists" as different from "a feature ships". Read the branch line.
Grep, don't read. Presence is cheap to prove; absence is the thing you're hunting.
Decide the fallback before the decision, so a missing feature becomes a design change rather than a surprise.
Pin the commit. This project's README says its API is "stable but unversioned" and there is no OpenAPI description anywhere in the tree. There's no contract to compile against, so the commit hash is the contract — the endpoints I use go into my own docs, and upgrades get a regression pass.

The result

Six checks and about ten minutes of curl and grep corrected all three faulty assumptions in the spec before a single line of integration code existed. The project stayed viable — but
reshaped: no OIDC, manual payouts, a pinned commit, and a plan that no longer
depends on a feature sitting on an unmerged branch.

The spec wasn't wrong to be optimistic. It was wrong to be unverified, and that's
the cheapest thing on the whole project to fix.

Building on a self-hosted stack, and want a second pair of eyes on whether
those features will actually be there when you integrate? That's the kind of
review I do.

WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You

Hoelee — Tue, 03 Nov 2026 04:25:38 +0000

Originally published on my blog — read it there with the full context, code and updates.

If you self-host NocoDB and search for "nocodb sso self-hosted", you'll find a documentation page describing OIDC single sign-on, a set of environment variables, and forum answers telling you it works if you set them. All of that is technically true. None of it tells you the part that matters.

So here it is: NocoDB's OIDC SSO is a paid feature. On an unlicensed self-hosted build, the code path exists but abandons the process — and the failure isn't a polite "SSO is not enabled on this instance". The whole server exits.

Why this matters

Two audiences get bitten here.

If you enable it in production, you've built a denial-of-service trigger. The route that handles the OIDC callback is reachable without authentication — it has to be, that's how a login flow starts. On an instance where that route crashes the process, anyone who requests it takes your database UI down. Not degrades it, not returns an error page: kills the container. You don't need to be a target; a single curious visitor, or a link scanner that follows URLs, is enough.

If you're evaluating NocoDB as a platform, this is a licensing signal you should read correctly. The feature is present in the open-source image, the variables are enforced, the code ships. What's missing is permission. A feature that is implemented but not licensed looks identical to a feature that works, right up until you try it — and that's the whole trap.

What the docs say, and what the image does

NocoDB's own docs are clear if you read the licensing line: "OIDC SSO is available on NocoDB Cloud (Business plan and above) and licensed self-hosted deployments (Business plan and above)." The other half — that the unlicensed binary will not degrade gracefully — isn't documented, and can't be discovered from the docs, because it's a runtime fact about the image.

The environment variables are real. The ones I confirmed against the shipped image:

NC_SSO: oidc
NC_SSO_OIDC_ISSUER: https://auth.example.com/application/o/your-app/
NC_SSO_OIDC_AUTHORIZATION_URL: https://auth.example.com/application/o/authorize/
NC_SSO_OIDC_TOKEN_URL: https://auth.example.com/application/o/token/
NC_SSO_OIDC_USERINFO_URL: https://auth.example.com/application/o/userinfo/
NC_SSO_OIDC_CLIENT_ID: ...
NC_SSO_OIDC_CLIENT_SECRET: ...
NC_OIDC_PROVIDER_NAME: Hoelee SSO

Only NC_SSO and the provider-name variable appear in the public documentation. The six per-provider variables are undocumented but enforced: start the container with NC_SSO=oidc and no URLs and it refuses to boot:

Open ID SSO is enabled but missing required env keys

That's the detail that makes the trap convincing. Validation at startup means the feature is wired in, not vestigial. Nothing about that error message suggests the code path is unreachable without a licence — it reads like a configuration mistake you can fix.

How to test it without risking anything

Never test this on the instance you care about. A throwaway container with an in-image SQLite meta store is enough, and it takes about a minute:

docker run -d --name nc-sso-test -p 10399:8080 \
  -e 'NC_SSO=oidc' \
  -e 'NC_SSO_OIDC_ISSUER=https://auth.example.com/application/o/your-app/' \
  -e 'NC_SSO_OIDC_AUTHORIZATION_URL=https://auth.example.com/application/o/authorize/' \
  -e 'NC_SSO_OIDC_TOKEN_URL=https://auth.example.com/application/o/token/' \
  -e 'NC_SSO_OIDC_USERINFO_URL=https://auth.example.com/application/o/userinfo/' \
  -e 'NC_SSO_OIDC_CLIENT_ID=test' \
  -e 'NC_SSO_OIDC_CLIENT_SECRET=test' \
  -e 'NC_OIDC_PROVIDER_NAME=Test SSO' \
  nocodb/nocodb:<your-tag>

Note the deliberate omission of NC_DB — with no external database configured, the container spins up its own SQLite store, so it needs no credentials, touches no production data, and can be deleted with docker rm -f nc-sso-test.

Watch the startup, then request the SSO route once:

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:10399/auth/oidc
docker ps -a --filter name=nc-sso-test

What I got, twice — the second time with a ?workspaceId= parameter to rule out a bad request shape:

### UNCAUGHT EXCEPTION ###
unhandledRejection TypeError: _0x418c23 is not a function at docker/index.js:1:27483406

...and the container is no longer running. Exit code 1, from an unauthenticated GET. Reproduced on 2026.09.0 with the same result both times. It isn't a configuration error, and it isn't a request the client got wrong.

Note the log line that explains it, from startup:

No license key found — running in CE mode

The other half of the licensing story

If your reaction is "fine, I'll buy the licence", read this before you budget for it, because there's a prerequisite that isn't obvious either:

Instance ID unavailable — PostgreSQL is required for enterprise licensing

The community edition I run uses MySQL as its metadata store, and MySQL cannot generate the instance ID that licensing depends on. So a licence isn't something you activate on the instance you have — it's a licence plus a metadata migration from MySQL to Postgres, which is its own project with its own risk, done on a live instance. NocoDB publishes a migration guide for exactly this, which tells you how common the situation is.

That isn't a complaint about the pricing model. It's the evaluation I wish I'd had before spending an afternoon: the cost of "just buy it" is licence plus a storage-engine migration for anyone running the default MySQL setup.

The fork that search results recommend — check it's alive first

Search results for NocoDB SSO without a licence point at brunostjohn/nocodb-oidc, a community fork that adds OIDC to the open-source build. It's a drop-in image and it's exactly the thing you want. It's also abandoned, and this is checkable in about two minutes:

# what version of NocoDB is the fork based on?
curl -s https://raw.githubusercontent.com/brunostjohn/nocodb-oidc/main/packages/nocodb/package.json \
  | python -c "import sys,json; print(json.load(sys.stdin)['version'])"
# 0.255.2

# when was it last touched?
curl -s 'https://api.github.com/repos/brunostjohn/nocodb-oidc/commits?per_page=1' \
  | python -c "import sys,json; print(json.load(sys.stdin)[0]['commit']['author']['date'])"
# 2024-10-29

A version from 2024 against an instance on a monthly release cadence is not a drop-in — it's a two-year downgrade, taking with it the schema migrations, the security fixes and the API behaviour your automations already depend on. The advice is only as good as the fork is maintained, and search results don't go stale when a repo does.

Check the fork's base version and last commit date before you plan around it. Two curl calls, and they replace a rollback you'd otherwise do at 1 a.m.

What to do instead

Decide whether you need SSO in the app, or in front of it. If what you want is "the internet can't reach this login form", a forward-auth proxy in front of the hostname gets you there with the open-source build. It's not single sign-on in the app, and the app still asks for its own password — I wrote up that construction, and the ingress trap that made my first attempt look successful while being a no-op, in The Forward-Auth Gate That Verified Perfectly — and Wasn't Live.
Keep the app's own login and lock the door at the edge. For an internal tool with a handful of users, this is what most people actually need.
If a licensed build is genuinely required, price the migration, not just the licence. Confirm which metadata store you're on first.

What I'd do differently

When a feature's availability is ambiguous, read it out of the image instead of the docs. The docs describe intent; the image is the truth. One docker exec ... printenv | grep -iE 'NC_|OIDC', one grep of the shipped bundle, and one throwaway container answered in twenty minutes what an afternoon of documentation reading couldn't.

And a blunter rule that generalises past NocoDB: an undocumented-but-enforced environment variable is a licensing boundary, not a feature flag. If a variable rejects your startup when it's incomplete, but the documentation hides half its siblings, the missing piece is almost never configuration.

The result

Five minutes to reproduce on a throwaway container, twice, with the container dead both times. One accidental finding I'd rather not have learned in production: an unauthenticated request to the SSO route is enough to stop the instance.

The upside is that I stopped trying to make the app do SSO, put the decision in front of the hostname where it belongs, and can now answer "can NocoDB do SSO for free?" in one sentence instead of an afternoon.

Want this for your business?

If you're evaluating self-hosted software, need single sign-on in front of internal tools, or want someone to check what your stack actually does versus what its documentation claims, that's the work I do.

WhatsApp: +60 12-797 2969 — tap to chat
Email: me@hoelee.com
Website: hoelee.com

I set up authentik single sign-on, self-hosted Docker stacks and reverse proxies for small businesses in Malaysia — and I write up what I learn while doing it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

The Forward-Auth Gate That Verified Perfectly — and Wasn't Live

Hoelee — Mon, 02 Nov 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

I finished the gate, ran the check, and it passed.

HTTP/1.1 302 Found
location: https://app.example.com/outpost.goauthentik.io/start?rd=...
x-powered-by: authentik

An unauthenticated request was being pushed to my authentik login page. Branded, correct, working. I was about to move on — except that my own rule for anything user-facing is to hit it the way a user hits it, from outside, over the public internet. So I did.

GET https://app.example.com/          → 200
x-powered-by: Express

The app answered directly. No login page, no redirect, no gate. The exact same hostname, one request path gated and one wide open, depending on where the request entered my network.

That was the whole day: not the SSO setup, which worked, but the fact that a hostname can be served by more than one ingress layer, and I had only gated one of them.

Why this matters beyond my one app

If you self-host anything behind a tunnel, a reverse proxy, or both, you have probably already made this assumption: "the reverse proxy is what serves my domains." It's an assumption that holds right up until it doesn't, and it fails in the most dangerous direction — everything you change looks correct, because the path you're inspecting is correct.

The failure mode is silent by construction. There is no error, no log line, no hint. Your configuration check passes because you're checking the configuration you edited. Meanwhile the real traffic — the traffic you were protecting — never touches it.

I've since found two other gates on the same host with the same latent problem, built the same way, waiting for someone to trust them.

The setup: SSO for an app that has no SSO

The app was a self-hosted database UI on my NAS (Docker, published on a host port, sitting behind nginx and a Cloudflare Tunnel — the usual shape of the infrastructure I manage). I wanted a login wall in front of it so that "reachable from the internet" wouldn't mean "anyone who guesses the hostname sees the login form and can start guessing passwords".

The app's own documentation advertises SSO. I'll keep this post focused on the ingress trap and put the licensing story in its own write-up, but the short version is that the advertised SSO is a licensed feature, the open-source build ships the code path but not the permission to use it, and the community fork that used to fill the gap hasn't been touched in two years. Twenty minutes of research, two abandoned approaches, one conclusion: the app would never do SSO for me, so the gate had to live in front of it.

Which is a normal, well-trodden thing to do. authentik calls it a proxy provider, and the flow is:

A login flow for this app — its own title, its own background, so a user who lands on the login page knows which system they're being asked to authenticate for.
A proxy provider — the object that knows the app's external hostname (https://app.example.com) and its internal address (http://app-container:8080).
An application linking the two, attached to the embedded outpost. The outpost is the component that actually receives traffic, checks for a session, and proxies the request to the app's internal host after login.
The ingress — pointing that hostname at the outpost instead of at the app.

Steps 1–3 live in authentik and are configuration. Step 4 is the one that decides whether any of it is real, and it's the one I got wrong.

Two smaller things from steps 1–3, since they cost me time:

A partial PATCH on a proxy provider is rejected. Updating one field returns:

{"internal_host": ["Internal host cannot be empty when forward auth is disabled."]}

The serializer re-validates the object, and a proxy provider without an internal host is invalid — so a one-field update fails as if you'd cleared a field you never touched. Send the identity fields together with the field you're changing:

curl -s -X PATCH "$AUTH/api/v3/providers/proxy/$PK/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"proxy","external_host":"https://app.example.com",
       "internal_host":"http://app-container:8080","skip_path_regex":""}'

authentik API tokens can expire mid-session. After working for an hour, every call started returning 403 {"detail":"Token invalid/expired"}. Note the status code: 403 means "we know who you are, and you may not do this" — the response body is the only thing that tells you the token died. If you cache a token in a script, re-read it from the database when calls start failing instead of assuming a permissions problem.

The trap: I verified the layer I changed

Here is the check I ran, from the host itself:

curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}' \
  -H 'Host: app.example.com' https://localhost/
# 302 -> https://app.example.com/outpost.goauthentik.io/start?rd=...

This is a good check. It proves the nginx vhost serves the outpost, that the outpost recognises the hostname, that the provider is linked to the application, and that the login flow is alive. Everything I had configured was proven correct in one command.

It says nothing whatsoever about whether public traffic reaches that vhost.

Here's why. My hostnames are served by two independent ingress paths:

Path	What serves it	Where it lands
Direct WAN IP → nginx (443)	the vhost I edited	the outpost ✅
Cloudflare Tunnel (`cloudflared`)	a rule inside the tunnel's own config	the app's port, directly ❌

The tunnel's ingress is a small ordered list evaluated per hostname:

app.example.com     -> http://app-container:8080     ← straight to the app
*                   -> http_status:404

That rule was written years earlier, when the goal was simply "make this reachable". It never mentioned nginx, because it never needed to. So my beautiful vhost sat off to the side of the real request path, gating a route that public traffic doesn't take.

The two paths even answer on the same hostname, so nothing about the URL hints at the difference.

What finally made it obvious was a response header. The gated check returned x-powered-by: authentik. The public check returned x-powered-by: Express. Same URL, same second, two different servers answering. When you suspect you're looking at the wrong layer, read the headers — they name the software that answered, which is exactly the question you're asking.

To see your own tunnel's rules rather than guessing (Cloudflare's API, not the dashboard):

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/cfd_tunnel/$TUNNEL_ID/configurations" \
  -H "Authorization: Bearer $CF_TOKEN" \
  | python -c "import sys,json; [print(r.get('hostname','*'), '->', r.get('service','')) for r in json.load(sys.stdin)['result']['config']['ingress']]"

git.example.com   -> http://10.0.0.10:6880
app.example.com   -> http://app-container:8080
bot.example.com   -> http://bot-container:8080
*                 -> http_status:404

There it is, in one line, after a day of looking at nginx.

The permission wall on the way to the fix

Editing that rule needs a token with Account → Cloudflare Tunnel → Edit. Mine didn't have it, and no zone-scoped token can substitute: tunnels are account-scoped objects, so a Zone-scoped token fails the read as well, with 1001 Not authorized. If you only have zone-level tokens, open the dashboard instead: Zero Trust → Networks → Tunnels → your tunnel → Public Hostnames and change the rule's service to your outpost.

For me the correct target was the outpost on the same Docker network as cloudflared:

app.example.com  ->  http://authentik-server:9000

The outpost reads the Host header (which the tunnel preserves) to pick the application, then proxies to the internal host from the provider config. That's the whole change: one service value. Everything I'd built in authentik was already correct — it was simply never on the path.

Optional SSO: gate the UI, leave the API alone

Once the gate is really in front of the traffic, the next problem is the API. A hostname-level gate can't tell a script from a browser, so token-only API calls get redirected to a login page — they don't fail with a JSON error, they get HTML.

If you want SSO for humans and no SSO for machines, authentik's proxy provider has a per-path exemption. The UI calls it Unauthenticated Paths; the API field is skip_path_regex:

{"skip_path_regex": "^/api/.*"}

Verified result on my instance, same hostname, same session:

Request	No session	Result
`/` and `/dashboard`	yes	`302` → authentik login
`/api/v1/health`	yes	`200` from the app
`/api/v2/tables/.../records` + API token	yes	`200` with real data

Two caveats worth knowing before you rely on it. This exemption applies at the outpost, so unlike a path split at the CDN it works on every ingress path. And it is not per-user: the gate applies to the request, not the person. There is no "some users can skip SSO" — either a path is exempt for everyone, or everyone needs a session.

I used a simpler form of the same idea first: move every machine caller off the public hostname before the gate exists. My automation was calling the app through the public URL and being routed out to Cloudflare and back into my own network for no reason. Pointing it at the container name (http://app-container:8080) shortened the path, removed a failure mode, and meant the gate couldn't break it. If a login wall takes down your integration, the integration was depending on the door being unlocked.

The honest part: this is not single sign-on

I have to be straight about what a forward-auth gate gives you, because "SSO" oversells it.

The outpost knows who you are. The app does not. Unless the app can consume an upstream identity — header-based auth, or native OIDC/SAML — it will show its own login form after the gate lets you through. So the user experience is: authentik login, then the app's login.

For a user who is already signed in to authentik, the first step is often silent. But it's still two systems, two session lifetimes, and two places a password can be wrong. In my case the app has no header auth and its own OIDC is a licensed feature, so the gate would have been an extra door bolted in front of the existing one — no reduction in logins at all.

That's a legitimate trade for some setups: keep the app's login for its own users, and stop the internet from reaching the login form. It's a poor trade if what you actually wanted was "my team types one password". Which meant the honest answer for me was to turn the whole thing back off.

Reverting it properly (and the arithmetic that saved ten other apps)

Undoing the gate is where I nearly caused a much bigger outage than the one I was preventing.

The nginx change was one line in a large generated vhost file. The obvious revert is to substitute the port back:

# DON'T — this is the version that would have hurt
sed -i 's#proxy_pass http://localhost:10000;#proxy_pass http://localhost:10380;#' vhost.conf

Before running it, I counted how many server blocks point at that same port:

grep -c 'proxy_pass http://localhost:10000;' vhost.conf
# 11

Eleven. My app was one of them. Ten other applications are gated through that outpost, and a blanket substitution would have silently un-gated every one of them at once — including the ones doing real authentication work. The "fix" would have looked successful and removed authentication from most of my infrastructure.

What I did instead:

Snapshot the current state first, so the revert was itself reversible: cp -a vhost.conf vhost.conf.bak-$(date +%Y%m%d-%H%M)-revert.
Find the exact line by diffing against the backup taken before the original change. That diff showed one changed line, which is proof of exactly how much I had touched.
Edit that line only, then re-diff against the original backup and require the result to be identical before going further.
Update the second file that mirrors the same setting — the NAS's own reverse-proxy database, which regenerates the vhost on DSM events. I edited the single key and asserted the hostname before writing, never restoring the whole file.
nginx -t then reload.

And then a detail that cost me another twenty minutes of doubt:

nginx -s reload && sleep 2 && curl -sk -H 'Host: app.example.com' https://localhost/
# still 302 -> /outpost.goauthentik.io/start

The revert looked like it had failed. It hadn't. A request issued microseconds after the reload can still be answered by the old worker while it drains. The re-test a few seconds later returned 200 x-powered-by: Express — direct to the app, gate gone.

If a config change appears not to apply, re-test after a beat before you go looking for a second cause. With a request that fast, "the config is wrong" and "the old process is still answering" are indistinguishable.

What I'd do differently

Enumerate every ingress path for a hostname before building anything on top of it. My checklist is now four questions, asked in this order:
- Is the hostname proxied by a CDN/tunnel? If yes, its rules are the real ones — check them first, not last.
- What does the reverse proxy vhost say?
- Does the platform mirror that vhost somewhere else that can regenerate it?
- Is there another hop (VPS, second proxy, vendor tunnel) with its own copy of the routing table?
Verify from outside, and verify which software answered. curl from the host with a Host: header proves your config is self-consistent. Only a request from the public internet, plus x-powered-by/server headers, proves it's in force.
Find out whether the app can consume an upstream identity before you build the gate. If it can't, you're adding a login, not removing one — a decision to make deliberately, not to discover afterwards.
Count before you substitute. Any port or hostname you're replacing globally probably appears in more places than you think. grep -c costs a second and prevents an outage.
Snapshot, diff, edit one line, re-diff. The backup's only job is to answer "how much did I change?" — and a diff answers it exactly.

The result

The gate worked. It was also invisible to every real user, and after working out what it actually delivered — a second login in front of an app that already had one — I turned it off again: one field cleared, one application unlinked, one line restored, verified by a request from outside my network.

The two checks that would have caught the whole thing on day one take about thirty seconds:

curl -sI https://app.example.com/ | grep -i '^x-powered-by'   # who is actually answering?
curl -s  https://app.example.com/ -o /dev/null -w '%{http_code}\n'  # what does a stranger get?

If the answer isn't what your configuration says it should be, you're editing the wrong layer. That's worth knowing before you spend the day on it — and definitely before you tell anyone the gate is live.

Want this for your business?

If you need single sign-on in front of internal tools, a self-hosted app exposed safely to the internet, or someone to audit which ingress path is actually serving your domains, that's the work I do.

WhatsApp: +60 12-797 2969 — tap to chat
Email: me@hoelee.com
Website: hoelee.com

I set up authentik single sign-on, hardened reverse proxies and Cloudflare tunnels, and self-hosted Docker stacks for small businesses in Malaysia — and I write up what I learn while doing it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

Hoelee — Sun, 01 Nov 2026 04:25:47 +0000

Originally published on my blog — read it there with the full context, code and updates.

I built a monitoring stack, wrote a reasonable set of rules, and then spent an
afternoon deleting most of my own thresholds. Every one of them was a percentage, and
every one of them was wrong in the same specific way.

The three alerts that taught me

1. "Filesystem above 90% used." It fired on a 17 TB volume that had ~500 GB free.
Nothing was wrong. Nothing was about to be wrong. On a volume that size, a single
large backup can swing the percentage by several points — the ratio is noise, and the
fact that 500 GB is still available is the actual state of the world.

2. "Memory above 90% used." Fired constantly on a host with 4.8 GB available.
Linux uses free RAM for page cache and gives it back on demand; MemTotal - MemFree
describes your filesystem cache, not your risk. The number that predicts an OOM is
MemAvailable. (Even if one alerts on MemAvailable exceeding 90% capacity, on a 64 GB host it still fires an alert when 4.8 GB of usable headroom remains.)

3. "CPU steal above 25%." Fired on a VPS that had been running its mail stack
perfectly at 41% steal. Steal means the hypervisor is busy — it is a capacity signal,
not a failure signal, and 41% steal on a 7-vCPU box was simply what that host costs.
The rule produced alerts that were always true and never actionable.

The common thread: I was alerting on a ratio, and the ratio doesn't know how big the
thing is.

Alert on the consequence, not the ratio

The test I now apply to every threshold: if this condition persists, what will actually
happen? That answer is almost always expressible in absolute units.

Instead of	Alert on	Because
Filesystem > 90%	Free space < N GB	"Can I still write?" is the question
Memory > 90% used	`MemAvailable` < 512 MB	about to OOM, not "cache is warm"
CPU steal > 25%	Steal > 50%	capacity cost vs. starvation
Load average > N	`load1 / cores > 2`	load is meaningless without core count

In PromQL, before and after:

# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90

# AFTER — capacity left, on the volumes that hold data
min by (instance, mountpoint) (
  node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}
) < 25e9

# BEFORE — page cache counted as "pressure" (misusing MemFree)
(1 - node_memory_MemFree_bytes / node_memory_MemTotal_bytes) * 100 > 90

# AFTER — the actual OOM precursor based on kernel-reclaimable MemAvailable
node_memory_MemAvailable_bytes < 512 * 1024 * 1024

Two details that make these rules survive contact with a real dashboard:

Keep the comparison in the threshold condition, not inside the expression. Query
min by (mountpoint) (node_filesystem_avail_bytes{...}), then let the alert rule
compare it to 25000000000. If you bake < 25e9 into the PromQL and leave an
inherited > 90 in the rule, the rule still evaluates — but it displays a nonsense
threshold, and the next person to read it has to reconstruct your intent from two
contradictory conditions.

Exclude the mounts that are views of other mounts. A "free space" rule is only as
honest as its selector. On my own setup, one NAS volume appears three times (as
/volume1, as /opt, and again on another host as a CIFS re-mount), and unRaid's shfs
union mount reports avail=0 — a rule without exclusions would either fire forever or
double-count capacity. One explicit selector beats a clever generic one:

node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}

Noise is not harmless

The cost of a bad threshold isn't the alert itself — it's the training. Every alert
that fires while nothing is wrong teaches you to skim, then to ignore, and then the one
that matters arrives in a channel you've stopped reading. My rule count went down
while coverage went up: fewer, each tied to a named consequence.

Thresholds are personal, shapes are not

The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB
available before I care about memory, 50% steal before a VPS is being starved. Yours
will differ with your hardware and your tolerance.

What transfers is the shape:

absolute units over ratios,
the consequence written into the alert's summary,
a selector that states exactly which mounts it speaks for,
and a rule count small enough that you still read every message.

The result

The same stack that was producing three permanent, uninformative warnings now runs 16
rules that stay silent when the infrastructure is healthy — and one of them immediately surfaced a
NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been
hiding behind a 99%-full giant that I'd learned to ignore.

Want this for your business?

If your monitoring sends you alerts you've learned to ignore, or you have none and would
rather find out about a full disk from a rule than from a failed backup, I set up
self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email)
with thresholds tied to what actually breaks.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)

Hoelee — Sat, 31 Oct 2026 04:25:13 +0000

Originally published on my blog — read it there with the full context, code and updates.

You open the dashboard you use every day and it's empty. Not "one panel broke" empty —
every panel, top to bottom, says No data. You check Prometheus and everything is
healthy. This post is the specific, non-obvious cause I hit, and the way my own
verification convinced me the dashboard was fine while it wasn't.

The searchable question

My Grafana panels show No data but the data is in Prometheus — why?

The usual answers (wrong datasource, wrong time range, rate() on a counter that
resets, missing scrape target) did not apply. Here is the checklist that narrowed it
down, in the order that costs the least time:

# 1. is the target actually being scraped?
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}'

# 2. does the metric exist right now?  (bypasses Grafana entirely)
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric'

# 3. is the panel's expression sound?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})'

All three were green: 7 targets up, 25k series ingested, and the panel's raw
expression returned data when I ran it by hand. So the problem was not the metrics,
the scrape, or the PromQL — it was the variable layer between them.

The actual cause: a variable that filtered on itself

The dashboard had a host picker. Two variables chained: $nodename (which machine)
and $node (its instance label). The definition of the first one was:

# BROKEN — the variable filters on its own value
label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename)

On first load $nodename has no value, so the selector becomes
nodename=~"" — which matches nothing. A variable whose query returns no rows has
no options, so it stays empty. $node is then defined against $nodename:

label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

…which is also empty, so every panel filtering on $node matches no series. Grafana
isn't lying when it says No data: the panel query genuinely has no data, because the
variable that scopes it resolved to nothing.

The fix is to stop the self-reference and give each variable an explicit saved default:

# FIXED — no self-reference, one hop per variable
# $nodename
label_values(node_uname_info{job="unraid-host"}, nodename)
# $node
label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

Then save current values for both so the dashboard opens populated instead of
depending on a click. (This is also why the same 41-panel layout can be cloned to
three hosts and just work.)

Why my verification missed it

This is the part worth stealing. I "verified" the dashboard by expanding each panel's
expression by hand, substituting the variable values I knew were correct
($nodename=unRaid, $node=host.docker.internal:9100), and asserting the query
returned points. Thirteen of those checks passed. The dashboard was still blank,
because the bug was upstream of the substitution: Grafana's own resolution of the
variables was empty.

A verification that substitutes the very value under suspicion cannot detect that
value failing to resolve. To catch it, read the values Grafana actually saved and
expand with those:

curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \
  | jq '.dashboard.templating.list[] | {name, current: .current.value}'

The correct check is on the variable query itself — run what Grafana runs:

# what does $nodename's label_values() actually return?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result'

Empty there means the dashboard cannot possibly render, no matter how healthy the data is.

The second trap: `$__all` is not `.*`

While automating panel checks, a variable set to All reports its current value as the
sentinel string $__all — not a regex. Expanding panel expressions naively turns
name=~"$__all" into a literal that matches nothing, so a perfectly healthy
container dashboard "fails" every panel. Resolve $__all to the variable's allValue
(usually .*) before expanding, or you'll chase a bug that only exists in your checker.

Not every empty panel is a bug

After the fix, out of the 41 total layout panels, 21 of the 25 core metric panels returned data. The remaining four were empty by design and
worth naming honestly in the panel description:

PSI panels (Pressure, Pressure Stall Information) need /proc/pressure, which the on-premise NAS storage node's kernel doesn't expose. The metric cannot exist there — the same panels on a host that does export node_pressure_* render fine.
Root filesystem panels whose expression carries fstype!="rootfs" can never match on a host whose / is a ramdisk, which is what unRaid's root is.

Distinguishing "broken because of a bug" from "empty because the metric cannot exist on
this host" is the difference between a real fix and a wild goose chase.

What I'd do differently

Never define a template variable that filters on itself. One hop per variable.
Always save explicit current values for variables a dashboard depends on, so a fresh load doesn't rely on a populated dropdown.
Verify with Grafana's saved values, not my idea of them. The masking is invisible in the test result and obvious in the UI.
Describe intentional gaps in the panel description. Future-you will otherwise spend an afternoon "fixing" a panel that was never meant to work there.

The result

All three hosts now run the same instrumented layout — a 41-panel host dashboard (with 21/23/25 of the 25 core metric panels returning data respectively) and a
10-panel container dashboard each — and the two structurally-empty classes documented in place instead of silently
confusing whoever opens them next.

Want this for your business?

If you have Grafana dashboards that nobody trusts — or servers and a NAS with no
monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container
metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that
have quietly stopped showing data.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Stop Writing Agent Prompts for Deterministic Work

Hoelee — Fri, 30 Oct 2026 04:25:11 +0000

Originally published on my blog — read it there with the full context, code and updates.

One of my scheduled jobs failed 29 times in a row. Nobody told me, because
it was designed to speak only when it had something to report — and a run
that dies before it can speak looks exactly like a run that found nothing
wrong.

The job was a version check: ask npm for the latest published version of a
CLI I use, compare it with the one installed here, and tell me if there is a
newer one. That is the entire task. It is two strings and an inequality.

I had built it as an agent job, because that was the pattern I had in my
head at the time. Every tick, the model would read the injected script
output and decide whether to write a notification. It worked for a while.
Then my global default model changed, and the job — which records the model
it was created with — stopped running entirely. It printed [drift_skip]
and exited. Twenty-nine times.

The fix was not a better prompt. It was deleting the model from the loop.

The rule

If the stop condition is a comparison — a string, a number, a status code — a model in that loop can only add failure modes.

There is no judgment to make in "is 3.8.50 newer than 3.8.50". The model
was not deciding anything the code could not; it was adding a dependency (a
model route, a provider, a config snapshot, a token bill) and a new way to
fail. So I rewrote the checker as 125 lines of Python with an explicit exit
path for every branch, and flipped the job to script mode: the script's
stdout is delivered as the notification, with no LLM anywhere in the path.

It has run clean ever since. Anthropic's own guidance says the same thing in
one line — use scripts for deterministic work; running a script is cheaper
than reasoning through the steps — but the cost was never the interesting
part for me. The reliability was.

Where the model does earn its place

Of the eleven loops I run, eight now have no model in the path. The three
that do all handle something I genuinely cannot express as code:

Is this email important? A mailbox produces promos, DMARC reports, container alerts and a customer asking for a quote. Fetching mail is deterministic. Ranking it is not.
Is this shop voucher about to overspend? The arithmetic is trivial; deciding what a finding means for money I have already committed is not.
Is this project stalled? Reading two markdown files is a script's job. Noticing that the plan has not moved in three weeks is not.

The test I apply to a new loop: write down what the check would print on
success and on failure. If both answers are a number or a string, it is a
script. If the answer is "it depends what it says", it needs a model — and
then it needs a verifier I trust more than the agent, because I am no longer
able to predict the output.

The part that actually bit me

The failure mode was not the drift. It was the silence.

A loop that speaks only when something is wrong is the only kind of loop I
can tolerate — I do not want a daily status report from eleven jobs. But
"silent because healthy" and "silent because it crashed before it could
speak" are the same observation from my side of the phone. Twenty-nine runs
of evidence were sitting in the job's execution log the whole time; I was
trusting the absence of a notification instead of reading the history.

So every script in the fleet now has a second exit path: exit 0 means
measured, here is what I found, and exit 1 means I could not measure —
and exit 1 is never silent. Missing config, unparseable state, an unexpected
exception: all of them print a line and exit 1. That one change would have
surfaced this bug on the first run instead of the twenty-ninth.

If you have a loop that has been "quiet for a while", go and read its last
ten runs before you conclude it is working. That advice cost me three weeks.

The short version

Deterministic check → script. Judgment → model. Do not mix them for convenience.
Give the silent-on-success convention one exception: "I could not measure" must always be loud.
Read the runner's own execution history, not just your notifications.

Want a version of this for your own site or shop — uptime, certificates,
backups, mail that only pings you when it matters?
WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

DEV Community: Hoelee

Authentik 2025.8 to 2026.8: The Breakable Parts Nobody Warns You About

What is authentik, and why run it?

Why upgrade at all

Pitfall 1: Portainer is the source of truth, not the compose file

Pitfall 2: the storage mount moved (2025.12)

Pitfall 3: RBAC migration leaves a poisoned session table

Pitfall 4: trusted proxies are now opt-in (2026.8)

Pitfall 5: the one that broke SSO — authorization_flow vs authentication_flow

What I'd do differently

The result

Want single sign-on for your business?

Upgrading n8n v1 to v2: Seven Deprecations in One Log File

Why it matters

Start here: n8n tells you what's wrong at boot

1. Your env values can now fail schema validation

2. WEBHOOK_URL is deprecated for N8N_WEBHOOK_URL

3. Internal task runner mode is going away — and mine was already broken

4. Task timeout drops from 300s to 60s

5 & 6. Two compression-node limits shrink (2 GiB → 256 MiB, 5000 → 1000 entries)

7. The storage path renames in v3 — and you have a volume mounted at the old one

The one outside the deprecation block: my AI sandbox disabled itself

What I'd do differently

The result

Running automation you'd rather not babysit?

Upgrading CodeIgniter 4.6 to 4.7: The Breaking Changes the Guide Misses

Why this matters

What I checked first

The two fatals that were not in the guide

1. Config\App::$permittedURIChars

2. Config\Format::$jsonEncodeDepth

How to find the rest before they bite you

Also worth knowing before you upgrade

What I'd do differently

The result

The JPA Field That Turns a Silent Lost Update Into a 409

The problem: an edit that vanishes with no one to blame

What I tried first, and why it failed

The fix: a @version field, a contract, and a conflict

1. The entity carries a version column

2. The version is part of the API contract

3. The service checks it inside the transaction that owns the write

4. The handler maps the exception to a 409

5. The whole path, end to end

Why a conflict is the correct answer, not a retry

What the rest of the reference service proves

Hardening This Pattern for Production Rollout

The result

When a Self-Hosted Model Gateway Tripped Trust & Safety

The error that sent me the wrong way

Why a shared token is not a shared key

Task dispatch instead of routing

The economics survived, because the workload splits

💬 Let's Compare Notes

Embedding Fonts So a PDF Looks the Same on Every Machine

The requirement was one sentence

Rule one: read the fonts back out of the finished PDF

Rule two: static instances only, and the reason is in the parser

Rule three: the engine speaks two weights, so map onto two slots

Rule four: do not subset against a character set you do not control

Rule five: make missing glyphs a build failure, not a surprise

What I would do differently

The result

How I Vet an Open-Source Dependency Before Betting On It

The problem, phrased as the question I'd actually search for

What I tried first, and why it wasn't enough

The six checks that actually settled it

1. Read the repo's vital signs before any feature claim

2. Grep for the feature instead of reading for it

3. Read the target branch line at the top of a feature doc

4. Read the data model, not the feature list

5. Check which paid tier gates the feature you need

6. Check that the money rail works in your country

Then: ask the fallback question before you need it

What I'd do differently

The result

NocoDB SSO Is a Licensed Feature — and the Env Vars Won't Tell You

Why this matters

What the docs say, and what the image does

How to test it without risking anything

2. `WEBHOOK_URL` is deprecated for `N8N_WEBHOOK_URL`

1. `Config\App::$permittedURIChars`

2. `Config\Format::$jsonEncodeDepth`

The second trap: `$__all` is not `.*`