DEV Community: Han-co

Deployment isn't the finish line: validating and monitoring credit models

Han-co — Fri, 10 Jul 2026 08:48:54 +0000

Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)

Up through Part 6, we've talked about building a good model and then designing policy with it. In finance, though, this is where the real work begins.

In Part 0 I mentioned in passing that "a model starts aging the moment it ships," and that "there's a reason things are slow." This piece is where I unpack that reason. In other fields, deploying a model feels like crossing the finish line; in credit, deployment is closer to the starting line. If building it took a few weeks, validating, overseeing, and watching over it afterward takes months — sometimes years.

Here's what this piece covers: having someone else validate the model independently before deployment, deploying itself carefully, and — after deployment — keeping watch and retraining it once it ages. And, wrapping all of it, documentation and governance.

The builder shouldn't be the validator

First, a concept: model risk. It's the umbrella term for the losses, reputational damage, and regulatory exposure that arise when a model is wrong or misused. A single model flows all the way through to provisioning and capital (Part 0), so when it's wrong, the whole firm wobbles. That's why finance treats a model as an asset that carries risk, and the framework for managing it is called model risk management (MRM).

There's one principle of MRM worth remembering: the person who builds the model and the person who validates it must be different people. This is usually called the three lines of defense. The first line is the team that develops and runs the model; the second is validation and risk management, independent of the first; the third is internal audit. "I validated my own model and it came out great" may fly in other fields, but not in finance. The builder has a hard time seeing the weaknesses of their own model. So validation is always done by a different set of eyes.

What validation looks at

When an independent validator receives a model, they look at roughly five things.

First, conceptual soundness. Do the methodology, the assumptions, and the variables make sense in the first place? Even if performance looks good, a relationship that defies common sense — "higher income means higher default risk" — is a coincidence in the data, not a signal you can trust.

Second, quantitative validation. Confirm discrimination, calibration, and stability in numbers — the three axes from Part 5. Is the AUC good but the PD wrong? Does it stay stable as time passes?

Third, sensitivity and stress testing. Does the score stay put when you nudge the inputs a little, and how does the model behave under a bad scenario like a recession? A model that only fits well in good times collapses exactly when you need it.

Fourth, benchmarking. Does this complicated model actually improve on a simpler one, or on the model already in place? If it's complex and doesn't improve much, that complexity is just risk.

Fifth, reproducibility. Run it again on the same code and data — do you get the same result? A model you can't reproduce can't be validated or audited.

You don't switch it on all at once: shadow mode

Passing validation doesn't earn a model an immediate spot in production, either. The standard safeguard when introducing a new credit model is shadow mode.

Shadow mode means running the new model alongside the live one, computing its scores without letting them drive any real decision. You approve and reject with the existing method, while recording — for months — what the new model would have decided. Only after you've confirmed the new model runs stably and doesn't diverge wildly from the existing results do you actually switch over.

In a similar spirit, you might compare a champion and a challenger in parallel, or do a gradual rollout that turns the model on for a subset of customers first. Either way, there's one thing in common: a rollback plan laid down in advance so you can undo it the instant something goes wrong. The principle isn't "switch on the high-performing model fast" — it's "switch on safely, even if it's wrong."

Models age: monitoring and retraining

After deployment it isn't over — the longest stretch is only just beginning. It's the aging I mentioned in Part 0. While a model trained on 2024 data scores 2026 customers, the economy, rates, and customer behavior all change. The distribution shifts, and the model slowly ages.

So after deployment you keep watching from several angles. First, how far the inputs have drifted from training time, with PSI and CSI — the stability metrics from Part 5. You track performance too, of course. But there's a difficulty peculiar to credit here. Since a default label is only settled 12 to 24 months later (Part 0), you can't see performance decay in real time. So you run backtests — matching predictions against actuals — belatedly, as labels accumulate, and in the meantime you catch warning signs early with leading indicators like PSI. You also watch whether the predicted default rate matches the actual, whether one particular segment is drifting badly, and whether the data pipeline has quietly broken.

And once you judge the model has aged, you retrain it. When to retrain — the trigger — is decided in advance too: when the distribution has moved a lot, when performance has dropped, when the policy changed, when a new product launched. The point is not to retrain on a whim, but to define the cadence and the procedure ahead of time and follow that governance. Because changing a model is itself a risk.

The documentation is the model

Finally, it's about leaving a written record of everything about the model. This part is surprisingly the crux.

Every model comes with development documentation: what it's for, what data and what methods and assumptions built it, what its limits are, and what the validation results and monitoring plan look like. To that you add an audit trail recording when what changed, plus version control and change management.

Here's the crux: documentation isn't cleanup you do after the model is finished — it's part of the development process. When the regulator asks "why this model?", when internal audit asks "what's the basis for this decision?", the documentation is the answer. A model you can't explain on paper is unusable in finance, no matter how well it performs.

So, there's a reason things are slow

In Part 0 I said the startup reflex of "ship it fast" doesn't travel well here, that there's a reason things are slow. This piece has been the explanation of that reason. Someone other than the builder validates; you switch on carefully; you watch it age and retrain; and you leave all of it in writing. It looks like a hassle, but it isn't a burden regulation forced on you — it's the infrastructure of trust. Because a single model judges one person's credit and can shake the firm's capital.

Predicting well is the entry ticket (Part 5); handling causality is skill (Part 6); and honestly validating the model you built and keeping it alive for the long haul — that is trust. Next time, I'll continue to the other axis of that trust: interpretability, the demand that you be able to explain why the model decided what it did, along with fairness and regulation.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Where do rejected applicants go? Reject inference and rejectkit

Han-co — Tue, 07 Jul 2026 07:32:23 +0000

Originally published at han-co.com · Code: GitHub · Part of the "Deep Dive" strand of my Credit & Finance Data Science series. (The original also has a hand-drawn diagram.)

In Part 4 I touched on reject inference briefly. The point was that a model built on approved customers alone becomes biased once you apply it to the full applicant pool. This post is a record of writing that reject inference in code. I bundled the classic techniques together and, above all, built a library that also evaluates "whether this correction actually helps on my data." It's called rejectkit, and it's now on PyPI and GitHub.

You will never know how the rejected applicants turned out

Picture a loan review. An applicant arrives, you look at their information, and you either approve or reject. For the ones you approve, you learn the outcome a few months later: pay well and it's good, fall behind and it's a default. But the ones you reject were never given a loan, so there's nothing to repay. You will never know how they would have turned out.

That's where the trouble starts. To build a model that predicts whether next year's applicants will fall behind, you need labeled data, and the only labels you have belong to the people you approved. But approved customers are a skewed sample: they got through precisely because they looked fine in the first place. A model trained on them alone has a distribution that's out of step with the full set of applicants who actually show up at your door.

It's the same as a doctor who concludes "my treatment works great" from the patients who came back. The ones it didn't help never returned, so they never make it into the data. This is exactly the selection bias from Part 0 and the reject inference from Part 4.

And yet the Python tooling was empty

Reject inference has been a standard topic in credit risk for decades. And yet, when I went looking, the Python tooling was empty.

R has a package called scoringTools, but it isn't even on CRAN, only on GitHub. Python's scorecard libraries (scorecardpy, optbinning, toad) do the WOE/IV binning and logistic scorecards from Part 4 well, but they don't touch reject inference at all. What was left was one-off research code written for papers.

So I built rejectkit. There were two goals. One was to bundle eight classic techniques behind a single scikit-learn-style API. The other, the more important one, was to provide a benchmark that measures "whether this correction actually helps on my data." The truth is that "the value of reject inference is doubtful, and no technique is always superior" has been the field's long-standing conclusion. A paper literally titled "Reject inference — can it work?" was already out in 1993. So the real message of this library isn't "trust it and use it," it's "measure first."

Eight techniques, and the assumption that decides everything

The ways of bringing rejects back into training fall into roughly three families.

First, methods that manufacture labels for the rejects and fill them in (the augmentation family). Score the rejects with the approval model and cut at a threshold to assign labels (simple); split one person into a good version and a bad version, two rows with weights (fuzzy); multiply the default rate by score band by a weight to encode the practitioner's assumption that "a reject is worse than an approved customer at the same score" as a number (parcelling); or borrow the default rate of nearby approved customers with similar features (extrapolation).

Second, methods that correct without manufacturing labels. Train a selection model that separates approvals from rejects and give approved customers an inverse weight (IPW reweighting, the same logic as the bias correction from Part 2), or take econometrics' Heckman control function and fold it into the classification problem as an extra feature.

Third, semi-supervised learning. Leave the rejects as unlabeled data, attach pseudo-labels only to the high-confidence ones, and repeat the retraining loop (self-training).

Which technique works comes down, in the end, to a single assumption: what the rejection depends on.

If rejection depends only on observed features (MAR), a well-built model actually isn't all that biased.
If rejection also depends on the unobserved outcome (MNAR), say a past reviewer screened out bad applicants using information that isn't in the data, then a model that saw only approved customers is the most biased of all.

And here's an important limitation. The augmentation family leans on the biased approval model to guess the rejects' labels, so it can't pull itself out of a strong MNAR situation on its own. Which is why the right question isn't "which technique should I use" but "does reject inference even help in my situation right now," asked first.

How do you measure whether it helps

The fundamental difficulty of reject inference is this: rejects have no ground truth, so you can't grade the correction directly.

rejectkit sidesteps this. It takes data where you know every outcome, deliberately picks some rows to play the role of rejects, and hides (masks) their labels. Then it grades how well each technique recovers those hidden labels, scored on an untouched test set. The "Masked" in the name is this label hiding.

The key metric is auc_recovery. Zero means on par with the naive model that used approved customers only; 1 means recovery all the way to the oracle that used every label; negative means it made things worse.

You pick how rejections are generated from three options: mar, which depends only on features; mnar, which also depends on the hidden outcome (the harshest); and cutoff, which approves in order of lowest predicted risk (closest to a real credit policy).

So does it help: synthetic data vs. real data

I ran it on clean synthetic data with MNAR. The oracle is 0.820 and naive is 0.749, but the reject-inference techniques barely clear naive and several actually lose ground. fuzzy scrapes break-even at +0.001, parcelling degrades to -0.116, and only Heckman just about holds the naive line. When selection depends on the outcome (MNAR), it plays out exactly as theory predicts. Reject inference is no free lunch.

So let's check on real data. I applied it as-is to the Kaggle Home Credit dataset (about 300,000 rows, 8% default rate).

Model	AUC	auc_recovery
oracle (all labels)	0.741	1.00
naive (approved only)	0.568	0.00
parcelling	0.582	+0.084
extrapolation	0.582	+0.079
heckman	0.580	+0.071

Here it's the opposite. Ignore the rejects and you collapse from 0.74 to 0.57, and reject inference actually recovers 7–8% of that loss. What was nearly useless on synthetic data is worth having on real data. And even with the same dataset, switch to mar or cutoff and naive is already oracle-grade, so there's no room left to recover (auc_recovery comes back as NaN).

The conclusion lands in one place. Whether reject inference helps depends on the data, which is why you have to confirm it with a benchmark before you use it.

How to use it

Fitting reject inference onto a single model goes like this.

from sklearn.linear_model import LogisticRegression
from rejectkit import RejectInferenceClassifier

# X_accept, y_accept: approved customers and outcomes (1=default) / X_reject: rejects (features only)
clf = RejectInferenceClassifier(
    estimator=LogisticRegression(max_iter=1000),
    method="parcelling",
    method_params={"uplift": 1.3},   # assume rejects are ~30% worse than the same score band
)
clf.fit(X_accept, y_accept, X_reject)
pd_bad = clf.predict_proba(X_new)[:, 1]

Measuring whether it helps first is one line.

from rejectkit import MaskedRejectBenchmark

bench = MaskedRejectBenchmark(selection="mnar", accept_rate=0.6, random_state=0)
print(bench.compare(["fuzzy", "parcelling", "reweighting", "heckman"], X, y).round(4))

Inputs accept pandas, polars, and numpy alike. It also ships with diagnostics (per-feature PSI) and visualizations for seeing how far the approval and rejection distributions diverge.

Wrapping up

Train a credit model on approved-customer data alone and you get sample-selection bias. That's the problem from Part 0 and Part 4.
rejectkit bundles eight classic techniques for correcting it behind one API.
The real differentiator is the benchmark that measures whether the correction helps on your data. Reject inference isn't a cure-all and can even be harmful, so the core idea is to measure first rather than trust and use.
The same principle applies beyond credit, to any selection-bias setting where you only see outcomes for those who got through: hiring performance, or fraud detection where only the flagged cases get labeled.

And the most trustworthy answer to reject inference is, in fact, a different one entirely: an experiment that deliberately approves a small random slice to secure real outcomes. It hands you fact instead of an estimate. That thread continues into causal inference and experiments in Part 6.

Installing is one line.

pip install rejectkit

GitHub: github.com/HangilKim11/rejectkit
PyPI: pypi.org/project/rejectkit

Feedback, issues, and PRs welcome.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Beyond prediction: causal inference and experiments in credit

Han-co — Tue, 07 Jul 2026 07:09:24 +0000

Originally published at han-co.com · Part of the "Basics" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the text below is identical.)

So far this series has mostly been about getting "who is risky" right. Building good features, scoring with trees, checking whether those scores are well calibrated. But in practice the harder question, the one with real money riding on it, is a little different.

If we raise a limit, will defaults go up? If we chase a delinquent customer, will we recover more? If we cut the rate, will fewer people churn?

These aren't prediction questions, they're causal ones. In Part 0 I described the core of this field as "estimating the unobserved counterfactual," and this is exactly where that counterfactual walks in through the front door. And the moment you can answer these questions, a credit data scientist graduates from "the person who guesses who's risky" into "the person who designs policy."

Prediction and causation are different questions

Prediction asks: what is this customer's probability of default? Causation asks: if we raise this customer's limit, how much does that probability change? These are completely different questions, and the ways you answer them are different too.

The fundamental reason causation is hard is that you can never see two outcomes for the same person at once. What would have happened to this customer, whose limit we raised, if we hadn't raised it is something we will never observe. We only ever see half of the world. This is the general form of the story from Part 0, where I said we never learn how the customers we rejected would have turned out.

The catch is that in credit, the decisions we make are themselves interventions in the outcome. Whether we approve, how large a limit we grant, what rate we set all directly affect that customer's default. So the data in our hands isn't a world we sat back and observed, it's a world we already shaped by intervening in it.

Here's where a common trap appears. The data shows that customers with higher limits default less. So if we raise limits, will defaults fall? No. We simply handed the higher limits to customers who already looked creditworthy. The limit didn't lower defaults; a hidden cause, good creditworthiness, produced both the high limit and the low default at the same time. A hidden cause that moves two outcomes together like this is called a confounder. (I checked this directly with public data in the Deep Dive piece.)

Mistake a correlation for causation when you set policy, and no amount of data will keep you from getting the direction wrong. This is why in credit, causal inference isn't optional, it's mandatory.

The gold standard is an experiment

The way to turn a counterfactual from a guess into a fact is actually simple: randomize the intervention. Split the target customers at random into two groups, raise the limit for one and leave the other alone. Because the split is random, the two groups have the same creditworthiness and the same income on average. So whatever difference opens up later in default rate or profit is due to the limit and nothing else. This is a randomized controlled trial (RCT), and it's the gold standard of causal inference.

Credit has one special form of this experiment: deliberately approving, at random, a small share of applicants you would otherwise have rejected. In Part 0 and the rejectkit piece I said we can never learn how rejected customers actually turn out, and this is the one way to turn that counterfactual into a fact. Reject inference guesses at that blank space with assumptions; an experiment just lets you observe the answer. That's why I said the most trustworthy solution to reject inference is, in the end, a small randomized approval.

It isn't free, of course. Approving bad customers on purpose is a loss, and a randomized limit increase carries risk too. So you don't swing big, you run experiments small, with a controlled design. How large a sample you need to detect a difference statistically, whether to randomize at the customer level or the branch level, where the loss has to cross before you stop, and even the lag from default labels only settling 12 to 24 months later all have to be set before you begin.

When you can't experiment: quasi-experiments

You can't always run an experiment. Often it's too slow, too costly, or ethically or legally out of bounds. When that's the case, you approximate causation as best you can from observational data you've already accumulated. This is called a quasi-experiment.

First, propensity scores and inverse-probability weighting (IPW). This is a way of making the treated group and the untreated group resemble each other on their observed characteristics. It's the same logic as the bias correction from Part 2, or the reweighting in rejectkit. If you gave larger limits to customers with good creditworthiness, in effect you pair up and compare people of similar creditworthiness.

Second, regression discontinuity (RDD). This one fits credit especially beautifully. Underwriting has a cutoff score. If 630 is the line, an applicant at 631 and one at 629 are essentially the same person, yet only one gets approved. If the outcome jumps sharply across that boundary, that jump is the causal effect of the intervention that is approval. The underwriting cutoff becomes, in itself, a natural experiment.

Beyond these, the toolbox holds more: difference-in-differences (DiD), which compares before and after a policy change, and instrumental variables (IV), which use an outside factor that only affects the intervention as a lever. Quasi-experiments all lean, though, on an assumption of the form "there is no unobserved confounding." So the order is clear. Experiment when you can; when you can't, use a quasi-experiment but always state which assumptions you're standing on. And the honest move is to check, with a sensitivity analysis, whether the conclusion flips when those assumptions wobble a little.

Python has a few tools that help with this process. DoWhy lets you treat causation as a flow of "state your assumptions, estimate, and then refute by testing how much the result shifts if those assumptions are wrong." It essentially has you write your assumptions down in code and go knock on them yourself, which pairs well with the sensitivity check I just mentioned. EconML collects the latest methods for estimating treatment effects, especially the heterogeneous treatment effect (HTE) that varies from person to person (uplift, just below), such as double machine learning and Causal Forest. CausalML specializes in uplift modeling and targeting, handling everything from meta-learners to the Qini evaluation that comes later, in one place.

Beyond the average: uplift

So far we've been asking whether raising a limit increases profit on average. But the real question in practice goes one step further: for whom should we raise it?

Raising a limit for a customer whose profit won't grow either way just adds risk, which is pure waste. Conversely, raising it for a strong customer who couldn't spend for lack of headroom generates incremental profit. Estimating, at the individual level, this extra difference an intervention creates for a given person is uplift modeling. It's not the same as simply predicting who will respond. You're looking not for the person who would have acted anyway, intervention or not, but for the person the intervention actually changes.

The use in credit is clear. Raise the limit only for customers where the increase creates incremental profit, and intervene only on the delinquents where a chase actually lifts recovery. As for techniques, there are things like the T-learner, the X-learner, Causal Forest, and uplift trees.

One word of caution. You must not evaluate uplift with ordinary AUC. Since what we're trying to get right isn't whether someone defaults but the effect of the intervention, you have to look at dedicated metrics like the Qini curve or Uplift@k.

Here's the difference. AUC measures how well you rank whether this person will default, or respond. But what uplift wants is a ranking of how much the intervention changes this person, so what's being measured is different from the start. On top of that, since you can't see both the intervened and the un-intervened outcome for one person at once, there is no per-person ground truth at all, no answer key to grade one person at a time the way an ordinary metric does. So the Qini curve detours around this by working with groups. As you intervene on customers starting from the ones the model scored highest, it plots, cumulatively, how much faster the outcome gap between the treatment and control groups piles up than it would if you'd chosen at random. Uplift@k is the incremental gain piled up by the time you've reached the top k%. In a word, where AUC measures "how well do I guess who will default," Qini measures "when I intervened in the order I chose, how much more did I actually make than intervening on no one at all."

In Part 5 I said that when the objective differs the metric has to differ too, and this is where that principle applies most dramatically.

So, what expertise in this field really means

I'll say again that prediction is the price of entry and causation is the expertise. Every decision we make in underwriting is an act of intervening in the outcome, and only when you can estimate that effect honestly can you design policy.

To lay out the order: experiment when you can, even in a small way. When you can't, approximate with a quasi-experiment but state which assumptions you're standing on. Credit in particular has a natural experiment in the underwriting cutoff, so don't leave regression discontinuity going to waste. And beyond the average effect, use uplift to choose whom to intervene on.

This way of thinking is the root solution to reject inference, and the foundation for limit policy, collection strategy, and pricing. In the next part I'll carry this into governance and monitoring: how you validate, audit, and operate the models and policies you've built this way.

The record of debiasing the limit-causation question by hand on public data is in the Deep Dive, so if you'd like to see the concept confirmed hands-on, that's a good one to read alongside this.

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/

Can Google's TabFM beat GBM in credit? I tested it on public data

Han-co — Sun, 05 Jul 2026 13:33:15 +0000

Originally published at han-co.com · Code: GitHub · Part of the "Review" strand of my Credit & Finance Data Science series. (The original has hand-drawn diagrams; the numbers below are identical.)

This is the first piece in the "Review" strand of the series. It's where I take a newly released tool and, through a practitioner's eyes, measure whether it's actually useful for credit work.

Google recently released TabFM: feed it a table as-is and it predicts with no training and no tuning, and it claims that "zero-shot, it beats even a well-tuned GBM." If that really holds in credit risk — a place GBMs have dominated for years — it's a big deal. So I pitted it against a carefully built GBM on public loss data. To give the conclusion first: under the conditions I ran, a well-tuned GBM came out slightly ahead. But this was not a head-to-head under the same conditions the paper claimed its win, so I can't declare a winner — and more to the point, the key finding is that TabFM came right up against that best with no feature engineering and no tuning.

What TabFM is

TabFM is a foundation model for tabular data that Google released at the end of June 2026. The core idea is zero-shot. It doesn't retrain per dataset, tune hyperparameters, or engineer features. Instead it reads the training data in as a single context, and prediction is done in one forward pass. This is called in-context learning. Much as GPT takes a handful of examples in the prompt and answers, TabFM takes the whole table as a kind of prompt and answers on the spot.

What makes this possible is an enormous amount of pretraining done in advance. But unlike images or text, there isn't nearly as much public tabular data, so Google synthesized hundreds of millions of tables with structural causal models (SCMs) and pretrained on those. The learned architecture has three parts. It attends alternately over rows and columns to capture relationships between features, compresses each row into a dense vector, and then predicts on top of that via in-context learning. Think of it as an evolution of the TabPFN line, an earlier attempt at a tabular foundation model.

The selling point is clear: no feature engineering, no hyperparameter tuning. In other words, the effort you pour into a GBM in practice drops to zero. And Google reported that on the TabArena benchmark this zero-shot model outranks painstakingly tuned supervised models — GBM-family models in particular — on ELO. That benchmark gathers 38 classification and 13 regression datasets, ranging from 700 to 150,000 samples. The model is out now on Hugging Face and GitHub, and Google says it will land in BigQuery within a few weeks, callable from a single line of SQL (AI.PREDICT).

Why the experiment: can it be used on credit losses

I had one question. This model that supposedly beats GBM on general benchmarks — does it do the same on credit losses?

In an earlier piece I concluded that on tabular credit data it's tree-based boosting, not deep learning, that wins. TabFM claims to overturn exactly that conclusion, so it was worth re-checking. And credit isn't like a general benchmark. Default is a rare event; you need not just the ranking but the probability to be accurate; and you have to be able to explain the reasons behind a decision. Winning on general data doesn't guarantee winning on credit. So I checked directly, on public data, whether zero-shot TabFM beats a well-tuned GBM in credit too.

Data and models

The data is UCI's Taiwan credit-card default dataset: 30,000 people, 23 features, a 22% default rate. The main features are the last six months of delinquency status plus billed and paid amounts, and it's known in the literature as a signal-limited dataset where standard models tend to stall somewhere around AUC 0.77 to 0.78.

The heart of a fair comparison is separating the model from the features. Compare a "feature-engineered GBM" against "raw TabFM" directly and, even if you see a gap, you can't tell whether it's the model or the features. So I lined up several baselines side by side.

Name	What it is
GBM raw	LightGBM, no features, no tuning (the effort floor)
GBM tuned	LightGBM + engineered features + Optuna tuning
CatBoost / XGBoost	Strong baselines with features + tuning
TabFM zero-shot	Out of the box, single forward pass (the main event)
TabFM ensemble	Paper preset (derived features + calibration)

The GBM-family models got features like delinquency dynamics added and were tuned with Optuna. TabFM was left out of the box, with zero tuning. I trained every model at the natural rate so the probabilities would match the actual default rate. For metrics I looked, as credit demands, at both discrimination (ROC-AUC, PR-AUC, KS) and calibration (Brier, ECE). Since default is 22%, the imbalance isn't extreme, but I also checked how well the rare defaults are caught with PR-AUC. I validated with stratified 5-fold.

Results

Arm	ROC-AUC	PR-AUC	KS	ECE ↓	Time
GBM tuned (LightGBM)	0.789	0.566	0.443	0.010	548s
XGBoost	0.789	0.565	0.439	0.009	102s
CatBoost	0.788	0.566	0.444	0.011	1179s
TabFM zero-shot	0.785	0.558	0.441	0.022	503s
GBM raw	0.779	0.554	0.429	0.013	0.5s
TabFM ensemble	0.774	0.540	0.418	0.018	268s

There are three things to read off this.

First, a well-tuned GBM (0.789) edges out TabFM zero-shot (0.785). All three trees sit above TabFM, and the ordering is the same on PR-AUC too (TabFM 0.558, tuned GBM 0.566). The gap is 0.4pp, inside the fold standard deviation (0.006), so statistically it's close to a tie — but the direction is consistently GBM on top. "Zero-shot beats a tuned GBM" did not hold on this data.

Second, compare no-effort against no-effort and the story is different. TabFM zero-shot (0.785) clears raw GBM (0.779) — with no features and no tuning. As a fast baseline it's genuinely attractive.

Third, calibration is on par too. Trained at the natural rate, the trees also produce well-matched probabilities (ECE 0.010), and TabFM comes in slightly behind at 0.022. Neither is badly off.

And the three boosters converge between 0.7885 and 0.7891. That means the ceiling on this data — where changing the model and adding features and tuning still doesn't lift you — is around 0.79. Neither side got above it.

To sum up: at least in this experiment, TabFM didn't beat a well-built GBM, it came close to it with no effort. And you have to keep in mind that the conditions weren't the same.

Limits of this experiment

These results are only valid within a few conditions.

It's one dataset. A single Taiwan credit-card set can't guarantee generalization. On other loss data the ranking could change.
The signal is limited. With 23 features and a 0.79 ceiling, there wasn't much room for a model to pull ahead in the first place. On data with many features and rich signal it could come out differently.
One seed, and I couldn't do out-of-time validation. With no trustworthy time column I split by random stratification, but a real credit model is more rigorously validated by splitting on time.
TabFM ran on an 8GB GPU. Because of that I couldn't properly run the ensemble setup noted in the model limits below, so the TabFM numbers in the table above should be read as a lower bound.

Limits of the model itself

Leaving the experiment aside, here's what catches when you try to bring TabFM into practice.

It's a black box. Being a foundation model, there are no coefficients and no clear rules. For credit underwriting, where you have to disclose the reason for a decision and explain it to regulators, it's hard to use as-is. You can bolt on post-hoc explanations like SHAP, but the model itself doesn't become the explanation the way a scorecard does.
It needs a high-end GPU. The weights alone are 6.5GB, so without a GPU inference is more than ten times slower, and the paper preset (ensemble, large context) needs a 16GB-plus GPU to run properly. That's a sharp contrast with a GBM, which runs fine on a single CPU.
There's a ceiling on data and feature size. In-context learning reads the whole training set in like a prompt, but attention grows with the square of the context. So millions-of-rows credit data, or data with very many features, is hard to load as-is. That's exactly why this family of models was built for small- to mid-sized tables in the first place.
Inference is heavy. A saved GBM scores in one line, but TabFM re-reads the training data every time it predicts. In high-volume real-time scoring, that cost becomes a burden.
It's still new. As a just-released model, it has no track record in practice and no cases of regulatory acceptance.

What's next, and you too

TabFM is available on Hugging Face now, and within a few weeks it should be callable from a single line of SQL in BigQuery. The barrier to entry drops that much, so if you have a big GPU or the BigQuery integration opens up, it'd be worth running the ensemble preset properly and re-measuring on larger data. I'm especially curious how it does on small-sample data, feature-rich data, and time-based validation. This piece's conclusion is only the story of one Taiwan credit-card round, after all.

Here's my conclusion. In this one experiment GBM came out slightly ahead, but I didn't compete under the conditions the paper claimed its win (large, diverse data; an ensemble preset; ample GPU), so I can't declare that it "can't win." What is certain is this: it's a remarkably fast baseline that lands within 0.4pp of that best with no features and no tuning. For prototyping or a first baseline it's attractive right now, and for a production credit model that needs peak performance, accurate probabilities, and explanations, a well-tuned GBM is still the safe choice for now. It's too early to say the tree-based conclusion has fallen, but there's now a candidate that comes close with no effort.

Appendix: code and data

Data: UCI Default of Credit Card Clients (Taiwan), public data
TabFM source: Google Research blog · Hugging Face · GitHub
Code: github.com/HangilKim11/blog-research/tree/main/tabfm-credit
Reproducibility: stratified 5-fold

I write han-co.com, a blog on credit and finance data science, in the language of practice. New posts by email: https://han-co.com/en/blog/