Casey Zhang - DEV Community

Casey Zhang

Nov 9

Six of Your Forty Tasks Should Never Have Been Scored: A Task-Admission Gate for Agent Benchmarks

#ai #testing #opensource #programming

7 min read

Casey Zhang

Nov 8

Calibrate the Task Before the Model: A Spec-Only Floor for Agent Benchmarks

#ai #testing #python #opensource

1

5 min read

Casey Zhang

Nov 7

Your Benchmark Forgot the Box: Attribute the Wobble Before You Blame the Model

#testing #ai #devops #programming

7 min read

Casey Zhang

Nov 6

Stop Peeking at Run Five: A Sequential Stopping Rule for Free-Tier Agent Benches

#ai #testing #python #devtools

7 min read

Casey Zhang

Nov 5

Grade the Grader First: A κ-Gate for Coding-Agent Benchmarks

#ai #testing #python #opensource

6 min read

Casey Zhang

Oct 27

Kill the Runner Mid-Patch: A Fault-Injection Bench for Agent Sessions

#testing #ai #opensource #performance

5 min read

Casey Zhang

Oct 26

Your Benchmark Has No Cost Axis: Score Free Endpoints by Quota per Accepted Patch

#benchmark #ai #testing #python

6 min read

Casey Zhang

Oct 20

Two Runs, Six Points Apart: Measure the Re-Run Spread Before You Compare Endpoints

#ai #testing #python #programming

7 min read

Casey Zhang

Oct 19

Free Model Endpoints Need a Contract Bench, Not a Leaderboard Screenshot

#ai #testing #opensource #api

7 min read

Casey Zhang

Oct 18

Your Coding-Agent Benchmark Needs a Run Manifest, Not Another Leaderboard Row

#ai #testing #python #opensource

6 min read

Casey Zhang

Oct 17

Free Tiers Are a Moving Target: Benchmark Them With a Frozen Envelope, Not a Screenshot

#ai #testing #python #performance

6 min read

Casey Zhang

Oct 16

Balance the Outcome Ledger: A Publication Gate for Coding-Agent Benchmarks

#ai #testing #devtools #opensource

1

7 min read

Casey Zhang

Oct 11

Keep the Denominator Frozen: A Checker for Agent Pass Rates

#ai #testing #python #tutorial

7 min read

Casey Zhang

Oct 10

Add a Negative-Control Slice Before You Believe a Coding-Agent Score

#testing #ai #python #tutorial

1

7 min read

Casey Zhang

Oct 9

Name the Failure Class Before You Rank a Coding Agent

#ai #testing #python #tutorial

10 min read

Casey Zhang

Oct 8

Reproduce the Score Before You Cite It: A 20-Task Harness You Can Run on Free Infrastructure

#ai #testing #python #opensource

6 min read

Casey Zhang

Oct 8

Refuse the Percentage Until the Run Envelope Is Frozen

#python #testing #ai #tutorial

6 min read

Casey Zhang

Sep 24

Measure the Public-Hidden Gap Before You Cite a Coding-Agent Score

#ai #testing #python #tutorial

8 min read

Casey Zhang

Sep 23

Measure Assertion Drift Before You Trust a Coding-Agent Score

#ai #testing #python #productivity

1

7 min read

Casey Zhang

Sep 22

Write the Benchmark Card First: Dataset, Metrics, and Controls for Coding Agents

#ai #testing #python #opensource

8 min read

Casey Zhang

Sep 21

Four Fields That Make a Coding-Agent Score Citeable

#ai #testing #python #programming

7 min read

Casey Zhang

Sep 20

Score the Patch, Not the Story: A Coding-Agent Evaluation Schema

#ai #python #testing #programming

2

8 min read

Casey Zhang

Sep 19

Don't Average Pass Rates Across Unequal Token Budgets

#ai #testing #python #productivity

1

7 min read

Casey Zhang

Sep 18

Make a Coding-Agent Score Unprintable Until the Controls Pass

#ai #testing #python #opensource

8 min read

Casey Zhang

Sep 17

Freeze a Holdout Before You Quote a Coding-Agent Score

#ai #python #testing #programming

9 min read

Casey Zhang

Sep 16

Label Infra Failures Before You Rank Coding Agents

#ai #testing #python #productivity

7 min read

Casey Zhang

Sep 14

Plant Bugs Your Agent Must Find: A Self-Test for Coding-Agent Benchmarks

#ai #testing #devtools #programming

6 min read

Casey Zhang

Sep 13

Match the Human Clock Before You Quote Superhuman Coding

#ai #testing #python #productivity

9 min read

Casey Zhang

Sep 12

Replay the Same 24 Tasks Twice Before You Rank Two Coding Agents

#ai #python #testing #productivity

7 min read

Casey Zhang

Sep 11

Break Agent Ties With Cost Per Fix and a Bootstrap Interval

#ai #python #testing #productivity

7 min read

Casey Zhang

Sep 9

Hash the Suite. Lock the Controls. Then You May Quote a Number.

#python #ai #testing #agents

7 min read

Casey Zhang

Sep 8

Refuse Every Agent Score That Ships Without a Protocol Card

#python #ai #testing #opensource

8 min read

Casey Zhang

Sep 6

Your Coding-Agent Score Needs a Dummy Baseline and a Size Split

#ai #testing #python #agents

7 min read

Casey Zhang

Sep 5

Don't Ship the Pass Rate. Ship the Failure Mix.

#ai #testing #agents #python

7 min read

Casey Zhang

Sep 4

Freeze the Retry Budget Before You Trust an Agent Pass Rate

#ai #testing #python #agents

8 min read

Casey Zhang

Sep 4

Build an LLM Cache Proxy in 40 Lines to Cut Token Costs 83%

#llmcache #python #sqlite #tokencost

6 min read

Casey Zhang

Sep 3

Your Coding Agent's Pass Rate Isn't a Benchmark Until the Oracle Is Frozen

#ai #testing #python #opensource

7 min read

Casey Zhang

Sep 2

Benchmarking an AI Coding Tool? Start With the Data You Actually Trust

#ai #benchmark #programming #llm

5 min read

Casey Zhang

Sep 1

Your Prompt A/B Test Is Probably Random Noise: A Paired-Test Harness for Free-Tier LLMs

#ai #llm #testing #python

5 min read

Casey Zhang

Aug 31

Benchmarking AI Code Generators: A Reproducible Method You Can Run on a Free Server

#ai #testing #benchmark #llm

5 min read

Casey Zhang

Aug 30

Cold Starts Will Eat Your Free LLM Tier: A Reproducible Benchmark

#benchmark #serverless #llm #testing

4 min read

Casey Zhang

Aug 29

The Token Allowance Trap: A Task-Price Benchmark Before You Build on Free LLM APIs

#ai #llm #python #testing

4 min read

Casey Zhang

Aug 28

Measure Your AI Reviewer Before You Trust It: A Seeded-Defect Benchmark

#ai #testing #python #productivity

5 min read

Casey Zhang

Aug 26

Benchmarking LLM Agents Without the Marketing Math: Dataset, Metrics, and Controls

#ai #llm #testing #opensource

5 min read

Casey Zhang

Aug 25

From git log to Release Notes: A Free-Tier Agent Case Study

#ai #agents #python #opensource

5 min read

Casey Zhang

Aug 24

Crash-Proof Batch LLM Processing: A SQLite Job Queue on a Free Server

#sqlite #llm #python #devops

1

5 min read

Casey Zhang

Aug 23

Building a Free-Tier GitHub Issue Triage Agent: A Case Study

#ai #agents #python #opensource

5 min read

Casey Zhang

Aug 17

Cheap Model Hype Is Not a Benchmark: Run a 30-Minute Agent Gate Before You Swap

#ai #agents #testing #python

4 min read

Casey Zhang

Aug 17

Cheap Model Hype Is Not a Benchmark: Run a 30-Minute Agent Gate Before You Swap

#ai #agents #testing #python

4 min read

Casey Zhang

Aug 14

Tool-Permission Gate: The Only MiniMax H3 Test That Matters

#ai #agents #testing #llmsafety

5 min read

Casey Zhang

Aug 10

Pin Your Agent's Behavior: Snapshot Tests for Tool-Call Sequences When You Swap Models

#ai #testing #agents #python

5 min read

Casey Zhang

Aug 7

AI Agent Boundary Testing: Fake-Tool Harness on a Free Server

#aiagents #security #promptinjection #python

6 min read

Casey Zhang

Aug 7

I Gave My Coding Agent Fake Tools on Purpose: A Boundary-Test Harness You Can Run on a Free Server

#ai #agents #security #python

5 min read

Casey Zhang

Aug 5

Stop Guessing: A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Repo

#ai #testing #productivity #tooling

5 min read

Writing Debut