DEV Community: Ishank Choudhary

billion-context Hands-On: Compressing Coding-Agent History to Fit a Small Context Window

Ishank Choudhary — Mon, 09 Nov 2026 12:13:31 +0000

Long agent sessions keep failing the same way for me. The window fills up with test logs and file dumps, the harness runs its own auto-compaction, and the agent forgets the constraint I gave it an hour earlier. So when billion-context reached 5,427 GitHub stars about three months after the repo was created, I wanted to see what it actually does to a real session, measured in tokens and in what the agent still remembers afterwards.

This post covers what I installed, the one bug I ran into, the numbers from a real tool-calling session, and where I think the project's claims hold up and where they don't.

What billion-context is

billion-context (github.com/ranxianglei/billion-context, MIT plus an attribution clause) is a local proxy, bili, that sits between a coding agent and its model API. It works with Anthropic, OpenAI and Google-native requests, and it adds four tools to every request:

compress: fold a range of messages into a summary
decompress: restore a folded range
search_context: keyword search over folded summaries
acp_status: report how full the context is

The main difference from Claude Code-style auto-compaction is that the model decides what to fold, nudged by the proxy as the window fills. The proxy runs the fold on its side, keeps the originals in a local SQLite store, and from then on sends the folded view upstream. Summaries can be folded again (tier 2, tier 3), and the README calls this "an LSM tree over summaries."

The README makes three headline claims: "100K context is enough", "5× fewer tokens", and "month-long single sessions." The author's preprint in the repo reports 174,327 model calls over 4.5 months. A benchmark folder labeled "independent third-party" reports 79.2% vs 83.2% for full context on LongMemEval-S at 71% lower cost. Both live inside the project's own repo, so I count them as claims, not evidence.

The repo is also changing very fast: 565 commits between 2 Nov and today, and npm latest is 0.1.213. I tested that version.

Setup

Installing took about five seconds:

npm install -g billion-context --prefix=~/.local
bili --version   # 0.1.213 (133f3525c)

I had a Free Tier Gemini key and no other credentials, so I skipped the bili claude / bili codex launchers. Instead I used the universal path: start the proxy and prefix the upstream URL with /bili/. To get folds within an afternoon's budget, I pinned the window to 40K tokens:

{
  "port": 8787,
  "compress": {
    "modelContextLimit": 40000,
    "maxContextLimit": "60%",
    "nudgeGrowthTokens": 8000
  }
}

ACP_AUTO_UPDATE=0 bili start
curl -s http://127.0.0.1:8787/__bili/health

I set ACP_AUTO_UPDATE=0 on purpose. By default bili checks npm every three minutes and upgrades itself after a staggered delay. That's convenient for a fast-moving project, but I don't want it in a benchmark, and I'd think about it before putting this on a shared CI box.

The first bug: Gemini's OpenAI-compatible endpoint

My first harness used Gemini's OpenAI-compatible endpoint. The first request went through fine. Every request after a tool call failed with:

HTTP 400: Function call is missing a thought_signature in functionCall parts.

Gemini 3.x returns an opaque thought_signature with each tool call, and it has to come back in the next request. The same two-request script returned 200 when sent directly and 400 through the proxy, so the proxy is dropping that field on the chat-completions path. Switching to Gemini's native generateContent wire fixed it:

BASE = "http://127.0.0.1:8787/bili/https://generativelanguage.googleapis.com/v1beta"
r = requests.post(f"{BASE}/models/gemini-3.5-flash-lite:generateContent",
                  json=body,
                  headers={"x-goog-api-key": KEY, "x-acp-session": "s2"})

If you run Gemini 3 behind an OpenAI-style client, test multi-turn tool calls before anything else.

The test session

I wrote a small agent loop with a single bash tool and pointed it at a fresh clone of python-slugify. I scripted 11 user turns, the kind I'd actually type:

ticket SLUG-482 for a client, Python 3.8 compatibility, no new dependencies
run the tests, read slugify.py, explain max_length vs word_boundary
a standup decision: keep the - separator and add max_words: int = 0
a CLI flag, a CHANGELOG entry, a README update
mid-session: staging port 7431, reviewer Priya Raman, Unicode refactor pushed to Q1 because of a Django adapter risk

The model was gemini-3.5-flash-lite. For the baseline, I ran Gemini's countTokens on every uncompressed request prefix, using the same system prompt and tool schema without the proxy. That gives an exact "what would I have sent" number for each of the 41 requests.

One thing about the agent itself: Flash-Lite never edited a file. It read code, ran tests, and then claimed in its final answer that it had implemented the feature. The git tree was clean. That's a model problem, not a compression problem. The fold summary was actually more accurate than the model's answer, because it listed the feature under "Open objectives."

The numbers

The session took 41 client requests and 184 seconds of wall-clock time. That includes three back-offs from Free Tier 429s (250K input tokens per minute); summed model latency was 45.9 seconds. The proxy added a median of 10 ms of local processing per request (max 26 ms over 50 requests).

Folds: 2 events, 7 summary blocks of 111–284 characters each
Fold 1 (turn 6): next prompt went from 27,853 to 9,396 tokens; ~14,095 tokens reclaimed
Fold 2 (turn 10): 26,778 to 12,046 tokens; ~9,515 reclaimed
Final request: 13,431 tokens via bili vs 38,816 uncompressed (2.9× smaller)
Peak prompt: 27,853 vs 38,816
Total input over the session: 663,321 vs 758,108 tokens (12.5% less)
Fixed overhead: the first request was 4,243 tokens upstream against a 180-token uncompressed prompt, so the injected tools and instructions cost about 4,060 tokens on every request

This is the first block the model wrote:

[Compressed conversation section] — Repo Exploration
- User request (m00001): ticket SLUG-482 for client Northwind.
  Constraints: Python 3.8 compatibility, no new dependencies.
- Explored repo layout and located main `slugify()` in
  `slugify/slugify.py:126` taking 16 parameters.
- Ran pytest via `python -m pytest`: 133 passed, 0 skipped.

That's 284 characters standing in for a few thousand tokens of find output, a full pytest log, and an inspect.signature dump. The user messages themselves stayed verbatim, because the proxy protects them.

Did the agent still remember?

I asked two quizzes at the end of the session, once through the proxy (folded view) and once directly with the full history.

Easy set (10 questions about things I told it): ticket ID, client, Python version, separator decision, new kwarg, test file, first test result, port, reviewer, postponed refactor. Through bili: 10/10 on 13,821 prompt tokens. Full history: 10/10 on 39,114. Since user turns aren't folded, this is the case the design is built for, and it held up.

Hard set (8 questions about tool output): the line number of slugify(), the module marked "DO NOT MODIFY", the CLI --algorithm choices, the last commit hash, the top CHANGELOG version, the pytest/Python versions, the parameter count, and the CLI test files.

Through bili with tools disabled: 4/8. It answered "unknown" for the details that existed only in folded tool output.
Through bili with search_context/decompress allowed (streaming): 6/8. The model ran five search_context queries ("slugify.py", "CLI", "algorithm", "CHANGELOG", "git log") and then answered. The "algorithm" search matched zero summaries but five folded originals, so the search covers raw history too, not only the summaries.
Full history, direct: 6/8, with different misses: it got the line number wrong (195 instead of 126) and named the wrong CLI test file.

So the folded agent matched the full-history agent, but only by paying for retrieval: six upstream rounds and 89,958 input tokens (72,497 of them cached), against one 39K call. Three practical notes. First, on the non-streaming lane the search_context output came back to my client as the response text, with no answer after it, so plan on streaming. Second, the search receipts ("Found 2 block(s) for ...") show up in the client-visible text stream, as do fold receipts. Third, in an earlier attempt the model decompressed a block to /tmp/acp-decompress-b6-*.txt and then tried to grep it with its own bash tool, so decompress-to-file assumes the agent can read local files.

What it cost

At Gemini 3.5 Flash-Lite list prices ($0.30/M input, $0.03/M cached, $2.50/M output):

bili: 77% of input tokens were cache hits, about $0.067 for the session
Uncompressed, assuming perfect prefix caching: about $0.039 (estimate)
With no caching on either side: bili $0.205 vs $0.234

On a 39K-token session, compression lost on cost once caching is counted. Every fold rewrites the prompt prefix, and the proxy's own log said so: "the whole prefix rewrites and the prefix cache restarts from scratch on the next request." On top of that, the ~4K-token tool overhead is charged on every call. The savings come from staying under a window you'd otherwise overflow, and from long sessions where uncompressed history grows without limit. My session was neither, so treat 12.5% as a lower bound for a short session, not as evidence for the "5×" claim.

What I'd tell my team

Where it's useful:

Long-running agent sessions on small-window or cheap models, where hitting the window is the real failure
Teams that want folds they can audit and undo; bili export <session> dumps the folded view and every block summary
Setups that already run a local proxy; median overhead was 10 ms

Where I'd hold off:

Short tasks under ~50K tokens. With default settings (50K growth gate, nudge at 75% of the window) my session would never have folded, and the tool overhead would only have added cost.
Gemini behind an OpenAI-compatible client until the thought_signature drop is fixed
Anything user-facing without reading the license. It's MIT plus a required attribution line on your home page, docs, or About page.
Pinned production environments. Hundreds of commits a week plus self-update on by default means you need ACP_AUTO_UPDATE=0 and a pinned version.

The design idea is good: let the model decide what's stale, keep the originals, and make folds reversible. In my run that held up on the questions that matter most, the constraints and decisions I gave it, at a third of the final context size. The "billions of tokens" claim is a separate test that needs a session thousands of calls long. I'll run that one with a stronger model before I put it in front of real agent traffic.

Originally published on Medium: https://medium.com/@ishank.iandroid/billion-context-hands-on-compressing-coding-agent-history-to-fit-a-small-context-window-7cdbac4bf6b8

Gemini Omni 1.2 Flash Generates and Edits 15-Second Videos in the Gemini API for $0.07 a Second

Ishank Choudhary — Sat, 07 Nov 2026 04:54:28 +0000

On November 5, 2026, Google made Gemini Omni 1.2 Flash (gemini-omni-1.2-flash) generally available across the Gemini API and Google Cloud. Two weeks after shutting down Veo 3.1 Preview in the developer API on October 22, Google is positioning Omni Flash as its default model for generating, editing and extending video with synchronized audio.

Three headline changes stand out:

Native 15-second clips, up from 10 seconds in Omni 1.1 Flash, selectable in one-second steps from 3 to 15 seconds.
A 30% price cut on video output tokens, dropping 720p generation from about $0.10 per second to roughly $0.07 per second ($12.00 per million video output tokens, down from $17.50).
Multi-turn conversational editing and stronger reference adherence, accepting up to 6 reference images and refining clips across turns without starting over.

One date in the release notes matters if you already ship on Omni 1.1: on December 3, 2026, every API call to gemini-omni-1.1-flash will automatically route to gemini-omni-1.2-flash.

I spent Friday evening wiring gemini-omni-1.2-flash up on Linux with Python 3.12, google-genai 3.0.0, raw REST calls and ffprobe. Because my AI Studio key is on the Free Tier, where Omni 1.2 enforces a hard zero-quota wall, I couldn't render paid clips from my own prompts. Everything else on the wire was testable right now, and the first bug hit in 90 milliseconds.

What I tested and where the Free Tier stops

My setup and test boundary:

Environment: Linux x86_64, Python 3.12.14, google-genai 3.0.0, ffprobe and a Free Tier AI Studio GEMINI_API_KEY.
What ran live ($0.00 spend): model discovery (models.list, models.get), input token accounting (countTokens) across text, reference images, uploaded 720p video and WAV audio, Files API upload latency, pre-quota schema validation on /v1beta/interactions, model-routing checks, stateful chaining and container inspection of Google's raw omni_*.mp4 API samples.
What blocked (HTTP 429): all four video generation calls I sent (5s 360p draft, 6s 720p alley tracking shot, 15s 720p timecoded clip via URI delivery, and 6s 2-image reference shot) returned Rate limit exceeded for model gemini-omni-1.2-flash (limit: 0 input tokens per minute on Free Tier) in 0.41 to 0.48 seconds.

Omni 1.2 Flash has zero Free Tier quota: 0 RPM and 0 RPD. Even a 3-second 360p draft (~$0.07) requires Paid Tier 1 billing enabled (8 RPM, 40 RPD, doubled from Omni 1.1). If you want to try 15-second clips manually without API billing, Google Vids includes them at no extra cost on Google and Workspace accounts.

Because I didn't run on a paid project, I haven't measured live video render times or visual drift on my own prompts. Google also published zero benchmarks comparing 1.2 against 1.1, so treat quality claims as untested until you run your own prompts.

Bug #1: Google's Python quickstart 404s out of the box

Gemini Omni models run on Google's stateful Interactions API (/v1beta/interactions), not generateContent, even though GET /v1beta/models/gemini-omni-1.2-flash (0.17s) still lists ["generateContent", "countTokens"] under supportedGenerationMethods.

If you install google-genai 3.0.0 and run the quickstart from Google's Omni docs:

from google import genai

client = genai.Client()
interaction = client.interactions.create(
    model="gemini-omni-1.2-flash",
    input="A marble rolling fast on a chain reaction style track, continuous smooth shot.",
    response_format={"type": "video", "duration": "5s", "resolution": "360p"},
)

It fails in 0.09 to 0.46 seconds with a 404:

NotFoundError: Error code: 404 - {'error': {'message':
"Model 'gemini-omni-1.2-flash' not found. Did you mean 'gemini-omni-1.1-flash'?",
'code': 'not_found'}}

"Did you mean 'gemini-omni-1.1-flash'?" is a red herring: gemini-omni-1.1-flash 404s on that path too (0.20s).

Inspecting client.interactions.sdk_configuration shows why. In google-genai 3.0.0, the Interactions sub-client defaults to Globals(api_version='v1') (POST /v1/interactions), where neither Omni 1.1 nor 1.2 exists. Both live only on v1beta. (Importing google.genai on Python 3.12 also emits two SyntaxWarning escape warnings in types.py.)

Passing api_version="v1beta" fixes the 404:

interaction = client.interactions.create(
    api_version="v1beta",
    model="gemini-omni-1.2-flash",
    input="A marble rolling fast on a chain reaction style track, continuous smooth shot.",
    response_format={"type": "video", "duration": "5s", "resolution": "360p", "delivery": "uri"},
)

Right behind that 404 sits a second SDK catch. When client.interactions.create(api_version="v1beta", ...) hits a 429, even a permanent limit: 0 Free Tier error, its internal retry loop retries silently for over 90 seconds per call before raising RateLimitError. Two calls hung my script for over three minutes before I killed it. Cap retries in CI so bad keys fail fast.

What I measured on the live API

Even without paid video output quota, several parts of the gemini-omni-1.2-flash stack respond on a Free Tier key.

1. Real input token counts (`countTokens`)

Omni 1.2 Flash bills inputs at $1.50 per million tokens across text, images and video. Calling POST /v1beta/models/gemini-omni-1.2-flash:countTokens returned HTTP 200 in 0.18 to 0.52 seconds:

14-word text prompt: 16 TEXT tokens (0.19s), costing $0.000024.
Two reference PNGs (1280x720 mug card and 800x800 robot sheet) plus 11-token prompt: 527 tokens (11 TEXT + 516 IMAGE, or 258 tokens per image, 0.18s), costing $0.00079.
Six reference PNGs (the maximum allowed) plus 16-token prompt: 1,564 tokens (16 TEXT + 1,548 IMAGE, 0.20s), costing $0.00235.
Uploaded 5.0-second 720p 24fps MP4 with AAC audio plus edit prompt: Files API upload took 1.13s and reached ACTIVE in 3.49s (2 polls). countTokens returned 1,488 tokens in 0.30s: 13 TEXT + 1,315 VIDEO (263 video tokens/s) + 160 AUDIO (32 audio tokens/s), or 295 media tokens/s. That is $0.00223 for 5s, or about $0.0066 for a 15s input clip.

Input tokens are rounding error. Six reference images or a 15-second source video cost under two-thirds of a cent; over 99.5% of your bill is video output.

2. Schema checks that fail fast (`HTTP 400`)

POST /v1beta/interactions validates its request schema before checking quota, returning HTTP 400 in 0.18 to 0.21 seconds:

Duration needs "s": "duration": 5 fails; pass "3s" through "15s" (defaults to "10s").
Four resolutions: "480p" fails, listing '360p', '720p', '1080p', '4k'.
Two aspect ratios: "1:1" fails, listing '16:9' (default) and '9:16'.
Five task enums: "video_upscale" fails, listing 'text_to_video', 'image_to_video', 'reference_to_video', 'edit', 'extend'. Google advises omitting task unless prompting picks the wrong mode, and deprecates <FIRST_FRAME> and <IMAGE_REF_0> control tags.

3. Stateful chaining and silent model routing

To test multi-turn state lookup, I created a Turn 1 interaction on /v1beta/interactions with gemini-3-flash-preview (HTTP 200 in 3.22s, returning v1_Chds...), then passed that previous_interaction_id into gemini-omni-1.2-flash on Turn 2. A fake ID failed pre-quota with HTTP 400 in 0.32s, while the real v1_... ID passed state lookup across models and hit the Omni 1.2 quota gate in 0.88s. Avoid setting store=false on Turn 1 if you plan to edit later, since disabling storage breaks previous_interaction_id.

Sending "model": "gemini-omni-flash-preview" (shut down earlier) returned a 429 naming gemini-omni-1.1-flash (0.52s), confirming the gateway already aliases retired Omni IDs server-side ahead of the December 3 1.1 to 1.2 switch.

Output cost math and the 4 MB payload trap

Google bills Omni 1.2 Flash video output at $12.00 per million tokens (text and thinking tokens at $9.00 per million):

360p: 1,931 tokens/s, or $0.023/s ($0.12 for 5s, $0.35 for 15s)
720p (default): 5,792 tokens/s, or $0.070/s ($0.42 for 6s, $0.70 for 10s, $1.04 for 15s)
1080p (upsampled): 8,688 tokens/s, or $0.104/s ($0.52 for 5s, $1.56 for 15s)
4K (upsampled): 17,376 tokens/s, or $0.209/s ($1.04 for 5s, $3.13 for 15s)

Nine 15-second drafts at 360p ($3.13) cost the same as one 15-second render at 4K ($3.13). Draft at 360p or 720p, refine via previous_interaction_id, and render 1080p or 4K last. Batch API is unsupported.

Inspecting four raw omni_*.mp4 samples from Google's API docs (encoder: Google) and three launch-blog demos with ffprobe showed why delivery settings matter:

Container specs: raw outputs are H.264 (yuv420p) at 24 fps with 48 kHz stereo AAC at 2.37 to 4.54 Mbps.
The 4 MB inline limit: inline base64 responses cap out at 4 MB. Three of Google's four raw 720p doc samples (5.42 MB at 10s, 5.65 MB at 20s, 5.78 MB at 20s) exceed 4 MB; only one 10s clip (3.08 MB) fits. Always pass "delivery": "uri" for 10-to-15-second 720p+ clips and poll GET /v1beta/files/{id} until ACTIVE.
Provenance (SynthID + C2PA): alongside invisible SynthID watermarking, all four raw API MP4s carry a C2PA JUMBF manifest (jumdc2pa) signed by Google Media Processing Services under Google C2PA Root CA G3. The FFmpeg-reencoded blog demos (Lavf62.12.102) lost their C2PA box, while SynthID survives in the frames.
One doc slip: omni_misty_mountains_1080p.mp4, embedded right under Google's resolution: "1080p" example, is actually 1280x720 in ffprobe.

Four doc contradictions and what to fix before December 3

Reading Google's Keyword post, Cloud blog, Gemini API docs and Cloud Agent Platform model card side by side surfaced four mismatches:

15 seconds does not apply to every mode: text, single-image, reference-to-video and edits reach 15 seconds (with 11-to-15s clips limited to 1 3s reference video instead of 3), but first-and-last-frame interpolation stays capped at 3 to 10 seconds. Passing "15s" with two keyframes returns an invalid argument error.
Audio references are gated: the DeepMind blog pitches 1-second voice cues, the Cloud blog says audio references need allowlisting, and the Gemini API docs mark audio uploads unsupported—even though countTokens already tokenizes WAV input at 32.5 AUDIO tokens/s.
Extended length (30s vs 40s): Gemini API docs cap multi-turn extension at 30 seconds total, while the Cloud model card implies up to 40 seconds.
Output token ceiling (65,536 vs 260,640): models.get lists outputTokenLimit: 65,536, yet 15s at 720p needs 86,880 tokens and 15s at 4K needs 260,640, matching the Cloud card's 260,640 limit.

Google also notes rough spots on fast sports action, rapid script changes and complex physics, no support for system instructions, temperature, top_p or negative prompt fields, and EEA/UK/Switzerland blocks on editing uploaded video.

If you run gemini-omni-1.1-flash today, four updates get you ahead of the December 3, 2026 automatic reroute:

Pass api_version="v1beta" in google-genai 3.0.0 and cap retries so quota errors fail fast.
Default to "delivery": "uri" on 10-to-15-second clips at 720p+ to avoid the 4 MB inline cap.
Cap UI durations at 3–15s for standard generation and 3–10s for keyframe interpolation and extensions.

4. Strip deprecated `<FIRST_FRAME>` and `<IMAGE_REF_0>` tags and test `gemini-omni-1.2-flash` on staging now, banking the 30% price drop on your own schedule.

Originally published on Medium.

Aura Autopilot Review: An Open-Source AI Agent That Playtests Your Godot, Unity and Unreal Games

Ishank Choudhary — Thu, 05 Nov 2026 12:55:31 +0000

I hid 4 bugs in a Godot game. It played with real keys and found all 4 in 4 minutes, plus one I didn't know about. Then I compared it to a plain LLM.

Originally published on Medium.

Every game dev I know has the same quiet dread: the bug a player finds in the first five minutes that nobody on the team ever hit.

So when Aura open-sourced Autopilot, an Apache-2.0 agent that plays your game to find bugs, I cleared an evening. The pitch is bold: about 43% more accurate and roughly 2x faster than other AI agents at real playtesting tasks.

I wanted to see it with my own eyes. Here's what I ran, what it found, and what it cost.

What Aura Autopilot actually is

Autopilot is not one big model clicking around. It's four parts working together:

An engine plugin for Unreal Engine 5 (5.4â€“5.8), Unity 6 or Godot 4.7+. It reads the game's state and presses real keys and mouse buttons, the way a player would.
An agent service, a Python program on your machine. It talks to the plugin over a local WebSocket, runs model-written code in a sandbox and keeps memory for each game.
A coding agent (OpenCode by default, or Claude Code or Codex). It reads your source and writes a small "harness": the list of moves that make sense for your game, like "dodge left" or "open inventory".
A decision model called Jev, from TypeSafe. During play it picks one of those moves roughly every 0.25 seconds. Fast "reflexes" like dodging run as plain code so they don't wait for the model.

The clever bit is the harness. Aura's team found that giving a model raw keys ("press W", "click") plays badly, while game-specific moves play well. So instead of hand-writing those moves for each game, the coding agent writes them for you.

Then it remembers. Each project gets a memory folder with a copy of your code, notes, harness code that worked, and past reports. Later runs skip the research and start from a harness that already works.

You give it one of four kinds of prompts:

Explore: "Explore the crafting system in Lvl_Main."
Test: "Go through this level and tell me everything that looks wrong."
Verify: "Verify that the tree shakes when you hit it with an axe."
Goal: "Kill the boss while keeping your health above 20%."

Where the 43% and 2x numbers come from

I always read the method before I quote a number. Here it is, from Aura's own blog:

They tested on Easy Survival RPG V5 in Unreal Engine 5.8.
That was 7 tasks: 5 verify tasks and 2 goal tasks, like "craft a stone hatchet and use it to kill a zombie".
Each tool got one cold attempt with empty memory, on the same Windows machine, all using Claude Opus 5.5 on medium effort.
Accuracy: Autopilot 7/7, Claude Code 5/7, and an unnamed "leading Unreal/Unity AI agent" 4/7.
Mean time: Autopilot 262s, Claude Code 437s, competitor 508s.
Cheating (teleporting, spawning items, god mode) counted as a fail. Both of Claude Code's goal runs cheated.

So the "43% more accurate, ~2x faster" headline is against that unnamed competitor: 100% vs 57%, and 262s vs 508s. Against Claude Code it's 100% vs 71%, and about 1.7x faster.

That's a real result, but a small one: seven tasks, one game, one attempt each, and the vendor ran it. I'd treat it as a promising signal, not a law.

My test setup

My setup:

Machine: a Linux box with a real X display. Officially Autopilot ships for Windows and macOS, but the source runs anywhere Python 3.12 does, and the Godot addon is pure GDScript.
Engine: Godot 4.7.2, the Linux build.
Game: "Dodge the Creeps", the classic open-source Godot demo.
The trap: I planted 4 bugs a player would notice:
- The score keeps ticking after you die.
- The Start button never comes back after Game Over, so you're stuck.
- The player can walk 240px off the right edge of the screen and hide there.
- Moving diagonally is about 1.4x faster than moving straight.
I removed the git history so no tool could just read my diff.

Install from source was smooth:

Cloned the repo, then uv venv and uv pip install -e ".[dev]". Under a minute.
Ran the test suite, which uses a simulated game with no keys and no network: 492 passed, 5 skipped in 4m50s.
Copied addons/autopilot into the game and enabled the plugin.
Signed in to Aura, started the agent service with autopilot serve --aura, then launched the game with --playtest=explore --playtest-minutes=5.

What happened when I ran it

I gave it the laziest prompt possible: explore the game for bugs, five minutes of play. No hints about what I'd planted.

Here's how the run went, start to finish:

The game window opened and the plugin connected to the agent service.
The coding agent read my 10 project files and wrote a harness: moves like "start game", "hold right against the edge", "move diagonally", "steer away from the nearest creeps" and "wait on the game-over screen".
Before playing, it ran a quick probe to check that each move actually worked.
Then Jev took over and played one 150-second episode with real key presses. I watched it start a game, dodge, push into the right wall and die around 29 seconds in.
After the death it kept trying to start a new game. It tried 10 times, and nothing happened.
It wrote a report, saved notes to the game's memory and exited cleanly.

Total wall time: about 4 minutes from launch to exit (222 seconds of agent time). It also saved a video recording and a frame-by-frame flipbook of the episode.

What it found

All 4 of 4 planted bugs, each one backed by numbers from the run:

Walking off screen: the player reached x=720 on a 480px-wide screen. It saw this five separate times while holding right.
Score after death: the score went from 16 at death to 146 two minutes later, with no game running.
Stuck after Game Over: the Start button stayed hidden for 120 seconds, and Enter did nothing in 10 tries. It called it a softlock, which is exactly right.
Fast diagonal: it measured about 580 px/s diagonally against 400 px/s straight, so roughly 41% faster.

It also found a fifth issue I hadn't planted: the hidden player moves when you hold a key on the title screen, before any game starts. That's a real bug in the demo. It flagged one more thing from code reading only (D-pad mappings that look swapped) and was honest that it hadn't checked that one in play.

No false alarms. Every claim pointed to a moment in the trace.

What it cost

The run went through my Aura plan, and the report logged about $1.11 of model usage.

One surprise: I'd set the coding agent to OpenCode with Gemini, but in Aura mode the report shows Aura ran it on Claude Opus 5.5 for me. So on Aura you don't need a Claude account to get Claude doing the code-reading part. If you bring your own keys instead, you choose the model and pay that provider plus TypeSafe for Jev.

A few things to know before you try it:

I couldn't test the autopilot connect claude or connect codex path. That hooks Autopilot into your own Claude Code or Codex, and I don't use either plan.
Run sharing is on by default. I turned it off with AUTOPILOT_SHARE=0 and --playtest-share=0. Do the same if your game is private.
Without an Aura account or a TypeSafe key, it stops about 20 seconds in. Jev is the player, and there's no swapping it out.
A small Linux gotcha: if something already uses port 8765, start the service with --port and pass --playtest-url to the game. That cost me two 2-minute timeouts.

The baseline: just ask an LLM to read the code

For comparison, I gave Gemini the full source of the bugged game in one prompt: "play it in your head and list every bug a player would hit."

gemini-3-flash-preview (31.6s, about 6.7K tokens in):

Found all 4 planted bugs, with the right file and line for each.
Listed 5 extra issues. A couple were fair, like the sprite staying upside-down after moving down and then sideways. The rest were guesses or design choices, like the trail particles always emitting.

gemini-3.1-flash-lite (25.6s):

Found 1 of 4: the score that keeps counting.
Missed the stuck Start button, the off-screen hiding spot and the fast diagonal.
Padded the list with timing "races" that don't really happen in play.

Both runs were free on the Gemini free tier.

Autopilot vs a plain LLM

On my little test, here's how they compare:

Bugs found: Autopilot 4/4 plus one real extra. Gemini Flash 4/4 plus five extras, mostly noise. Flash-Lite 1/4.
Time: Autopilot about 4 minutes. Gemini about 30 seconds.
Cost: Autopilot about $1.11. Gemini $0 on the free tier.
Evidence: Autopilot gave me measured numbers, a trace and a video. Gemini gave me line numbers and a confident guess.

So on speed and cost, the plain LLM wins easily, and that's fair: my bugs were all visible in the code. If that's your kind of bug, a good model reading your source is hard to beat.

But the difference I felt was trust. Gemini said I could hide off-screen. Autopilot walked there and showed me x=720. Gemini mixed real bugs with design choices, and I had to sort them myself. Autopilot's list had nothing to throw away, and it found a real bug I didn't know the demo had.

And code-reading has a hard ceiling. It can't catch bugs that only show up in play, like physics, timing, level layout or a door you can clip through. That's the space Autopilot is built for, and my test didn't even reach it.

My result doesn't confirm Aura's "43% more accurate, 2x faster" claim, and it doesn't refute it. That was against other playing agents on 7 Unreal tasks. What I can say is that on a cold run, with a lazy prompt, on Linux, which isn't even officially supported, it caught everything I hid.

Should you try it?

My take:

Yes, if you ship on Unreal, Unity or Godot and have repeatable QA checks ("can I still open the inventory after a new game?"). Verify and goal prompts with pass/fail exit codes fit straight into CI.
Use a plain LLM first for cheap code review. It's faster and free. Bring Autopilot in for what code can't prove.
Expect it to get better per game. The memory design is the real idea here. My next run starts from a harness that already works.
Read the benchmark as a start. Seven tasks is a demo, not a verdict. Run it on your own game.

The repo is at github.com/ramenvr/autopilot. If you try it on your own project, tell me what it finds. I'd love to hear about a bug that only shows up in play.

Originally published on Medium: https://medium.com/@ishank.iandroid/aura-autopilot-review-an-open-source-ai-agent-that-playtests-your-godot-unity-and-unreal-games-081fb9246738

HackerShelf: The Books Hacker News Recommends, Rebuilt from Karpathy's HackerTube with Cursor

Ishank Choudhary — Tue, 03 Nov 2026 04:56:38 +0000

On 1 Nov, Andrej Karpathy shared HackerTube (https://hackertube.org/). He had Claude filter 43M Hacker News comments from 20 years down to 6,454 recommended videos, and he said the site took about 2 hours with Opus 5.5 in Claude Code. He also published the data as a public zip.

As a Lead SWE, my first question was whether I could pull off the same trick with a different topic and a different tool. I picked books, because HN comment threads are basically one long reading list. I picked Cursor, because that's the tool I use every day.

The result is HackerShelf: https://ishankdev.github.io/hackershelf/ (code: https://github.com/IshankDev/hackershelf).

How Karpathy's pipeline works

His DATA.md lays out the method, and I copied it:

Scan every HN comment from 2007 to 2026 in the open-index/hacker-news dataset for YouTube links. The comment needs at least 100 characters of its own text.
Keep a video only if at least 3 different people mentioned it in at least 3 different months.
Claude Sonnet 5.5 labels each mention with a stance, topics, a reason and a quote.
Sonnet writes a card for each video, then puts it into 1 to 3 categories.
Opus 5.5 in Claude Code runs the whole thing.

He did a few steps by hand: a Google Cloud API key for YouTube metadata, a VPS and a domain.

My version: HackerShelf

Same idea, but for books:

Source: HN comments from Jan 2024 to Aug 2026 from open-index/hacker-news, 9,437,609 comments in total
Pre-filter: DuckDB regex for book links (Amazon, Goodreads, Open Library, Bookshop, ISBNs, O'Reilly, Manning, PragProg, No Starch) and phrases like "must read", "worth a read" and "book called". This took about 9 seconds and left 11,519 candidates.
Extraction: gemini-3.5-flash-lite in 288 batches of 40 found 6,076 book mentions.
Resolve: Open Library matched 2,212 works. Using Karpathy's rule (at least 3 people, at least 3 months), 151 books were left.
Cards: one Gemini call per book, so 151 calls
Site: a static site on GitHub Pages

The final shelf has 151 books, 723 mentions and 601 unique recommenders, sorted into 9 categories.

Time: Karpathy's 2 hours vs. mine

Being honest, my total was longer than his.

Planning and data checks: 7:47 to 7:50 PM IST
Cursor build (pipeline code, the full site and deploy setup): about 40 minutes, 7:56 to 8:37 PM IST
Gemini extraction: about 20 minutes, about 8:50 to 9:11 PM IST
Open Library resolve and filter: about 44 minutes, 9:24 to 10:08 PM IST, with no LLM. This was the slowest step.
Cards: 509 seconds, 10:09 to 10:17 PM IST
Site data: under 1 second, at 10:17 PM IST

From start to finish that's about 2.5 hours of wall clock (7:47 to about 10:17 PM IST). That includes my own manual steps: creating the repo, giving Cursor GitHub access and turning on GitHub Pages.

Cursor's part, writing the actual code, took about 40 minutes. Most of the remaining time went to calling APIs and waiting on rate limits.

Cost

Tokens: 1,968,177 in and 623,943 out
What I paid: $0, on Gemini's free tier. I hit 429s and handled them with backoff and retry.
Paid-rate equivalent: at $0.30 in and $2.50 out per 1M tokens, about $0.59 + $1.56 = about $2.15
Cursor usage cost: I don't know this number yet, so I won't guess.

What went wrong

The first Cursor run shipped sample data. The site worked, but it only had 9 placeholder books. The real data had to come from a separate run of the LLM stage.
Too many candidates. The first broad filter matched about 392k comments that might mention a book, which was far too many for a free tier. Tighter link and phrase rules cut that down to 11,519.
The model paraphrased. 470 of the extracted "quotes" weren't actually word for word in the source comment. I didn't trust them, so I blanked them.
Open Library isn't perfect. One top-10 entry resolved as "1984 (adaptation)" by Michael Dean instead of Orwell's novel. Fuzzy title matching is the weak link here.

Does it hold up?

I spot-checked 10 books and 83 mentions against the original HN comments. 83 out of 83 had the right author, and every quote was verbatim in its source comment. For example:

"I recommend reading The Mom Test, it's a great book about how to navigate this early validation stage"
"Robert Nystrom's Crafting Interpreters [0] is a really fantastic book on the topic."

The top 10 books Hacker News recommends (2024 to 2026)

Designing Data-Intensive Applications, Martin Kleppmann: 13 recommenders, 14 mentions
Amusing Ourselves to Death, Neil Postman: 11 recommenders, 11 mentions
The Making of the Atomic Bomb, Richard Rhodes: 11 recommenders, 12 mentions
Permutation City, Greg Egan: 11 recommenders, 11 mentions
The Mom Test, Rob Fitzpatrick: 10 recommenders, 10 mentions
A Philosophy of Software Design, John Ousterhout: 9 recommenders, 11 mentions
Blindsight, Peter Watts: 11 recommenders, 11 mentions
Chip War, Chris Miller: 9 recommenders, 9 mentions
1984: 10 recommenders, 10 mentions (Open Library mismatched this one, see above)
Children of Time, Adrian Tchaikovsky: 11 recommenders, 11 mentions

The ranking uses a score that weighs strong recommendations above passing mentions, which is why a book with 9 recommenders can rank above one with 11.

What I took away

Karpathy's recipe works beyond videos. The "3 people, 3 months" rule is a surprisingly good filter for noise.
An AI coding tool can write the code in under an hour. The time goes into data, rate limits and the manual steps.
Don't trust LLM-extracted quotes until you check them against the source. 470 were quietly wrong.

Credits

All credit for the idea and method goes to Andrej Karpathy and HackerTube. The comment data comes from open-index/hacker-news under ODC-By 1.0. The comments themselves are still their authors' work, and every mention on HackerShelf links back to the original comment.

Originally published on Medium.

NVIDIA Nemotron 3.5 Bolt E4B: Running an Offline Voice Agent With Tool Calling on a CPU

Ishank Choudhary — Wed, 28 Oct 2026 04:46:48 +0000

NVIDIA released Nemotron 3.5 Bolt E4B on October 23. The pitch is a small model tuned for agents: it calls tools, follows instructions, and is small enough for laptops, Jetsons and other edge hardware. The model card's benchmarks are tested on H100s, RTX 5090s and DGX Spark. I wanted to see what happens without any of those.

So I took the official 4-bit GGUF, ran it on a plain 8-vCPU Xeon VM with 16 GB of RAM and no GPU, and built a small voice assistant around it. It listens to a WAV file, picks a tool, calls it, and speaks the answer back. Everything below is from my own runs. I've kept the raw CSVs for every number.

What Bolt E4B actually is

A few facts from the model card and the files themselves, because the name hides some of them:

Architecture: Gemma 4 E4B, post-trained by NVIDIA with reinforcement learning (GRPO via NeMo RL) for tool use, instruction following and multi-turn chat. About 8B total parameters and about 4B "effective" ones.
License: OpenMDW-1.1 on NVIDIA's release. The base Gemma 4 weights are Apache 2.0.
Context: 64K tokens. Text in, text out. There's no audio in the model, so a voice agent still needs separate speech-to-text and text-to-speech.
Weights: the official repo is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB, with a single Q4_K_M GGUF. NVIDIA also ships a BF16 checkpoint for vLLM and a DFlash speculative-decoding drafter aimed at GPUs.
Recommended serving: llama.cpp with --jinja, sampling at temperature 1.0 and top_p 0.95, and a chat-template switch for "thinking" on or off.

The first surprise: the "4GB" file is 6.18 GB on disk. I opened the GGUF and summed the tensors:

Tensor group	Size
Per-layer embedding tables (mostly Q8_0)	3.05 GB
Transformer blocks (Q4_K + Q6_K)	2.40 GB
Token embeddings	0.71 GB

The per-layer embeddings are lookup tables: each token only reads a few rows. That's what makes the "4GB" label fair at run time, as you'll see in the memory section. You still need about 6.2 GB of disk space and download time, though.

The setup

Hardware: 8 vCPU Intel Xeon (AVX-512, VNNI), 16 GB RAM, no swap, no GPU.
Runtime: llama.cpp CPU build b11913, using llama-server as an OpenAI-compatible endpoint.
Speech-to-text: faster-whisper, base.en and small.en, int8 on CPU.
Text-to-speech: Piper with the en_US-lessac-medium voice.

The server command was nothing special:

llama-server -m Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf --alias nemotron-bolt \
  --host 127.0.0.1 --port 8000 -c 8192 -np 1 -t 8 --jinja --metrics

The model loaded in 2.5 to 6 seconds. My first sanity check was 17 Ã— 23, and it answered 391.

Raw speed on 8 CPU cores

llama-bench, 8 threads, 3 repetitions:

Test	Tokens/s
Prompt processing (512 tokens)	194.6 Â± 32.8
Generation (128 tokens)	10.9 Â± 2.8

About 11 tokens per second of generation is slow next to a GPU, but it's enough for an assistant whose replies are one or two sentences. Prefill speed matters more than I expected. My system prompt plus five tool schemas came to about 500 tokens, and the very first request took 6.4 seconds before anything came back. After that, llama.cpp's prompt cache kept the prefix warm, and only the new user message needed processing.

Test 1: tool calling, 20 prompts Ã— 3 runs

I defined five mock tools, the kind a voice assistant would have:

get_weather(city)
calculate(expression)
create_event(title, date, time, duration_min)
set_timer(minutes, label)
control_light(room, state, brightness)

The system prompt gave the date (Tuesday 2026-10-27, IST) and asked for short spoken replies. I wrote 20 prompts: three per tool, plus five where the right move is not to call any tool. Those were a joke, a definition, a "thanks", a trivia question, and "order me a pizza", which no tool can do.

Each response was scored on two things: did it pick the right tool (or correctly pick none), and were the arguments exactly right?

Thinking off (the mode I'd use for voice), three full runs:

Metric	Result
Right tool, or correctly no tool	60 / 60
Exactly correct arguments	57 / 60 (95%)
Median time to first token	0.87â€“1.15 s
Median end-to-end per prompt	2.1â€“3.6 s
Median generated tokens	20

Some results I liked:

"Set a timer for 45 seconds" became minutes: 0.75.
"An hour and a half" became duration_min: 90.
"Dim the bedroom to 30%" became state: "on", brightness: 30. It didn't treat "dim" as "off".
"18% GST on 4999" became 4999 * 0.18.
None of the five no-tool prompts triggered a tool. For the pizza request it said plainly that it couldn't order food, rather than forcing the request into create_event.

The miss was the same in all three runs. "Put sprint planning on my calendar for Friday at 11 for an hour" got the right title, time and duration, but the date was 2026-11-06, a week too late. With today being Tuesday the 27th, "Friday" should be the 30th. Relative-date math is where a 4B model shows its size.

Thinking on, one run of the same 20 prompts:

Metric	Result
Right tool	20 / 20
Exactly correct arguments	20 / 20
Median end-to-end per prompt	13.1 s
Median generated tokens	150

With reasoning on, it got Friday right (2026-10-30). That cost about 350 characters of reasoning and 17.6 seconds for that one prompt. The first token still arrives fast, but it's a reasoning token rather than an answer. Overall that's roughly 5x slower per turn for one extra correct argument out of 20. For a voice assistant on CPU, I'd keep thinking off and handle dates in code: pass the model a resolved calendar ("Friday = 2026-10-30") in the prompt, or let the tool parse "Friday" itself.

One llama.cpp detail if you're measuring this yourself: tool-call output streams in late, once the call has been parsed. So TTFT for a tool call is close to the time the whole call is ready. For plain-text replies, the first token showed up in 0.36 to 0.6 seconds.

Test 2: a full offline voice turn

Next I wired up the whole loop:

Piper speaks the user's request into a WAV file, as a stand-in for a microphone.
faster-whisper transcribes it.
Bolt picks a tool (thinking off).
The mock tool runs.
Bolt turns the tool result into a one-sentence reply.
Piper speaks the reply into another WAV.

Six utterances, timed stage by stage. Medians:

Stage	Whisper base.en	Whisper small.en
Speech-to-text	0.35 s	0.85 s
LLM: choose tool + args	2.39 s	1.85 s
LLM: spoken reply from tool result	1.48 s	1.51 s
Text-to-speech (Piper)	0.08 s	0.07 s
Whole turn	4.08 s	4.58 s

The LLM is nearly all of the turn time. Piper is basically free: it made a 4-second reply in 70 to 100 ms. Whisper is cheap too. That means most of the latency budget goes to two model calls: one to pick the tool and one to phrase the answer.

What the transcripts showed:

base.en heard "Pune" as "Pooner". Bolt still called get_weather(city="Pune") correctly.
base.en turned "Tell me a one-line joke about compilers" into "Tell me what you want to do." Bolt sensibly listed its abilities, but the joke never happened. small.en got the sentence right, and Bolt told the joke.
"Book a dentist appointment tomorrow at four thirty p m" became date: 2026-10-28, time: 16:30, which is correct.
The 18%-of-4,999 answer was "about 900" in one run and "899.82" in another. At temperature 1.0 the wording varies, so if exact numbers matter, put them in your reply template.

My takeaway: small.en is worth its extra half second, because recognition errors cost far more than ASR latency. Since the model handles fuzzy input well, I'd spend the speed budget on the ASR side.

Memory: where the "4GB" name comes from

On Linux, llama.cpp memory-maps the GGUF, so top can show a scary number. Right after load, RSS was about 9.6 GB, because the file pages had just been read. Once the system had been running for a while, the server process looked like this:

Measure	Value
Anonymous (actually allocated) memory, 8K context, 1 slot	3.8 GB
RSS after the page cache settled	4.1 GB
Same server with 16K context and 4 slots (anonymous)	3.7 GB
Voice stack (Whisper small.en + Piper) in a separate process	0.66 GB

So about 4 GB of RAM for the model, plus about 0.7 GB for speech. The large embedding tables mostly stay on disk until a token needs a row. That matches the name, and it's a key reason this model fits small devices. Just plan for the 6.2 GB download and the disk space.

Limitations, honestly

CPU only. I didn't test the DFlash speculative drafter or any GPU, so these numbers are a floor, not a ceiling. NVIDIA's GPU numbers will look very different.
Small test set. 20 prompts Ã— 3 runs is a smoke test, not BFCL. For a fuller picture, NVIDIA reports 67.3 on BFCL-v3 and 74.1 on ToolSandbox for this Q4_K_M build.
Single tool call per turn. I didn't test parallel or chained tool calls.
Synthetic audio. My "microphone" was Piper's own voice, which is clean speech. Real rooms, accents and background noise will make Whisper's job harder.
Dates. With thinking off, it misread "Friday" in every run. Resolve dates in code.
Variance. My first run happened while other jobs were finishing, and its median turn was 3.6 s against 2.1 s in the clean third run. Run your own benchmarks more than once.

Would I use it?

For a local assistant that turns speech into a handful of well-defined actions (timers, lights, calendar, lookups), yes. On a CPU-only VM it picked the right tool 60 out of 60 times, filled the arguments correctly 95% of the time, and completed a whole spoken turn in about four and a half seconds, with no network calls and about 4.7 GB of RAM including speech.

Where I'd be careful: anything that needs arithmetic on dates or times, long open-ended answers at 11 tokens per second, and noisy audio with a tiny Whisper model. Each of those has a simple fix: resolve dates in code, keep replies short, and use small.en.

Next I want to run the same loop on a Jetson-class board with the DFlash drafter and see how far that 4.5 seconds drops.

Originally published on Medium.

Claude Duet vs Grok Bot vs Muse vs Dots: Always-On AI Agents Compared (2026)

Ishank Choudhary — Sun, 25 Oct 2026 04:47:55 +0000

Over six weeks, every big AI lab has shipped the same idea. You don't get a chatbot that waits for you to type. You get an agent with its own computer that works on a schedule, connects to your apps, and asks before it does anything risky.

I'm a Lead Software Engineer, and I already run one of these every day for my writing workflow. So when Anthropic's Claude Duet launch post passed 41,785 likes and 5.16M views in under a day, I wanted to know how the four actually compare. Hype reels weren't going to answer that.

How I tested. I live in India. I have ChatGPT Go, not Pro, and no Claude subscription. Meta's Muse isn't available in my country. So:

Grok Bot (xAI): tested hands-on. I've used it daily for weeks, and I ran the specific tests below for this post.
Claude Duet (Anthropic): docs-based, not tested. I used Anthropic's product page, launch article, help centre and demo.
Muse (Meta): docs-based, not tested. I used Meta's newsroom post and help centre.
Dots (OpenAI): docs-based, not tested. I used OpenAI's launch post and help centre.

I checked every price on the official pages on 22 Oct 2026.

What "always-on" actually means

All four share the same basic design:

A cloud computer of its own. It has a browser and sometimes a shell, so the agent can work while your laptop is shut.
Proactive runs. Scheduled briefs, monitors and "tell me when" triggers.
Connectors. Mail, calendar, docs and code, all through your own accounts.
An approval gate. It drafts, and you press the button for anything that sends, pays or deletes.

They differ on price, where they're available, and how much of that computer you actually get to use.

The comparison

Price (official pages, checked 22 Oct 2026)

Claude Duet: no extra cost on Claude Pro and Max during the research preview. It uses your plan's usage limits, and its background checks count against them. (claude.com, Duet help article)
Grok Bot: from $30/mo (Premium, via X Subscription), with Super at $100 and Ultra at $200. At the 11 Aug 2026 beta launch there was no standalone price. Access came bundled with Cursor Premium Teams ($120/seat), Cursor Ultra ($200) and SuperGrok Heavy ($300), which is where the "$120–300" number in the reels comes from. (x.ai/bot, x.ai news)
Muse: free tier with a weekly limit (card required), Power $20/mo and Maximum $100/mo. (Meta help centre)
Dots: needs ChatGPT Pro, or the Business Premium/Enterprise beta. Free, Go and Plus are excluded. The cheapest Pro tier is $100, which is ₹10,699/mo on the Indian App Store. (openai.com, OpenAI help centre)

Availability

Duet: US-only research preview with a waitlist. Not on Team or Enterprise plans. Works in the iOS/Android apps and on claude.ai.
Grok Bot: generally available on paid plans, and I use it from India.
Muse: US first, plus Canada and Mexico per Meta's coverage. Not available in India. Works in the app, on the web and in WhatsApp.
Dots: launched 29 Sep 2026 with a gradual rollout. Pro is excluded only in the EEA, Switzerland and the UK, so India qualifies on paper.

Its own computer

Duet: a cloud VM with its own browser. You can't yet watch or take over that browser, and it checks in every few hours.
Grok Bot: a full persistent Linux box with a shell, a browser desktop you can watch or take over, and files that stay between sessions. (Tested, results below.)
Muse: the "Muse Secure VM". Logins happen there, and Meta's line is that Muse never sees your passwords.
Dots: its own cloud computer, running GPT-6 Astra.

Approvals

Duet: asks before it sends, pays, signs or deletes. It can make payments you approve (Anthropic's example is a $640 permit fee), but it can't move money between accounts or trade securities, even with approval.
Grok Bot: outbound email and messages become a draft card, and nothing leaves until I press Send. Risky actions on its computer get an automatic safety review. (Tested, see below.)
Muse: payments go through one-time Link by Stripe cards.
Dots: approval before consequential actions, per OpenAI's help centre.

Connectors

Duet: "more than 6,000 connectors and plugins".
Grok Bot: MCP connectors (Gmail, Notion, GitHub, X and more), plus anything its browser can reach.
Muse: Meta apps plus partner integrations (no public count).
Dots: 4,000+ apps.

Memory

Duet: carries over your Claude memory, skills and apps.
Grok Bot: persistent memory and saved skills across chats, plus the files on its box.
Muse: personal context across Meta apps (details are thin in the docs).
Dots: builds on your ChatGPT account (memory specifics not spelled out in the launch docs).

What the reels got wrong

"Duet never sees your passwords." That's Meta's claim about Muse. Anthropic says logins run in your own cloud computer "so sensitive details stay private", which is close but not the same promise.
"Grok Bot launched at $120–300." Those were bundled plans, not a standalone price. It now starts at $30/mo.
"Duet can't move money." Half right. It can't move money between accounts, but it can make a payment you approve.
Everything else I checked about Duet holds up, including Pro/Max inclusion, the morning brief, meeting prep, follow-ups, 6,000+ connectors and the US-only waitlist. The A/B-test proposal only appears in the demo.

Grok Bot, hands-on

Every test was read-only or sandboxed. Nothing was sent, posted or deleted.

Its own computer

Installed a Python CLI (pip install --user cowsay) in 0.8 s, then ran it again in a later session. It persisted.
Started a 12-minute background job and walked away. Result: all 12 one-minute steps ran and the job finished on its own (started 09:52, done 10:11 IST). Honest caveat: the box sat idle for about 8 minutes mid-run, so the wall-clock time stretched from 12 to about 19 minutes. Pass, but don't count on exact timing.
Produced a CSV of these results and a PNG chart on the box, ready to hand back in chat. Pass.

Connectors (read-only)

Gmail: searched "Medium" over the last 7 days and got back about 201 matching emails (25 per page) in about 4 s. Correct.
Notion: searched "Medium" and found my "Medium Topics" page at the top in about 4 s. Correct.
GitHub: listed my 5 most recently pushed repos, with push dates, in about 4 s. Correct.
X: read the Claude Duet launch post's public metrics (41,785 likes, 5,155,927 views, 3,174 reposts, 13,141 bookmarks) in about 3 s. Correct.

That's 4/4 connector reads correct, each in under 5 s.

Routines (timing)

I scheduled a one-shot routine for 10:05 AM IST. It fired at 10:07:58 IST, 178 s late, and wrote its result to a file on its own. It fired, but with about a 3-minute drift.

Approval before send
By design, any outbound email or message becomes a draft card in my chat. It shows the recipient, subject and body, and nothing goes out until I press Send. Declining discards it.
In weeks of daily use, every outgoing email or message has stopped at a draft card that only I can send. When the agent hits a login or a 2FA prompt, it hands me its screen rather than typing a password, and actions it can't undo, like deleting things, wait for my explicit yes. I never saw it send anything on its own.

Verdict by use case

You already pay for Claude Pro/Max and live in the US: join the Duet waitlist. You pay nothing extra, your memory carries over, and it has the biggest connector catalogue. The trade-off is preview limits and no control of its browser.
You want a real computer you can watch, script and take over, from anywhere: Grok Bot. At $30/mo it's the cheapest paid entry that's available outside the US, and the box behaves like a dev machine.
You want something cheap or free inside WhatsApp and Instagram: Muse, if you're in a supported country.
You already pay for ChatGPT Pro: Dots is the natural pick. At ₹10,699/mo it's hard to justify just for the agent.

My honest take: these products have converged fast. Approval gates and cloud computers are standard now. The real differences are where you live, what you already pay for, and how much of the agent's computer you're allowed to touch.

Duet, Muse and Dots sections are docs-based, not tested. Prices checked on official pages 22 Oct 2026.

Originally published on Medium.

ScaleDown CDMs: Fast, Cheap Classify, Extract and Summarize Models Tested

Ishank Choudhary — Sat, 24 Oct 2026 04:48:11 +0000

Last week ScaleDown launched Constrained Decoding Models (CDMs). The launch post got about 1.8M views. The pitch is simple: most production LLM calls don't need prose. They need a label, a field, or a short summary. So instead of a general model writing an answer token by token, each CDM does one job, reads your input once, and returns a typed result. Pricing is $0.05 per million input tokens (OpenRouter lists $0.04 with a 20% promo), and output is free.

The launch post made big claims: "200x faster, 467x cheaper than frontier LLM APIs" and "77.2% on DecisionBench". A thread from Bitroot pushed back the next day. Their point was that the 467x has no stated baseline, DecisionBench has no public source, and nobody should trust a classifier result without a few hundred labelled examples and confidence intervals. That's a fair critique, so I built the test around it.

Here's what I ran, what held up, and what didn't.

The setup

I'm a Lead SWE, so I tested this the way I'd vet any vendor before routing production traffic to it: public labelled data, the same prompts for every engine, and raw results saved to disk.

Classify: 300 CLINC150 test utterances (10 intents plus 50 out-of-scope) and 110 Bitext customer-support tickets across 11 categories.
Extract: 50 SROIE receipts (company, date, address, total) and 30 LinkedIn job posts (company, top of pay range, and work type as an enum).
Summarize: 6 QMSum meeting transcripts (4k to 9k tokens), 2 long ones (31k tokens), and one 369k-token stress doc.
Baseline: Qwen3 1.7B running locally on Ollama (8-core CPU, no GPU), with the same instructions and JSON-schema-constrained output. I didn't have a frontier API key on this box, so for frontier models I did list-price math from the official pricing pages (checked 22 Oct 2026).

Everything goes through one OpenAI-compatible base URL. Classify also has a Jev-compatible Decisions endpoint, which is the one I used most:

import httpx, os
r = httpx.post(
    "https://api.scaledown.xyz/v1/scaledown",
    headers={"x-api-key": os.environ["SCALEDOWN_API_KEY"]},
    json={
        "state": {"text": "how do I reset my pin"},
        "questions": {"intent": {"type": "choice", "criteria": {
            "pin_change": "wants to change or reset their PIN",
            "balance": "asks for an account balance",
            "oos": "none of the above / out of scope",
        }}},
    },
)
print(r.json()["answers"]["intent"])
# {'type': 'choice', 'choice': 'pin_change', 'confidence': 0.979, ...}

Each response includes a usage.cost field. Mine always worked out to exactly $0.05 per million billed input tokens.

Classify: the strongest result

CLINC150 (n=300, 10 intents + out-of-scope)   accuracy   95% CI          OOS recall
ScaleDown Decisions (classify-1)              97.3%      94.8 - 98.6%    98%
ScaleDown /classify (rubrics)                 96.7%      94.0 - 98.2%    100%
Qwen3 1.7B local                              90.7%      86.8 - 93.5%    86%

Bitext support tickets (n=110, 11 categories)
ScaleDown Decisions                           92.7%      86.3 - 96.3%
ScaleDown /classify                           90.9%      84.1 - 95.0%
Qwen3 1.7B local                              62.7%      53.4 - 71.2%

The confidence intervals don't overlap with the small local model on either set. On CLINC150 the out-of-scope handling impressed me most. It correctly rejected 49 of 50 out-of-scope queries, and only 6 in-scope queries were wrongly pushed to "oos". That's the failure that hurts in a real router.

I didn't need any prompt engineering. I wrote one plain-English line per label, and that was it.

Latency, sequential, measured from a cloud box in India: p50 162 ms, p95 397 ms for Decisions, and p50 134 ms for /classify with reasoning: false. The ticket set had a longer tail (p95 about 1.0 s).

Extract: good spans, and the misses are interesting

Extract takes a standard response_format JSON schema. Each property's description is the search criteria, and every value comes back with a confidence score and character offsets into your input.

schema = {"type": "object", "properties": {
    "company":    {"type": "string", "description": "the hiring company's name"},
    "max_salary": {"type": "string", "description": "the top of the stated salary or pay range"},
    "work_type":  {"type": "string", "enum": ["Full-time", "Contract", "Part-time"],
                   "description": "employment type"},
}}
client.chat.completions.create(
    model="extract",
    messages=[{"role": "user", "content": job_post}],
    response_format={"type": "json_schema", "json_schema": {"name": "job", "schema": schema}},
    extra_body={"top_n": 1, "context_chars": 0},
)

                        field acc   valid JSON   spans verbatim   p50 latency
Receipts (50 SROIE)
  ScaleDown Extract     89.5%       100%         100%             1.43 s
  Qwen3 1.7B local      89.5%       100%         94%              3.55 s
Job posts (30)
  ScaleDown Extract     90.0%       100%         n/a              0.60 s
  Qwen3 1.7B local      81.1%       100%         n/a              2.76 s

Per field on receipts, ScaleDown got 100% of totals right, compared with 78% for the local model. Totals are the field you actually care about. It was weaker on company names (80% vs 88%).

Every span ScaleDown returned appeared verbatim in the source text. The citations aren't decoration. You can highlight exactly where a value came from, which is what Bitroot said to check.

I read every miss, and most were judgment calls rather than hallucinations:

The SROIE gold "date" often includes the time (21 MAR 2018 17:59). ScaleDown returned just the date.
It returned KING'S CONFECTIONERY S/B without the registration number the label includes.
On job posts it sometimes returned the parent company instead of the brand (ManpowerGroup for an Experis post, HCA Healthcare for MountainView Hospital).
One real error: on one bakery delivery receipt it picked the buyer's name and address instead of the seller's. That template appeared 4 times in my random sample, so it cost 8 field points on its own.

Valid JSON was 100% on both engines, but that's partly because I constrained the local model's output with a schema. The bigger difference is that ScaleDown hands back offsets and a confidence score with no extra work.

Summarize: fast and faithful, but watch the query focus

For summaries I passed the QMSum query as the system instruction and the transcript as the user message.

QMSum meetings, 4k-9k tokens (n=6)   p50       p95       ROUGE-L   summaries w/ invented facts
ScaleDown Summarize                  0.65 s    0.89 s    0.230     0 of 6
Qwen3 1.7B local (CPU)               24.3 s    34.1 s    0.201     2 of 6

I checked every number and name in each summary against the transcript. ScaleDown's held up, including the details: a 2.39 average rating, the drop from 15 to 12.50 euros, and the 50 million euro profit target were all there in the source. The local model said one rating used "1 to 5" when the meeting used a seven-point scale. It also called an over-budget prototype "reasonably priced".

Two weaknesses showed up:

Query focus. In one meeting it summarised the presentation instead of answering "what did the PM question". On both 31k-token transcripts it summarised an adjacent part of the meeting rather than the exact question. It's a summarizer first and a QA model second.
The 1M-context claim. I sent a 369k-token document made of 12 concatenated meetings and asked for one or two sentences per meeting. It accepted the input and returned in 14.2 s. The summary, though, covered only one of the two distinct meeting series in the document and skipped the other entirely. So it can ingest that much, but don't rely on coverage at that length.

Is it really 200x faster?

Not against anything I could measure, and not according to ScaleDown's own docs.

Against a 1.7B model on my CPU, it was 1.8x faster for short classification, 2.5 to 4.6x for extraction, and about 37x for 6k-token summaries, where CPU prompt processing dominates. A local CPU model isn't a frontier API, though. ScaleDown's own head-to-head benchmark page reports 2.2x to 13.9x faster than each provider's lineup average (2.4x vs OpenAI, 3.8x vs Anthropic). That's a solid speedup, but it's an order of magnitude below the launch post's 200x. I couldn't find what "200x" was measured against.

Is it really 467x cheaper?

The price is real. Every call billed exactly $0.05 per million input tokens, and output was free. The ratio, though, depends entirely on the baseline, and there's a catch: ScaleDown bills more input tokens than your raw prompt. Label criteria, schemas and internal prompts are counted. My classify calls billed about 481 tokens, while the same task as a plain LLM prompt was 172 tokens. Extract calls billed about 1,380 tokens against 571.

Here's the cost per 1,000 calls at list price, using my measured token counts and leaving out reasoning tokens:

                         classify   extract   summarize (6k tok)
ScaleDown CDM            $0.024     $0.069    $0.317
gpt-5-nano               $0.012     $0.057    $0.333     (0.5x - 1.0x)
gpt-5.4-nano             $0.044     $0.204    $1.300     (1.8x - 4.1x)
gpt-5.4-mini             $0.163     $0.752    $4.860     (6.8x - 15x)
Claude Haiku 4.5         $0.210     $0.931    $6.394     (8.7x - 20x)
Claude Sonnet 4.6        $0.631     $2.793    $19.18     (26x - 61x)
gpt-5.5                  $1.089     $5.015    $32.40     (45x - 102x)
Claude Fable 5           $2.102     $9.311    $63.94     (87x - 202x)

So against the cheapest small models, ScaleDown costs about the same, and on short classification gpt-5-nano is actually half the price. Against flagship models it's 45x to 200x cheaper. You only reach 467x if the frontier model also spends a few hundred reasoning tokens per call. For example, gpt-5.5 classifying with 500 reasoning tokens works out to roughly 670x. That's realistic for a reasoning model doing routing, but it isn't stated anywhere in the launch post. ScaleDown's own docs are more modest: 89x cheaper than the OpenAI average and 161x cheaper than the Anthropic average.

The /classify endpoint accepts up to 50 chunks per request and bills the label set once. If you classify at volume, that removes most of the token overhead I measured.

Limitations: I didn't benchmark Compress or concurrent load this round; my free credits ran out after ~1M input tokens, since the 50M free tokens were a launch-window promo. After that I got 402 Insufficient credits, so budget for a top-up before you run a real eval.

What Bitroot got right

The skeptic thread was right on every point I could check:

467x has no stated baseline. I can reproduce it only by assuming an expensive reasoning model.
I still found no public source for DecisionBench's 77.2%.
Price is the easy part to verify, and it checks out.

Their main advice was to build a test set of 300+ labelled examples and report intervals, and that advice is what made this test worth anything. With n=300, the gap between 97% and 91% is real. With n=30 it wouldn't be.

My take

I'd route classification traffic to ScaleDown today. It was the most accurate engine I tested, with clean out-of-scope handling, sub-200 ms p50 latency, and no prompt tuning. Extract is a strong default when you need provenance, because every value comes with an offset you can highlight. Summarize is fast and didn't invent facts in my sample, but test its query focus on your own data, and don't trust its coverage of 300k+ token inputs.

The marketing ratios are a different story. If you already run gpt-5-nano, you won't save much. If you run flagship or reasoning models for routing and extraction, the savings are large and real.

Test it on your own data. A few hundred labelled examples will tell you more than any launch post, mine included.

graphify: Turn Your Codebase into a Knowledge Graph for AI Coding Agents

Ishank Choudhary — Mon, 19 Oct 2026 04:55:12 +0000

graphify has been all over my timeline this week: it went past 126k GitHub stars, got into Y Combinator and showed up on the Vercel Marketplace. The pitch is simple. Type /graphify in Claude Code, Codex or Cursor, and your agent queries a graph of your code instead of reading files one by one.

"Saves tokens" is an easy claim to make and a hard one to check, so I installed it on a plain Linux box and pointed it at a real repo. No cloud trial, no API keys. Here's what happened.

What graphify actually is

graphify is an open-source (Apache-2.0) Python CLI plus an agent skill. It parses your project with tree-sitter and writes three files to graphify-out/:

graph.html, an interactive force-directed graph you open in a browser
GRAPH_REPORT.md, a summary with "god nodes", communities and surprising connections
graph.json, the full graph your agent queries through explain, path and query

Code extraction is deterministic and local. An LLM only gets involved for docs, PDFs, images and video, or for naming communities, and only if you configure a backend.

Setup

My test box runs Debian 13 with Python 3. The PyPI package name has a double y:

python3 -m venv venv
venv/bin/pip install graphifyy      # installed 0.9.92
git clone --depth 1 https://github.com/encode/httpx testbed/httpx

For the testbed I picked httpx: a real, well-known Python HTTP client with about 17,750 lines of Python plus docs and a changelog. It's big enough to be interesting and small enough that I know roughly what the answers should be.

Building the graph: 2 seconds, 0 tokens

You don't need an assistant for the code-only build. The update command does it straight from the CLI:

cd testbed/httpx
graphify update .

Re-extracting code files in . (no LLM needed)...
  AST extraction: 101/101 uncached files (100%) [8 workers]
[graphify watch] Rebuilt: 2233 nodes, 4453 edges, 166 communities

A fresh build took 2.1 seconds. After I edited one file, an incremental rebuild re-parsed only that file and took 1.6 seconds. The report's own footer says Token cost: 0 input · 0 output.

The output is not small: graph.json is 1.7 MB, graph.html is 1.9 MB and GRAPH_REPORT.md is 24 KB. Markdown docs got pulled in as structure too, so CHANGELOG.md and docs/api.md show up as nodes.

What the report got right

The "god nodes" list is honestly a good first page for a new contributor:

1. Response - 170 edges
2. Request - 111 edges
3. Changelog - 79 edges
4. Client - 69 edges
5. AsyncClient - 51 edges
6. BaseClient - 45 edges

That's httpx's real core. 88% of edges were EXTRACTED (read directly from the source) and 12% were INFERRED, with an average confidence of 0.91. I like that every edge carries this tag, because it tells you what to trust.

"Surprising connections" was less useful. Every entry was a changelog or doc heading linked to Client, which is true but not surprising.

Querying: where the token savings come from

This is what your agent actually runs. I counted every output with tiktoken's cl100k encoding:

graphify explain "httpx/_client.py::Client"
graphify path "httpx/_client.py::Client" "DigestAuth"
graphify query "how does a request get sent through the transport"

explain on Client came to 628 tokens. It listed 69 connections with file and line numbers, like --> HTTPTransport [uses] [INFERRED] httpx/_client.py:L731. Reading _client.py itself costs 13,659 tokens.
path returned a 3-hop chain in a single line: Client <-contains- _client.py -imports_from-> _auth.py -contains-> DigestAuth.
query came to 1,728 tokens. It ran a BFS that found 536 nodes but showed only 82, and printed a clear TRUNCATED warning telling me to raise --budget to 14720 to see everything.

So yes, for "where is X and what touches it" questions, a graph lookup is roughly 20x cheaper than dumping the file. What it doesn't do is replace reading the code once the agent has to change a function. The graph points you to the right lines; it doesn't contain them.

Two rough edges

Ambiguous names. graphify explain "Client" refused to answer, because Client matched both the class in httpx/_client.py and a heading in docs/api.md. The error tells you to use the path::symbol form, which works, but docs being indexed alongside code means common names collide a lot.

Community labels with a small local model. Without an API key, the 166 communities are named after a representative file or symbol (test_content.py, AsyncByteStream). graphify can rename them with an LLM, Ollama included:

pip install openai     # the Ollama backend needs it; the first run told me so
graphify label . --backend=ollama --model=qwen3:1.7b

On my box, with a 1.7B model on CPU, this ran for more than 11 minutes without finishing, and I killed it. The code graph doesn't need this step at all, but if you want nice labels, give it a real model or an API key. I saw the same kind of small-model limit when I ran Anatomy's skill on a 1.7B local model.

Is it worth installing?

For code, yes. It's free, local and fast, and explain/path give an agent a cheap map before it starts opening files. I'd add graphify update . to a git hook and let the agent use the graph first.

What I didn't test: the /graphify skill inside a paid assistant, the semantic pass over PDFs and images, and the hosted Graphify Cloud product with its "80% token savings" claim. My 628-vs-13,659 number is one lookup on one repo, not a benchmark of a whole agent session.

Repo: github.com/Graphify-Labs/graphify

PlxNative: A Fast Native Plex Client for LG webOS TVs, Built in Rust

Ishank Choudhary — Sat, 17 Oct 2026 04:51:58 +0000

Most smart-TV apps are web pages. On LG's webOS that usually means JavaScript running inside a browser engine, which then hands its output to a compositor before anything reaches the screen. PlxNative is a Plex client for LG webOS TVs that takes that layer out. Its developer, Gleb Linnik, writes in the README: "No Chromium, no JavaScript, no web view. It just draws."

This is an explainer, built from the project's README, its docs, its source and its release notes. I don't own an LG TV, so nothing here was tested on a real television. What I did run is the project's own Linux desktop simulator against its mock Plex server, and that section near the end shows exactly what that does and doesn't tell you.

What PlxNative is

An unofficial Plex client for LG webOS TVs. It is not made by or affiliated with Plex or LG; the project says so on its site and in TRADEMARKS.md.
Open source under GPL-3.0-or-later, © 2026 Gleb Linnik. The PlxNative name and brand artwork are excluded from the licence.
One person's project. The README calls it "one person's spare time".
Current release: v0.8.0, published on GitHub on 7 October 2026. Nightly builds already report 0.9.0-nightly.
Young and active. The repository was created on 3 July 2026, and when I checked on 16 October it had 969 stars and 95 forks, with commits landing the same morning.
Free. There's no paid tier. You need a Plex account and a Plex Media Server, and the FAQ confirms a free Plex server works.

Why it exists

The origin story is short. The developer's 2019 LG TV still had a great picture, but browsing a Plex library on it "felt slower than it needed to." Rather than tune a web app, he wrote a native one. In his words: "I use it every day to watch things off my server in the next room."

The site makes the argument with a pair of stacks. A typical TV app goes:

The app (a web page) -> JavaScript -> Browser engine -> Compositor -> OpenGL -> GPU -> Screen

PlxNative goes:

The app (native code) -> OpenGL -> GPU -> Screen

The claimed result is 60 fps on that 2019 set. I couldn't measure that, and the project itself says the simulator can't either. More on that below.

The architecture

Mostly Rust, with a thin C shell

The repository is overwhelmingly Rust. A rough line count on the current main gives about 576,000 lines of .rs files against about 7,300 lines of C and C++. That count includes tests and generated data, so treat it as a ratio, not a measure of features.

The structure is spelled out in rust-modules/Cargo.toml. It's a Cargo workspace whose root crate is linked into a small C app as a static library:

# Static library linked into the C app (hybrid migration): each ported module
# exposes the same C ABI its src/*.h declares, so the C code calls it unchanged.
[lib]
name = "plxnative_modules"

The workspace is split into layer crates: base, machine, net, platform, gfx, ui, plex, telemetry, data, session, media, appkit, screens and storage. docs/module-layers.md describes how they depend on one another.

A GPU-drawn interface

The UI goes through SDL2 and OpenGL ES (both are listed in the repo's topics), so there's no DOM and no layout engine. The design language is translucent "glass" panels over your own posters and backdrops, with text sized for reading across a room. The docs folder holds design notes on liquid glass, backdrop blur and a hardware budget for the glass effect, which gives a sense of how much of the work goes into making that look cheap enough for TV silicon.

Video on the TV's own decoder

Playback doesn't decode video in software. The README says the app "hands video to the same silicon the built-in apps use". In the code, that goes through a video-sink trait in the platform crate. On the TV it's implemented by a StarfishSink, which talks to LG's Starfish media APIs, and on the desktop by a HostSink.

What it can direct-play, per the README:

H.264 and HEVC, including 4K and 10-bit
Dolby Vision profiles 5 and 8
E-AC-3 with Atmos
AV1 or VP9 (10-bit VP9 only on webOS 4 sets), on TVs whose codec table lists that decoder

Anything else is transcoded by your Plex server. Before you press Play, the detail page shows whether a title will play directly or be converted on the server. The app also ships its own LGPL build of FFmpeg.

Features

From the README, the site and the 0.8.0 release notes:

Sign in on the TV by scanning a QR code with your phone, then pick a Plex Home profile. PIN-protected profiles are supported.
Browse your own and shared libraries, with Continue Watching, Recently Added, sort, filters (including unwatched-only) and an A–Z rail.
Search across every server you can reach, with results grouped by kind.
A full player with resume, seek and scrub, chapters, audio and subtitle tracks (image subtitles included), Skip Intro, Skip Credits, Up Next with auto-advance, and progress synced back to Plex.
New in 0.8.0: subtitle search from inside the player, ASS/SSA subtitles that keep their styling, browsing Plex collections, Boost dialog and Normalize loudness on servers with Plex Pass, a Settings section for playback, and Spanish and Belarusian translations.
Works without internet after the first sign-in. The app talks to your server directly, so the library keeps working on your home network when the internet is down.
Privacy defaults. Crash reports and usage analytics are two separate opt-in questions, both off until you say yes. The README says no title, search term, subtitle line, server name or address can appear in any report. The app talks to your Plex server, to plex.tv to sign in, and to discover.provider.plex.tv for cast biographies.

0.8.0 also changes one security behaviour. Some home servers only answer over plain HTTP. Earlier versions refused to send your Plex sign-in that way. 0.8.0 will connect, but only after asking "Connect without encryption?", and only for a server on the same network at a private address after every secure route, Plex's relay included, has failed.

How to install it

PlxNative isn't in the LG Content Store yet; the site says it's "in the process of review" with no confirmed date. There are two routes, and neither needs root.

Route 1: LG Developer Mode

The installation guide needs an LG TV on webOS 4.0 or newer, a Plex account with a server, and a Mac, Windows or Linux computer on the same network. In short:

Create or sign in to an account on LG's developer site.
Install the Developer Mode app on the TV, sign in, turn on Dev Mode Status (the TV restarts) and then Key Server.
Note the TV's IP address and passphrase, and connect to it from webOS Dev Manager on your computer.
Install the .ipk directly, or install Homebrew Channel first and get PlxNative from its catalogue.

For a manual download, verify the package first:

sha256sum -c ipk.sha256

The catch is renewal. Developer Mode sessions expire, and if one lapses LG removes the apps installed through it. The guide's advice is to open the Developer Mode app and press EXTEND before the remaining time runs out, with a calendar reminder. An expired session can't be extended.

Route 2: Homebrew Channel

If your TV already runs Homebrew Channel, open it, find PlxNative (a 19.6 MB download, listed as version 0.8.0) and select Install. Updates arrive there too, and the Channel checks the download's checksum before installing.

Nightlies

Adding https://plxnative.com/nightly/repo.json to Homebrew Channel installs PlxNative Nightly as a separate app with its own tile and sign-in. The README is blunt about these builds: "It is not tested on a television."

Limits and current status

The README has a section titled "The honest scope", and it's worth taking at its word:

Movies and TV shows only. No music, no photos, no Live TV or DVR.
No typing in server addresses. Servers come from your Plex account.
Relay and secure-only servers are supported but "haven't been watched end to end."
Not all TVs work. It needs webOS 4.0 or newer. A 2016 set on webOS 3.4.3 has been tested and works.

The known issues list is specific:

Some 2019 sets on LG's k5lp or k3lp chip, installed through Developer Mode, won't play video. The sandbox withholds a device the video path needs; Kodi and Moonlight hit the same wall. Without root there's no fix.
2018 sets on webOS 3.9.3: sign-in and browsing work, but the prebuilt v0.8.0 never starts video (#249). A source build at or after commit 60f99c7 fixes it.
webOS 3.x restarts the stream on every seek instead of jumping in place.
Sets on the k6hp chip show a crash in opt-in reports that the developer can't reproduce without the hardware (#174).

Building the simulator on Linux

Since I couldn't put it on a TV, I built the desktop simulator the docs describe. The box runs Debian 13 on x86_64 with 8 vCPUs and no GPU. The only prerequisites were the SDL2, SDL2_ttf, Mesa, CMake and pkg-config dev packages plus a current stable Rust toolchain:

git clone https://github.com/GLinnik21/plx-native && cd plx-native
make sim-linux SIM_TDIR=rust-modules/target-sim

   Compiling plxnative-modules v0.8.0 (/workspace/plxnative/repo/rust-modules)
    Finished `release` profile [optimized] target(s) in 3m 08s
real    3m57.100s

That gave a 16 MB plxnative-sim binary on the first try. The extra 49 seconds went into fetching and building the libass stack (FreeType, FriBidi, HarfBuzz and friends).

The project's screenshot tool then boots the simulator against tests/mock_pms.py. That's a synthetic Plex server that also stands in for plex.tv, so no account, token or real library is involved. I ran it headless under Xvfb with software OpenGL:

python3 tools/demo_library.py derive
LIBGL_ALWAYS_SOFTWARE=1 xvfb-run -a -s '-screen 0 1920x1080x24' \
  python3 tools/screenshots.py --bin rust-modules/target-sim/release/plxnative-sim \
  --out shots --only home-shelves,library-grid,search,detail,signin

Each scene settled in 2.2 to 4.9 seconds. One hiccup: the home scene also renders the website's link-preview card, and that step failed because the box lacked the file utility, so I captured home-shelves instead.

The sign-in screen is the QR flow, here pointed at the mock:

The Movies grid, with the sort and filter chips and the A–Z rail, browsing the project's demo library of openly licensed films:

And a title page, with the "Direct Play" label next to the runtime:

Be clear about what this proves. The simulator runs the same UI and Plex code as the TV build, and it boots, signs in to a fake server and draws every screen I asked for. But docs/building.md says plainly that it "cannot answer frame rate (different GPU), text rasterization, or anything about LG's decoder and video plane. Those need the set." The Linux build also has no video path at all; the Makefile notes that Play "intentionally reaches the host seam's existing 'no video path' result." So the speed and the hardware decoding, the two things PlxNative exists for, are still unverified by me.

Who it's for

If you run Plex at home and own a fairly recent LG TV, PlxNative is a well-documented alternative to the official app. Expect a project that tells you about its rough edges in the README. You'll need to be comfortable with Developer Mode renewals or Homebrew Channel until the LG Content Store listing lands. If you need music, Live TV or a set from the known-issues list, the README itself says the official app will serve you better.

If you're a developer, the repository is interesting on its own terms: a Rust UI stack drawing straight to GPU on TV-grade hardware, with design and performance notes for almost every decision. The simulator builds in a few minutes on plain Linux if you want to look around.

Site: https://plxnative.com/
Source: https://github.com/GLinnik21/plx-native

I Ran Anatomy's /anatomy Skill on a 1.7B Local Model. The Kit Passed Every Check; the Figure Was 738 Bytes.

Ishank Choudhary — Wed, 14 Oct 2026 04:59:13 +0000

Anatomy went from a new repo on 8 October to 6,632 GitHub stars by Tuesday morning, 13 October (I checked the GitHub API at 10:31 AM IST). It's an MIT-licensed Agent Skill for Claude Code that also runs in OpenCode, Codex and, as of this week, Antigravity. You type /anatomy plus an idea, and the agent is supposed to invent a physical machine whose working is that idea, then build it as an interactive isometric SVG figure out of dozens of small shaded parts.

The demos are lovely: a Raptor engine that comes apart into 11 assemblies, a ripple tank with WebGL caustics, a dial gauge you can slide. I wanted to know two things the demos can't tell me. Does the machinery underneath actually hold up on a plain Linux box? And how much of the result is the skill, and how much is the model driving it?

Short version: the kit is real, fast and honest about itself. The skill, though, is written for a strong model, and a small local one doesn't come close. Here's exactly what I ran.

What I tested and what I didn't

The box is Debian with 8 vCPUs, 16 GB of RAM (about 4 GB free during the test) and no GPU. There was no Claude Code, no Codex and no cloud API key on it. So the agent was OpenCode 1.19.0 pointed at a local qwen3:1.7b through Ollama on CPU.

I tried to pull qwen3:4b as a step up. The download failed on this box's network ("redirect target not allowed"), so 1.7B was the only model I could run.

That shapes everything below. I tested:

The install path, with the skills CLI the README recommends
The kit on its own: building the examples, the test suite, the geometry audit, and the headless-Chrome checks
Three real /anatomy prompts in OpenCode, taken from the README and the skill's own evals

I did not test Claude Code or Codex, or any frontier model. I also skipped the React output, the 3D (beta) mode, the WebGL figures driven by an agent, and the new community-gallery publishing flow. The figures in the README were almost certainly made with a much stronger model than mine, so please read the agent results as "what happens at the bottom end," not as a verdict on the skill.

Install: one Node catch and a different folder

The README's quick start is one line:

npx skills add wheresryan22/anatomy -g -a opencode -y

On the box's Node 20.19.2 it crashed immediately:

npm WARN EBADENGINE   required: { node: '>=22.20.0' }
SyntaxError: The requested module 'fs/promises' does not provide an export named 'glob'

Anatomy itself only needs Node 20.10+, but the skills CLI (1.7.2 here) needs 22.20+. The README does mention this in a collapsed section, but it's easy to miss. On Node 22.20.0 the same command finished in 7.6 seconds. It showed a security summary ("Gen Safe, Socket 0 alerts, Snyk Low Risk") and copied 13 MB.

One surprise: it installed to ~/.agents/skills/anatomy, not ~/.config/opencode/skills/anatomy as the README's install table lists for OpenCode. OpenCode found it anyway, since it also reads the .agents folder:

opencode debug skill
# ... "name": "anatomy", "location": "/home/box/.agents/skills/anatomy/SKILL.md"

The kit on its own: fast, deterministic, and it checks itself

Before involving any model, I ran the parts that don't need one. This is where Anatomy impressed me most.

The examples rebuild from Node scripts with zero npm dependencies:

node examples/dial-indicator/build.mjs
node examples/desk-computer/build.mjs
node examples/arcade-cabinet/build.mjs
git status --short     # empty

All three exited 0, and git status was clean afterwards. The rebuilt pages were byte-identical to the committed ones, which is exactly what CONTRIBUTING promises. The Raptor engine, the heaviest example, took 115 seconds to build, in line with the README's "a few minutes."

The test suite ran 225 tests in about 38 seconds:

# tests 225
# pass 220
# fail 5

All five failures were react-*.test.mjs files that require React, which isn't installed. The README says React is only needed for React output, so this is a setup gap rather than a bug. I'd still like those tests to skip cleanly instead of failing.

The geometry audit is the standout. It doesn't just render the figure. It proves that nothing passes through anything and that every overlap is painted in the right order:

$ node examples/arcade-cabinet/build.mjs --audit
audit · 15 tubes · 243 solids · 349 items · 1 depth states · 1290 overlaps · 1290 order rules · 11.0 s
audit passed

The headless-Chrome checks worked too, once I set CHROME_PATH:

$ node scripts/capture.mjs examples/dial-indicator/dial-indicator-light.html --out dial.png
saved dial.png
clip 395,64 650x485 · readout "block 2 · reads 13.8 px" · scrolls sideways: no
console: clean

$ node scripts/lines.mjs examples/dial-indicator/dial-indicator-light.html
line ends: 0 of 367 ends on 23 lines float
console: clean

The capture took about 4 seconds. That readout ("block 2 · reads 13.8 px") comes from the same model that moves the gauge needle, which is the skill's core promise: every number in the figure comes from the figure's own model.

This is what a finished figure looks like, straight from the repo's smallest example and captured by the skill's own script:

Running /anatomy in OpenCode with a local model

The skill is a 31,600-byte SKILL.md, plus nine reference guides totalling about 300 KB that the agent is meant to open as needed. When you run the command, OpenCode injects SKILL.md into the prompt. Ollama's logs put the first request at roughly 7,700 input tokens before the model had written a word. On CPU, just reading that prompt took 45 to 70 seconds, and generation ran at about 4 to 5.5 tokens per second.

I hit two snags before any figure work began.

1. opencode run sat at "init" for 13 minutes. It was waiting on stdin. Adding < /dev/null fixed it:

opencode run --model ollama/qwen3:1.7b --command anatomy \
  "explain how a token bucket rate limiter works, interactive. Standalone HTML, light theme." < /dev/null

2. The first real run died after five minutes:

ProviderHeaderTimeoutError: Provider response headers timed out after 300000ms

OpenCode waits 300 seconds for response headers by default, and a slow local model with a big prompt can blow past that. Setting headerTimeout: false and timeout: false in the provider's options fixed it. That was the only fix I tried.

Then I ran three prompts from the README and the evals.

Token bucket rate limiter (standalone HTML, light). This one ran for 16 minutes and exited 0, but it wrote no file. The model handed the whole prompt to OpenCode's explore subagent, which came back empty. It then spent 1,560 tokens thinking and never answered. By the end, the context was 15.8k tokens.

CDN edge cache, hit vs miss, TTL 60 s (just the SVG). This one took 8 minutes 23 seconds and two turns, and it did write a file. The file was 738 bytes: three plain circles, two of them labelled "Hit" and "Miss", plus two "60s" labels. It never opened the kit, an example, or a reference. It ran no build or check, even though the skill's workflow ends with "verify it like a critic." It used CSS-style rotateX(-30deg) inside an SVG transform attribute, which SVG doesn't support. Chrome ignored those transforms, so the Hit and Miss circles landed stacked on top of each other in the top-left corner, mostly off the canvas. It also wrote the file into the skill's own install folder (~/.agents/skills/anatomy/examples/cdn-cache.svg), not the project directory. Its closing message: "File written successfully... No further action required."

Database connection pool: 10 connections, a queue when all are busy (standalone HTML, dark). This one took 10 minutes 37 seconds and again wrote into the skill's examples/ folder. The page is a plain 2D SVG on a black background. The "pool of 10 connections" is two circles drawn at the same spot, followed by the comment . The queue is one circle and . No isometric camera, no kit, no readout, no motion, and no check. Its last line: "The file was written successfully."

Here's that CDN file as Chrome renders it, at the same size it was written:

To be fair, this is a 1.7-billion-parameter model on CPU being asked to follow a 7,700-token craft manual. The skill tells the model to collect true numbers, invent an object, write a parts list, build the geometry with the kit, and then audit and screenshot every joint. That's a long agentic loop, and a model this small doesn't hold onto it. What the runs do show is that nothing in the skill makes up for a weak model. The kit only helps if the agent actually calls it.

What worked and what broke

Worked:

The kit: zero dependencies, examples rebuild byte for byte, the Raptor builds in under two minutes
The geometry audit: 1,290 overlaps checked against 1,290 order rules in 11 seconds, and it passed
The Chrome checks: screenshots with live readouts, a line-end check, clean consoles
Skill discovery: OpenCode listed anatomy right after install, even in an unexpected folder

Broke or surprised me:

npx skills add crashes on Node 20, even though Anatomy itself supports Node 20.10+
The CLI installs to ~/.agents/skills/anatomy, which differs from the README's OpenCode row
5 of 225 tests fail without React, instead of skipping
opencode run hangs without < /dev/null, and its 300-second header timeout kills slow local runs. Both are OpenCode issues, not Anatomy ones
With qwen3:1.7b, none of the three runs produced a real figure. One wrote nothing. One wrote a 738-byte SVG with invalid transforms. One wrote a 1,049-byte page whose pool of 10 connections is two circles and a comment saying "add 10 circles". Both files landed in the skill's own install folder, and the agent never ran a single check

Should you use it?

If you write docs or technical posts and you already run Claude Code or Codex with a strong model, yes, it's worth trying. The best part isn't the prompt. It's the kit underneath, which is deterministic and checks itself in ways I haven't seen in other "make me a diagram" tools. Start by building examples/dial-indicator and reading its roughly 300 lines. That tells you more about the output than any screenshot.

If you were hoping a local model could drive it, not at 1.7B. The skill is long, and the workflow is long, and the README says it's "slow on purpose." Budget a long session with a capable model, and expect to read the PNGs it makes.

Not tested here: Claude Code, Codex, any frontier model, the React output, the 3D mode, agent-built WebGL figures, and gallery publishing. If you've run it on a strong model, I'd love to hear how long one figure took you and what it cost in tokens. The new community gallery says it records both.

I Pointed REA at a 100 MB Electron App. Here's What It Found and the Two Preload Files It Missed.

Ishank Choudhary — Mon, 12 Oct 2026 04:58:58 +0000

On Sunday morning, 11 October, REA was number one on GitHub's daily trending page, with 25,793 stars gained in a single day. The repo showed 76,998 stars and 16,440 forks. The pitch is simple: "Reverse engineer anything with agents." You install an MCP server, and your coding agent can take apart native binaries, Electron apps, websites, Android APKs and more, then show you the evidence.

I'm wary of anything that gets that famous that fast, so I skipped the README demos. I installed it on a plain Linux box, pointed it at a real 100 MB Electron app, and then built a tiny app where I know every right answer, to see how well it grades itself.

Short version: the static JavaScript analysis is genuinely good and fast. It also missed something I could see with grep, and what it asks of an agent is heavy. Details below.

What I tested and what I didn't

The box is Debian 13 with 8 vCPUs and no GPU. There was no Claude Code, Codex or cloud API key on it, so no agent drove REA. I called it from its own CLI and from a small script that speaks raw MCP over stdio. That tests the tools, not an agent's judgment about when to use them.

REA lists a lot of targets: native binaries, .NET, APKs, firmware, websites, process capture. Native work needs Hopper, Ghidra or IDA, and I had none of the three on this Linux box (the Hopper features are Mac-only). So I stuck to the one target that needs only Node: JavaScript and Electron apps, analyzed statically.

The real target was PI-Desktop v0.16.1, the Electron app from my previous article. Its app.asar is 100 MB and holds 459 relevant files. I also built a five-file toy Electron app, because I can't grade a tool on an app whose source I haven't read.

Install: fast, with one Node catch

REA wants Node 22.19+, 24.11+ or 26+. The box shipped Node 20, so I grabbed 22.20 from nodejs.org. The box has no xz, so I unpacked the tarball with Python. Then:

npm install rea-agents@latest
# added 144 packages, and audited 145 packages in 3s   (node_modules: 96 MB)
npx rea --version
# 6.4.0

The version matters. npm lists 31 versions since the package appeared in July, and GitHub shows releases 6.1, 6.2 and 6.3 all on 9 October. 6.4.0 landed on 11 October, the day I tested. The README itself says it "changes quickly", so pin a version.

Then rea doctor:

healthy: false
  node     ok    22.20.0
  host     FAIL  unsupported_host   debian 13
  hopper   FAIL  missing_analysis_engine
  ghidra   FAIL  GHIDRA_INSTALL_DIR is not set

The supported hosts are macOS 12+, Ubuntu 24.04+, Fedora 41+, Nobara, Arch, CachyOS and Omarchy, so Debian 13 gets a red mark. It didn't stop anything I ran. Static JavaScript analysis worked fine on the "unsupported" host. I also ran it once under Node 20, where doctor flagged the version, and the analysis still ran. I wouldn't rely on that, but it's good to know the check is stricter than the code.

I didn't want setup touching the box's real config, so I ran it against a throwaway home directory:

HOME=/tmp/fakehome npx rea setup --client claude_code --dry-run   # prints the plan
HOME=/tmp/fakehome npx rea setup --client claude_code --yes
# appliedActions[2]: configured_claude_code, installed_skill

The dry run listed exactly two changes before anything was written. The result was a small mcpServers.rea entry (npx -y rea-agents@6.4.0 mcp) and a skill folder with a SKILL.md and references. Small, clear and reversible. That's how I like setup to behave.

The scan: 52 seconds, 3.1 million AST nodes

time npx rea analyze-javascript-application app.asar --format json > out.json
# real 0m52.2s     out.json: 59.2 MB

REA read 459 files (24.2 MB of text), parsed 421 JavaScript files, walked 3,114,427 AST nodes and had zero parse failures. For a 100 MB bundle, 52 seconds is fine.

Then I checked its Electron summary against the unpacked source, using plain grep:

Here is where it matched:

Browser windows: REA said 5. There are five new BrowserWindow( calls (four in one chunk, one in index.js).
contextBridge APIs: REA said 2. The preload scripts call exposeInMainWorld for piDesktop and pluginBridge.
IPC: REA said 4 main-process handlers, 6 renderer transmissions and 5 renderer listeners. My grep counts for ipcMain.handle/on, ipcRenderer.invoke/send/sendSync and ipcRenderer.on come to the same numbers: 4, 6 and 5.
Sender checks: it reported 4 places where the main process looks at the sender of an IPC message. I didn't verify each of those by hand, so I'm only passing the number along.

And here is what it got wrong:

Preload entrypoints: REA said 0. The app has two preload files, preload/index.cjs and preload/plugin-panel.js. The windows load them through code like join(getModuleDirectory(import.meta.url), "../preload/index.cjs"). My guess is that this helper call is the reason REA couldn't resolve the path statically, but I haven't confirmed that.
Exposed API members: REA said 0. Both bridges obviously expose members, so this zero follows from the missing preload link.

REA is also candid about what it dropped. The result carries 744 limitations. For 365 of them, the text says: "Expanded application relationships for [file] exceed the parent's result budget; complete static facts were retained on the file node." In plain terms, on a big app it keeps counts and per-file facts but throws away some of the connections between them.

That cost me the most useful feature on the real app. I asked it to trace anything touching the channel pi-plugin-panel:

seed_matches: []
coverage.status: no-match
"No graph entity matched the literal seed; this is not evidence that the feature is absent."

The string pi-plugin-panel-drop is right there in the source. The graph just didn't have channel nodes for it. The tool says so, which I respect, but you'd still be stuck.

The toy app: where it shines

So I wrote a five-file Electron app (main.js, preload.js, index.html, renderer.js, package.json) with known contents:

one window with a preload script
one contextBridge API with four members
three literal IPC channels: notes:read, notes:write, open-external
one loop that registers job:sync and job:export from a variable, and a preload call that builds "job:" + topic
two handlers that check the sender, through a small helper function

I analyzed it as written, then again after esbuild --minify, which squashes main.js into one 732-byte line with one-letter variable names.

On both versions, REA's counts were identical, and nearly all of them were right: 1 window, 1 preload, 1 bridge API with 4 members, 3 literal channels, 2 dynamic channel operations, 4 main handlers, 4 renderer transmissions, 3 of them paired with their handlers (the fourth is the dynamic one). Each run took about two seconds.

The tracer is the highlight. I fed the evidence back in with trace-application-feature and a seed of channel: "notes:read". It returned the chain I wanted, on the minified bundle: preload.js invokes the channel notes:read, and the channel is handled by the notes:read handler. It even kept the dynamic channel honest. In the minified code the label is "job:"+e, the actual expression with the renamed variable, rather than a made-up string.

The one miss on the toy: sender validation. I wrote two handlers that call a checkSender(e) helper, which looks at e.senderFrame.url. REA reported 0 sender-validation observations. That's the kind of check you'd want a security-minded tool to notice. A check hidden behind a helper function got past it.

The part nobody puts in the README: what it costs an agent

REA exposes 170 MCP tools. I opened a stdio session, sent initialize and tools/list, and measured the reply:

init: 1.4 s    serverInfo: {"name":"rea","version":"6.4.0"}
tools/list: 170 tools in 0.3 s
payload: 1,954,704 bytes

That's about 1.95 MB of JSON before the agent does anything. By the usual rule of thumb of four characters per token, that's roughly half a million tokens, though that's my estimate and clients may defer or trim tool lists. For scale, I run the local model I use for tests with a 16k context. The project's own docs mention a related problem: GitHub Copilot CLI with gpt-4.1 can block chat with REA's full schema profile, and they suggest a "compact" profile. I tried it with REA_MCP_INPUT_SCHEMA_PROFILE=compact. The list shrank to 1,772,775 bytes, which is 9% smaller. Helpful, not a cure.

This is why I didn't try to drive it with a small local model. The tool list alone wouldn't fit in the context window, so any result would say more about my setup than about REA.

The tool call itself is heavy too. I called analyze_javascript_application on the same app.asar over MCP with detail: "summary":

call: 60.5 s    response: 1,350,868 bytes (text part: 672 KB)

A "summary" that is 672 KB of text is a lot for an agent to read. If you use REA through an agent, I'd expect the agent to spend a lot of its budget on REA's output. I'd test it with your exact client before you commit.

Native binaries: stopped at the door

I did try the native path, to see how it fails. Pointing rea analyze at PI-Desktop's 20 MB Rust host binary:

code: provider_unavailable
candidate_ids: ghidra, hopper, ida

That's a clear message with a clear fix, and I'd have been disappointed by anything vaguer. rea inspect-artifact on the same file still worked without a provider. In 1.7 seconds it returned the format (ELF) and a SHA-256 that matches what sha256sum printed, plus a suggested next step: open_binary, which needs one of the three engines. So you get identification, not understanding, until you install a disassembler. I didn't, so I can't tell you how good the native side is.

What worked and what didn't

What worked:

Install in 3 seconds, and a setup that shows its plan before writing anything
Static analysis of a 100 MB Electron app in about 52 seconds, with zero parse failures
Counts for windows, bridges and IPC that matched my grep on the real app
Identical, correct results on plain and minified code in my toy app
An honest evidence trail: limitations listed, "no match" is not claimed as "absent"

What didn't:

Missed both preload files on the real app, which zeroed out the exposed-API count
Dropped connections on 365 files because of its own result budget, so the trace on a real channel found nothing
Missed a sender check hidden behind a helper function
A 1.95 MB tool list and a 672 KB "summary", which an agent has to carry
doctor goes red on Debian 13 even though the JavaScript path ran fine

What I didn't test: Hopper, Ghidra or IDA, websites and browser capture, APKs, firmware, runtime capture, and above all an agent actually using the tools to "build a similar feature", which is the headline pitch. That claim needs a frontier model and a real session, and I didn't have one.

Should you try it?

If you work with Electron or bundled JavaScript, yes. The pairing of fast static analysis and a graph you can trace is useful, and the toy-app result on minified code impressed me more than any star count. Run it on an app you already understand first, the way I did, so you learn where it's blind before you trust it on one you don't.

If you plan to hand it to an agent, go in with care. Count the tokens in the tool list, try the compact profile, and keep an eye on how much an analysis result takes from the context window.

As for the trending page: stars tell you people are curious, not whether the tool matches a given job. I couldn't check the claim that it jumped from about 9k stars in four days, because GitHub only gives star timestamps to signed-in API calls. What I could check is that on Sunday morning it was real, MIT-licensed, released four times in three days (6.1 to 6.4), and good at most of the job I gave it. It also missed two files that grep found in a second.

I'd like to see this on a Mac with Hopper and a frontier model driving it. If you've run REA that way, tell me what held up.

I Ran PI-Desktop's Plan, Goal and Subagent Modes on a 1.7B Local Model. Here's What Broke.

Ishank Choudhary — Sat, 10 Oct 2026 04:59:13 +0000

Pi, the open-source coding agent engine, reached 1.0 on 1 October. Three days later, on 4 October, PI-Desktop shipped v0.16.1, the first release built on Pi 1.0.0. PI-Desktop is a desktop GUI for that engine: Electron on the front, a Rust "host core" underneath, and the Pi agent packages running in a Node sidecar.

One thing to clear up first. PI-Desktop is a community project by vastsa, not something from the Pi team. It's LGPL-3.0, has about 6.4k GitHub stars, and its README calls the 0.16.x line an "Early Preview". Several other projects use the "pi-desktop" name, so if you go looking, start from the repo and the official docs.

I didn't want to write a feature list. I wanted to know what actually happens when you install it on a plain Linux box and run the three things that set it apart from a terminal agent: Plan mode, Goal mode and subagents.

The honest caveat up front

There was no cloud API key on my test machine. So I ran everything against qwen3:1.7b through Ollama, on CPU only: Debian 13, 8 vCPUs, no GPU. That model managed about 250 tokens/s on prompt processing and about 8 tokens/s on generation.

That is a stress test, not a fair benchmark. A 1.7B model is the weakest setup the app would ever see. Some of what broke is the model's fault, and I'll say which parts. Other things broke regardless of the model, and those are the more useful findings.

The test project was lukeed/clsx v2.1.1: 21 files and 32 passing tests. Small enough for a tiny model to have a chance.

Install: fast and uneventful

Linux gets an AppImage, a .deb and an .rpm (glibc 2.35+, so Ubuntu 22.04+, Debian 12+ or Fedora 36+). I used the AppImage, extracted it instead of mounting it so libfuse2 wasn't needed:

wget https://github.com/vastsa/PI-Desktop/releases/download/v0.16.1/PI-Desktop-0.16.1-linux-x86_64.AppImage
chmod +x PI-Desktop-0.16.1-linux-x86_64.AppImage
./PI-Desktop-0.16.1-linux-x86_64.AppImage --appimage-extract
./squashfs-root/AppRun --no-sandbox

What I measured:

Download: 164.8 MB, and the SHA-512 matched latest-linux.yml.
Extract: about 3 seconds, 414 MB on disk. No missing shared libraries according to ldd.
First launch: the window appeared in 1.56 s cold and 0.54 s warm.
Memory: about 630 MB RSS across the Electron, host-core and sidecar processes, before the agent did anything.
--no-sandbox was needed only because the extracted tree's chrome-sandbox isn't setuid. The installed packages shouldn't need it.

Inside the bundle you get Chromium 150, the Rust pi-desktop-host-core binary, the Pi sidecar, a bundled models.dev catalog of 8,344 models, and two built-in plugins (a browser and a file manager).

The first screen is a calm "What can I help you build?" with a Get started checklist. The composer has three chips that matter for this test: the mode (Agent / Plan / Goal), the permission (Ask every time / accept edits / Auto) and the model.

Hooking up a local model

There's no dedicated Ollama tile in 0.16.1. Local models go in through Settings → Agent → Models → Add provider → Custom endpoint, which takes "any OpenAI- or Anthropic-compatible URL".

Base URL:   http://127.0.0.1:11434/v1
API key:    ollama        (dummy)
API format: OpenAI Chat Completions

Fetch list found the model and the provider showed "Connected · 1 model found". The provider list is long, too: subscription sign-ins (Claude Pro/Max, ChatGPT, GitHub Copilot, SuperGrok/X Premium and others) plus around 30 API-key providers.

A smoke test ("Reply with exactly: PI-Desktop smoke test OK") worked, but it took 1 minute 54 seconds, and it showed me the first real number:

The app's system prompt plus tool schemas came to about 4.1k tokens (Ollama cached 4,098 of 4,717 prompt tokens).
The context ring in the composer read 96% after one turn, because I'd set Ollama to a 16k context and the custom model's context window showed as "— —" in the UI.
The reasoning setting appended /think to my prompt (Qwen's reasoning switch), and the model echoed it back in its answer.

Lesson one: when you add a local model, open the model's Advanced settings and set the Context Window yourself. With 16k of context, the harness takes a quarter of it before you type anything.

Plan mode: it worked, but the plan was fiction

I opened the clsx folder as a project, set the chip to Plan with "Ask every time", and asked it to plan a test for nested arrays and falsy values.

The mechanics were solid:

The plan was saved as a real file, .pi/plan/add-nested-array-falsy-values-tests-for-clsx-20261007-1108.md (315 bytes).
A plan card appeared with Open plan, Reject and Approve (Ask), and the File Manager pane opened showing the markdown.
The chat got an automatic, sensible title.

The content was not. The plan said to add test files in /work/clsx/tests, a path that doesn't exist. The run log showed "3 issues · 4 tools": a search failed, a read failed, the first SubmitPlan failed and the second one succeeded. The app reported "Processed for 5m 19s", but wall-clock time from sending the prompt to seeing the card was about 17 minutes. The context meter sat at 95%.

That's mostly the model. A 1.7B model with a quarter of its context taken by instructions invents paths. But the workflow around it (a separate artifact, an explicit approval gate) is exactly what I'd want to put in front of a stronger model.

Goal mode: the permission chip changed on its own

Goal mode is meant to write a .pi/goal/*.md file with an outcome and acceptance criteria, wait for approval, then switch to Agent and work until it can report which criteria it verified.

I clicked the mode chip twice to get to Goal. Then I noticed the permission chip next to it had silently switched from "Ask every time" to "Auto". I hadn't touched it. For a mode whose whole job is to run autonomously toward a target, that's the one setting I'd expect it never to loosen on its own.

The run didn't go well:

The first model call took 4 min 26 s.
No goal file and no acceptance criteria were ever written.
The model went looking in /work/tests again.

The safety net did work: even under Auto, a Grep on /work/tests stopped with "Accesses a path outside the session workspace", rated LOW RISK. I denied it. The model then tried to Edit /work/tests/new-test.js, which failed. I stopped it at about 12 minutes with nothing verified, and the chat had been auto-renamed in Chinese ("新增嵌套假值测试用例", roughly "add nested falsy value test cases").

The failed run is the model's fault. The permission flip and the Chinese title happen in the app, and they're worth reporting to the maintainers. Fairly, the docs do say Plan and Goal are "contract modes, not strict read-only security profiles". I'd still check that chip every time you switch modes.

Subagents: the one clean win

PI-Desktop ships five built-in subagents, each with a defined tool set:

Explorer: Read, Glob, Grep, Bash
Code reviewer: Read, Glob, Grep (read-only)
Test runner: Read, Glob, Grep, Bash
Fixer: adds Edit and Write
UI designer: adds BrowserPreview, Edit and Write

You can add your own as markdown files in ~/.agents/subagents/ (up to 16). Delegation uses a Task tool that runs in the background and returns a delegation ID, with TaskWait, TaskList and TaskStop alongside it, and up to 10 at once per session. Subagents only work in Agent mode.

In Agent mode I asked: "Use the test-runner subagent to run npm test and report the result."

This went well:

The UI drew a small delegation graph: Main agent → test-runner, "Coordinating 1 delegated task".
The permission prompt said clearly that it was "Asked by the test-runner subagent", showed the exact command (npm test) and labelled it HIGH RISK. Running a shell command is the right thing to flag.
After Allow once, the subagent finished in 2 min 44 s (3 steps). The whole turn took 5 min 55 s.
The result was exit 0, 32/32 passed, which matches what I got running npm test in a shell myself.

The subagent's own transcript opened in a side pane marked read-only ("Subagents are driven by the main agent"). And the chat was renamed again, to "npm测试成功" ("npm test succeeded").

The lesson: give a small model a narrow, well-defined job and the harness carries it. Ask it to plan open-ended work and it falls apart.

Two findings that have nothing to do with the model

1. "OS keychain" isn't quite true yet. The landing page and the README say credentials live in the OS keychain. I ran gnome-keyring with Secret Service available, saved the Ollama provider, and zero items went into the keyring. Instead, two files appeared:

~/.pi-desktop/secrets/<sha>.bin    48 B   mode 644
~/.pi-desktop/secrets/.machine-key 32 B   mode 600

The project's own storage spec explains it: the shipped backend is an AES-256-GCM file store keyed by a machine key that "host-core generates once and keeps beside them", and the OS keychain backend is one "that neither host-core nor Electron main implements today, so a same-user process that can read the data directory can also decrypt the secrets." The spec is honest about this. The marketing isn't. If you put real cloud keys in it, treat the ~/.pi-desktop folder as sensitive.

2. The window only redraws when the mouse moves. On my Xvfb/xfwm4 desktop, timers and streaming output froze until I moved the cursor. That could be specific to my headless setup, but it made the long local runs feel even longer.

The rest, briefly

MCP Market: Settings → Agent → MCP has a Market tab that merges a built-in catalog with the official MCP registry. It listed Memory, Sequential Thinking, Filesystem, Playwright, Context7, Fetch, Git, Time, Serena and others, most marked Verified, each with its npx/uvx command. Some descriptions showed up in Chinese. As of v0.16.1, tools from user-added MCP servers need approval.
Session import: it can import sessions from Claude Code, Codex, OpenCode and Pi.
Scheduled tasks: hourly, daily or weekly, but they only run while the app is open.
No telemetry and no required account, according to the README.

What a bigger cloud model would likely change

I didn't test this, so treat it as an expectation, not a result. With a capable cloud model and a large context window, I'd expect:

The 4.1k-token system prompt to stop mattering. At 128k+ context it's a rounding error, not 25%.
Plan mode to produce real paths, since a stronger model would actually read the repo before planning.
Goal mode to write its acceptance criteria and run through them in minutes, not stall after 12.
Total time per turn to fall from minutes to seconds.

What I don't expect to change: the permission chip flipping to Auto, the keychain wording, the Chinese auto-titles and the redraw bug. Those are in the app.

Verdict

PI-Desktop v0.16.1 installs in seconds, starts in under two, and its Plan, Goal and subagent workflows are thought through: separate artifacts, approval gates, clear risk labels, a visible delegation graph. Subagents did their job even on a 1.7B model on CPU.

It's also clearly an early preview. Check the permission chip when you switch to Goal, don't trust the "OS keychain" line yet, and set your local model's context window by hand.

If you run it with a frontier model, I'd like to know whether Goal mode holds up. That's the test I want to see next.

Originally published on Medium.

DEV Community: Ishank Choudhary

billion-context Hands-On: Compressing Coding-Agent History to Fit a Small Context Window

What billion-context is

Setup

The first bug: Gemini's OpenAI-compatible endpoint

The test session

The numbers

Did the agent still remember?

What it cost

What I'd tell my team

Gemini Omni 1.2 Flash Generates and Edits 15-Second Videos in the Gemini API for $0.07 a Second

What I tested and where the Free Tier stops

Bug #1: Google's Python quickstart 404s out of the box

What I measured on the live API

1. Real input token counts (countTokens)

2. Schema checks that fail fast (HTTP 400)

3. Stateful chaining and silent model routing

Output cost math and the 4 MB payload trap

Four doc contradictions and what to fix before December 3

4. Strip deprecated <FIRST_FRAME> and <IMAGE_REF_0> tags and test gemini-omni-1.2-flash on staging now, banking the 30% price drop on your own schedule.

Aura Autopilot Review: An Open-Source AI Agent That Playtests Your Godot, Unity and Unreal Games

What Aura Autopilot actually is

Where the 43% and 2x numbers come from

My test setup

What happened when I ran it

What it found

What it cost

The baseline: just ask an LLM to read the code

Autopilot vs a plain LLM

Should you try it?

HackerShelf: The Books Hacker News Recommends, Rebuilt from Karpathy's HackerTube with Cursor

How Karpathy's pipeline works

My version: HackerShelf

Time: Karpathy's 2 hours vs. mine

Cost

What went wrong

Does it hold up?

The top 10 books Hacker News recommends (2024 to 2026)

What I took away

Credits

NVIDIA Nemotron 3.5 Bolt E4B: Running an Offline Voice Agent With Tool Calling on a CPU

What Bolt E4B actually is

The setup

Raw speed on 8 CPU cores

Test 1: tool calling, 20 prompts Ã— 3 runs

Test 2: a full offline voice turn

Memory: where the "4GB" name comes from

Limitations, honestly

Would I use it?

Claude Duet vs Grok Bot vs Muse vs Dots: Always-On AI Agents Compared (2026)

What "always-on" actually means

The comparison

Price (official pages, checked 22 Oct 2026)

Availability

Its own computer

Approvals

Connectors

Memory

What the reels got wrong

Grok Bot, hands-on

Verdict by use case

ScaleDown CDMs: Fast, Cheap Classify, Extract and Summarize Models Tested

The setup

Classify: the strongest result

Extract: good spans, and the misses are interesting

Summarize: fast and faithful, but watch the query focus

Is it really 200x faster?

Is it really 467x cheaper?

What Bitroot got right

My take

graphify: Turn Your Codebase into a Knowledge Graph for AI Coding Agents

What graphify actually is

Setup

Building the graph: 2 seconds, 0 tokens

What the report got right

Querying: where the token savings come from

Two rough edges

Is it worth installing?

PlxNative: A Fast Native Plex Client for LG webOS TVs, Built in Rust

What PlxNative is

1. Real input token counts (`countTokens`)

2. Schema checks that fail fast (`HTTP 400`)

4. Strip deprecated `<FIRST_FRAME>` and `<IMAGE_REF_0>` tags and test `gemini-omni-1.2-flash` on staging now, banking the 30% price drop on your own schedule.