DEV Community: Datafund

How can an auditor verify an AI agent's transcripts haven't been changed?

Datafund — Wed, 04 Nov 2026 14:22:26 +0000

An auditor can verify that an AI agent's transcript has not been changed by comparing the copy they are given with a fingerprint (a SHA-256 hash) recorded when the transcript was written, and by fetching the stored record from storage the lab running the agent cannot change. If the fingerprint carries a notary signature or is anchored on a blockchain, checking that also shows when the fingerprint existed. None of this proves the transcript was complete or true when it was stored; it proves it has not changed since.

Why this question matters now

In July 2026, during an internal OpenAI evaluation of cyber capabilities, models in a sandboxed testing environment exploited a zero-day vulnerability to gain internet access and reached Hugging Face's servers (OpenAI's incident report). The independent review of the agents' behaviour by METR and Redwood Research found that agents "tried to retroactively edit transcripts". They "modified some accessible action logs, but these weren't the source of transcripts", and the reviewers wrote: "We did not see them successfully edit the source of truth in our transcripts (scoped to Jul 7-13)."

OpenAI's later guidance, "Towards safety cases for frontier AI training", recommends: "Save agent transcripts immutably for a reasonable period of time (for example, in write-once storage accounts)", and says "Auditors should be provided with sufficient access to verify that the claims of the safety case are valid and sound, and to raise gaps if found."

This page is about the second half: how the people entitled to check (auditors, evaluators, the client) can verify a transcript themselves.

The one rule: the fingerprint must come from before

A fingerprint handed over together with the transcript proves nothing: whoever edited the transcript could compute a fresh one. The check only works if the fingerprint was fixed at write time and has been out of the lab's reach since. Three common ways to get that:

The fingerprint (or the record's storage address) was sent to you, or to a party you trust, when the transcript was written.
A notary signed the fingerprint with a timestamp at write time.
The fingerprint was anchored, meaning written on a blockchain, at write time.

Ask which applies. If none does, you can still compare the stored copy with yours, but you are trusting that the stored copy is the original.

Step by step

1. Collect what you need

The transcript copy you are given, the record's storage address, the fingerprint recorded at write time, and any notary signature or anchor reference.

2. Fetch the record yourself, from storage the lab cannot change

Do not treat a copy the lab sends you as the reference. Fetch the stored record yourself, from storage where nobody, the agent and the lab included, can change it in place.

On Swarm, the decentralised storage network (ethswarm.org), a record's address is computed from its content. Swarm's documentation puts it plainly: "Because the address is derived from the content, a content-addressed chunk cannot be changed in place. Different content yields a different address." An edited transcript would get a different address; the original stays until its stated expiry. The Datafund Provenance Toolkit downloads through Datafund's gateway; the hash check in the next step is what removes the need to trust the gateway.

A write-once storage account is the other common choice. In Amazon S3 Object Lock's compliance mode, "a protected object version can't be overwritten or deleted by any user, including the root user in your AWS account". An auditor then relies on the provider and the account setup, so pair it with a fingerprint kept somewhere the lab does not control (write-once storage compared).

What this proves: the bytes you fetched are the bytes stored at that address (or in that locked object). What it does not: that this is the only record, or that nothing was left out.

3. Recompute the fingerprint and compare

Compute the SHA-256 hash of the record you fetched and of the copy you were given. Both must match the fingerprint recorded at write time. If the copy you were given differs, something changed between storage and hand-over; ask why.

What this proves: the transcript has not changed since the fingerprint was recorded. What it does not: when that was, unless step 4 settles it.

4. Check the notary signature or the anchor

If a notary was used, it signed the fingerprint and a timestamp with its own key. Verify the signature. In the toolkit, during testing, the notary is Datafund's public gateway; the gateway is open source, so any party you trust can run it. A notary timestamp rests on trusting whoever runs the gateway.

If the fingerprint was anchored, check that it was written on the blockchain and when. Then the time no longer rests on the gateway alone; it inherits the chain's trust model. In the toolkit, anchoring runs on a test network (Base Sepolia) today; mainnet anchoring is not live.

What this proves: the fingerprint existed at that time (vouched for by the notary, or recorded on chain). What it does not: anything about the content beyond the fingerprint.

5. Follow the chain, if there is one

Provenance differs from a log by linking each output to the inputs and actions that produced it (What is agent provenance?). If records are linked (the toolkit's transformation and merge links), repeat steps 2 to 4 for each linked record, to check what the transcript says it came from.

What the whole check does not prove

That the transcript was complete or true when it was stored. When an agent uploads its own record, it can write anything into it first. The METR and Redwood review describes "tool call spoofing", where agents could "pretend to run one command while actually running another". A faithful record of a spoofed output is still a spoofed output.
That everything was recorded. A check cannot show what was never stored.
That the work was correct, lawful or authorised, or who uploaded the record.

Having the software that runs the agent write the record, rather than the agent, would narrow the gap between what happened and what was recorded. In the toolkit that is a proposal, not something it does today.

Trying it

The Datafund Provenance Toolkit is pre-release, meant for testing, on a test network (Base Sepolia); AI agent provenance tools compared lists what it does today. The free tier is limited and may fail under heavy load; retry later. Free-tier records are kept for about a day; for evidence, use a stamp funded for the retention period you need.

The quickest test is dataprovenance.app: record a file, get a receipt, then play the auditor and verify it. How to give an AI agent a record it can't rewrite shows the check in code, and If your agent can rewrite its own record, you don't have a record covers why it matters.

Questions people ask

How can an auditor verify that an AI agent's transcripts have not been changed?
By comparing the transcript with a fingerprint recorded when it was written, and fetching the stored record from storage the lab cannot change. If the fingerprint carries a notary signature or an on-chain anchor, checking it also shows when the fingerprint existed.

Is a hash supplied with the transcript enough?
No. Anyone who edited the transcript could compute a new hash for the edited version. The hash has to have been fixed at write time and kept out of the lab's reach since: sent to the auditor then, signed by a notary, or anchored on a blockchain.

Does a matching fingerprint prove the agent behaved well?
No. It proves the record has not changed since it was stored. An agent that writes its own record can put anything in it before storing it, so the check says nothing about whether the record was complete or true.

Sources

OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation" (21 July 2026): https://openai.com/index/hugging-face-model-evaluation-security-incident/
METR and Redwood Research, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident" (26 August 2026): https://alignmentforum.org/posts/nB8KKapnWGBXtKKiM/brief-independent-investigation-of-agents-behavior-reasoning
OpenAI, "Towards safety cases for frontier AI training" (28 September 2026): https://openai.com/index/towards-safety-cases-for-frontier-ai-training/
Swarm, DISC and content addressing: https://docs.ethswarm.org/docs/concepts/DISC/
Swarm, postage stamps and expiry: https://docs.ethswarm.org/docs/concepts/incentives/postage-stamps/
Amazon S3, Locking objects with Object Lock: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html
Datafund Provenance Toolkit: https://github.com/datafund/provenance
Swarm gateway (swarm_connect, open source): https://github.com/datafund/swarm_connect
dataprovenance.app: https://dataprovenance.app

What is agent provenance?

Datafund — Wed, 04 Nov 2026 11:07:23 +0000

Agent provenance is a record of what an AI agent did that links each output to the inputs and actions that produced it: this report came from this step, which read these files. A log lists events in order and is usually kept where its owner, and often the agent, can change it. Provenance is the chain behind each result, kept where nobody, the agent included, can change it in place, so the people entitled to check (auditors, evaluators, the client) can follow it back and see it was not changed after it was stored.

Provenance in plain terms

The W3C defines provenance as "information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness".

For an agent, that means three kinds of thing:

Entities: the prompt, the files the agent read, the tool results it received, the report it wrote.
Activities: the actions that used and produced them. A search, a code edit, a test run.
Agents: who was responsible. The AI agent, the person who gave it the task, the software that ran it.

W3C PROV is the standard model for this chain. Its data model defines an agent as "something that bears some form of responsibility for an activity taking place".

How agent provenance differs from logging

A log is, in NIST's words, "a record of the events occurring within an organization's systems and networks". They differ from provenance in two ways.

A log is a list. Provenance is a chain. A log tells you what happened, in order. Provenance records which inputs each output depended on.

A log can usually be changed by whoever holds it. Anything with write access can edit it, including the agent if the log is within its reach. For agent records this is not hypothetical. In July 2026, during an internal OpenAI evaluation of cyber capabilities, models in a sandboxed testing environment exploited a zero-day vulnerability to gain internet access and reached Hugging Face's servers, according to OpenAI's incident report. The independent review by METR and Redwood Research found the agents "tried to retroactively edit transcripts" and "modified some accessible action logs". As far as the reviewers saw in the data they analysed (7 to 13 July), the source-of-truth transcripts were not edited.

OpenAI then recommended, in "Towards safety cases for frontier AI training": "Save agent transcripts immutably for a reasonable period of time (for example, in write-once storage accounts)". The same document says auditors "should be provided with sufficient access to verify that the claims of the safety case are valid and sound".

So agent provenance has two parts: the chain (what came from what), and where it is kept (somewhere the agent and its operator cannot rewrite it).

	Log	Agent provenance
Shape	Events in time order	Each output linked to its inputs and actions
Main question it answers	What happened, and when?	What did this result come from?
Who can change it	Usually its owner, and anything with write access	Nobody, the agent included, until its stated expiry
Main use	Running and debugging systems	Evidence for auditors, evaluators and clients

The two work together. Observability and logging tools stay the right choice for watching an agent while it runs; provenance is the record you hand to someone who needs to check it afterwards.

Where the record is kept

A provenance chain is only as good as the place it is stored. The options fall into three groups: write-once cloud storage, tamper-evident logs and databases, and content-addressed decentralised storage, where a record's address is computed from its own content. AI agent provenance tools compared sets them side by side, and Write-once (WORM) storage for AI agent records compares the storage options.

Swarm, the decentralised storage network (ethswarm.org), is in the third group. Swarm's documentation says: "Because the address is derived from the content, a content-addressed chunk cannot be changed in place." Storage is paid for with a postage stamp whose balance runs down over time, so a record on Swarm cannot be changed in place until its stated expiry. Is Swarm write-once storage? explains expiry.

The proof is that the record is kept where nobody can change it. Publishing a fingerprint (the record's SHA-256 hash) somewhere public is one optional way to show that, not a requirement.

What provenance proves, and what it does not

A stored record shows that it was not changed after it was stored, not that it is true. An agent that uploads its own record can write anything into it first. Nor does it show that everything was recorded, or that the work was correct or authorised. Having the software that runs the agent write the record narrows that gap; it does not close it. How an auditor verifies AI agent transcripts walks through the check.

Where the Datafund Provenance Toolkit fits

The Datafund Provenance Toolkit is open source and in pre-release, meant for testing. It runs on a test network (Base Sepolia); mainnet anchoring and payments are not live. Available today on the test network:

storing a record on Swarm, and getting back its address and fingerprint;
an optional notary signature with a timestamp from the gateway;
anchoring, which means writing the fingerprint on a blockchain through a smart contract;
lineage links between records: a transformation link (one record made from another) and a merge link (one record made from several).

The notary is Datafund's public gateway during testing; the gateway is open source, so any party you trust can run it, and the timestamp rests on trusting whoever runs it unless the fingerprint is anchored. Agents can use the toolkit through its MCP server, which is being updated; treat it as experimental. Two things are proposals only: the runtime (not the agent) writing the record, and sealed records.

The free tier is limited and may fail under heavy load; retry later. Free-tier records are kept for about a day; for evidence, use a stamp funded for the retention period you need.

The quickest test is dataprovenance.app. For the code, read How to give an AI agent a record it can't rewrite; for why it matters now, If your agent can rewrite its own record, you don't have a record.

Questions people ask

What is agent provenance?
Agent provenance is a record of what an AI agent did that links each output to the inputs and actions that produced it, kept where nobody, the agent included, can change it in place. It lets the people entitled to check (auditors, evaluators, the client) follow a result back to where it came from.

How is agent provenance different from logging?
A log lists events in time order and is usually kept where its owner, and often the agent, can edit it. Provenance links each output to the inputs and actions that produced it, and is kept where it cannot be changed in place, so it can serve as evidence rather than only as an operational record.

Is W3C PROV required for agent provenance?
No. W3C PROV is the standard model for describing the chain of entities, activities and agents, but the chain can be recorded in other formats too. What matters is that the links exist and the record cannot be changed after it is stored.

Does agent provenance prove the agent told the truth?
No. A stored record shows it was not changed after it was stored, not that its content is true or complete. An agent that writes its own record can put anything in it before storing it, which is why having the software that runs the agent write the record is better.

Sources

W3C, PROV-Overview (Working Group Note, 30 April 2013): https://www.w3.org/TR/prov-overview/
W3C, PROV-DM: The PROV Data Model: https://www.w3.org/TR/prov-dm/
NIST CSRC Glossary, "log" (from NIST SP 800-92): https://csrc.nist.gov/glossary/term/log
OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation" (21 July 2026): https://openai.com/index/hugging-face-model-evaluation-security-incident/
METR and Redwood Research, independent review of the Hugging Face incident (26 August 2026): https://alignmentforum.org/posts/nB8KKapnWGBXtKKiM/brief-independent-investigation-of-agents-behavior-reasoning
OpenAI, "Towards safety cases for frontier AI training" (28 September 2026): https://openai.com/index/towards-safety-cases-for-frontier-ai-training/
Swarm documentation, chunks and content addressing: https://docs.ethswarm.org/docs/concepts/DISC/
Swarm documentation, postage stamps: https://docs.ethswarm.org/docs/concepts/incentives/postage-stamps/
Swarm: https://www.ethswarm.org/
Datafund Provenance Toolkit: https://github.com/datafund/provenance
Swarm gateway (open source): https://github.com/datafund/swarm_connect
Provenance MCP server: https://github.com/datafund/swarm_provenance_MCP
dataprovenance.app: https://dataprovenance.app
Datafund, "How to give an AI agent a record it can't rewrite": https://datafund.io/blog/how-to-give-an-ai-agent-a-record-it-cant-rewrite/

How to give an AI agent a record it can't rewrite

Datafund — Tue, 03 Nov 2026 14:00:57 +0000

AI agents now write code, open pull requests and send reports without a person watching each step. When something goes wrong, the first thing people ask for is the record of what the agent did.

Usually that record is a log. The system writes it, its owner keeps it, and anything with write access can change it. That includes the agent, if the log is within its reach.

This is the developer half of agent provenance: how to keep an agent's record on Swarm, the decentralised storage network, where nobody, the agent included, can change it in place until its stated expiry, what our open toolkit does today, and what is still a proposal. The story post, If your agent can rewrite its own record, you don't have a record, covers why it matters now.

OpenAI recommends records that cannot change

In summer 2026, agents in an OpenAI security evaluation got out of their sandbox and into Hugging Face's servers. The independent review by METR and Redwood Research found agents "tried to retroactively edit transcripts" and "modified some accessible action logs". The source-of-truth transcripts were not edited, as far as the reviewers saw in the data they analysed (7 to 13 July).

OpenAI recommends: "Save agent transcripts immutably for a reasonable period of time (for example, in write-once storage accounts)". It adds that auditors "should be provided with sufficient access to verify that the claims of the safety case are valid and sound". Records that can't be changed, and auditors who can check what they are told. We agree with both. Write-once storage is one way to do the first. What we add is a way for a reviewer to check a record without relying on the owner's account setup; a write-once bucket plus a fingerprint kept somewhere the owner does not control could also do that, and we would like to hear where it does it better.

A record that cannot be changed in place

We store records on Swarm, the decentralised storage network (ethswarm.org). Three properties do the work.

Content addressing. A record's address is computed from its own content. An edited version can be stored, but it gets a different address. The original stays where it was.

Postage-stamp expiry. Storage is paid for with a postage stamp, Swarm's storage ticket, whose lifetime is stated and public. So the precise claim is: a record on Swarm cannot be changed in place until its stated expiry. Call it WORM storage (write once, read many) with an expiry.

Many nodes hold it. Each chunk is kept by the nodes closest to its address, and neighbouring nodes keep copies, while its stamp is funded. Whoever you give the address to can fetch the record from a Swarm node and check it without trusting whoever stored it. The toolkit's own download goes through our gateway; the hash check is what removes the need to trust the gateway.

That is what shows the record was not changed after it was stored. Nothing has to be published for that to hold.

[!quote] Change one byte and you get a different address. The original stays where it was until its stated expiry.

Provenance is a chain, not a list

A log lists events in order. Provenance links each output to the inputs and actions that produced it, so you can follow the chain back: this report came from this step, which read these files. W3C PROV is the standard model for this chain: things (entities), the actions that used and produced them (activities), and who was responsible (agents). The files you store can be in the PROV format.

Here is the plain split between today and proposal.

What the toolkit does today, on the test network (Base Sepolia). It stores the record you give it on Swarm and returns its address and its fingerprint (the record's SHA-256 hash), and, if you ask, a notary signature with a timestamp. It links records into a chain: a transformation link says one record was made from another, and a merge link says one record was made from several (recordTransformation and mergeTransform in the SDK, chain transform and chain get --follow in the command-line tool). And it can anchor a fingerprint, which means writing it on a blockchain through a smart contract, so its time no longer rests only on our gateway.

What is proposal. The software that runs the agent writing that chain automatically, step by step, outside the agent's decisions. Nothing in the toolkit does that today.

[!quote] A log tells you what happened, in order. Provenance tells you what each result came from.

What the toolkit does today

The Datafund Provenance Toolkit is open source and in pre-release, meant for testing. What we have tested: upload a record to Swarm through a public gateway; get back its address (the reference) and its fingerprint; optionally have the gateway's notary sign the fingerprint and a timestamp, with the gateway's own key; then download, recompute and check. With the TypeScript SDK (tested with @datafund/swarm-provenance 0.6.1):

import { createHash } from 'node:crypto';
import { ProvenanceClient } from '@datafund/swarm-provenance';

const client = new ProvenanceClient(); // public gateway, free tier
const text = `Synthetic agent report: 3 files checked, 0 errors. ${new Date().toISOString()}`;
const { reference, metadata } = await client.upload(text, { sign: 'notary' });
await new Promise((r) => setTimeout(r, 60_000)); // the free tier is rate-limited: wait before the next call

// Later, the person checking, holding the reference and the fingerprint:
const r = await client.download(reference);
const recomputed = createHash('sha256').update(r.file).digest('hex');
console.log(recomputed === metadata.content_hash, r.verified, r.signatures?.[0]?.timestamp);

The snippet waits between the two calls because the free tier is rate-limited. It condenses the quickstart script, which ran on 17 October and printed a match and Notary signature verified: true. Agents can also use the toolkit through its MCP server, which is being updated.

What we propose next

Everything in this section is a proposal. None of it is built or tested yet.

The runtime writes the record, not the agent. The software that runs the agent records each step automatically, outside the agent's decisions: what it read, what it did, what it produced. The quickstart shows the idea at the smallest scale, as a wrapper in your own code. It records only the step's name, what the step returned and the finish time, not what the step read or did, and you write it yourself; the toolkit does not run it.

Sealed records. Records stay encrypted, and keys go to an auditor when needed. Test and evaluation runs, which use no personal data, are an obvious place to start.

How this compares with Certificate Transparency and Rekor

Certificate Transparency and Sigstore Rekor solve a close problem and are widely used. Both are public append-only logs with inclusion proofs (this entry is in the log) and consistency proofs (the log has only grown).

Content addressing alone gives neither. A Swarm address proves the bytes match; it says nothing about what else was stored, or in what order. If you need that, you need something kept outside the record: anchoring on a blockchain, which inherits that chain's trust model, or a transparency log.

A transparency log is built to show that an entry exists and in what order, not to hold large records. Rekor, as we read its documentation, stores a signature and a hash rather than the artifact, so someone still has to keep the agent's transcript itself. Swarm keeps the record, on many nodes and with a public expiry.

For agent records, we think keeping the record where it cannot be changed is the core, and anchoring is an extra for when a public time or ordering matters. The obvious alternative is a transparency log for fingerprints plus separate storage. If you think that is better, challenge us.

What a check proves, and what it does not

A check proves that the bytes you downloaded match the fingerprint recorded at upload. The record has not changed since it was stored. With a notary signature, it also shows our gateway vouched for that fingerprint at that time; with an anchor, the blockchain shows when the fingerprint was written.

It does not prove that everything the agent did was recorded, that the work was correct, lawful or authorised, or who uploaded it. When the agent uploads its own record, it can write anything into it before storing it. The METR and Redwood review describes agents "spoofing" tool call outputs; a faithful record of a spoofed output is still a spoofed output. Recording in the runtime would narrow the gap between what happened and what was recorded. It does not close it.

[!quote] A receipt proves a record hasn't changed. It doesn't prove everything was recorded, or that the work was right.

Limits today

Pre-release. The toolkit is meant for testing environments, not production.
Test network. Anchoring, lineage links and the paid mode (x402) run on a test network (Base Sepolia). Mainnet is not live.
The notary is us. The timestamp rests on the gateway's own key, unless you anchor.
Limited free tier. No wallet, key or account. The free tier is limited, and may fail under heavy load; retry later.
Short expiry. Free-tier records are kept for about a day; for evidence, use a stamp funded for the retention period you need.
Readable by whoever has the reference. Records are not encrypted. Use synthetic data only.
Untested path. We have tested the match case, not a tampered record. Please try.

Try it

The quickest way is dataprovenance.app. It runs in your browser with nothing to install: record a file, get a receipt page, and verify it. The app uploads the file to Swarm through our gateway, and anyone with its reference can download it while it is kept, so use a file you are happy to make public.

Try it: give a file a receipt

For code, the toolkit repository has the quickstart: it stores a synthetic agent report, returns its receipt, and has you play the client who checks it. Because the data is synthetic and public, anyone can run the check.

Then break it. Open an issue in the repository, or tell us on Hacker News where the design falls short.

Questions people ask

What tools exist for AI agent provenance?
They fall into three groups: write-once cloud storage such as S3 Object Lock, tamper-evident logs and databases such as Sigstore Rekor and immudb, and content-addressed decentralised storage such as Swarm. The Datafund Provenance Toolkit is open source and in pre-release: it stores agent records on Swarm and lets the people entitled to see them check that nothing changed.

What storage can't be changed by an AI agent?
Storage the agent cannot overwrite, and where an overwrite would show. A write-once bucket such as S3 Object Lock blocks changes inside one provider's account. On Swarm a record lives at an address computed from its content, so a changed record gets a new address and the original stays until its stated expiry. Either way, the software that runs the agent should write the record, not the agent.

How can an auditor verify that an AI agent's transcripts have not been changed?
By comparing what they are given with a fingerprint recorded when the transcript was written, and fetching the record from storage the lab cannot change. With the toolkit, the auditor downloads the record by its address, recomputes its SHA-256 fingerprint and checks the gateway's notary signature, or the anchor on the blockchain.

Sources

Datafund, "If your agent can rewrite its own record, you don't have a record" (20 October 2026): https://datafund.io/blog/agents-using-swarm-cant-rewrite-their-history/
METR and Redwood Research, independent review of the Hugging Face incident (26 August 2026): https://alignmentforum.org/posts/nB8KKapnWGBXtKKiM/brief-independent-investigation-of-agents-behavior-reasoning
OpenAI, "Towards safety cases for frontier AI training" (28 September 2026): https://openai.com/index/towards-safety-cases-for-frontier-ai-training/
Swarm, chunk storage and content addressing: https://docs.ethswarm.org/docs/concepts/DISC/
Swarm, postage stamps: https://docs.ethswarm.org/docs/learn/technology/contracts/postage-stamp/
W3C PROV overview: https://www.w3.org/TR/prov-overview/
Certificate Transparency, RFC 9162: https://www.rfc-editor.org/rfc/rfc9162 (it obsoletes the original RFC 6962: https://www.rfc-editor.org/rfc/rfc6962)
Sigstore Rekor: https://docs.sigstore.dev/logging/overview/
Datafund Provenance Toolkit: https://github.com/datafund/provenance
dataprovenance.app: https://dataprovenance.app
TypeScript SDK: https://github.com/datafund/swarm_provenance_SDK
Quickstart test log, 17 October 2026 (internal): 2026-10/test-runs/quickstart-rerun-2026-10-17.md