DEV Community: Adan Potyondy

Bad data costs real money. Here is the HubSpot to Supabase workflow that fixed it.

Adan Potyondy — Thu, 05 Nov 2026 19:36:21 +0000

In September I worked with a customer on a data quality problem and recorded a short demo of the general case. Contacts come out of HubSpot, get cleaned and deduplicated, and only the rows that pass land in Supabase.

The interactive demo is on the original post. Here is a separate 6-minute live build of the same flow; it matches duplicates with fuzzy (Jaro-Winkler) matching rather than the exact email match described below.

The problem the customer brought us

Their sales team lives in HubSpot. Their product and reporting live on a Supabase database. A nightly job copied contacts from one to the other, and it copied everything: the same person three times under three spellings of their email, trailing spaces in names, companies typed in four different ways, and records with no usable email at all.

None of that looks dramatic in a CRM. It looks dramatic in the results. Duplicate contacts get counted twice in pipeline reports, two reps reach out to the same person in the same week, and the product team builds features for a customer base that is smaller and cleaner than the one the numbers describe. Bad data does not stay in the database. It shows up in decisions.

What the demo shows

The recording is a walk through the general case, not the customer's data. It uses the Spanish version of the Neblex workspace, which is the language the customer works in; every screen in Neblex Integration Fabric is available in English, German, and Spanish.

Pull the contacts from HubSpot. The HubSpot connector reads contacts with the properties the customer cares about. The flow can run on a schedule or start when a contact is created or changed.
Clean the rows. Emails are lower-cased and trimmed, names lose their stray spaces, and company names are normalized so "Acme", "ACME Inc" and "acme, inc." stop being three companies.
Remove duplicates. The Remove duplicates step matches records on the normalized email and drops the repeats. It can also keep every record and tag each one as unique or duplicate, with a pointer to the match, so a review branch can look at the borderline cases.
Check quality before writing. The Assert data quality step checks the batch against the rules the customer chose, such as required fields and an email address on every row. If a rule fails, the run stops with the failing records and the rule named, so nothing questionable reaches the database and someone can fix the source in HubSpot instead of quietly inheriting the problem downstream.
Insert into Supabase. Only the rows that passed are written to the Supabase table, with the run recorded end to end.

That last point is the one the customer cared about most. Every run shows each step, what stopped the run and why, and how many records were written; with run data kept for replay, the records themselves are there too. If a number looks wrong in a report, the answer is in the run history, not in a guess.

Why we clean at the boundary

It is tempting to fix data quality in the database with a cleanup script that runs once a month. It never stays fixed, because the source keeps producing the same problems. Cleaning at the boundary, in the flow that moves the data, means every record is checked every time and the source system gets the feedback it needs.

It also means the cleanup is not a separate tool. Bulk data processing and data sync and change data capture run on the same platform, with the same connectors and the same audit trail as everything else. When the customer later wants to push a cleaned company list back into HubSpot, that is another flow, not another product.

What this means for your team

If your reporting and your CRM disagree, the fix is rarely a bigger dashboard. It is usually a clean, checked path from one system to the other, with the failures visible instead of silently loaded.

Frequently asked questions

Does the cleanup change the data in HubSpot?

No. The flow reads from HubSpot and writes to Supabase. When a quality rule fails, the run stops and names the records and the rule, so the team can correct them in HubSpot, and the next run picks up the corrected contacts. Writing corrections back to HubSpot would be a separate flow you choose to turn on.

How are duplicates decided?

By the field you choose, or the whole record. In the demo it is the normalized email address. Matching can be exact or fuzzy with a similarity threshold, and instead of dropping duplicates the step can tag them for a review branch.

Can this run as contacts change instead of on a schedule?

Yes. The HubSpot connector has 46 polled triggers, checked as often as every minute, so a flow can start when a contact is created or changed and check the new record on its way through.

Does it work with a plain PostgreSQL database instead of Supabase?

Yes. Supabase is PostgreSQL underneath, and the same flow writes to any PostgreSQL, MySQL, SQL Server, or Snowflake target through their connectors.

Maria Soria works in customer success at Neblex. Originally published at neblex.com.

Headless iPaaS: what an AI assistant should and shouldn't do in your integration platform

Adan Potyondy — Thu, 05 Nov 2026 18:05:46 +0000

A headless iPaaS is an integration platform you can fully use without its visual interface: through an API, a command line, or an AI assistant connected over MCP. Done right, the assistant builds, validates, tests and publishes integrations, and a person keeps the secrets, the production publish and the access grants.

The term is new, and vendors use it for very different things. Some let an assistant run a workflow somebody else built. Others let it build the whole integration and push it to production.

This post compares the designs by what an assistant can actually do at each stage of an integration's life, and what should still need a person. At the end you'll find ten questions to put to any vendor.

What is a headless iPaaS?

An integration platform whose capabilities don't depend on its screens.

The term comes from content management. A headless CMS is a back end that stores content and serves it through an API, with no display layer of its own. Applied to an iPaaS (integration platform as a service), it means anything you can do in the visual builder, you can also do through an API.

What changed in the last year is who calls that API. The Model Context Protocol (MCP), an open standard for connecting AI applications to outside systems, is now supported by Claude, ChatGPT, VS Code, Cursor and others. A platform with an MCP server can be driven from the same assistant your developers already use to write code.

How useful that is depends on what the server exposes.

Three ways a platform can work with an AI assistant

You'll run into three designs, and they differ in how much of the job the assistant can finish.

Let's use one job as the running example, with the same CRM as our HubSpot contacts post. In Claude Code you type: "When a HubSpot deal closes, create the customer in our billing database and post the deal to the #sales Slack channel."

Assistant inside the visual builder

The platform ships its own assistant on the canvas. It turns your sentence into draft steps: a HubSpot trigger, a database insert, a Slack message. Then a person checks the mapping, saves, tests and publishes in the browser.

That works well for people who live in the builder. But you can't use it from the coding assistant you already have open, and it stops at the draft.

Action-only MCP server

The platform exposes actions, or workflows that already exist, as tools. Your assistant can post to Slack or kick off a workflow someone already built, but it can't create or change one.

In our example, the assistant could send a test Slack message. Somebody still has to build the integration itself in the builder.

Full-lifecycle headless platform

The platform exposes the same capabilities its visual builder uses. Your assistant lists the connectors and the connections you already have, reads the HubSpot and database operations, writes the workflow, validates it, tests it on a sample deal, reads the result, fixes the mapping and asks to publish.

A person still steps in at a few deliberate points, covered below. Everything else happens in one conversation, and the result opens in the visual builder like anything else.

What an assistant needs at each stage of the integration lifecycle

Building is one stage of many. An integration gets discovered, built, tested, published, run, debugged and eventually retired. An assistant that only covers the first two leaves the rest to a person.

Stage	What the assistant needs	In-builder assistant	Action-only MCP	Full-lifecycle headless
Discover	List connectors, their operations and schemas, and existing connections	Within the canvas	Only the exposed actions	Yes
Build	Create and edit the workflow definition	Draft only	No	Yes
Validate	Check the definition before saving	When a person saves	No	Yes
Test	Run on sample data without publishing, and read step results	A person runs it	No	Yes
Publish	Make it live, with approval where required	A person	No	Yes, with approval for protected environments
Run and observe	Start runs, read run history and step events	Varies	Trigger only	Yes
Debug	Read the failed step's input and error, fix, re-test	Varies	No	Yes
Retire	Unpublish, delete to a trash, restore	A person	No	Yes

In our example, the problems show up during testing. A deal without a company attached, or a currency the billing table won't accept, shows up in the test run's step events. An assistant that can read those events fixes the mapping and tests again in the same conversation.

One that can't hands you an error message to chase. And after a failed production run, those same events tell you which step to re-run from. We covered why that matters in Replay from the failed step, not from record one.

Guardrails: what should still need a person

Handing an assistant write access to the system that moves your customer data is not a small decision. Keep the high-impact steps with a person.

The standards agree. The MCP specification says there "SHOULD always be a human in the loop with the ability to deny tool invocations," and that servers must implement proper access controls.

OWASP lists "excessive agency" among the top risks for LLM applications: too much functionality, too many permissions or too much autonomy. Its fix is least privilege plus human approval for high-impact actions.

For an integration platform, that comes down to five things:

Secrets stay with a person. The assistant can create a connection, but a person enters the API key or completes the OAuth sign-in in the browser. The secret never passes through the conversation.
Building needs a strong sign-in. Authoring and publishing should require a person's sign-in with a second factor, not a long-lived API key sitting in a config file. A key can be allowed to run workflows. It shouldn't be able to change them.
Access is scoped. Whoever grants access picks the projects and environments the assistant can reach, and what it may do there: operate, author or publish. Revoking it takes effect immediately.
Production publishes wait for approval. The assistant can build and test freely in a development environment. Publishing to production waits until a person approves it in the browser.
Everything is reversible and logged. An edit that collides with someone else's is rejected instead of overwriting it, deletes go to a trash you can restore from, and every run leaves events you can read, with secrets scrubbed.

The assistant does the work. A person keeps access, secrets and the production publish.

None of this stops the assistant from doing the work. In our example, a person steps in exactly three times: to grant access, to enter the HubSpot and database credentials, and to approve the production publish.

Common mistakes when evaluating AI access

Most of them come from treating "it has an MCP server" as the answer.

Only checking the build. A demo that turns a sentence into a workflow is the easy part. Ask to see the assistant test it, read a failed step and fix it.
Letting the assistant hold the keys. If connecting a system means pasting its API key into the chat or a config file, that key now lives in the assistant's context and logs.
Using one environment. If the assistant builds where production runs, every test writes real records. Keep development, test and production separate, and gate the last one.
Granting everything. A grant that covers every project and every permission is exactly the "excessive permissions" OWASP warns about. Start with one project and the author scope.
Assuming the visual builder goes away. It shouldn't. Somebody will review the assistant's work, and most people do that in the builder.

Questions to ask before you let an AI assistant build in your integration platform

They work with any vendor, and the answers tell you which design you're really getting.

Can the assistant create and edit integrations, or only run existing ones?
Does it have the same capabilities as the visual builder, or a subset? Which parts are missing?
Can it test an integration without publishing it, and read each step's input, output and error?
Can it read run history and debug a failed run?
How are credentials entered? Does a secret ever pass through the assistant?
What sign-in does building and publishing require? What can a plain API key do?
How is access scoped: by project, by environment, by action?
What needs a person's approval, and where does that approval happen?
What happens when the assistant and a colleague edit the same integration at the same time?
Can deletes be undone, and is every change and run logged?

What this means for IT and security

Treat the assistant as its own user with its own grant, not as an extension of the developer's account.

Give it one project in a development environment, the author scope, and no publish rights to production. Then watch what it asks a person for. A well-built platform sends you a browser link for every credential and every production publish, and for nothing else.

Where Neblex fits

Neblex Integration Fabric is a full-lifecycle headless platform. The visual builder and the MCP server work with the same building blocks, so an assistant connected to Neblex can do what a person does in the builder, under the same rules.

Any MCP client can connect, including Claude Code, Codex, Cursor, VS Code and Windsurf.
The assistant works across flows, forms, apps, data tables, transform modules, data models, AI agents and connections, not just workflows.
It discovers connectors and their operations, validates before saving, tests before publishing, runs flows, reads secret-scrubbed run events and checks flow health.
Secrets never go through MCP. Creating a connection returns a link where a person enters them.
Building and publishing require a user-approved sign-in with a second factor. A static API key can inspect and run the flows it's granted, and can never create or publish.
On the consent screen, the person picks the scopes (operate, author, publish), the projects or flows, and the environment. Revoking access cuts it off immediately.
Publishing into a protected environment waits for a person's approval in the browser.
An edit that collides with someone else's is rejected, so the assistant re-reads and re-applies its change instead of overwriting. Deletes go to a trash, and a restore re-checks dependencies.

Two things stay in the browser on purpose: entering secrets, and promoting work between environments. That's how we build and test our own flows: from Claude Code, in a development environment, then promoted to production in the GUI.

So before your team connects an AI assistant to its integration platform, find out two things. Can it test and debug as well as build? And which steps still go to a person? Those answers tell you whether a headless iPaaS will speed up your integration work or just move the risk somewhere else.

Sources

Headless content management system, Wikipedia
What is the Model Context Protocol (MCP)?, modelcontextprotocol.io
MCP specification: Tools (2025-06-18), modelcontextprotocol.io
LLM06:2025 Excessive Agency, OWASP Top 10 for LLM Applications, OWASP Foundation, 2025
MCP guide, Neblex Integration Fabric docs

Last checked: October 15, 2026.

Frequently asked questions

What is a headless iPaaS?

An integration platform you can fully use without its visual interface, through an API, a command line, or an AI assistant connected over MCP. The visual builder usually stays; it just stops being the only way in.

Can an AI assistant like Claude Code build integrations in an iPaaS?

Yes, if the platform exposes building and not just running. Check that the assistant can discover connectors, create and validate a workflow, test it and read the run results. Some platforms only let an assistant trigger existing workflows.

Is it safe to give an AI assistant write access to an integration platform?

It can be, if the platform limits what the assistant can reach and keeps the high-impact steps with a person. Look for access scoped per project, a strong sign-in for building, secrets entered by a person, approval before publishing to production, and a full run log.

What's the difference between an iPaaS MCP server and a headless iPaaS?

The MCP server is the connection. What matters is what it exposes. Some only let an assistant run existing actions or workflows. A headless iPaaS exposes the same capabilities as the visual interface, so an assistant can build, test, publish and debug.

Does a headless iPaaS replace the visual builder?

No. An assistant can build an integration that you then open, review and change in the visual builder. Headless adds a way in. It doesn't take one away.

Daniel Wiegand is the founder of Neblex. Originally published at neblex.com.

On-prem runtime security: what leaves your network

Adan Potyondy — Mon, 02 Nov 2026 19:57:17 +0000

Does data leave your network with an on-prem integration runtime? It depends on the design. With the Neblex On-Prem Worker, our local integration runtime (often called an on-prem integration agent), the Neblex cloud gets run status, diagnostics, record counts and integrity receipts over one outbound encrypted connection on port 443. Records go only to the systems each flow is built to reach. AI agents work the same way: an agent, its reasoning model and its embedding model can all run on premises, just as flows do.

The two questions a security review asks

Before an integration runtime goes inside the network, a security review asks what it will send out and what it needs from the outside. Here is how the On-Prem Worker answers both: what it sends, what it listens on, and what it does when the internet drops.

The control plane in the Neblex cloud coordinates; each On-Prem Worker works inside its own network and sends only run reports to the cloud. Records reach the systems a flow uses, including SaaS apps, directly from the On-Prem Worker, never through the Neblex cloud.

How much memory, CPU and disk it uses is covered in What does an on-prem integration runtime cost your server? We measured ours.

What goes to the Neblex cloud?

Run status, diagnostics, record counts and integrity receipts. Record contents stay with the On-Prem Worker, unless a flow shows something to someone through the Neblex cloud: a webhook answer, a form page or an App result.

The On-Prem Worker runs flows next to your systems and keeps records on your disk; one outbound connection carries run reports to Neblex.

The On-Prem Worker opens one outbound, encrypted WebSocket (wss) connection to the Neblex cloud on port 443, over TLS 1.3 with a post-quantum key exchange. It opens no port to the internet.
It proves its identity inside that connection with two keys created on its own machine, which never leave it: the key for the certificate Neblex issues it, and a post-quantum key.
Its only listener on a standalone install is a local console bound to 127.0.0.1, which other machines can't reach. Members of an On-Prem Worker cluster also accept a link from the other members of the same cluster.
Every run on it is reported in local custody: the cloud gets a compact run report, and the records stay with the On-Prem Worker.

What goes to your own systems?

Whatever each flow is built to send, directly. The On-Prem Worker connects to every system a flow uses on that system's own port: HTTPS on port 443 for most SaaS APIs, 5432 for a hosted PostgreSQL database, and the usual ports for local databases, queues and file servers. A flow that writes contacts and deals to a CRM calls the CRM's API; a flow that loads rows into a cloud PostgreSQL database connects to it on port 5432.

Those records leave your network because you chose that destination, and they travel from the On-Prem Worker straight to it, never through the Neblex cloud. Plan outbound firewall rules around your flows' destinations as well as the one connection to the Neblex cloud.

Where are records held during a run?

On your machine. Records do pass through the On-Prem Worker, and a large batch is buffered in its local run journal on your disk until the run is settled. They don't go to the Neblex cloud.

The On-Prem Worker purges the journal once the run is settled, so batch data doesn't accumulate on disk. Neblex doesn't store your records; anything that must persist lives in your own database.

What happens when the internet connection drops?

The run keeps going, and only the status reports wait. The flow executes on the On-Prem Worker, not in the Neblex cloud, so the connection to Neblex isn't in the data path.

The On-Prem Worker keeps reading from and writing to every system it can still reach.
It keeps its run reports in its local run journal and delivers them once it reconnects.
The run's final status reaches the Neblex cloud with the rest of those reports.

A step that needs the Neblex cloud itself, such as a form a person fills in, waits for the connection and then continues. No step fails because the Neblex cloud is out of reach.

Recovering from a failed step, rather than an outage, is covered in Replay from the failed step, not from record one.

The On-Prem Worker also keeps a local, encrypted copy of the scheduled, polling and streaming flows placed on it, so it starts them itself while the cloud is out of reach. Flows started by a webhook, a form or a person in the app come in through the Neblex cloud, so they start once it's reachable again.

Other local runtimes also keep running when their platform is out of reach. Ask any vendor to show you what theirs does.

Can AI agents and their models run on premises?

Yes, the same way flows do. An AI agent runs on the On-Prem Worker as a step in a flow, and both models it depends on can run inside your network too: the large language model (LLM) that does the reasoning, such as an open-weight model behind an OpenAI-compatible or Ollama endpoint, and the embedding model that turns your documents and records into vectors for the agent's knowledge base. Those vectors are stored in your own PostgreSQL database with pgvector, and the agent's long-term memory is stored only in your own database.

Set up that way, an AI step sends no more out of your network than any other step: prompts, model responses, records, documents and embeddings stay on premises, and the Neblex cloud gets the same run status it gets for any other flow.

If an agent uses a hosted provider for either model instead, whatever is sent to that model goes to that provider. Pick one whose data residency fits your rules, or keep the model local.

What to ask any vendor

Does the integration run on your machine, or are records relayed to the vendor's cloud to be processed?
Which ports does the runtime open, inbound and outbound?
Which destinations and ports will your flows need opened outbound, and does any of that traffic pass through the vendor?
What exactly is in its run reports?
What happens to a running job when the internet drops, and to a scheduled one?
Where is an in-flight batch held, and when is it deleted?
Can AI agents, and both the reasoning and embedding models they use, run inside your network the same way flows do?

Ask to see the answers in the runtime's logs and network connections, not only in the documentation.

Sources

On-Prem Workers, Neblex Integration Fabric docs
Offline controls, Neblex Integration Fabric docs
AI agents guide, Neblex Integration Fabric docs
On-Prem Worker architecture and run reporting, as designed and documented by Neblex engineering.

Last checked: October 30, 2026.

Frequently asked questions

Does an on-prem integration runtime need inbound firewall ports?

Not all of them do. The On-Prem Worker opens one outbound, encrypted WebSocket connection to the Neblex cloud on port 443 and opens no port to the internet. On a standalone install, its only listener is a local console bound to 127.0.0.1, which other machines can't reach.

Does my data pass through the vendor's cloud?

It depends on the design. Some runtimes relay records to the vendor's cloud for processing; others run the integration locally. The On-Prem Worker runs the flow on your machine and sends Neblex run status, diagnostics, record counts and integrity receipts, not record contents, unless a flow shows something to someone through the Neblex cloud, such as a form page or an App result.

What happens to a running job if the runtime loses its internet connection?

With the On-Prem Worker, it keeps running. The flow executes on the On-Prem Worker, which keeps its run reports in its local run journal and delivers them after reconnecting. Ask any vendor what their runtime does in this case.

Can AI agents run on an on-prem runtime?

With Neblex, yes, just as flows do. An AI agent runs on the On-Prem Worker, and both its reasoning model (LLM) and its embedding model can run inside your network, with the agent's knowledge base in your own PostgreSQL with pgvector and its memory in your own database. Set up that way, all processing stays on premises and no AI provider outside your network sees your data.

Adan Potyondy leads go-to-market at Neblex. Originally published at neblex.com.

What does an on-premise integration runtime cost your server? We measured ours

Adan Potyondy — Thu, 22 Oct 2026 17:50:12 +0000

We measured on-premise resource usage for the Neblex Worker, our local integration runtime, across 15 everyday flows and three large workloads. Idle, it used about 25 MB of memory. Fourteen of the 15 flows averaged 29 to 94 MB, and a 1 million row load into a cloud database peaked at about 450 MB. CPU stayed low except during heavy matching.

The question IT and security teams ask first

A request we hear often from prospects is to move data from a database inside their network to a cloud warehouse, on a schedule, without opening that database to the internet. That means installing software inside the network: an agent or a runtime. Before anyone discusses the warehouse, IT and security teams ask what that software will cost their servers and what it can reach.

The usual answer is a page of minimum requirements. We wanted to answer with measured numbers, so we ran the Neblex Worker through a week of tests, from small event flows to million-row loads, and recorded what it used. This post gives the numbers, the method and the limits.

This post answers the first question for a local runtime. What the runtime sends out of your network, and what happens when the internet drops, are covered in a separate post.

How we measured on-prem integration agent resource usage

All tests used synthetic or test data, generated by us, on our own hardware, between October 2 and 8, 2026.

Deployments: we ran the runtime two ways, as a background service and in a container capped at 4 CPU cores and 4 GB of RAM. The test machine also hosted the PostgreSQL, Valkey (an open-source, Redis-compatible cache) and Kafka instances it read from and wrote to, so the runtime competed for memory more than it would on a dedicated server.
The workloads: 15 flows deployed to the runtime (events, files, CRM syncs and database loads); a 1 million row load from a local PostgreSQL database into a cloud database, with data-quality checks and masking; fuzzy duplicate matching over about 970,000 records; and, to isolate the effect of batch size, a plain copy from PostgreSQL to Valkey at 1,000 to 5 million rows.
The tools: standard process tools sampled the runtime's memory and CPU every half second, and the container's own memory statistics separated process memory from file cache. Start and finish times came from the runtime's log, and the 15 flows' averages came from the Neblex console.

Your records, connectors and hardware will shape your own numbers; the patterns are what carry over.

What does the runtime use when it's idle?

Very little. The runtime is a single executable, about 74 MB on macOS and 92 MB on Linux x64.

It needs no separate language runtime, such as Java, .NET or Node.js, and it starts no child processes. Builds are published for Linux x64, macOS on Apple Silicon and Windows x64.

Measure (idle)	Value
Install	One executable, about 74 MB (macOS) or 92 MB (Linux x64)
Runtime's own memory	22 to 27 MB
Resident memory (RSS)	27 to 35 MB
CPU	Under 1% of one core
Listening ports	One, on loopback only

For a security reviewer, the small number matters less than what it implies: there is no separate runtime to patch and no helper processes to account for.

What does the runtime use, by kind of flow?

It depends on what the flow does. Memory follows how much the flow holds at once, CPU follows how much it computes per record, and run time follows the destination.

Kind of flow	What we ran	Runtime memory	Runtime CPU (one core = 100%)
Idle	Connected, no runs	22 to 27 MB	Under 1%
Everyday flows	15 flows: MQTT, Kafka and AMQP events, CRM syncs, file transfers, database loads	29 to 94 MB average for 14 flows; 122 MB for one bulk load	0 to 31% average
Load to a cloud database	1 million rows, local PostgreSQL to a cloud PostgreSQL database, with checks and masking	Peak 320 to 445 MB; median 53 to 118 MB	12 to 26% average
Local bulk copy	1,000 to 1 million rows, PostgreSQL to Valkey	41 MB to about 250 MB	About 22% average
Heavy computation	Fuzzy duplicate matching over about 970,000 records	Peak 534 to 947 MB (resident)	93 to 102% average
One very large batch	5 million rows	About 420 MB, plus 1.7 GB of journal on disk	22 to 24% average; brief peaks near one core

The sections below take each row in turn.

What do everyday flows use?

Most use about as much memory as the idle runtime. We deployed 15 different integrations to one Neblex Worker, covering events, file transfers, CRM syncs, cache updates, database loads and an AI step, which gives a broad sample of how the runtime behaves. The Neblex console reports CPU and memory for each flow, averaged across its runs, on synthetic or test data. Run times come from the runtime's log, from start to finish of each successful run.

Flow (what it does)	Avg CPU (one core)	Avg memory	Avg run time
Local Postgres customers to Supabase (bulk load)	19.1%	122 MB	Varies by batch size*
AMQP order events to CRM deals, with dead-letter queue	4.3%	94 MB	0.4 s
Postgres customers: AI industry tagging with an LLM	1.7%	41 MB	19.6 s
Local Postgres customers to Valkey cache	30.9%	40 MB	8.8 s
E-commerce new order to CSV file on FTP (ERP handoff)	2.1%	39 MB	0.7 s
Valkey usage counters to cloud Postgres daily usage, with quarantine	26.8%	38 MB	3.0 s
MQTT machine alarm to work-management task, with quarantine	5.5%	37 MB	1.2 s
MQTT plant telemetry to Postgres history, with quarantine	6.1%	37 MB	0.9 s
Kafka signup events to CRM contacts, with dead-letter topic	4.5%	36 MB	1.0 s
MQTT machine alarm to service-desk maintenance request, with quarantine	4.7%	35 MB	1.4 s
CRM company changes to Valkey account cache	4.3%	32 MB	0.5 s
Postgres customer data-quality check to work-management task	6.1%	31 MB	1.4 s
Postgres new customers to CRM contacts sync	1%	31 MB	0.4 s
FTP contacts file to CRM contacts import	0%	30 MB	0.1 s
Nightly FTP contacts file to Supabase, with data-quality checks	6.1%	29 MB	4.1 s

*A 1 million row load took 402 to 1,240 seconds, from 5,000 down to 100 rows per insert statement.

Fourteen of the 15 flows averaged 29 to 94 MB, close to the runtime's idle memory. Only the bulk load from local Postgres went above 100 MB. The event-driven flows, reading from MQTT, Kafka and AMQP, averaged 4 to 6% of one core. Twelve of the 15 finished in under 5 seconds on average; the AI tagging flow's 20 seconds is mostly spent waiting for the language model, with CPU under 2%.

What does a 1 million row load to a cloud database use?

Less than 450 MB at its peak. Moving data from a database inside the network to a cloud database is the request we hear most, so we ran it at full size: 1 million synthetic customer rows from a local PostgreSQL database into a Supabase table, which is hosted PostgreSQL, over a home internet line (about 175 Mbps down and 44 Mbps up).

Each row went through four data-quality checks, and its email was hashed and its phone number cut to the last four digits before it left the machine. Rows that failed stayed in a quarantine table on the local network. We repeated the load four times, with 100, 1,000, 5,000 and 10,000 rows per insert statement.

The runtime's own memory peaked at 320 to 445 MB across the four runs and spent most of each run near its median of 53 to 118 MB.
CPU averaged 12 to 26% of one core.
Every run loaded exactly 976,000 rows and kept 24,000 in local quarantine. The fastest took 402 seconds.
The internet line was never the limit: uploads averaged about 5 to 7 Mbps of the 44 available.

A later post covers the checks, the masking and how batch size changed the timing.

What does heavy computation cost?

About one CPU core, and more memory. Comparing records with each other is a different kind of work from passing them through, so we ran fuzzy duplicate matching on the runtime: about 970,000 customer records, grouped by zip code, with names and streets scored for similarity inside each group. The candidate pairs stayed in a local table.

First run, 970,000 records: 1,510 seconds. CPU averaged about 102% of one core, peaking at 217%, and resident memory peaked at 534 MB.
Repeat with extra preparation, 976,000 records, on the same machine with the runtime running as a background service: 1,683 seconds. CPU averaged 93%, and resident memory peaked at 947 MB.

If you plan matching, scoring or heavy transformation on the runtime, size for this case rather than for the everyday flows.

Why the destination sets the pace

In every bulk run, the runtime spent most of its time waiting for the destination, not computing.

We built the Valkey flow to write one record per command and wait for each reply, so the local copy ran at roughly 7,500 to 9,400 rows per second. Raising the flow's concurrency setting to 4 and then 64 didn't change that rate. That fits what Valkey's documentation on pipelining describes: when a client sends one command at a time, round-trip time is the limit.
Inserting into the cloud database one row per statement took about 0.35 seconds per round trip, under 3 rows per second. Grouping thousands of rows into each statement brought the full 1 million rows down to under 7 minutes.

Reading was far faster than writing, so millions of rows were waiting to be written at once. The runtime absorbed that gap: its memory stayed flat while the waiting rows sat in its local run journal on disk, covered below.

For large loads, batching writes is the lever, with any agent or runtime. The reassuring part for a server owner is the other side of the same finding: a runtime that is mostly waiting uses little CPU.

How do memory and CPU change as the batch grows?

Memory grows with the batch the runtime holds at once; CPU stays low. To isolate batch size we used the plainest flow we had, a copy from PostgreSQL to a local Valkey cache.

Rows in one run	Duration	Peak memory	Average CPU (one core = 100%)
1,000	about 20 s	about 41 MB (RSS)	not averaged
100,000	12.3 s	about 121 MB (RSS)	about 23%
1,000,000	106 s	about 250 MB*	about 22%

*From a repeat 1 million row run on varied data (116.6 s), measured as the process's own memory including compressed pages.

The 1 million row run took 106 seconds, about 9,400 rows per second. The runtime held about 250 MB during the run and dropped to about 70 MB around 20 seconds after it finished. CPU averaged about 22% of one core at every size.

Where does a large batch live: memory or disk?

On disk. During a 5 million row batch, the runtime's own memory stayed flat at about 420 MB while it wrote the in-flight batch to its local run journal, about 1.7 GB or roughly 350 bytes per row.

The run journal is a durable local record of the batch that lets a run recover from an interruption. The runtime's logs describe it as hash-chained, sealed and flushed to disk with fsync.

The runtime purges it once the run is settled: after that run, its data folder went from about 1.7 GB back to about 1 MB. Neblex doesn't store your records; anything that must persist lives in your own database. Plan disk for the journal, roughly the size of the largest batch you expect to move.

How does the runtime allocate memory and disk?

It sizes itself to the deployment it runs in, rather than to a fixed setting. Installed as a service on a server it reads the machine's resources; in a container it reads the container's limits, not the host's.

Memory sets how much work it accepts. At start, the runtime checks the memory available to it, including a container's limit, and sets how many jobs it will take on. The limit scaled with memory: on the larger test machine it allowed several times as many jobs as in the 4 GB container, and capped at 1 GB it accepted no work at first. CPU didn't change it.
Disk holds what is in flight. It reserves disk for its run journal at start, from about 5 GB to about 25 GB depending on the deployment in our tests, and writes large in-flight batches there, so its memory stays flat while a slow destination catches up.
The journal is emptied as runs settle, so the disk use is temporary.

For sizing, our tests ran comfortably within 4 CPU cores and 4 GB of RAM, and we treat that as the practical minimum for a production install. Add disk for the journal as described above. Published minimum requirements for the runtime are being finalized by our engineering team.

What to plan for when you size it

Size for your heaviest flow, not the average. Everyday flows sit close to idle; matching, scoring and heavy transformation keep about one core busy and can approach 1 GB.
Plan disk for the run journal, roughly the size of the largest batch you expect to move.
Batch your writes. Throughput follows how the flow writes: one record per command waits on every round trip, while grouping rows per statement took a 1 million row load from an estimated four days to under seven minutes.
Account for record size. Our test records were about 275 to 290 bytes; larger records raise memory, journal size and run time in proportion.
Give it room. Capped at 1 GB, the runtime accepted no work at first; 4 CPU cores and 4 GB of RAM is a comfortable starting point.

Whatever on-prem integration agent you evaluate, measure its resource usage before and during a real job, and ask the vendor exactly what the agent sends out of your network.

Frequently asked questions

How much memory does an on-premises integration agent need?

It depends on the agent and the workload.

In our tests on synthetic and test data, the Neblex Worker used about 25 MB idle, 29 to 94 MB on average for 14 of 15 everyday flows, and at most about 450 MB while loading 1 million rows into a cloud database. Heavy duplicate matching peaked near 950 MB. Our tests ran comfortably within 4 CPU cores and 4 GB of RAM.

How much disk does an on-premises integration runtime need?

Enough for its run journal. Large in-flight batches are written there and purged once the run settles; a 5 million row batch took about 1.7 GB, roughly 350 bytes per row. At start the Neblex runtime also reserves journal space, about 5 GB to 25 GB depending on the deployment in our tests. Plan disk roughly equal to your largest batch.

How much CPU does an on-premises integration agent use?

Less than most people expect, because it spends most of its time waiting for the systems it connects. In our tests, everyday flows averaged 0 to 31% of one core and a 1 million row cloud load 12 to 26%. Heavy computation is the exception: fuzzy duplicate matching kept about one core busy.

How do I measure an agent's memory correctly?

Sample the process while a representative job runs. On macOS, use the process footprint, because resident memory leaves out compressed pages. On Linux in a container, look at anonymous memory in the cgroup's memory.stat, because the container total includes reclaimable file cache.

Sources

Neblex internal tests, October 2 to 8, 2026: synthetic data generated by Neblex, on Neblex hardware (a shared test machine, and a container capped at 4 CPU cores and 4 GB of RAM), with timings from the runtime's log and per-flow averages from the Neblex console.
Valkey documentation: Pipelining
Valkey documentation: Bulk loading
Linux kernel documentation: Control Group v2, memory.stat
ps(1) and footprint(1) manual pages.

Last checked: October 7, 2026.

Adan leads go-to-market at Neblex. Originally published at neblex.com.