Neon Expands Backend Primitives Database Authentication Storage Compute AI Gateway

We're doing something we've never done before. Firebolt is going open source. Not open-core. Not a free tier with strings. Open source - with a vision of becoming the project teams can build their most ambitious workloads on, AI-native or customer-facing. Data lakes were supposed to be open. But every real-time system that touched one quietly dragged in its own closed format, closed indices, closed ingest path. "Real-time" and "open" became a choice. We think that's a false choice. So our first move, validated by leaders at some of the world's largest tech companies: The world's first real-time data lake built entirely on open source technologies. → No engine-native formats → No proprietary catalogs → Built on DuckLake + Vortex + PostgreSQL + object storage → Engine-agnostic indices on top → Deployable anywhere — including GPUs What it actually unlocks: → Real-time ingest from Kafka straight into the lake - faster than any proprietary real-time system, because Vortex is the better format and our engine is built for it → Point DuckDB at the same lake, run vector search on a partition - still faster → Load the same lake onto a GPU for inference. No copies. No conversions. No re-ingestion. One lake. Many engines. CPU and GPU. Zero lock-in. Its not a pitch, its a philosophy. The real-time data lake belongs in the open. Curious how it works - especially the GPU side? Reach out and we can share more details. #Firebolt #OpenSource #RealTimeData #DataLake #AINative #DataEngineering #GPU

Breaking the KV-Cache Memory Wall: How to scale LLM context without blowing up your GPU bill. If you’ve ever tried to run production-level generative models over long context windows, you’ve run headfirst into a physical bottleneck: the Key-Value (KV) Cache. While compute demands stay flat during the decoding phase, memory scaling is brutally linear. Push into massive context sequences, and your GPU high-bandwidth memory (HBM) completely saturates, triggering latency spikes and Out-of-Memory (OOM) faults. The brute-force fix? Throw more expensive hardware at it.  The engineering fix? Implement a Heavy Hitters Oracle (H2O). In my latest write-up, I break down the bare-metal optimization layers of dynamic cache pruning. Runtimes don't need to retain every token history—they just need to preserve the critical "attention sinks" and heavy hitters. By strategically discarding low-contribution tokens, we can keep the memory footprint completely flat. What's inside the breakdown: 1. Hardware Topology: Tracing data movement from GPU registers to garbage collectors. 2. Pen-and-Paper Trace: A manual walkthrough of tracking running attention scores online. 3. System Stability: Why protecting the very first "anchor token" is non-negotiable to prevent model hallucinations. 4. PyTorch Implementation: A zero-dependency code block simulating driver-level tensor slicing. If you want to understand how modern inference engines achieve up to a 5x throughput boost under complex workloads, grab a pencil and jump in. 👇 The link to the full article (and a downloadable print-ready workbook PDF to trace the math yourself) is provided below! #MachineLearning #LLMOps #AIEngineering #DeepLearning #PyTorch #Transformers

💻 Tech Thursday: Experimenting with self hosted LLMS    With all the buzz around data sovereignty and LLM providers increasing their prices, open-source models are becoming more relevant. With open-source models, all data can stay on your local machine, and there are no extra costs for hosted compute.     Recently, Ollama even added integrations into widely used coding agents like Codex and Claude. So switching is as easy as three commands!      # install    curl -fsSL https://lnkd.in/ewrPbZ-D | sh      # pull a model    ollama pull granite4.1:8b      # use the model in your favorite agentic coding CLI    ollama launch codex     Unfortunately, we’ve experienced that the smaller models that can run on your laptop aren’t able to do reliable tool calling yet. So requesting them to inspect open code changes, for example, will unfortunately fail.     Still, it is amazing how easy it is to set up, and for general questions it works quite well!    Our next step will be experimenting with larger open-source models on hosted GPU’s!     We will definitely be keeping a close eye on developments in the open source LLM space and are curious to hear about other people’s experiences! 

Our inference latency spiked to 3s under real traffic. Team's first instinct: throw more GPUs at it. We doubled the budget. P99 latency: still 2.8s. The actual bottleneck turned out to be a 200ms synchronous database call in feature retrieval. Every prediction was blocking on a Postgres query that should have been cached at the edge. We fixed it with a Redis layer in front of the feature store. Latency dropped to 180ms. We kept the original GPU budget and gave the doubled headroom back. The uncomfortable thing about scaling ML in production: most of the cost problems are not model problems. They are the boring infrastructure underneath. Database calls, network hops, serialization overhead, retry storms when one downstream service blinks. The model wants to be fast. The system around it has to let it be fast. If your inference is slow, profile the system before you scale the model. The cheapest fix is almost never the one a vendor wants to sell you.

A Codex bug could age your SSD in 6 months. 640TB/year writes from local logs. 37TB of SSD writes in 21 days. A reported Codex logging issue was writing aggressively to a local SQLite database, annualized to roughly 640TB/year. The scary part is not just SSD wear; it is the pattern. As AI agents become long-running workers on our machines and inside companies, small background mistakes can silently become infrastructure problems. Autonomy without observability, caps, approvals, and kill switches is not productivity. It is operational risk with a better interface. #Codex #OpenAI #SoftwareEngineering #DevTools

🚀 Microsoft announced GPU acceleration for Fabric Data Warehouse. As an example, when running TPC-H 100 at high concurrency, the results are genuinely hard to believe , my first reaction was: "BS !!! This is physically impossible " Performance is an order of magnitude faster than the CPU equivalent I was fortunate to get early access and run some independent benchmarks. It is not every day that you witness a shift in technology this significant. "Any sufficiently advanced technology is indistinguishable from magic" Maraki Ketema #MicrosoftFabric #OLAP #GPU #DataWarehouse #Performance #SQL #CUDA #NVIDIA research paper : https://lnkd.in/gRyFhFgP forbes article :https://lnkd.in/gj46p2iw

Adam Szymański and Tyler Akidau have a paper accepted at VLDB 2026 on something that's been running in production in our Oxla query engine for two years and now underpins Redpanda SQL. As core counts push into the hundreds, intra-process shuffle becomes the bottleneck in parallel query engines. Batch partitioning materialises all intermediate data and introduces a global barrier with poor cache locality. Channel-based streaming avoids materialisation but synchronisation cost scales linearly with core count. Neither holds up on modern hardware. The ring-buffer shuffle uses lock-free atomic slot acquisition into fixed-size batch groups, achieving amortised O(1) synchronisation per batch and O(M) memory independent of input size, with no global barrier. On a 72-core GraceHopper it outperforms channel streaming by up to 44% and batch partitioning by up to 79%. At 192 cores on a dual-socket Graviton4 those gaps grow to over 100% and over 300% respectively. Worth noting: the paper also characterises where it doesn't win. On chiplet architectures with partitioned L3 caches the shared atomic counter becomes a cross-die bottleneck and channel streaming stays competitive. That kind of honest benchmarking is rare. https://lnkd.in/exFmcatn #DatabaseEngineering #QueryEngineering #ParallelComputing #SystemsResearch #VLDB

If you are into local LLMs, I think DwarfStar is a real sign of where this is heading. Local, consumer-run LLMs are not new. Ollama is a super successful project and I have been using it for about two years. Small models got genuinely useful too. Try Gemma 4, it is insanely good for its size and runs on pretty much any Mac. DwarfStar is the next level. It was built by Salvatore Sanfilippo, the guy who created Redis, and his approach is different from Ollama. He took an open source frontier model, DeepSeek V4, and took it apart into its building blocks. Then he quantized it cleverly: he shrank the less important parts (lower precision, smaller size) and kept the important parts intact. The result is a near-frontier model that runs on consumer hardware. The hardware will still set you back $7,000 or more, but before this it was not just expensive, it was impossible. With some governments restricting access to models, and open source steadily closing in on the closed labs, running a frontier-level model on your own machine is not a dream anymore. And the part I care about most: your data stays yours. See my detailed technical take in the post below.

As we are facing huge server issues due to 1000+ everyday incremental users, there are many features not performing well since today morning. The issue will be resolved by 5.00 pm today. Regards, Technical Team Richa AI.

🚀 Concurrency Series — Part 1 🧵 Concurrency vs ⚡ Parallelism One common misconception: Concurrency and parallelism are the same thing. They are not. ⚡ Parallelism means multiple tasks run at the same time. Example: If your machine has 16 CPU cores, it can run up to 16 CPU-heavy tasks truly in parallel. 🧵 Concurrency means many tasks are in progress, even if they are not all running at the same instant. Think of a web server. A request thread may: 🔹 Receive request 🔹 Validate input 🔹 Call another service 🔹 Wait for DB/network response While that thread is waiting on I/O, the CPU can run another thread. That is where concurrency helps. It keeps the CPU busy while some work is waiting. ✅ Works well for I/O-bound systems: 🌍 Web servers 🚪 API gateways 🗄️ DB clients 📩 Messaging systems But concurrency is not free. More threads can mean: 🧠 More memory 🔁 More context switching 📋 More scheduling overhead 🔒 More synchronization complexity 🐞 More race conditions / deadlocks Now think about CPU-bound workloads: 🖼️ Image processing 🎬 Video encoding 🤖 ML inference If you have 16 CPU cores, creating 100 compute-heavy threads usually will not make it faster. Why? Because the bottleneck is the CPU. Extra threads mostly compete for the same cores. 🎯 Key takeaway Concurrency is not about creating more threads. It is about keeping system resources busy without overwhelming them. ✅ I/O-bound? Concurrency helps. ✅ CPU-bound? Cores are the limit. ✅ Too many threads? Performance can drop. Next post: Redis event loop. 💬 Why does Redis perform so well without one thread per request? #Concurrency #BackendEngineering #SystemDesign #Parallelism