If you're new to Neon Postgres, I've put together a quick little explainer video on what exactly "Serverless Postgres" means.

Post #4 in my observability series is live.   Last week: We have seen how bad observability impacts organization.   This week: Let's look at various key components of observability.   In this blog, I have tried to cover three key pillars and a key fourth pillar "Event".  https://lnkd.in/e-7fQwKN Comment if you see any other factor!!   #Observability #SRE #DevOps #PlatformEngineering #EngineeringLeadership Minutus Computing

We scaled a FastAPI service from 2 pods to 8.   Throughput didn’t change. No errors. 😑   No alerts.  🤔 Just… no improvement. 🥲 The issue wasn’t Kubernetes. It was one early decision: synchronous database calls where async should have been used. But this isn’t an “always use async” post. Both have a place. Knowing when to use each is what separates systems that scale from ones that quietly bottleneck. Have you run into this before? https://lnkd.in/eVhQyBPW

S3 as a File System? The Gap Just Closed ☁️📂 For years, we’ve managed a "data friction" tax: the gap between S3’s massive scale and the local Linux file systems that many of our favorite tools require. We built complex pipelines, duplicated data to NFS filers, and managed the overhead just to keep our legacy and high-performance apps running. That era is ending. In this week's Data Engineering Weekly, we dive into a major shift in distributed systems: S3 Files. As Werner Vogels recently highlighted, S3 is evolving to provide native NFS access, allowing you to mount buckets directly with sub-millisecond latency. Why this changes the game for Data Engineers: 🔹Native NFS Access: Applications can now work with S3 data directly using existing tools—zero code changes required. 🔹High Performance Caching: By using a two-tier model, small active files are cached on EFS infrastructure for speed, while large reads stream directly from S3. 🔹The Stage & Commit Model: Instead of a PUT for every byte, changes are batched every ~60 seconds. This "Git-like" approach slashes S3 request costs and prevents versioning bloat. 🔹S3 as the Source of Truth: Even with simultaneous API and NFS modifications, S3 remains authoritative, with conflicts safely moved to a lost+found directory to prevent data loss. Whether you are running AI/ML training, processing genomics data, or migrating legacy apps, the ability to use S3 as a shared file system at scale is a massive win for infrastructure simplicity. Read the full deep dive in Data Engineering Weekly: 👉 https://lnkd.in/gaTf3W5r Inspired by Werner Vogels’ latest insights on the changing face of S3: 🔗 https://lnkd.in/gwUewJ-a Which of your legacy pipelines will you simplify first with native NFS access? Let’s discuss in the comments! 👇  #DataEngineering #AWS #S3 #DistributedSystems #CloudInfrastructure #SRE #DataEngineeringWeekly

I recently worked on optimizing a backend system where API latency had become a serious bottleneck. At peak load, response times were inconsistent — affecting user experience and overall system reliability. Instead of scaling blindly, I focused on identifying the real issues: • Inefficient database queries (missing indexes, heavy aggregations) • Repeated calls to external services • Lack of caching for frequently accessed data Here’s what I did: → Optimized MongoDB queries and added proper indexing → Introduced caching for high-read endpoints → Reduced unnecessary external API calls → Refactored parts of the service for better separation of concerns Result: ~40% reduction in latency and much more stable performance under load. One thing I’ve learned: Scaling isn’t always about adding more resources — sometimes it’s about fixing what’s already there.

If you want to learn how databases really work behind the scenes, check out this free YouTube series by Ben Dicken He has covered the entire "Database Internals" book videos where you will learn: • How databases store data using B-Trees • How storage engines work • LSM Trees and write optimization • Distributed systems and leader election • etc Over 10 hours of free content. Highly recommended for anyone into backend development. checkout: https://lnkd.in/gNNSbeWZ #Databases #BackendDevelopment #SoftwareEngineering

AWS shipped something last week that I think is a bigger deal than people are giving it credit for. It's called S3 Files. The short version: you can now mount any S3 bucket like a regular file system. ~1ms latency on cached data, the same bucket reachable through both the S3 API and POSIX, and it's already live in 34 regions. The reason this matters is that most tooling — data science notebooks, build systems, log processors, even the bash scripts you wrote five years ago — expects a normal file system. Until now you'd copy data out of S3 onto EFS or a local disk, run your job, and copy it back. S3 Files just deletes that step. And if you can co-locate it next to compute, there's no data egress at all. One nice side effect: agents get easier. A coding agent already knows ls, cat, and grep, so instead of writing a custom S3 tool wrapper, you can point it at a mount path and let it work the way it already wants to. The bigger thing I'd zoom in on is the pattern. Object storage stays the source of truth, and a faster layer sits on top as a cache. We use the same shape at Mixpeek for vectors — S3 Vectors underneath, Qdrant on top, retrieval pipeline on top of that. Same thesis, applied to two different abstractions: object storage is truth, the fast layer is a cache. mixpeek.com #AWS #S3 #CloudInfrastructure #DataEngineering #AIInfrastructure #ObjectStorage

𝙀𝙫𝙚𝙧 𝙏𝙝𝙤𝙪𝙜𝙝𝙩 𝘼𝙗𝙤𝙪𝙩 𝙃𝙤𝙬 𝙄𝙣𝙨𝙩𝙖𝙜𝙧𝙖𝙢 𝙃𝙖𝙣𝙙𝙡𝙚𝙨 𝙈𝙞𝙡𝙡𝙞𝙤𝙣𝙨 𝙤𝙛 𝙇𝙞𝙠𝙚𝙨 𝙖𝙩 𝙩𝙝𝙚 𝙎𝙖𝙢𝙚 𝙏𝙞𝙢𝙚? You tap ❤️ on a post. At the exact same moment, thousands of others do the same. Yet somehow… 👉 The like count updates smoothly 👉 The app doesn’t crash So what’s happening behind the scenes? Here’s a simplified view: 1️⃣ Distributed Architecture Instead of a single server, the system runs across multiple machines, sharing the load of incoming requests. 2️⃣ Load Distribution Traffic is intelligently routed so that no single server gets overwhelmed during spikes. 3️⃣ Data Partitioning (Sharding) User and post data are divided across multiple databases to scale efficiently with massive traffic. 4️⃣ Eventual Consistency The like count might not sync globally at the exact same moment… 👉 But it becomes consistent within milliseconds. 5️⃣ In-Memory Caching Hot data like like-counts are stored in fast systems (like Redis) for quick reads and updates. 6️⃣ Asynchronous Processing (Queues) Instead of processing everything instantly, actions are queued and handled reliably in the background. 💡 The real takeaway: It’s not about processing one request instantly… 👉 It’s about handling millions of actions reliably without breaking the system. That’s where solid system design makes all the difference. #SystemDesign #DistributedSystems #Scalability #BackendEngineering #SoftwareEngineering #TechExplained

Most “Simple” APIs Fail the Same Way, Here’s What Actually Breaks First I took a minimal Flask API (one endpoint + PostgreSQL) and load tested it with k6 until it started to crack. The goal wasn’t to build a high‑performance system, but to understand how a typical service degrades under load and what scaling actually means. Setup: • Flask app with Gunicorn (4 workers) • PostgreSQL with a single indexed table • k6 running on a local machine • One endpoint: GET /data → simple SELECT query What Broke First (and Why): • CPU saturation - At ~350 req/s, CPU spiked to 100%. Response times increased linearly, why because the API logic itself request parsing, JSON serialization became the bottleneck before the database even saw the traffic. • Database contention - After increasing workers, the bottleneck shifted. PostgreSQL showed high wait events and connection queueing. Why? because Concurrent connections overwhelmed the database’s pool. Even simple reads slowed down under concurrency. Tradeoffs I Observed: • Caching → reduced DB load by ~70% but added ~200MB memory overhead • More workers → improved throughput until CPU became the limit again • Database indexes → lowered per‑query latency but didn’t solve concurrency spikes Have you load tested your own services? What was the first bottleneck you hit? #BackendEngineering #DevOps #SystemDesign #Performance #LoadTesting

🚀 HOW TO TAMING S3 SHUFFLE AT SCALE In the latest edition of Data Engineering Weekly, we break down the Zero-to-One Guide to optimizing your data pipelines. 🛠 What’s Holding Your S3 Shuffle Back? 🔹Over-reliance on Local Disk: Traditional shuffle relies on local storage, which leads to "Disk Full" errors and prevents independent scaling of compute. 🔹Ignoring S3 Request Limits: Without proper partitioning and pacing, you’ll hit S3 503 Slow Down errors. 🔹Poor Data Serialization: Inefficient formats like Java default increase payload size, driving up network I/O costs and slowing down transfers. 🔹Lack of Adaptive Execution: Static partitioning fails with data skew. You need Adaptive Query Execution (AQE) to handle skewed keys and coalesce small partitions. 🔹Neglecting Cost Monitoring: API calls (PUT/GET) can cost more than the storage itself. Without lifecycle policies to clean up temporary files, your AWS bill will snowball. 💡 Deep Dive: This breakdown was inspired by the insightful work of Manoj Babu and Meghanath Macha in their recent article on optimizing S3 shuffle performance. 🔗 Read the full breakdown here: https://lnkd.in/gPmjEuDv #DataEngineering #AWS #S3 #BigData #CloudComputing #Spark #DataArchitecture #DataEngineeringWeekly