How 100K+ Order Exports Were Silently Killing Our Logistics SaaS Database — And How I Fixed It

When we first built order exports, it worked simply: user clicks export → backend queries orders → generates Excel → sends the file back. Perfect for a few hundred orders.

Then our customers scaled. Exports of 50K, 80K, eventually 100K+ orders. That's when everything broke.

Three failures hit at once. Exports took 15–20 minutes to process — but HTTP connections don't wait that long. The browser timed out, the user got an error, and received nothing. Worse, the backend kept working anyway, burning CPU and RAM on a job nobody would ever receive. And loading 100K orders into memory at once caused RAM spikes that crashed the server when multiple users exported simultaneously.

Users were stuck staring at a browser tab for 15 minutes, only to get a timeout error.

The Background Processing Solution: Decoupling Requests

When a user clicks export now, the backend does one thing pushes a job to a message queue and returns "queued" in milliseconds. The user is free to keep working immediately.

A dedicated worker picks up the job in the background, fetches orders in small chunks instead of all at once, builds the Excel file progressively, then uploads it to cloud storage. When it's done, the user gets a notification with a download link. The file sits in storage ready to download whenever they want.

The HTTP request is done in milliseconds. The heavy work runs quietly in the background, completely free from any timeout constraint.

But Then the Database Broke

The background worker was now successfully processing 100K+ order exports, but it was hammering the main database to do it. Chunked processing meant dozens of sequential batch queries per export. Multiple exports running in parallel meant dozens of concurrent batch queries hitting the same database that every other part of TechShip depended on.

Shipment lookups slowed down. Tracking updates lagged. Webhook processing queued up. The export worker was starving the rest of the platform.

The fix was straightforward: spin up a read replica dedicated exclusively to export workers. The primary database handles all writes and real-time reads for the live application. The replica handles nothing but export queries, isolated, so a heavy export job can run as many batch queries as it needs without touching production traffic at all.

One replica. One job. Zero impact on the rest of the platform.

Is your SaaS doing this?

Slow exports killing your database?

Background jobs competing with live traffic is a silent killer. Catch it before your users do.