How we kept a large supplier catalogue import from taking Medusa down

The client problem

A supplier sends a catalogue file. It contains products, variants, prices, stock references and the inevitable inconsistencies that appear when data has crossed several systems before reaching commerce.

For the merchant, the request sounds simple: “Can we import this into Medusa?”

The difficult part is not reading a CSV. The difficult part is keeping the commerce platform available while the file is being checked, transformed and turned into real catalogue records. The import may take longer than an HTTP request. Some rows will be wrong. An operator will want to know whether the job is moving. If it stops, the team needs a way to continue without starting blindly from the beginning.

That was the client problem behind the import system we built for a B2B marketplace. Medusa gave us the commerce engine—the product, variant, price and inventory concepts—but it did not automatically provide the operating model required for large supplier files. We added that layer around it.

The result is not a clever CSV endpoint. It is a controlled journey from an external file to a trustworthy commerce catalogue.

The business problem hidden inside “import this file”

Large catalogue imports create four risks at the same time.

First, they compete with the storefront. If an import consumes all available memory, holds a database transaction for too long or monopolises the application process, shoppers and operators pay the price.

Second, supplier data is rarely ready to publish. A row can be syntactically valid and still be commercially unusable: a product may have no usable identifier, a variant may refer to a missing parent, a price may use the wrong currency or two rows may collide on the same SKU.

Third, an import is an operational process. Somebody needs to answer basic questions: Has the file been accepted? How many rows have been inspected? What failed? Can we stop the remaining work? Can we resume it?

Finally, the destination is not a flat products table. A Medusa catalogue is a graph of related commerce records. Products have variants; variants have prices and inventory relationships; marketplace products need seller ownership. Going faster is pointless if the fast path leaves that graph incomplete.

This changes the architectural question. Instead of asking, “How do we parse a large CSV?”, we ask, “How do we move a large, imperfect supplier catalogue into commerce without turning the import into a platform event?”

Why one synchronous request is the wrong unit of work

The most tempting implementation accepts a file, reads every row and creates every product before returning a response. It is attractive because the code looks linear. It is also the wrong contract for a long-running import.

A browser request has a short and fragile lifetime. A connection can close. A proxy can time out. A process can restart. None of those events should decide whether a catalogue exists.

We therefore separate acceptance from execution.

When the platform accepts a file, it creates a durable import record and hands the heavy work to a background workflow. The operator gets an import identity immediately. The system can now report status independently of the browser session that started the job.

That one decision produces an important business property: the import becomes manageable. It can take the time the data requires while the storefront continues serving its own traffic.

A four-stage journey from file to catalogue

The system is easier to understand as four successive stages.

1. Accept the file and create a durable job

The uploaded object and its import record become the stable hand-off between the user interface and background processing. The UI does not need to keep a request open, and the worker does not depend on a browser remaining connected.

This is also where access control matters. An operator must only be able to view or control imports that belong to the relevant seller context. Operational tooling is part of the product; it cannot be a hidden administrative shortcut around marketplace boundaries.

2. Parse the input and stage raw rows

The parser does not try to turn every line into a complete Medusa product immediately. It converts the file into durable staging rows.

In our implementation, staging is flushed to PostgreSQL in groups of 1,000 rows. The bulk insert uses PostgreSQL’s set-oriented capabilities rather than issuing one insert for every line. This reduces database round trips and, more importantly, gives the import a durable intermediate state.

Staging is not a technical waiting room. It is the place where the platform can preserve the supplier’s input, record row-level errors and decide what each row means before publishing it into the live catalogue.

3. Process pending rows in bounded batches

The product-creation phase repeatedly selects a limited number of rows that are still pending. In this platform, the processing chunk is 500 rows.

Each chunk is a deliberate unit of work. It is large enough to benefit from set-based database operations, but small enough to limit the impact of a failure. After a batch has been committed, progress is durable. The next batch does not need the full file in memory and does not reopen work already marked as created.

This is a crucial distinction: the complete import is not one enormous transaction. Each batch has its own transaction. If processing stops later, the system retains the completed work and the pending staging rows show what remains.

4. Build the complete commerce graph

The fast path still has to respect Medusa’s model.

Our bulk processing creates and links the records required for products, variants, prices, inventory and marketplace ownership. This is where commerce knowledge matters. A generic data pipeline can insert rows quickly; a commerce import must preserve the relationships that make products purchasable, priceable, searchable and attributable to the correct seller.

The business promise is not “we inserted data fast.” It is “the resulting catalogue behaves like a catalogue created through the platform’s normal domain model.”

Why PostgreSQL staging changes the operating model

Without durable staging, the parser and the product creator are tightly coupled. If product creation slows down, parsing has nowhere safe to put its output. If the process restarts, the platform may not know which rows were accepted, which were published and which still need work.

With staging, the system gains a ledger of the import.

Each row can carry its source position, current status and diagnostic information. The product workflow can ask for the next pending group instead of reconstructing state from an in-memory array. The operator interface can derive progress from persisted facts rather than an optimistic spinner.

This also creates a clean place for validation. Checks that apply to many rows—duplicate identifiers, missing relationships or invalid combinations—can be expressed as set-based database operations. The database is good at comparing sets. The application does not need to pull an entire catalogue into JavaScript simply to discover that two lines share the same business key.

For a merchant, this is the difference between an upload feature and an import capability. The former accepts bytes. The latter knows what happened to the data.

Progress that an operator can actually use

“Processing” is not a useful status when a job lasts long enough for someone to wonder whether it is stuck.

We persist progress during both staging and product processing. That allows the interface to show a meaningful lifecycle: the file has been received, rows are being staged, catalogue records are being created, some rows have failed or the job has completed.

Row-level diagnostics are equally important. A supplier should not have to correct an entire file because a small number of lines are invalid. The useful output of validation is a repair list: which row failed, why it failed and what the operator can change.

This is not merely better UX. It reduces the operational cost of every catalogue update. The team can act on a bounded set of problems instead of treating the import as a black box.

Cancellation means “stop safely,” not “pretend nothing happened”

Long-running jobs need a stop control. But cancellation has to match the way work is committed.

Because this import commits one batch at a time, cancellation stops further eligible processing while preserving the work already completed. The operator sees a stable state: completed batches remain in the catalogue, pending work remains visible and no new batch is launched.

That contract matches the operational decision the person is trying to make. They can stop the remaining workload, inspect the import record and decide whether to correct the source data, continue later or manage the products that have already been created.

The button is useful because its meaning is precise.

Resume from durable work instead of restarting blind

The same staging ledger supports recovery.

The seller-scoped resume route re-enqueues processing for an import and selects rows that remain pending. Rows already marked as created are skipped. That means an interruption does not force the operator to re-upload the file and hope duplicate detection catches everything.

The operator does not have to reconstruct the job from memory. The platform remembers completed and pending units of work and can continue from that state. Recovery becomes part of the normal import lifecycle instead of an emergency database exercise.

Put a clear boundary around every heavy stage

Staging and product processing are bounded: rows are flushed and consumed in explicit chunks. This protects the application from carrying the full catalogue through the workflow.

File acquisition is a separate architectural choice. A platform can stream a very large remote file into object storage before parsing, or accept a simpler buffered path when supplier files are contractually capped. What matters is making that input contract explicit instead of allowing an unknown file size to become an unknown infrastructure risk.

Once the file is in controlled storage, the remaining stages can operate against durable references and bounded row groups. The import then has a measurable capacity model from acquisition through catalogue publication.

What Medusa gave us—and what we added

Choosing Medusa was an architectural decision because it gave the project a coherent commerce engine without dictating the complete operational product around it.

We reused Medusa’s product, variant, pricing, inventory and workflow foundations. Around those foundations, we added the capabilities the client’s supplier model required:

• durable import sessions;
• streamed or bounded parsing paths;
• PostgreSQL staging records;
• bulk, set-based catalogue creation;
• marketplace ownership links;
• visible progress and row-level errors;
• seller-scoped cancellation and resumption.

This is the value of an extensible commerce platform. We did not have to rebuild checkout and catalogue fundamentals in order to solve a very specific B2B data problem. Nor did we have to force that problem into a generic import screen designed for smaller, cleaner files.

A practical decision framework for your own imports

Before implementing a large catalogue import, answer these questions in order.

1. What keeps the storefront safe? Decide where CPU, memory and database capacity are bounded, and whether heavy work runs separately from customer-facing requests.
2. What is the durable hand-off? Define when an upload becomes an import job that can survive a disconnected browser or restarted process.
3. What is staged? Preserve enough source data and row identity to validate, diagnose and resume work.
4. What is the batch boundary? Choose a measurable unit that limits transaction duration and failure impact.
5. What makes a row publishable? Separate file syntax from business validation and catalogue-domain rules.
6. What does progress mean? Report persisted facts—accepted, staged, processed, failed—not just activity.
7. What exactly do cancel and resume promise? Describe their semantics in terms an operator can safely act on.
8. How is the full commerce graph created? Verify products, variants, prices, inventory and ownership links together.

The answers will vary by catalogue and infrastructure. The sequence should not.

The larger lesson

The most important move was not a particular batch size or SQL statement. It was turning a fragile upload request into an observable operating process.

Medusa provided the commerce model. We extended it with a durable import system shaped around the client’s supplier reality: files that take time, data that needs review, operators who need control and a storefront that must remain available.

That is how we approach commerce architecture at WeAreSouk. We start with the business problem that the platform does not solve automatically. Then we design the missing capability so that it fits the commerce model, the operational team and the failure modes of the real world.

If catalogue onboarding is becoming the bottleneck in your Medusa programme, we can map the import journey—from supplier file to published product—and identify where staging, validation, operator controls or domain integration need to change.