- A supplier catalogue can be technically imported and still be commercially unusable.
- The products exist. Their SKUs, descriptions and specifications are present.
- Manual classification is accurate but slow.
- For a multi-vendor procurement platform built on Medusa, we needed a third option.
The client problem
A supplier catalogue can be technically imported and still be commercially unusable.
The products exist. Their SKUs, descriptions and specifications are present. But thousands of items have no reliable place in the navigation. A maintenance buyer looking for a valve, a circuit breaker or an appliance part cannot discover it through the categories they understand.
Manual classification is accurate but slow. Sending one synchronous AI request per product looks faster, until browser timeouts, rate limits, partial failures and inconsistent answers turn the operation into a support problem. Letting a model invent categories is worse: the catalogue appears organised while its taxonomy quietly fragments.
For a multi-vendor procurement platform built on Medusa, we needed a third option.
We built an asynchronous categorisation pipeline that uses the live catalogue taxonomy as its boundary. Elasticsearch retrieves plausible categories for each product. Those candidates become a constrained OpenAI Batch request. The request artefact is streamed through object storage instead of held in application memory. Results are streamed back, validated against the original candidates and applied in bounded groups. Exceptions remain visible and retryable.
The AI does the repetitive classification work. Medusa remains the authority over products, categories and publication.
The client problem was discovery, not artificial intelligence
The business objective was simple: make a newly imported supplier catalogue browsable.
Procurement catalogues are especially difficult to classify. Titles are terse. Supplier terminology varies. A product called a “kit” may belong under plumbing, appliance repair or access control depending on the specifications. Manufacturer descriptions often assume knowledge that a buyer does not have.
The customer did not need an AI demo. They needed products to land in useful, existing categories so buyers could find them and operators could review the uncertain cases.
That framing changed the architecture.
We did not ask a model, “Which category would you create for this product?” We asked the platform first, “Which categories from our approved taxonomy could plausibly fit?” The model’s job was to decide among those candidates or decline.
This makes AI a component inside catalogue operations rather than a parallel source of truth.
Why a synchronous loop was the wrong shape
A synchronous categorisation endpoint couples too many lifecycles.
The browser waits for the backend. The backend waits for search and the model provider. One slow response holds the entire request. A network interruption leaves the user unsure whether work continued. Retrying the page can start the same products again. Progress exists only in logs or in the number of calls that happened to return.
Large catalogue work needs a durable identity.
The supplier launches a categorisation session. The platform either creates a new active session or returns the one already running. The request receives a tracked identifier and status URL, while a Medusa workflow continues the operation asynchronously.
The session exposes phases such as candidate enrichment, batch preparation, provider submission and result processing. The supplier sees accepted work rather than a spinner attached to one HTTP connection.
This distinction is fundamental: accepting the job is not the same as completing the catalogue.
One active session protects the supplier from duplicate work
Double-clicks, refreshed pages and impatient retries are normal user behaviour. They should not create competing AI runs over the same seller catalogue.
Before launching work, the platform cleans up stale sessions, takes a seller-scoped PostgreSQL advisory lock and finds or creates the active categorisation session inside one controlled transition.
If a session already exists, the route reuses it and does not launch a second worker. If no uncategorised products remain, it reports that there is no work. If a new session is created, only then does the workflow start.
This gives the product a clear promise: one supplier sees one current categorisation journey.
The implementation also caps each run. A large backlog is worked through in deliberate batches, and the dashboard reports how many eligible products remain. The platform does not pretend that one enormous provider request is the only way to process a very large catalogue.
Bounded runs make progress, retry and operational recovery easier to understand.
Start from canonical product information
Classification quality depends on the product evidence supplied to the model.
The pipeline reads title and description from Medusa, then resolves structured specifications from the platform’s product-attribute links. Legacy specification metadata is used only as a fallback when canonical attribute records are unavailable.
This matters because supplier imports rarely produce uniform descriptions. A title alone may be ambiguous. Attributes such as voltage, material, connection type, dimensions or appliance compatibility often contain the clue that separates two neighbouring categories.
Using canonical attributes also keeps categorisation aligned with the data buyers will later filter and compare. The AI is not analysing a hidden prompt-only representation of the product. It is working from the same structured product knowledge the commerce platform owns.
The pipeline therefore begins by assembling a reliable product description, not by sending whatever text happens to be easiest to fetch.
Retrieve candidates before asking the model
For each product, Elasticsearch searches the existing category catalogue.
The search uses the product name, description and specifications to retrieve plausible candidates. Returned category handles are mapped back to live Medusa category identifiers. Candidates that no longer exist in Medusa are discarded before the model sees them.
This retrieval step solves two different problems.
First, it narrows the decision. The model compares a relevant shortlist instead of reasoning over an entire taxonomy or free-form category space.
Second, it grounds the answer. Every acceptable output points to a category the platform already knows.
If search produces no usable candidate, the product is sent to review with a specific reason. The system does not ask the model to improvise merely to increase an automated success counter.
This is the core safety contract: retrieval proposes the allowed vocabulary; the model ranks or rejects it; Medusa validates the final identifier.
Process products with stable pages and bounded concurrency
Candidate retrieval is itself a large workload.
The pipeline reads products in pages using a stable identifier cursor. That choice matters because processing changes each product’s import status. Offset pagination over a shrinking result set can skip rows as completed products disappear from the query. A keyset cursor continues from the last identifier and remains stable while statuses change.
Within each page, Elasticsearch searches run with bounded concurrency. This keeps the operation moving without opening an unbounded number of search requests. Each product receives either a stored candidate set or a reviewable failure state such as no candidates or a search error.
The platform updates product metadata in bulk after the page instead of writing one record for every small step.
For the operator, these mechanics translate into a more valuable outcome: every product selected for the run eventually has a visible state. A search error does not silently disappear, and a product with no candidate is distinguishable from one that has not yet been processed.
Stream the JSONL artefact to durable storage
OpenAI Batch accepts a JSONL file containing one request per product.
The dangerous implementation is to build every line in a JavaScript array, join the array into one giant string and then turn that string into an upload object. The application process becomes the temporary owner of the entire artefact, exactly when catalogue size makes memory most precious.
Our pipeline opens a Medusa Cloud object-storage upload stream and writes each request line as products are prepared. It records the resulting bucket, object key, byte count and product count on a pending batch job.
The JSONL contains a stable product identifier, the selected model and a prompt built from the retrieved categories. The complete file lives in durable object storage, not in workflow state or one oversized in-memory value.
This separation also improves recovery. Preparing the requests and submitting them to the model provider are different stages with a persistent artefact between them.
Stream from storage to OpenAI Batch
A separate submission job claims pending categorisation artefacts.
It uses database locking to ensure workers do not submit the same pending job concurrently. Stale upload locks can be recovered, submission attempts are counted and retry state is written back to the job.
The worker opens a read stream from object storage and sends it to the OpenAI Files API through a streaming multipart request. This avoids an SDK conversion path that would first materialise the async stream as a large Blob inside Node.
Once the file has an OpenAI identifier, the worker creates a Batch request against the Responses endpoint and stores the provider batch ID. If a later submission attempt begins after the file upload succeeded, it can reuse the recorded input file instead of uploading the artefact again.
For the business user, this is invisible—and that is the point. A long-running catalogue operation should not require the supplier to understand storage, multipart bodies or provider file identifiers. Those controls exist so the visible workflow remains dependable.
Completion is an ingestion workflow, not a notification
When the provider finishes, a signed webhook or a polling job resolves the batch state.
A completion event is not treated as proof that the catalogue has changed. It triggers a dedicated Medusa ingestion workflow. Already completed jobs are skipped, while failed, cancelled or expired batches receive their own terminal state.
The ingestion workflow downloads the output as a stream and parses it line by line. Results are accumulated only into bounded buffers before database writes.
Each response is connected back to its product through the stable request identifier. The platform then reopens the product’s stored candidate set and validates the model’s recommendation.
This is where the provider’s answer becomes—or fails to become—a commerce decision.
The webhook says the external computation finished. The ingestion workflow decides what the platform is willing to accept.
Reject categories that were never offered
A model can return syntactically valid JSON and still produce an invalid business answer.
The returned category may not be present in the product’s retrieved candidate set. It may refer to an identifier that looks plausible but does not belong to the live taxonomy. The model may explicitly say the product is uncategorised because the source data conflicts or because none of the candidates fits.
The ingestion path treats those outcomes differently.
A recommendation is accepted only if its identifier appears in the stored candidate set. A nonexistent or out-of-set ID becomes a reviewable failure. A deliberate no-match answer preserves the model’s reason and confidence without creating a category link. Malformed lines are counted rather than allowed to terminate all useful results around them.
Valid assignments update product metadata and create the Medusa category relationship with conflict-safe insertion.
The model is allowed to choose. It is not allowed to enlarge its own authority.
Partial failure should produce a useful queue
Large AI operations will contain exceptions.
Some products have poor descriptions. Some have contradictory specifications. Search may find no candidate. A provider response may be malformed. A category may disappear between request preparation and result ingestion.
The pipeline preserves those distinctions in product and job state. The dashboard can separate “fix the source data” from “no taxonomy match,” a search failure or an invalid model response.
Successful assignments remain useful. Failed products can be reset for a targeted later run without erasing the categories already applied elsewhere.
This is better than two common extremes: rolling back an entire batch because one item failed, or declaring the batch successful while silently dropping the difficult rows.
The operator gets a smaller, intelligible review queue. Automation handles repetition; people spend attention where judgement or source correction is genuinely required.
AI classification does not equal publication
When a valid category is applied, a product that was not already live moves into a proposed state rather than being automatically published.
That preserves an important boundary.
Categorisation answers where a product belongs. Publication answers whether the complete commercial record is ready for buyers. Images, pricing, stock, descriptions, supplier approval and compliance may still require review.
The platform can therefore accelerate one expensive catalogue task without giving the model authority over the whole product lifecycle.
This is also why human-approved categories deserve protection during later processing. Automation should reduce unresolved work, not overwrite decisions that an operator has already validated.
The detailed human-authority contract belongs in a separate article, but the batch architecture respects it from the start.
Why Medusa was the right foundation
Medusa owns the products, category taxonomy, product-category relationships and publication states. Elasticsearch provides a searchable projection of that truth. OpenAI Batch supplies asynchronous inference over a constrained decision set.
Each component has one clear responsibility.
We did not move the catalogue into an AI service. We extended the Medusa product lifecycle with a specialised classification pipeline. Candidate retrieval begins from the live taxonomy. Every accepted result returns through Medusa validation. The final relationship is a normal category link that the rest of commerce can consume.
This is the architectural advantage of an extensible commerce engine: advanced AI capability can be added without surrendering ownership of the business model.
The model assists the catalogue. It does not become the catalogue.
A practical batch-categorisation checklist
Before categorising a large catalogue with AI, define these controls.
- Does each run have a durable session and visible progress?
- Can duplicate launches reuse one active seller job?
- Is the run capped so work, files and recovery remain bounded?
- Does the prompt use canonical product attributes as well as titles?
- Are candidate categories retrieved from the live taxonomy first?
- Can the model return only a candidate ID or an explicit rejection?
- Is the JSONL streamed to durable storage rather than assembled whole in memory?
- Can submission retry without rebuilding or re-uploading completed stages?
- Are result files ingested as a stream in bounded database batches?
- Is every returned category validated against the original candidate set?
- Do partial failures become a review queue with meaningful reasons?
- Does final product publication remain a separate human-controlled decision?
These controls turn generative output into an operable catalogue capability.
The broader lesson
The client problem was not “How can we use AI?”
It was “How can suppliers make a large imported catalogue discoverable without asking operators to classify every product by hand?”
We solved it by keeping the AI inside a narrow, evidence-based role.
Elasticsearch retrieves categories that really exist. OpenAI Batch makes the comparison asynchronously. Object storage carries the artefact without making the application own the whole file. Medusa validates every returned identifier, records the exceptions and preserves the product lifecycle.
The result is not uncontrolled automatic taxonomy. It is a catalogue operation that can process large backlogs, explain what happened and direct human attention to the products that genuinely need it.