Applying LLMs row by row to business data, with structured output and every failure marked
A pipeline step that runs a chosen language model over every row of a table, returns typed fields that match a schema the user defines, and marks each row as answered or failed.
- Partner
- A multinational advisory firm
- Period
- October 2024 to January 2026
- Model providers
- 5
- Five of the gateway's six: OpenAI, Anthropic, Google, Groq and xAI
- Rate limits tracked
- 2
- Requests and tokens per minute, per provider
- Status
- Every row
- Answered or failed, with the reason
The challenge
Teams wanted to classify, extract and summarize information across thousands of records. Free-text model answers could not be fed reliably into reports, databases or other systems.
What we built
An analysis step that sends each row to a selected model with a templated prompt and requires a structured answer. It paces calls by each provider's request and token limits, retries transient failures with backoff, and adds the generated fields to the original data as typed columns, with a status column on every row.
What was delivered
- Output schema defined by the user, with configuration checked before a run starts
- Pacing by each provider's request and token limits
- A status and error on every row, so a fallback value is never mistaken for an answer
Partner background
Our partner is a multinational advisory firm whose teams work with large volumes of documents, spreadsheets, databases and email. It wanted one internal platform where those teams could build data pipelines and AI agents themselves, instead of commissioning a new application for each need. The platform had to run inside the partner's Microsoft 365 and Azure environment, keep each team's work separate, and move work from experiment to production through controlled environments. This case study covers one part of that platform; nine case studies describe the whole.
The challenge
Free text does not fit a table
Model responses vary in shape and wording. Without a fixed structure, results need manual cleanup before anyone can sort, filter or load them.
Volume meets rate limits
Running a model over a large table means many calls in a short time. Providers limit both requests and tokens per minute, and individual calls fail for transient reasons.
Silent failures
If a failed row is filled with default values and nothing marks it, a report built on the table cannot tell a real answer from a placeholder.
Mistakes surface late
A prompt that references a missing field, or output fields that clash with existing columns, would otherwise fail halfway through a long and costly run.
Objectives
- Return model results as typed fields that downstream steps can rely on
- Stay within each provider's request and token limits
- Stop on a failed row by default, or contain the failure to that row and mark it
- Catch configuration errors before any model call is made
- Allow the model and provider to be chosen per step
Our role
CharCentric provided technical leadership and architecture within a multidisciplinary engineering team, and contributed directly to implementation. The platform was built over 16 months, from October 2024 to January 2026, as a Python and FastAPI backend on Azure.
Scope and timeline
The analysis step was developed between November 2024 and October 2025 as part of the platform's pipeline library, alongside its configuration model, prompt validation and logging.
Approach
Schema first
The user defines the output as a JSON Schema. The step converts it into a typed model and asks the model provider for structured output, so every answer either matches the schema or is treated as a failure.
Validate before spending
Before the first call, the step checks that prompt variables exist in the data, that templates are valid, and that generated field names do not collide with source columns. It also shows an estimate of the run's token use and cost.
Pace by the provider's limits
A limiter tracks both requests and tokens per minute for each provider and adjusts to the limits a provider reports in its response headers where it sends them, using configured limits otherwise. Calls route through the platform's model gateway. Concurrency is a separate setting, so teams can run many rows at once without exceeding a token budget.
Implementation
Prompt templates
Prompts combine fixed values with fields from each row, including nested fields. The same template runs across the whole table, which keeps results comparable.
Batch processing
Rows are processed in asynchronous batches. Transient failures are retried up to three times by default, with exponential backoff and jitter, honouring any Retry-After the provider sends. Every retry is logged.
Failure policy
Every output row carries a status and an error column. By default, a row that still fails stops the run with a clear error. With tolerate failure on, the row receives default values, is marked failed, and the run continues.
Output
Generated fields and the status columns are merged with the original row into a new typed table that writers, joins and other steps can use directly, including to filter out failed rows.
Tools and technologies
| Tool | Purpose |
|---|---|
| LangChain | Model initialization and structured output |
| Pydantic | Typed models generated from JSON Schema |
| OpenAI, Anthropic, Google, Groq, xAI | Selectable model providers |
| asyncio | Concurrent batch processing |
What was delivered
- Providers
- 5
- Rate limits tracked
- 2
- Retries by default
- 3
- Status column
- Every row
- A reusable analysis step available to every pipeline on the platform
- Typed, schema-checked output instead of free text
- Pacing by request and token limits, with retries and backoff
- A status on every row, and configuration errors caught before any cost
Why it matters
Language models become useful in operations when their output can be trusted by the next system in the chain, and that includes knowing which rows the model did not answer. Treating structure, pacing and failure as design decisions turns a model call into a dependable data step.
If your organization is planning a platform of this kind, or needs a specific part of one designed and delivered, we would be glad to discuss it.