A pipeline engine that runs, pauses and resumes data workflows
The orchestration core of a low-code platform, running visually connected steps in dependency order with breakpoints and resumable runs.
- Partner
- An international consulting group
- Period
- October 2024 to January 2026
- Step types
- 13
- Readers, transformers, aggregators, LLM analysis and writers
- Ways to run
- 3
- On demand, on a schedule or as a published endpoint
- From saved outputs
- Resume
- Partial re-runs without repeating earlier steps
The challenge
The platform let users connect data steps visually. Something had to run those graphs correctly, with branches and merges, and let users inspect and repeat parts of a run while building it.
What we built
An orchestration engine that runs each step once all of its inputs are ready, stores every step's input and output with its schema, supports breakpoints, and can rebuild typed data from a previous run to resume without repeating earlier steps.
What was delivered
- Fan-out and fan-in across steps, run in dependency order
- Breakpoints and per-step control over what is logged
- Runs started on demand, by cron schedule or through a published endpoint
Partner background
Our partner is an international consulting group whose teams work with large volumes of documents, spreadsheets, databases and email. It wanted one internal platform where those teams could build data pipelines and AI agents themselves, instead of commissioning a new application for each need. The platform had to run inside the partner's Microsoft 365 and Azure environment, keep each team's work separate, and move work from experiment to production through controlled environments.
The challenge
Graphs, not lists
Users connect steps freely. A join waits for two inputs, and one output can feed several writers. Running steps in the order they were drawn would produce wrong results.
Building is iterative
While designing a pipeline, users need to stop at a step, inspect the data and change something, without paying for every earlier step again.
Sensitive data in logs
Some steps handle data that should not be stored in run logs.
Objectives
- Run any valid graph of steps in correct dependency order
- Support breakpoints for inspection during design
- Resume runs from saved outputs
- Control logging of inputs and outputs per step
- Run pipelines on demand, on a schedule or as an endpoint
Our role
CharCentric provided technical leadership and architecture within a multidisciplinary engineering team, and contributed directly to implementation. The platform was built over 16 months, from October 2024 to January 2026, as a Python and FastAPI backend on Azure.
Scope and timeline
The orchestration engine was built between October 2024 and May 2025. Run controls, triggers and publication followed between January and November 2025.
Approach
Count inputs, release when ready
The engine counts the incoming connections of every step. Steps with none start first. Each time a step finishes, its outputs are passed on and the waiting count of each target drops. A step runs when its count reaches zero.
Store data with its shape
Every step output is stored with a JSON Schema of its data. When a run resumes, the engine rebuilds typed models from the stored schema, so resumed steps receive the same data types as in the original run.
One interface for every step
All steps share one interface and are created by a factory. A scaffolding command generates the class, input model, state and registration for a new step.
Implementation
Step library
Thirteen step types in five categories: readers for APIs, CSV, SQL databases, documents, Outlook and SharePoint; a Python code transformer; join and append aggregators; an LLM analysis step; and writers for SQL databases, Outlook and SharePoint.
Run controls
Breakpoints stop execution at chosen steps. Input and output logging can be disabled per step. Runs can be cancelled while in progress.
Triggers
Pipelines run on demand, on cron schedules managed in batches per pipeline, or through published endpoints tied to an environment.
Tools and technologies
| Tool | Purpose |
|---|---|
| Python and asyncio | Orchestration engine |
| Pydantic | Typed data between steps |
| PostgreSQL | Run records and stored step outputs |
| Redis | Trigger storage |
| FastAPI | Invocation and management endpoints |
What was delivered
- Step types
- 13
- Step categories
- 5
- Run triggers
- 3
- Resumed data
- Typed
- An orchestration engine for arbitrary step graphs
- Breakpoints, per-step logging control and cancellation
- Resumable runs with typed data rebuilt from stored schemas
- Scheduled and endpoint-triggered execution
Why it matters
A visual builder is only as reliable as the engine behind it. Getting ordering, typing and resumption right is what lets non-specialists build pipelines with confidence.
If your organization is planning a platform of this kind, or needs a specific part of one designed and delivered, we would be glad to discuss it.