A three-layer test suite that gates every release of an AI and data platform
Unit, integration and end-to-end tests for a Python backend that had none, run on every pull request and required to pass before a release can merge.
- Partner
- A multinational advisory firm
- Period
- 3 months
- Test layers
- 3
- Unit, integration and end-to-end
- Merge gate
- Every PR
- A failing test blocks the release
- Live checks
- Nightly
- Integration tests against the real services
The challenge
The firm's internal AI and data platform had a Python backend connected to PostgreSQL, ClickHouse, Prefect, Azure Blob Storage, internal authorization services and language models. No tests existed, so every release relied on manual checks.
What we built
A Pytest suite in three layers that runs in GitHub Actions on every pull request. Integration tests replay recorded responses so they give the same result every time, and a nightly job runs them against the live services. Language model features are checked for structure on every run and for quality against a reference set each night.
What was delivered
- A test suite that must pass before any release merges
- Repeatable integration tests for every external dependency in scope
- Language model features checked for structure and quality
Partner background
Our partner is a multinational advisory firm that built an internal AI and data platform for its teams. In its first version, the platform's backend depended on two databases, a workflow orchestrator, cloud storage, internal security services and language models. The firm wanted to release often without relying on manual checks before each release.
The challenge
No safety net
The backend handled API requests, database work and external integrations, but had no automated tests. Every change was checked by hand, or not at all.
Many external dependencies
Authorization services, Prefect, PostgreSQL, ClickHouse, Azure Blob Storage and language models each had to be exercised, and tests that call live services are slow and can change from run to run.
Model output varies
A language model rarely gives the same answer twice, so a test cannot simply compare its output with a fixed string.
Workflows, not just functions
The platform's value lies in multi-step workflows. Testing individual functions would not show whether a full workflow produced the right result.
Objectives
- Automated tests at every layer of the backend
- A suite that blocks a release when it fails
- Repeatable results from tests that touch external services
- A way to check language model features despite varying output
- A structure that new features and integrations can extend
Our role
CharCentric designed the test framework, wrote the three test layers and connected them to the release process over three months.
Scope and timeline
The work covered unit tests for services and modules, integration tests for every external dependency, end-to-end tests for the main workflows, and the continuous integration jobs that run them. It was delivered in three months.
Approach
One framework, three layers
Pytest runs all three layers with shared fixtures and conventions, so contributors move between layers easily and one command runs everything.
Record, then replay
Integration tests record each external service's responses once and replay them on every pull request, which keeps them fast and repeatable. A nightly job runs the same tests against the live services, so a change on the other side is still caught within a day.
Check structure always, quality nightly
On every run, language model features are checked for structure: the expected fields, types and length. Each night, their answers are scored against a small reference set, and a score below the agreed threshold fails the job.
Implementation
Unit tests
Unit tests cover API endpoints and the utilities that process, transform and validate data.
Integration tests
Database operations run against PostgreSQL and ClickHouse in disposable containers on every pull request. Integration tests also cover authorization and authentication services, Prefect task orchestration, language model calls and Azure Blob Storage uploads, retrieval and processing.
End-to-end tests
End-to-end tests run complete data flows from Prefect orchestration to storage in PostgreSQL and ClickHouse, the AI content generation feature, and meeting summarization from recorded input to delivered summary.
Release gate
GitHub Actions runs the suite on every pull request and reports coverage on it. Branch protection requires a passing run before anything merges.
Tools and technologies
| Tool | Purpose |
|---|---|
| Pytest | Test framework for all three layers |
| pytest-recording | Recorded responses for repeatable integration tests |
| Testcontainers | Disposable PostgreSQL and ClickHouse for each run |
| GitHub Actions | Runs the suite on every pull request and each night |
| PostgreSQL and ClickHouse | Databases under test |
| Prefect | Workflow orchestration under test |
| Azure Blob Storage | File storage under test |
| Language model APIs | AI features under test |
What was delivered
- Test layers
- 3
- Merge gate
- Every PR
- Live and quality checks
- Nightly
- Delivery
- 3 months
- A Pytest suite in three layers, run on every pull request
- Merges blocked when a test fails
- Recorded integration tests, with a nightly run against the live services
- Structure checks on every run and nightly quality scoring for AI features
Why it matters
A test suite only makes releases predictable when nothing can be released without it. Running it on every pull request, and keeping it repeatable enough that a failure always means something, is what lets a growing platform release often.
If your organization is facing a similar challenge, we would be glad to discuss it.