One gateway to 45 models from six AI providers, with guardrails and tracing
A single model layer that lets teams choose and change models without rebuilding their workflows, with routing, policy and tracing handled in one place.
- Partner
- A global consultancy
- Period
- October 2024 to January 2026
- Chat models
- 45
- Available to agents from one catalogue
- Providers
- 6
- OpenAI, Anthropic, Google, Groq, xAI and Perplexity
- Gateways
- 2
- Portkey or Helicone, selected by configuration
The challenge
Model choice changes quickly. Workflows written against one provider's interface become costly to move, and each team handling its own keys and policies leaves no consistent oversight.
What we built
A model configuration and gateway layer used by both pipelines and agents. Teams select a provider and model from one catalogue, requests route through Portkey or Helicone, guardrail policies attach per agent, and agent chains are traced in LangSmith.
What was delivered
- 45 chat models across six providers, plus a separate catalogue of reasoning models
- Gateway routing that can be switched between two products by configuration
- Consistent settings for temperature, seed and token limits
Partner background
Our partner is a global consultancy whose teams work with large volumes of documents, spreadsheets, databases and email. It wanted one internal platform where those teams could build data pipelines and AI agents themselves, instead of commissioning a new application for each need. The platform had to run inside the partner's Microsoft 365 and Azure environment, keep each team's work separate, and move work from experiment to production through controlled environments.
The challenge
Provider lock-in
Each provider has its own SDK, parameters and model names. Code written for one does not move easily to another when prices, quality or availability change.
No single point of control
Without a common layer, keys, usage and safety policies are handled differently in every workflow.
Visibility
Multi-step agents are hard to debug without a record of each model call and its inputs.
Objectives
- Offer many models through one consistent configuration
- Route model traffic through a gateway that can apply policy
- Keep the gateway product replaceable
- Trace agent chains for debugging and review
Our role
CharCentric provided technical leadership and architecture within a multidisciplinary engineering team, and contributed directly to implementation. The platform was built over 16 months, from October 2024 to January 2026, as a Python and FastAPI backend on Azure.
Scope and timeline
The gateway and guardrail integration was built between December 2024 and May 2025. The agent model catalogue was extended between June and September 2025.
Approach
Configuration over code
Models are described by provider, model name and a small set of behaviour settings. Changing a model is a configuration change, not a code change.
An interface for the gateway
Portkey and Helicone implement the same guardrails service interface. A factory selects one by configuration, so the gateway can change without touching agents.
Policies outside the code
Guardrail policies are defined in the gateway and attached to an agent by reference. Policy changes do not require a redeploy.
Implementation
Model catalogue
45 chat models across OpenAI, Anthropic, Google, Groq, xAI and Perplexity, from GPT-5 and Claude to Gemini, Llama, Grok and Sonar, plus a separate catalogue of reasoning models.
Behaviour settings
Temperature, seed for reproducible output, and maximum tokens are validated before a model is created.
Guardrails sub-component
When attached to an agent, it sets the gateway address and headers so every model call from that agent passes through the configured policy.
Tracing
Agent chains can be managed and traced in LangSmith, giving a step by step record of prompts, responses and timing.
Tools and technologies
| Tool | Purpose |
|---|---|
| Portkey | Model gateway and guardrails |
| Helicone | Alternative gateway |
| LangChain | Provider-neutral model interface |
| LangSmith | Chain management and tracing |
| Pydantic | Validated model configuration |
What was delivered
- Chat models
- 45
- Providers
- 6
- Gateway options
- 2
- Catalogue
- 1
- A single model catalogue shared by pipelines and agents
- Gateway routing with a replaceable gateway product
- Guardrail policies attached per agent by configuration
- Tracing for agent chains
Why it matters
Choosing a model is a decision organizations will revisit many times. A gateway and a shared configuration keep that decision cheap to change and keep oversight in one place.
If your organization is planning a platform of this kind, or needs a specific part of one designed and delivered, we would be glad to discuss it.