WHAT WE DO
AI Assurance
Independent validation of the AI and agentic systems you are putting into production. We don't build them, and we don't sell the models. We tell you, with evidence, how they behave and where they fail.
Why Independent
The team that built an agent is the wrong team to tell you whether it is safe to ship. Not because they are careless, but because they know what it is supposed to do, and the failures live in what nobody thought to ask. AI Assurance is a separate practice at QASource with no stake in the system passing.
It is also the one part of the company that will not take build work from the client it assesses. We do development for many clients. We do not do it for a system we are validating.
Our clients here are CTOs and VPs of Engineering putting AI into products and workflows, in healthcare, financial services and enterprise software, who need something to show their board, their regulator and themselves.
What We Validate
- Behavior Under Realistic Use: Does the system do what it is supposed to across the distribution of inputs it will actually see, including the ones the product team never wrote down?
- Failure Modes: Where it hallucinates, drifts, refuses, or acts outside its intended scope; how often; and what the consequence is when it does.
- Guardrails and Policy: Whether the controls that are supposed to constrain the system hold under adversarial and accidental pressure.
- Retrieval and Grounding: Whether the context retrieved is the right context, and whether the answer is actually grounded in it rather than in the model's own priors.
- Multi-agent Handoffs: Where work passes from one agent to another, and what is lost in the handover: state, intent, or the constraint that was supposed to travel with it.
- Adversarial Behavior: Deliberate attempts to make the system misbehave (prompt injection, jailbreaks, guardrail bypass), and which controls hold.
- Tool Use and Agent Actions: For systems that act on behalf of people: whether the right tool is called with the right arguments, and what happens when a call fails, when instructions conflict, or when the environment changes underneath.
- Data Integrity: Whether the pipelines feeding the system produce consistent inputs, and what happens when they don't.
- Regression Over Time: A baseline you can re-run when the model, the prompt, or the data changes, so "it worked in March" is a fact rather than a memory.
Judgment Is What's Being Tested
An automated evaluation tells you whether an agent produced the output you expected. It cannot tell you whether the agent did the right thing — whether the judgment was sound, the tone was acceptable, the escalation was warranted, or the outcome was one your customer would have tolerated.
Those are human assessments, and clients are now asking for them by name. QASource supplies trained validators who work through agents, workflows, and processes by hand, under a defined method.
How human validation works →We spent a decade automating human testers out of this work. The judgment that agents now have to be measured against is the one thing we never found a way to automate.
What You Receive
- A written assurance finding: scope, method, results, failure modes observed, residual risk, and what would change our conclusion
- Human validation results where judgment was assessed: rubric, scores, rater agreement, and the disagreements themselves
- The evaluation harness itself, so your team can re-run it
- Ground-truth datasets built for your system, owned by you
- Bounded runtime monitoring where the system's behavior can change after release
What We Don't Claim
We do not certify that an AI system is safe. No one honestly can. We state what we tested, what we observed, and how confident that makes us, within bounds we write down.
We do not audit for fairness or bias. That is a distinct discipline, and we will tell you who does it well.
We do not validate systems we built.
Bring Us the System You're Least Sure About
A scoped assurance assessment takes three to four weeks and produces a finding you can act on.