Testing artificial intelligence (AI) agents with real users
Do you already have an artificial intelligence (AI) agent, in pilot or in production? We test it with real people, on the channel they will use, before expanding its use. You get a reasoned decision: release it, narrow the scope, or go back to design.
We design and test reliable AI agents with real users. This page is for those who already have an agent: we start from the test and, if needed, continue with redesign. A trial in the vendor's environment, on clean dialogues prepared by the team, tells you whether a flow starts. It does not tell you what happens when a person writes in their own words, makes an incomplete request, or arrives while operators are busy.
Who it is for
- The Chief Information Officer (CIO), who is responsible for information systems and must say yes or no to going into production.
- Those who lead customer service and Customer Experience (CX), the customer's experience, and see the agent responding to their customers.
- Product leads who have an agent in pilot and must decide whether to extend it to other users or channels.
- Those who had an agent built by a technology partner and want a check on their own side. We work alongside those who built it, not in their place.
If the agent is still an idea, the starting point is the Agentic maturity assessment.
What we look at and how we work
We look at the agent's behaviour inside the service, activity by activity. The quality of the model in the abstract stays out.
- The request. Does the person understand what they can ask? Does the agent understand the request even when it is incomplete or poorly worded?
- The task. The task really closes for the person, and not only in the agent's reply.
- Errors and recovery. How the agent behaves in edge cases, with missing data or out-of-scope requests, and whether it can get the person back on track.
- The scope. Forbidden actions are not attempted, or they are stopped in a visible way.
- The handoff to a human operator. It triggers when it should, reaches the right person with enough context, and the agent stops acting after the handoff. We also test what happens when no one can take the case.
- Cases treated as resolved. We examine a sample: silent failures do not show up among the obvious ones.
- The accessibility of the journey. Understanding, correcting, asking for help, including with relevant assistive technologies. This check does not replace an accessibility audit.
How we work. We build the scenarios from the real service. We involve people with a stated profile relative to those who will use the service, not only the project team; if the service has operators, they take part too. We test on the real channel or the closest possible one, and we write down the differences from production. Sessions are observed; when the plan provides for it, crowd testing widens devices and contexts. We do not set a number of participants that is "enough" for everyone: what matters is coverage of the activities and edge cases, and clarity of the limits. Where participants come from, how many people and sessions, duration, delivery format, timelines and price are defined in the quote, on request. We do not give numbers on the page.
What you receive
- The outcome of each activity tested, with three possible answers: pass, fail, not tested. Each outcome is accompanied by what we observed.
- The conditions and limits of the test: who took part, on which channel, what was not tested.
- A reasoned recommendation: release, narrow the scope, fix and retest, or go back to design. The release decision remains yours and the service owners'.
- Redesign priorities, in order of harm to people and to the service.
Delivery format, timelines and price are defined in the quote, on request. We do not show a price on the page.
The test documents what was verified within a declared scope. It is not a certification and does not guarantee the absence of errors.
How it continues
For us, design and testing are one cycle, and the test opens the redesign. If the problems concern rules, delegation or the handoff to an operator, we bring them back into the service design with your team and those who build the agent, then we test again. If the agent's scope has never been written down, we start from a workshop on the Agentic Design Canvas, a method by Luigi Greco. If instead it emerges that the real question is further upstream — that is, where to use an agent — the next step is the Agentic maturity assessment.
Frequently asked questions
Is it enough to try the agent in the vendor's environment?
No. That environment is useful to see whether a flow starts. It does not tell you how the agent behaves on the real channel, with an incomplete request, with a busy operator, or with a case already opened in a previous contact.
How many people are needed to decide on release?
There is no number that works for every agent. The number for your case is agreed in the plan, based on the activities and edge cases to cover. Details and the quote are defined on request.
Can you test an agent built by another vendor?
Yes. We work alongside those who built the agent, with responsibilities agreed among you, the technology partner and us.
Does the test certify compliance with the European regulation on artificial intelligence (AI Act)?
No. It documents the observed behaviour and the limits of the test; the legal assessment remains with the competent functions.
Further reading
- What usability tests are (article)
- Agentic design, the new frontier of service design (article)
- Artificial intelligence won’t fix broken systems (article)
- AI Experience Continuity (page)
- Make an artificial intelligence agent reliable in production (solution)
- User testing (service)
- Crowd testing (service)
- Agentic Design Canvas, a method by Luigi Greco (article)
- Agentic maturity assessment (solution)
- Agentic design workshop (page)
Published work on this topic
Wide Group: user research and usability testing on brokers’ management software
Tell us which activities your agent should perform and on which channel. In the first message, do not include credentials or personal data that are not needed. We reply from our offices in Rome and Milan.
