Lessons from a real customer journey on building, evaluating, and governing enterprise AI agents.
Most AI projects start with a good idea. A team identifies a process that takes too much time, connects an AI model to the right data, builds a POC, and gets a promising result.
Then the harder conversation starts.
How will it work with our real business systems? Can we trust the answers? What happens when one agent isn’t enough? Where should a person step in? What can the agent access and do? And what happens to cost as more people start using it?
These are some of the questions we increasingly encounter when working with customers on enterprise AI.
We recently had a conversation with our customer Senaca Barnes, AI Solutions Architect at Kaneka Americas Holding, about what it takes to build and deploy AI agents in the enterprise. He shared his team’s experience building AI agents and multi-agent workflows, as well as how they approach evaluation, governance, and cost.
This article shares some of the practical lessons from that customer journey and a framework organizations can use when evaluating their own AI initiatives:
Start With the Business Problem, Not the Agent
One of the most common mistakes we see is starting with the technology. An organization selects an AI model, an agent platform, or a new framework and then looks for a problem that fits. We believe the better starting point is the business process.
An agent can be a good fit when a process involves multiple steps, requires some judgment, uses information from different sources, and has a clearly defined outcome.
But not every problem needs an agent.
If a process is straightforward and follows a fixed set of rules, traditional automation may be simpler and more reliable. An agent may also be the wrong choice when reliable data is unavailable, the desired outcome is unclear or there is no practical way to check the result.
The goal should not be to find a use case for an agent. The goal is to find a business problem where an agent can create meaningful value.
Do You Need One Agent or Several?
Once the use case is clear, the next question is how the work should be structured.
A relatively simple agent workflow might look like:
Goal → Reason → Use Tools → Act → Evaluate
A more complex process may require:
Complex Goal → Specialized Agents → Orchestration → Evaluation → Outcome
That is where a multi-agent approach can make sense.
The point is not to add more agents. It is to divide a genuinely complex process into responsibilities that can be handled, tested, and improved independently.
If one agent can reliably complete the task, adding several more may only increase complexity and cost.
A Real Example: Kaneka’s Evidence Review Process
Kaneka shared an example of a complex evidence-review process that was broken into six specialized steps as part of its AI agent journey. The workflow breaks the work into six specialized steps:
- Pulling relevant facts
- Drafting a summary
- Assigning roles
- Scoring the evidence
- Resolving disagreements
- Producing the final report
And there was still a person involved before the final report moved forward. The work was broken into smaller responsibilities, and each part had a purpose.
The simple takeaway is this: if a process involves very different kinds of work, it can make sense to give those jobs to different agents. It also makes it easier to understand where things are working, where they are not, and where a person needs to step in.
Connect Agents to Enterprise Data and Systems
An agent cannot be useful in an enterprise if it cannot access the information and systems required to do its job. That could mean documents, business applications, databases, APIs, or other enterprise sources.
This is why organizations need an AI-ready data foundation that can support both AI applications and enterprise AI agents.
In many organizations, that information is spread across systems such as SAP and other enterprise applications, Salesforce, SharePoint, third-party apps, and internal databases. You also need to decide:
- Which sources can the agent access?
- How reliable is the information?
- How is the information maintained?
- Can the user verify where an answer came from?
- What actions can the agent take with that information?
Kaneka’s technical knowledge use case is a good example. Users can ask questions about manufacturing and quality information in plain language and get answers linked back to the source documents using retrieval-augmented generation (RAG). The solution brings together Azure AI Foundry, Azure AI Search, Document Intelligence, Blob Storage, and Azure Functions.
The broader lesson is simple: an agent becomes much more useful when it can not only find information but also work with the applications and workflows people already use every day.
Build Evaluation into the Architecture
One of the hardest questions with an AI agent is also one of the simplest: How do you know the answer is good enough?
Traditional software generally produces predictable results. AI systems can produce different responses to the same request, and a response can sound convincing while still being incomplete or incorrect.
That makes evaluation an important part of the design. In one of Kaneka’s approaches, multiple models evaluate the same evidence independently. When their assessments disagree, another model acts as a judge to help determine the result. Human review remains part of the process.
This approach is often referred to as LLM-as-a-Judge or AI-assisted assessment. The broader lesson is that AI agent evaluation should not be something added after an agent has been built. Teams should define what a good response looks like and establish a way to measure it.
Keep Humans in the Loop
More autonomy does not mean removing people from the process. For many enterprise use cases, the right model is a combination of AI and human decision-making.
Kaneka’s document extraction workflow is one example. The system processes information from emails and PDFs, but a person remains part of the review process before the information moves forward.
The question is not whether humans should be involved. The question is where human involvement adds the most value.
Choose the Platform Based on the Use Case
Platform selection should come after the business and technical requirements are clear.
There are several platforms and approaches for building AI agents. The right choice depends on what you’re trying to build, the systems you need to connect, how much control you need, and who will maintain it.
The question shouldn’t be, “Which AI platform should we use?” It should be, “What are we building, and what does it need to do?”
Put Governance Around the Agent
As an agent gets access to more data and systems, governance becomes increasingly important.
A practical enterprise AI governance approach should address at least five areas.
- Data Access: Define exactly which data sources an agent can access — and which it cannot.
- Human Oversight: Identify where a person needs to review or approve an AI-generated result.
- Auditability: Make important agent decisions traceable, including the inputs, sources and outputs involved.
- Escalation: Give the agent a clear path to hand a task to a person when it is uncertain rather than encouraging it to guess.
- Ownership: Someone needs to be responsible for the business process and for addressing issues when an AI-generated result is wrong.
These controls become especially important as agents move from answering questions to taking actions.
Think About Cost Before You Build
A simple interaction might look like: User → Agent → Answer
A more complex workflow could involve: User → Agent → Tool → Data → Agent → Judge → Retry → Answer
Each additional step can introduce model calls, token usage, API calls, data processing, and infrastructure costs.
That means cost needs to be considered during architecture and design, not after deployment.
Teams should understand the expected cost per task and how that cost changes as the number of users and workflows grows. The objective is not simply to minimize cost. It is to find the right balance between quality, reliability, and cost.
Five Questions to Ask Before Building Your Next AI Agent
Before starting an AI agent project, ask five simple questions.
- What problem are we solving? Start with the business process and the result you want to improve.
- Do we actually need an agent? Maybe traditional automation is enough. Maybe a copilot is enough. Make sure an agent is solving a real problem rather than being used because it is the latest technology.
- Is one agent enough? If the process has several very different types of work, consider whether it makes sense to split the responsibilities.
- How will we check and control it? Decide what a good result looks like, how you will check it, and where a person needs to be involved.
- Can it work at scale? A solution that works for ten people in a demo may behave very differently when hundreds or thousands of people start using it.
Think about performance, reliability, cost, and ongoing maintenance from the beginning.
Have an AI Agent Use Case in Mind?
If your team has an AI idea, business challenge, or process you are considering for automation, bring it to us. Unvired is offering a complimentary 4-hour AI advisory workshop to help organizations explore an AI idea, use case, or business challenge.
We can use the session to understand the process, assess whether an agent is the right approach, explore the possible architecture, and discuss what a practical next step could look like. Schedule a Call with our AI Experts today!
