If you’re thinking about deploying an AI agent, the big question isn’t just “Can it do this?” It’s “Should it?” — and “Who is accountable when it does?”
AI agents are different from chatbots or simple automations. They can act: updating records, triggering workflows, approving requests, contacting customers, and making decisions. That means the risks are different, too.
In this post, we’ll look at how to decide what an agent should be allowed to do, where human approval still matters, and how to build a practical operating model before the technology goes live.
The first question isn’t “Can it?” — it’s “Should it?”
Organizations can be tempted to start with the technology: find a new AI capability, then look for somewhere to use it. But a better starting point is the process.
An agent shouldn’t be allowed to do something simply because it can.
Some tasks may only need a well-written prompt. Other processes may be unclear or broken and need to be redesigned before they’re automated. Adding an agent to a bad process won’t fix it; it may just help the problem move faster.
Before deciding what an agent can do, ask:
- What problem are we trying to solve?
- Is an agent the right approach, or would a simpler AI-assisted workflow work?
- Is the underlying process clear and reliable?
- Who will be affected by the change?
That last question points to another essential ingredient: change management. AI adoption isn’t only a technology project. People need to understand the new process, the intended outcomes, and the risks if governance is going to work in practice.
Decide what the agent can recommend, decide, and do
Once you’ve established that an agent is appropriate, define its boundaries. Be specific about what it may recommend, what it may decide, and what it may actually do.
Guardrails are important, but they aren’t the whole of governance. Governance also means defining decision rights, accountability, permissions, escalation paths, and the points where a human must step in.
A risk-based approach can help:
Lower-risk tasks: automate with clear rules
Routine tasks with limited impact and appropriate data controls may be suitable for more automation. Examples discussed in the episode included adding information to systems, answering routine emails, and supporting simple internal workflows.
Medium-risk tasks: use thresholds and escalation
Some decisions may be automated within defined limits, with exceptions sent to a person. Travel approval is one example: a low-cost, short trip might follow a streamlined process, while a more expensive or unusual request could require human review.
This middle ground matters. Sending every minor request through several layers of approval can create its own problems: people may become overwhelmed and stop reviewing approvals carefully.
Higher-risk tasks: keep meaningful human oversight
Actions involving sensitive data, access control, compliance, significant financial consequences, or decisions that affect people deserve stronger oversight.
The guest particularly highlighted the risks of collecting, changing, or rewriting sensitive data. An agent could expose information to someone who shouldn’t see it or change its meaning before it is shared. In these situations, human review can be essential.
The goal isn’t to forbid agents from doing useful work. It’s to delegate the right tasks while keeping appropriate protections around higher-impact decisions.
Who owns the outcome?
Accountability can become vague as soon as an agent is involved. The episode offers a practical starting point: the person who asks for an outcome is responsible for their use of it.
That doesn’t mean IT has no role. Technical teams may manage the platform, access, and controls, but they shouldn’t be left owning business decisions they didn’t make.
A clearer model separates responsibilities:
The business area
owns the use case and why the agent exists.
The data owner
sets boundaries for the data being used.
IT and security
manage technical controls, permissions, and monitoring.
The user
remains responsible for actions they initiate and outputs they send or approve.
Shared accountability needs clear ownership. Otherwise, it can become no ownership — and “everyone” may not be a useful answer when a customer, regulator, or board asks what happened.
Transparency matters here, too. Be open when AI has helped produce a report or complete a task. And if an error happens, explain what went wrong and how it will be addressed.
The guest described a municipal billing example in which an AI-related decimal error led to incorrect bills. The organization acknowledged the issue, retracted the bills, and corrected the problem. Being upfront about mistakes can build more trust than claiming a system is flawless.
Build informed trust, not blind faith
Trust and skepticism can coexist. An agent may perform well repeatedly, but that doesn’t mean people should stop checking its work. If it gets 20 things right, a user may be less likely to notice the 21st mistake.
To build informed trust, make the agent’s capabilities and limitations visible. People should be able to understand:
- What information the agent can access
- Which sources it uses
- What it can recommend, decide, or do
- What it cannot do
- When a person must review or approve an action
Be honest about limitations as well as benefits. Agents may struggle with unusual cases, have access to only some information, or need human review before taking a sensitive action.
And when errors happen, explain what happened, what was affected, what will be done to fix it, and how the organization will reduce the chance of it happening again. Honest handling of mistakes is part of building trust.
Make the operating model practical
Governance documents can look polished and still have little effect if they don’t fit the way people work. A useful operating model should be practical, clear, and tied to real ownership.
The business area where the agent lives should own the use case. If it’s an HR agent, HR should understand and manage its purpose and lifecycle. If it’s a sales agent, sales should do the same. IT and security still have an important role in the technical controls and data handling.
A practical model should answer:
Purpose:
What problem is the agent solving?
Scope:
What can it see, recommend, decide, and do?
Permissions:
Who can use it, approve its actions, and change its access?
Human oversight:
When must a person step in?
Monitoring:
What tests, logs, reviews, and measures show it is working as intended?
Retirement:
Who decides when the agent is no longer needed, and who turns it off?
Planning for retirement is easy to overlook. An agent shouldn’t remain active simply because no one has taken responsibility for decommissioning it.
Documentation also matters. If the people who created, use, or own an agent leave, someone else should be able to understand why it exists and take over its responsibilities.
Questions to ask before approving an agent
Before approving an AI agent, ask:
- What is it for?
- What is it allowed to see?
- What is it allowed to decide?
- What is it allowed to do?
- Which actions need a human to say yes?
- Who owns the outcome?
- Who can stop the agent?
- What evidence shows it is still behaving as intended?
If you can’t answer one of these questions, the agent may not be ready.
Information professionals should be involved early in these conversations — not brought in after deployment to manage the consequences. Their understanding of information, governance, and risk can help shape what organizations choose to automate and how they do it responsibly.
Frequently asked questions
What is the biggest risk with AI agents?
The risk is that agents can act. When an agent can trigger workflows, update records, or contact people, the consequences can move beyond a poor answer to a poor decision or action.
Should every AI agent require human approval?
Not necessarily. Some low-risk tasks may be automated with clear rules. Actions involving sensitive data, access, compliance, significant impact, or people may need human oversight.
Who should own an AI agent?
The business area using the agent should own its purpose and use case. IT and security support the technical controls, access, and monitoring.
How can organizations build trust in AI agents?
Be transparent about what the agent can do, which information it uses, where its limitations are, and how errors will be handled. People can make better decisions when they understand the system.
What should an organization do before deploying an AI agent?
Start by defining the problem and reviewing the process. If the process is unclear or broken — or if a simpler approach would work — an AI agent may not be the right solution.
The takeaway
Before an AI agent goes live, make sure its purpose, permissions, decision rights, human oversight, ownership, and monitoring are clear. Otherwise, automation may simply help risk move faster.
Need some help setting things up and checking you’re good to go? or retrospective help? Send us a message via the chat function at the bottom right of the screen to let us know what you’re dealing with, or contact us via email instead.
