Start with what remains human

When a small or medium-sized enterprise (SME) considers using an AI agent, it should begin by identifying what tasks will stay with humans. This approach helps define the project's scope, budget, and how its success will be measured. A clearly defined scope prevents the project from becoming too broad and unmanageable.

The challenge of multiple tasks

Many managers envision an AI assistant handling multiple tasks, such as responding to emails, qualifying leads, updating a customer relationship management (CRM) system, sending quotes, and summarizing meetings. However, each of these tasks involves different data sources and workflows, making testing and evaluation significantly more complex.

Testing and diagnosing errors

Testing an AI agent is done by observing how it handles specific cases. With multiple tasks, the number of possible combinations increases, making it harder to identify issues. When an error occurs, determining whether it stems from the agent's instructions, the data it uses, or a software integration becomes time-consuming, especially with more tools involved.

Human reactions to errors

A team may quickly lose trust in an AI agent if it makes a visible mistake, especially if the tool was presented as capable of handling all tasks. This can lead to the project being abandoned without a clear failure point. Each additional function the agent handles also introduces more security concerns, such as ensuring proper access controls.

Choosing the right task

A good candidate for AI automation is a task that is frequent, follows stable rules, and has errors that are easy to correct. If a task occurs only a few times a month, the time and effort needed to build and maintain the AI may outweigh the benefits. Frequent tasks provide enough data for testing and evaluation.

Understanding the rules

Before assigning a task to an AI, ask the person currently handling it if they can explain their decision-making process in one page. If the explanation includes many exceptions and uncertainties, the rules are not yet clear. An AI agent will follow these ambiguous rules without the same level of judgment as a human.

Handling errors responsibly

The key question is not whether an AI will make mistakes, but what happens when it does. An error in classifying an email might be quickly corrected, but a mistake in quoting a customer or scheduling a meeting could have more serious consequences. The ability to detect and correct errors before they affect customers is crucial for safe automation.

Existing software solutions

For common tasks like scheduling appointments or sorting emails, existing software solutions are available. These tools are maintained by their providers and billed monthly. It is reasonable to test one of these solutions on real cases for a few weeks before considering custom development.

When to build custom AI

Custom development is justified when the task involves tools that existing software cannot connect to, such as outdated internal systems. It is also needed if the rules are unique to the company or if data privacy concerns prevent using third-party platforms. In these cases, three key checks are necessary: where the data goes, what happens if the provider changes its terms, and who can modify the AI's behavior when needed.

Evaluating the budget

When comparing the cost of building a custom AI agent to using existing software, it is important to compare them on the same time frame, including maintenance. The real work begins after deployment, as real-world requests are more complex than test cases.

Tracking and adjusting performance

After deployment, it is essential to track every request the AI handles, noting what it did and whether corrections were needed. A weekly review of these logs can reveal where the AI struggles. Adjustments should be planned, with time set aside to refine the instructions and ensure the AI continues to perform reliably.

Ownership and documentation

Ownership of the AI's code, instructions, and scenarios is often overlooked. Contracts with AI model providers should clearly state who owns the data and systems involved. Proper documentation is necessary to allow for changes in service providers without rebuilding everything from scratch. An SME that cannot switch providers easily has not acquired a tool but has created a dependency.