AI is exciting. Urgent, even.
But after working with countless companies on AI adoption in my role as a Senior Solutions Engineer, I’ve noticed a few common challenges – regardless of company size, budget, or ambition. Too often, teams jump in with the right intentions and still end up with disappointing results.
The problem isn’t that AI doesn’t work. The problem is that AI done wrong wastes time, money, and trust – and most teams aren’t equipped to vet tools, ask the right questions, or structure implementation for success.
To help support teams evaluate and deploy AI with confidence, we just released The AI Agent Blueprint. It’s a practical roadmap for a moment when everyone’s trying to figure out what comes next.
In this post, I’ll break down my personal learnings of what companies consistently get wrong at the start of their AI journey and how to avoid those traps. Whether you’re evaluating a solution like Intercom’s Fin or just exploring the space, these are the lessons I wish every team had before they started.
Core concepts to help you vet AI solutions like an expert
Before we get into the common pitfalls, let’s cover a few key concepts. You don’t need to become an engineer to thoroughly evaluate AI Agents, but you do need to understand a few foundational terms. This knowledge will help you:
- Ask sharper questions during demos.
- Spot red flags in vendor pitches.
- Choose scalable, future-proof solutions.
- Guide internal alignment and buy-in.
- Build confidence in your final decision.
A little technical fluency goes a long way. Keep in mind these are just a few of the many terms out there. But here are the ones I’d suggest getting comfortable with today:
Retrieval-Augmented Generation (RAG)
RAG enhances generative AI by pulling in real-time, relevant information from your company’s data sources before generating a response.
Why it matters: Most AI tools claiming to “know your business” only use pre-uploaded or static training data. RAG-based systems dynamically search live sources like help centers, product docs, or internal wikis, making them far more accurate and adaptable (assuming your data hygiene and permissions are in good shape).
Easy way to remember: Think of RAG as an AI assistant with an open-book exam. Instead of relying only on memory (pre-trained data), it searches for the latest, most relevant information before responding. This makes RAG especially useful for AI Agents, customer support systems, and AI-driven search engines, ensuring responses are more accurate and up to date.
Vector search
Vector search enables AI to match by meaning, not just keywords. It converts both the user’s question and your documentation into numerical vectors and retrieves the closest semantic match even when the phrasing differs.
Why it matters: Without vector search, your AI may only work if the user phrases things “just right.” With it, users can speak naturally and still get the correct response.
Easy way to remember it: Vector search is like finding a song by its vibe, not its title. It works by intent, not exact match – essential for intuitive AI experiences.
Agentic AI
Agentic AI goes beyond answering simple questions; it can initiate actions, pursue goals, and carry out multi-step tasks.
Why it matters: Most AI tools today are passive. They only respond when prompted. Agentic AI drives outcomes. For example, Intercom’s Fin is evolving to handle actions like checking order status, triggering refunds, or escalating issues, all without human involvement.
Easy way to remember it: Agentic AI is like a rockstar project manager, not just a note-taker. It doesn’t just reply with information when simple questions are asked. It plans, acts, and follows through to get the job done.
MCP (Model Context Protocol) Server / Client
MCP is an emerging approach for managing AI agents at scale. It involves three core components:
- The model (the AI system itself).
- The context (what data and information it can access).
- The protocol (the rules for how it talks to other tools and data).
Why it matters: As AI gets embedded across your organization, centralized governance becomes critical. MCP ensures agents act within rules, respect permissions, and scale responsibly – without needing to hard-code logic into every use case.
Easy way to remember it: Think of MCP as a control tower for your AI agents. It manages what they know, what data they can use, and what boundaries they stay within.
Understanding concepts matter because they help you ask better questions and spot red flags during vendor evaluations. But understanding terminology alone isn’t enough.
Common mistakes I see teams make
Here are five mistakes I see even well-informed teams make, and some advice on how to avoid them.
Mistake #1: Treating all AI tools the same
The AI space is moving fast. It’s a constantly evolving landscape and full of buzzwords, which can create confusion. I often see teams treat “chatbots” and AI Agents as interchangeable, without realizing there’s a massive difference between things like:
- A legacy rules-based bot with generative copy slapped on top.
- A true agentic AI system that takes action, learns from context, and scales with your business.
If you don’t understand core terms like RAG, MCP, or the differences between LLMs and agentic AI, it’s nearly impossible to ask the right questions during your evaluation process. I’ve heard of too many teams buying solutions that are outdated or require heavy upkeep after deployment. Educating your team on the fundamentals gives you the confidence to separate real capability from flashy demos.
Mistake #2: Assuming you can build it in-house
There’s a real cost and complexity of building AI Agents internally – orchestration, retrieval systems, prompt chaining, governance, and more. It’s not just a weekend project. It’s a long-term infrastructure investment. And for most companies, it quickly becomes a distraction rather than a differentiator.
Many teams assume building their own AI Agent will be faster, cheaper, or more flexible than buying. On paper, it sounds reasonable – especially if you’ve got a strong engineering team, access to top-tier models, and a healthy budget. But in practice, that path is much harder than it looks.
I laugh as I write this because I’ve been there myself. Over the past few years, I’ve built a handful of AI apps in my free time. At first, it was thrilling. The early wins came fast, and the possibilities felt endless. But it didn’t take long for reality to hit: shipping something truly polished – even at a tiny scale – required far more time, infrastructure, and expertise than I had imagined.
At a company level, those challenges only grow. Building an AI Agent from scratch means committing to:
- Data chunking, embedding, and relevance tuning.
- Prompt chaining, context management, and hallucination reduction.
- Real-time retrieval architecture and RAG pipelines.
- Fine-tuning, model upgrades, and fallback orchestration.
- Security, permissions, audit logs, AI governance… and so much more!
Even top tier, well-resourced companies often end up circling back to buying after burning time, money, and momentum. The true cost of building isn’t just engineering – it’s also maintenance and velocity. Successful and innovative teams stick to their areas of expertise and bring in experts for the rest.
Mistake #3: Betting on the wrong vendor
I’ve seen many teams focus too narrowly on what a product looks like during the sales process – or assume the vendor will “figure it out later.”
That’s a risky bet in a space that’s changing this quickly. The result is often a tool that can’t keep up, requires constant hand-holding, or becomes too rigid to scale.
The best vendors continue to learn, evolve their tools, and drive more value over time. They are also constantly testing and releasing new features.
When evaluating vendors, ask:
- Is the vendor investing meaningfully in AI R&D?
- Does their team have a clear roadmap for improvement?
- Can this system adapt to your workflows without needing engineering support at every step?
- How much ongoing maintenance will be needed?
These questions separate vendors building for tomorrow from those selling yesterday’s technology. The AI landscape moves fast – you want a partner who’s staying ahead of it, not catching up to it.
Mistake #4: Ignoring your internal foundation
Now let’s assume you’ve chosen a vendor you feel confident in. But did you know even the best AI Agents need proper fuel? Your content is actually one of the most overlooked success factors.
Even with the right solution in place, they’re often disappointed by lackluster results – not because the AI isn’t capable, but because it doesn’t have enough high-quality material to work with. If your content is outdated, inconsistent, or hard to parse, the AI will struggle. Garbage in, garbage out.
I’ve seen teams buy best-in-class AI and still get stuck because they hadn’t invested in the inputs that make it powerful:
- A well-structured help center.
- Clear, detailed documentation.
- Internal process visibility (for things like internal AI/copilot).
- Robust APIs.
The good news? You don’t need to overhaul everything on day one. But clean and accessible content makes a world of difference.
Mistake #5: Expecting instant, perfect resolution rates
Another major misconception is expecting AI to resolve 100% of support conversations right out of the gate. In reality, no AI tool starts at perfection – and your team needs a clear understanding of how resolution rate works to set the right expectations.
For context, Fin typically resolves over 65% of support questions out of the box, with minimal training needed, and continues to improve month-over-month. But what makes a great AI implementation is not just where you start, but how you optimize over time. Things like improving content and identifying automation gaps all help drive resolution rate up over time.
And if you’re not tracking your current resolution rate today or don’t fully understand how your vendor defines it, you’ll struggle to see and gain value. My advice is to establish a baseline, set realistic targets, and measure consistently. Smart teams see resolution rate as a growth metric, not a fixed score.
Final thoughts
In my experience, companies that succeed with AI don’t just adopt tools – they implement futureproof systems that connect knowledge, workflows, and decision-making to drive real business outcomes.
- They don’t build everything from scratch.
- They don’t fall for flashy demos of stale technology.
- They partner with vendors who are already building what’s next.
So, if your team is exploring AI, whether you’re just starting or reconsidering your current stack, my advice is simple: start with the concepts and lessons outlined here, use them to evaluate your options, and choose partners who are building what’s next – not just what’s trendy.
And if you’re looking for a broader strategic roadmap to guide that journey, The AI Agent Blueprint is a great place to dive deeper. It lays out how to go from launching an AI Agent to building successful systems that scale and drive real business value.
Because AI isn’t just a trend. It’s a capability your business will depend on. And when done right, it can be your most powerful teammate.
