Is your AI initiative falling short despite the hype? The root cause often lies not in the AI technology itself but in how your architecture handles the Natural Language Understanding (NLU) layer. In this episode, we explore why treating AI as a bolt-on feature leads to failure and what leadership must do to embrace the fundamental paradigm shift required for success.

In this episode, you'll learn:

- Why legacy deterministic web app architectures break when faced with conversational AI

- The critical role of the NLU layer as the "brain" driving dynamic, user-led interactions

- How multi-intent queries, partial understanding, and fallback strategies redefine system design

- The importance of AI-centric orchestration bridging probabilistic AI reasoning with deterministic backend execution

- Practical architectural patterns like the 99-intents fallback and context management to improve reliability

- How to turn unsupported user requests into upsell and engagement opportunities

Key tools and technologies mentioned include Large Language Models (LLMs), function-calling APIs, AI orchestration layers, and frameworks from thought leaders like Keith Bourne, Ivan Westerhof, and Sunil Ramlochan.

Timestamps:

0:00 - Introduction & Why AI Projects Fail

3:30 - The NLU Paradigm Shift Explained

7:15 - User Perspective vs. System Reality

10:20 - Handling Multi-Intent & Partial Understanding

13:10 - Architecting Fallbacks & Out-of-Scope Requests

16:00 - Business Impact & ROI of Robust NLU Architectures

18:30 - Closing Thoughts & Leadership Takeaways

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

MEMRIQ INFERENCE DIGEST - LEADERSHIP EDITION

Episode: Why Your AI Is Failing: The NLU Paradigm Shift CTOs Must Understand

============================================================

Welcome to Memriq Inference Digest - Leadership Edition. I'm Morgan, and this podcast is brought to you by Memriq AI, a content studio building tools and resources for AI practitioners. If you want to stay ahead in the AI game, you can check us out at Memriq.ai.

Today, we're diving into a critical topic for decision makers: the impact of the Natural Language Understanding layer — or NLU — when moving from traditional deterministic web applications to AI-driven chatbot architectures that can engage users in natural, open-ended conversation.

Also, if you want to go deeper into this with diagrams, clear explanations, and even hands-on code labs, look up Keith Bourne's second edition on RAG and AI agents on Amazon. It provides significant insight into the architectures that support these concepts — and it's accessible for leaders who want understanding without drowning in jargon.

This episode is part of Memriq's broader effort to help practitioners and leaders improve their AI efforts. Here's the hard truth: not understanding the NLU layer and its architectural implications is a key reason many AI initiatives fail in production. We're covering these topics to help you avoid the mistakes that others are making.

That's worth repeating — these aren't theoretical concerns. They're the difference between AI projects that deliver real business value and those that stall out after the demo. Let's jump in.

Here's a hard truth that CTOs and technology leaders need to hear — if you're still architecting your AI features as tools bolted onto a traditional web application, you're setting yourself up to fail. And this is exactly why so many enterprise AI initiatives are falling flat.

That's a strong statement. What's going wrong?

Most technology leaders are treating AI like they treated every other new technology — as a feature to add, a component to integrate, another vendor to plug in. But conversational AI with an NLU layer isn't a feature. It fundamentally rewires how your entire application operates.

So they're applying the old playbook to a new game?

Exactly. In a traditional web app, you control everything. Users click buttons you designed, fill forms you specified, navigate paths you predetermined. The interface constrains what's possible. But the moment you introduce a conversational interface with natural language understanding, you've handed the steering wheel to your users — and most architectures simply aren't built for that.

And that's causing failures?

Massively. Patrick Chan, formerly a Google engineer and now CTO of Gentoro, describes how the NLU layer acts as the "brain" of these systems — it interprets what users want, decides what needs to happen, then triggers the right backend services. That's not a helper function sitting on the side. That's a core system actor making decisions your traditional architecture never anticipated.

So the architecture itself is the bottleneck?

It's worse than a bottleneck — it's a fundamental mismatch. Dr. Rosario De Chiara, who writes extensively on LLM-first design, puts it bluntly: the AI becomes "a decision-maker, not a helper." How users navigate through tasks becomes emergent rather than predetermined. If your architecture still assumes predetermined paths, you're fighting against the very nature of what you've built.

This is the wake-up call, then. Leaders need to stop thinking about AI integration and start thinking about AI-centric architecture.

That's the paradigm shift. And everything we'll discuss today flows from understanding that distinction.

Let's break down what this paradigm shift actually means for business leaders. When you introduce an NLU layer, you're not adding a component to your existing system — you're replacing the core interaction model of your application.

Walk us through the key concepts in plain terms.

In a traditional web application, interactions are deterministic — meaning predictable and repeatable. Ivan Westerhof, Chief Automation Officer at Scale Fast AI, explains it simply: "A website has a predefined scope. The user's actions are finite." Click button A, get result B. Every time. The interface itself enforces the boundaries of what's possible.

And that predictability is comfortable for organizations.

Very comfortable. You can test every scenario, guarantee outcomes, predict costs. But here's what changes with NLU — as Westerhof puts it, "the user defines their own journey and experience." You've given them a text box and said, "Tell me what you want." That's not a constrained input space. That's infinite possibility.

So the NLU layer has to interpret that infinite space?

Interpret it, classify it, extract the relevant details, and then route it to the appropriate backend service — all probabilistically, meaning it's making its best educated guess rather than following fixed rules. Mahesh Kumar, CMO of Acceldata, points out that the same question phrased differently might yield varied responses. That's fundamentally different from deterministic systems where identical inputs always produce identical outputs.

How should leaders be thinking about this?

They need to understand they're building a hybrid system now. The NLU layer provides probabilistic reasoning — it handles ambiguity, infers what users want, manages multi-turn conversations. But execution remains deterministic — your backend services have defined inputs, outputs, and business logic. The challenge is orchestrating between these two fundamentally different paradigms.

And this orchestration is what's missing from most current implementations?

Exactly. Many organizations are calling an AI API from within their existing architecture and calling it "AI-powered." But without a proper orchestration layer that can handle the translation between flexible understanding and rigid execution, manage context across conversations, and gracefully handle the inevitable mismatches — they're building on sand.

If you take away just one thing today: moving from traditional web apps with fixed, predictable flows to AI-driven chatbots means adopting an NLU layer that interprets open-ended user input and dynamically controls backend operations — and this requires rethinking your entire architecture, not just adding an AI component.

Let's set the stage for why this matters right now. A few years ago, customer-facing apps were built around deterministic designs — users clicked buttons, filled forms, and followed pre-set paths. That worked well for simple tasks but started to feel clunky as customer expectations evolved.

And the technology has matured enough to make this practical?

Exactly. The barriers to entry have dropped significantly, so more businesses are adopting these architectures. But the problem is that old deterministic architectures can't handle this level of ambiguity or complexity. They break when the user strays from the script.

The conversational interface and the AI co-create the interaction?

Exactly. Which makes conventional testing approaches inadequate. You can no longer enumerate all possible paths. For VPs and founders, the urgent question is: how do we evolve legacy applications to meet these new expectations without breaking reliability or compliance? The answer is adopting AI-centric architectures now — because early movers gain competitive advantage by delighting their customers and increasing engagement.

So the "why now" is a perfect storm — rising user expectations, mature AI tools, and competitive pressure all pushing businesses to rethink their digital interfaces.

Here's the fundamental difference leaders need to grasp. Traditional web apps operate in what we call a closed-world design — all user actions are predefined, and the system only handles anticipated scenarios. Think of it like a well-marked maze: users can only go where the design allows.

That's a significant mental model shift.

It is. At the heart of this is the NLU layer, which interprets what the user says, recognizes their intent, and identifies relevant details. But it doesn't stop there. The NLU layer also dynamically calls backend services based on that understanding.

What does this architecture actually require?

Let me walk through the essential components in business terms. First, you need intent prioritization logic — when a user expresses multiple requests or something ambiguous, the system needs rules for what to handle first.

What about memory?

Critical. The system must remember what's been discussed, what's been accomplished, and what's pending. Without this, every interaction feels like starting over — which frustrates users.

And backend integration?

Every service your orchestrator calls should be designed to return clear signals when something can't be fulfilled — not crash or return confusing errors. The orchestrator needs actionable information to formulate helpful responses.

What about governance and controls?

Multiple layers. Confidence thresholds to catch low-certainty interpretations before they trigger actions. Policy engines to enforce business rules — maybe certain actions require approval, or certain requests are off-limits. Oversight hooks for compliance-sensitive domains. And comprehensive logging of every decision for visibility into what this system is actually doing.

That's a substantial checklist.

It is. And that's why bolting an AI onto existing architecture fails. You're not adding a feature — you're building a fundamentally different kind of system that requires all these components working together.

Here's a concept that trips up a lot of business leaders, and it comes straight from the fundamental difference between deterministic and conversational interfaces. I call it the "User Perspective versus Reality" problem.

What's the core tension?

In your organization, you know your products and services deeply. You understand the capabilities, the limitations, the edge cases. That's your "reality." In a traditional web application, you present that reality to users through carefully designed interfaces — buttons, menus, workflows. Users can't ask for something that doesn't exist because there's no button for it.

The interface constrains the conversation.

Exactly. But the moment you introduce a conversational AI interface, your carefully controlled "reality" gives way to the user's perspective. And here's the thing — user knowledge exists on an enormous spectrum, from complete misunderstanding of your products to potentially knowing more than your customer service representatives.

That's a wide range to design for.

It's massive. A user might ask for a feature that doesn't exist, use terminology you don't recognize, combine requests in ways you never anticipated, or reference capabilities they assume you have based on a competitor's offering. In a traditional interface, none of these scenarios would surface. In a conversational interface, they're daily occurrences.

So the architecture has to account for this entire spectrum?

That's the key insight. An AI-centric application accounts for that entire spectrum of user understanding — from confusion to expertise — rather than just the predetermined paths you could previously dictate.

How does this manifest in practice?

Consider this scenario: a user asks "Can I use feature X on product B?" In your reality, only product A supports feature X. In a traditional app, there would be no way to even ask this question — the feature X option simply wouldn't appear on product B's interface. But in a chat interface, users will absolutely ask. And your architecture needs a strategy for this — not just an error message, but an actual strategy.

Building on that user perspective challenge, there's another layer of complexity that most architectures handle poorly — multi-intent queries and partial understanding.

What do you mean by multi-intent?

Users rarely ask single, clean questions. They combine requests. "Can I use feature X on product A? What about product B?" That's two related queries in one sentence. Or consider: "Book me a flight to London next Tuesday, reserve a hotel nearby, and I'll need a rental car too." That's three distinct requests bundled together.

And the system needs to handle all of them?

It needs to recognize all of them, prioritize them, execute them in a sensible order, and track which ones succeeded and which ones failed. If the flight books but the hotel doesn't have availability, the system can't just say "Done!" — it needs to report partial success and ask how to proceed on the hotel.

That's significantly more complex than handling single requests.

It is. And then there's context switching. A user asks about weather in London for their trip, then suddenly asks "What about flight delays at Heathrow?" A rigid system might fail because "flight delays" wasn't part of the original booking flow. But a well-designed system recognizes this as a related but new request and either handles it or gracefully redirects.

How should leaders think about supporting this?

The system needs to maintain a model of the conversation — what topics have been discussed, what requests are pending, what information has been collected. And the orchestrator needs flexibility to handle multiple requests from a single user message, then combine the results coherently.

What about when the system only partially understands a request?

This is where it gets interesting. A well-designed system doesn't just fail or guess — it acknowledges what it understood, acts on the confident parts, and asks clarifying questions about the uncertain parts.

So it's transparent about its understanding?

Exactly. "I can book your flight to London for next Tuesday. For the hotel, did you want something near the airport or in central London?" The system demonstrates competence on what it grasped while surfacing ambiguity constructively. Most current implementations don't have this sophistication — they either try to handle everything at once and fail, or force users into rigid one-request-at-a-time interactions.

Now here's where I want to challenge the defensive mindset most organizations bring to handling requests they can't fulfill. When a user asks for something you can't provide, most systems return some variation of "Sorry, that's not available." That's a wasted opportunity.

Wasted how?

Consider the cost of getting that user to this moment of interaction. The marketing spend, the product design, the email campaigns, the paid advertisements, PR efforts, direct sales outreach, customer incentives, follow-up nurturing — all of that represents significant investment in bringing this person to your application.

And they're asking for something.

They're 95% of the way to finding a solution. They've articulated a need. They've engaged with your system. And if their request doesn't match exactly what you offer, but you have something close — something that could fulfill their underlying need — you have a golden opportunity to present it.

So handling unsupported requests becomes a sales opportunity?

Exactly. If you have similar products with somewhat similar functionality — not exact matches, but in the range of fulfilling user needs — your system shouldn't say "Sorry, this isn't available." It should say, "Product B doesn't support feature X, but Product A does — and it also includes these additional capabilities that might interest you."

That's a fundamentally different approach.

It requires your system to be product-aware, to understand relationships between offerings, and to have enough context to make relevant suggestions. This isn't just error handling — it's intelligent redirection. And it dramatically changes the ROI calculation on your AI investment.

The system needs to know not just what failed, but what alternatives exist.

And present them persuasively. This is why the NLU layer and orchestrator need deep integration with your product catalog and business logic — not just your technical APIs. You're building a system that can recover gracefully and add value even when the initial request can't be fulfilled.

Let's compare approaches to help frame decisions. On one side, you have deterministic web apps: fixed interfaces, clear workflows, predictable outputs. They're straightforward to test, reliable, and great when tasks are simple and well-defined. But they lack flexibility and can frustrate users with complex needs.

How do you decide which path to take?

Use deterministic workflows when your processes are simple, compliance-heavy, or risk-averse — like financial transactions or regulated environments. Use AI chatbots when user needs are diverse, complex, or when natural, conversational interactions can unlock value — like customer support or sales assistance.

What about different frameworks and approaches in the market?

The Deepset Team provides flexible agent frameworks and emphasizes that this isn't binary — it's a spectrum from fully deterministic to fully AI-driven, and most production systems sit somewhere in between.

And the governance angle?

As Mahesh Kumar notes, "embedding an LLM in a business process means redefining how tasks are routed, governed, and interpreted" — not eliminating deterministic logic, but thoughtfully combining both paradigms.

That helps leaders frame decisions by business context rather than technology hype.

Let me translate the technical architecture into business terms. The NLU layer starts with a Large Language Model — think of it as an AI trained on vast amounts of text to understand and generate human-like language. When a user sends a message, the model interprets it to extract what the user wants and the relevant details.

So the AI isn't just generating text — it's taking action?

Exactly. But it's not magic. There's an orchestrator sitting between the AI and your backend systems. It manages the conversation flow, provides context to the AI — like previous interactions or customer history — and handles responses from backend services to keep the dialogue coherent.

What about the division of responsibilities?

This is crucial to understand. The AI handles understanding and phrasing — interpreting what users want and formulating natural responses. Your backend services handle execution and business rules — actually performing actions and enforcing constraints. This division is what allows the system to handle a wide variety of requests while maintaining reliability.

What about handling edge cases?

Fallback mechanisms are critical. When the AI detects a request is out of scope — meaning unsupported or unclear — it can decline gracefully, suggest alternatives, or escalate to a human. This is essential for maintaining trust and compliance.

Sounds powerful, but also complex to get right.

It is. But the payoff is a chatbot that feels intuitive, reliable, and capable of handling real conversations — not just scripted FAQs.

Let's talk about one of the most challenging aspects of AI-driven systems — handling requests that fall outside what your system can do. This is fundamentally different from traditional applications.

How so?

In a traditional interface, if a feature isn't available, the user simply doesn't see an option for it. Problem avoided. But in a conversational system, users will absolutely ask for things you don't support, in ways you never anticipated. And detecting these out-of-scope requests reliably is essential — but difficult.

What goes wrong when systems don't handle this well?

Two failure modes. False positives — rejecting something you could have handled. That frustrates users. False negatives — failing to recognize an unsupported request and either making something up or executing the wrong action. That causes real problems and erodes trust.

So what's the solution framework?

Multiple layers working together. First, fallback behaviors need to be built into the architecture explicitly. And here's a key architectural point — the logic for determining what's supported should live in your backend services, not in the AI. The service itself knows what it can and can't do and returns a structured result. The AI's job is to route correctly and formulate helpful responses based on that result.

Can you give an example?

The AI correctly interprets "I want feature X on product B" and calls the appropriate service. But the service itself knows product B doesn't support feature X and returns a structured "not supported" result with suggestions. The AI then uses that to formulate a helpful response. This keeps the probabilistic and deterministic responsibilities cleanly separated.

What about the 99-intents pattern we've heard about?

This is Ivan Westerhof's concept — deliberately designing your system to catch likely categories of unsupported requests and route them to specific, helpful responses. Rather than one generic "I don't understand" fallback, you design specific handlers for predictable categories of out-of-scope questions.

Like what?

A handler for competitor product questions might respond: "I can only help with our products, but here's how our offering compares." A handler for feature requests might say: "That feature isn't available yet, but I can note your interest." A handler for questions outside your domain entirely might offer to connect to a human.

So you're designing for failure cases proactively.

Exactly. And Westerhof emphasizes an important point — don't just ask users to "rephrase" when the issue isn't phrasing. If they clearly asked for something you don't support, acknowledge that directly and offer a path forward.

Now, the proof is in the business outcomes. When companies adopt AI chatbot architectures with robust NLU layers and proper orchestration, they see dramatic improvements in user engagement.

What about the ROI story?

There's a strong one. Fallback handling — when done right — can convert previously lost requests into upsell opportunities. That's direct revenue impact. Systems with proper multi-intent handling can complete complex transactions that would have required multiple sessions or human intervention before.

And the accuracy improvements?

Deployments using structured approaches with proper orchestration report significant reductions in misclassified requests — often in the 30-50% range compared to basic implementations. One study showed a 91% reduction in nonsensical responses — that's a game-changer for customer experience.

What about latency concerns?

Valid concern. These layers add overhead — typically 500 to 1500 milliseconds per user interaction. Customers won't wait several seconds for a reply. So optimization through caching and asynchronous processing is critical. But the capability gains usually justify the investment in performance tuning.

I have to ask the tough questions. AI chatbots sound great, but what can go wrong?

Yeah, what are the real risks?

First, unpredictability is the elephant in the room. Sometimes the AI misinterprets user intent or hallucinates — that's when it makes up facts or misleads. If your fallback mechanisms aren't solid, that risks poor user experiences or worse, brand damage.

What about the human element?

Human-in-the-loop controls are essential. You'll need people ready to step in when the AI can't handle a request, which means staffing and workflow changes. In compliance-critical environments, you may need human review for certain categories of decisions regardless of AI confidence.

And testing gets much trickier. Traditional QA approaches don't scale when user inputs are open-ended. You can't enumerate all possible paths anymore.

And as Sunil Ramlochan emphasizes, when requests fall outside what you can do, the system should provide alternative recommendations rather than dead ends. That requires more sophisticated response generation than most teams initially build.

So it's a delicate balancing act between innovation and risk management.

Exactly. Leaders must invest in governance from the start, or you'll pay later in lost trust and failed initiatives.

Let's talk about where this is happening right now. Customer support is a huge area — companies use AI-powered chatbots to understand complex, multi-topic queries that traditional systems can't handle.

So this isn't future talk — it's live in the wild and transforming user engagement across sectors.

Let's set up a scenario: a customer asks for a feature on a product your company doesn't support, combined with another request you can handle. How do different approaches fare?

In a traditional web app, you'd likely get a dead-end on the unsupported part — no way to even express that request.

A basic chatbot might respond with confusion or a generic fallback to everything, frustrating the user entirely.

But an advanced AI system using proper orchestration, the 99-intents pattern, and multi-intent handling can separate the requests — fulfill what it can, gracefully decline what it can't with a helpful explanation, and suggest alternatives. This keeps the conversation alive and builds trust.

The system recognizes the unsupported request, triggers appropriate fallback logic, but also completes the supported request successfully — demonstrating competence while being honest about limitations.

But what about the risk of the chatbot giving incorrect information?

That's where confidence thresholds and the division of responsibility come in — if the AI isn't confident, it asks for clarification or escalates rather than guessing. And the backend services, not the AI, determine what's actually supported.

So the key is balancing AI's flexibility with deterministic controls. Leaders should weigh these trade-offs carefully according to their risk appetite and brand values.

Here are practical recommendations for leaders. First, recognize this is an architecture decision, not a feature decision. You're not adding AI to your existing system — you're building a fundamentally different kind of system.

Ensure your backend services are designed to fail safely — they should return clear, actionable signals when requests can't be fulfilled, not crashes or confusing errors.

Implement the 99-intents pattern — design specific handlers for predictable categories of unsupported requests rather than relying on generic fallbacks.

Design your fallbacks as opportunities, not dead ends. When users ask for unsupported features, redirect them to alternatives that can fulfill their underlying needs.

Build for multi-intent queries from the start. Users will combine requests, and your system needs to handle, prioritize, and report on multiple requests within a single interaction.

Invest in context management. The system should remember what's been discussed and what's pending — without this, every interaction feels like starting over.

Blend AI's probabilistic reasoning with deterministic guardrails like policy enforcement to maintain compliance and brand safety.

Set up monitoring and logging from day one. Track failed requests, user frustration signals, and fallback rates to iteratively improve performance. The Deepset Team recommends beginning with more deterministic approaches and gradually allowing more AI autonomy as you learn.

Great checklist. Skipping any of these is like building a house on sand.

Before we move on, if you want to really deep-dive into this domain, Keith Bourne's second edition on RAG and AI agents is an excellent resource. It provides significant insight into these architectural concepts with diagrams and real-world examples — and it's accessible for leaders and engineers alike. Definitely worth a read.

Quick reminder — Memriq AI is an AI consultancy and content studio building tools and resources for AI practitioners. This podcast helps engineers and leaders stay current with the rapidly evolving AI landscape.

For more AI deep-dives, practical guides, and cutting-edge research breakdowns, head to Memriq.ai.

Despite all the progress, several big challenges remain. Handling truly novel or out-of-scope user requests without frustrating customers is still a tough problem — false positives and false negatives both cause issues.

Balancing AI autonomy with strict business rules, compliance, and safety is an ongoing challenge. As Mahesh Kumar notes, we're still learning how to redefine task routing and governance for AI-embedded processes.

Handling multi-intent queries and seamless context switching in real-time remains an active area of development. Dr. De Chiara points to the challenge that the interface itself can invent new flows — and we're still developing patterns to manage this emergent behavior.

Designing backend services that fail safely and provide clear, actionable signals is underexplored. Patrick Chan's work on user-aligned functions is a step forward, but more standardization is needed.

We also lack widely adopted standards and best practices for AI orchestration architectures, making implementation more custom than it should be.

And scaling monitoring and feedback loops to continuously improve system behavior at production scale is a major operational challenge.

Most critically, many CTOs still haven't recognized that this is a paradigm shift, not a feature addition. Until leadership understands they need to rearchitect rather than integrate, AI initiatives will continue to underperform.

My takeaway? CTOs, this is your wake-up call. If you're still treating AI as a bolt-on feature, you're architecting for failure. The NLU layer fundamentally changes how your application operates.

The shift from closed-world to open-world design — from predefined user paths to emergent, co-created interactions — demands rethinking architecture from the ground up. You're building an evolving conversation engine, not adding a feature.

Don't underestimate the complexity of multi-intent handling and partial understanding. Users combine requests, switch contexts, and express themselves ambiguously. Your system needs to handle all of it gracefully.

The solutions framework matters — services that fail safely, the 99-intents pattern for proactive fallback handling, and constructive redirection as the default. These aren't nice-to-haves; they're requirements for production reliability.

Turn your fallbacks into opportunities. When users ask for something you can't provide, don't give them a dead end — give them an alternative that fulfills their underlying need. That's where ROI lives.

The division of responsibility is key — AI handles understanding and phrasing, backend services handle execution and business rules. Keep that separation clean, and your system stays reliable at scale.

And finally, keep iterating. Monitor everything, learn from failures, and evolve your system in response to real user behavior. This is a living system, not a static deployment.

That wraps up today's deep dive into the NLU layer's impact on transitioning to AI chatbot architectures — and why it demands a fundamental rethink from business and technology leadership.

And thank you to you, our listeners. Remember, AI success isn't about the technology — it's about the architecture and the mindset. Stay informed, stay curious.

Catch you next time on Memriq Inference Digest - Leadership Edition. Cheers!