Agentic AI Chatbot Modernization for a Global Hospitality Organization
Improving question-answering success from approximately 60% to 90%+ on a shared test set of real customer questions about services, products, and content.
Overview
Following the modernization of its customer extranet, a global hospitality organization wanted to make information about its services, products, and content easier to find and use through conversational AI.
The organization initially launched a chatbot using a third-party SaaS platform. That implementation relied on older models, struggled to answer users' questions consistently, and experienced poor adoption.
Stackular rebuilt the assistant around a custom .NET orchestration service, Azure OpenAI models, the application's existing APIs, and an indexed documentation knowledge base. Rather than treating every question as a document search or a predefined intent, the new assistant determines which information source and application capabilities are appropriate for the request.
Using the same test set of questions drawn from actual production conversations, the team compared the previous chatbot with the rebuilt assistant. Question-answering success improved from approximately 60% to 90%+ on that shared evaluation set.
The follow-on project focused on the assistant's reasoning, retrieval, integration, and diagnostic capabilities while retaining valuable application services established during the earlier extranet modernization.
The Challenge: Answering Questions About Services, Products, and Content
The chatbot needed to support different kinds of requests within the same conversation. Some questions required current information from the customer-facing applications. Others concerned product features, service guidance, or supporting documentation. Some fell outside the assistant's scope or required professional judgment it was not qualified to provide.
Those questions could not all be answered using the same mechanism.
A question requiring current application information needed access to the correct records and user context. A question about how a product or service works needed relevant documentation. Neither should be answered by generating a plausible response from general model knowledge.
The existing implementation also depended on manually configured intents and integrations for different question types. As the range of questions expanded, that approach became difficult to extend. Limited visibility into the chatbot's internal decisions made it harder to distinguish a routing problem from missing information, an incorrect API request, or an answer-generation problem.
The organization needed more than a newer model. It needed an assistant whose access to information, choice of tools, and response behavior could be engineered and inspected.
Stackular's Approach: Reuse the Application, Rebuild the AI Orchestration
The existing platform already exposed useful application data, search, and recommendation capabilities through APIs. It also had an established identity system and a shared frontend component library.
Stackular retained that foundation and rebuilt the layer that connects a user's question to the appropriate application capability or document source.
The resulting implementation combines an embedded chat interface, a server-side .NET orchestrator, separate routing and reasoning models, capability-based tool selection, document retrieval, and conversation tracing.
This kept business logic in the application services rather than recreating it inside prompts. The assistant could use the platform's existing information and calculations without becoming a competing system of record.
Solution Architecture
Each question follows a bounded, traceable answer path
- Customer experienceEmbedded Customer Application Chat InterfaceConversational access within the existing application
- Identity and accessServer-Resolved User ContextValidated identity, organization, application scope, and language context
- Orchestration.NET Orchestrator + RoutingCapability selection and a bounded agent loop
- Read-Only ToolsApplication APIs for supported services and products
- Azure AI SearchSupporting content and document retrieval
- Scope-Boundary ResponseUnsupported requests and professional-judgment limits
- Grounded responseAzure OpenAI Answer GenerationAnswers use available tool results or retrieved supporting content
An Embedded Chat Experience
The chat panel is integrated into the customer-facing applications through the shared frontend component system. This places the assistant within the product experience rather than in a separate support application.
The native component follows the host application's visual language and connects to a dedicated backend service.
A Server-Side .NET Orchestrator
The orchestrator manages the request path between the chat interface, identity context, models, tools, and document retrieval.
Its responsibilities include resolving user context, applying scope decisions, selecting the relevant capabilities, sanitizing tool arguments, calling downstream services, and recording what happened during the turn.
The model can help interpret the question and choose among available tools. It does not define its own access permissions or replace the application's business rules.
Separate Paths for Live Application Data and Supporting Content
For questions supported by an application integration, the assistant uses live API tools to retrieve the relevant data. For questions about services, product features, or procedures covered by the knowledge base, it searches indexed documentation and uses retrieved passages to compose an answer with supporting citations.
This distinction is central to the architecture. Current business information remains in the application systems, while the document index provides supporting knowledge and guidance. The assistant only uses live tools where the corresponding application capabilities are available.
The assistant does not maintain a separate copy of live application records as its own operational database.
A Bounded Agent Loop
After routing identifies the relevant capability, the reasoning model selects tools and uses their results to construct the answer. The documented implementation limits the number of agent-loop iterations rather than allowing unrestricted execution.
This is the agentic element of the solution: the model can determine which supported information-retrieval steps are needed instead of relying exclusively on a manually configured response for every possible phrasing.
The assistant's tool catalog is read-only. It excludes operations that change application data. Requests to modify records are directed to the appropriate application workflow rather than executed by the assistant.
Routing Questions to the Right Capabilities
Routing became an important part of improving answer reliability.
The implementation first applies deterministic checks for certain requests, including greetings and scope-related responses. A keyword-based pre-filter then handles questions it can classify confidently with a relevant capability. Other requests are passed to a routing model.
The route determines what the assistant can access next. Supported application-data questions reach the relevant tools. Documentation questions reach the content index. Out-of-scope requests receive a boundary-setting response. Questions requiring professional judgment receive a distinct response explaining the assistant's limits.
The router also accounts for follow-up context so a conversation can retain its subject across turns.
This matters because the wrong route can make an otherwise answerable question fail. A live-data question sent to documentation search may return no useful material even when the application has the requested information. Similarly, selecting the wrong capability can hide the tool needed to answer the question.
Making routing decisions visible gave the team a concrete place to investigate these problems instead of treating every failure as a generic model issue.
Combining Tool Calling with Retrieval-Augmented Generation
The assistant uses different grounding mechanisms for different information needs.
For live application questions, tool calling retrieves information through the platform's APIs. For product documentation, retrieval-augmented generation uses relevant indexed passages as evidence for the answer.
These approaches complement one another. Document retrieval is useful for explaining a feature or procedure, but it is not a substitute for obtaining current application data. Conversely, an API response containing business records may not explain how a user should configure a software feature.
The design also keeps business calculations in application services. The assistant reports values returned by those services rather than inventing missing figures or calculating an unsupported average itself.
That separates language interpretation from business computation: the model helps determine what the user is asking, while the application remains responsible for authoritative data and calculations.
Preserving User and Organization Context
The assistant operates within a platform where user identity and organizational context affect access to live data.
The implementation validates the user's Auth0 token and resolves the relevant organization, application scope, and language context on the server through the application's user-context endpoint. It does not accept the request body as the authority for tenant identity.
Downstream calls forward the user's bearer token so existing application authorization remains part of the access path. Server-owned parameters are sanitized rather than allowing model-generated arguments to redefine the user's scope.
These controls apply to the live API path. They do not imply that every document in the knowledge base automatically inherits the application's record-level permissions; document access is a separate concern.
Making Answer Failures Traceable
An unanswered question can have several different causes. The assistant may choose the wrong route, fail to select a needed capability, omit a required API parameter, receive an empty response, or retrieve documentation that does not cover the question.
Those problems require different fixes.
Stackular added SQL-backed tracing that records the conversation turn, selected tier and capabilities, routing rationale, tool arguments, and downstream response status.
The records made it possible to investigate issues such as a capability being lost between turns, a required parameter not being sent, and an application-data question being incorrectly classified as a documentation request.
That visibility is an important change from simply observing whether the final answer looked right. The team can examine how the assistant arrived at the answer and identify the part of the system that needs attention.
Managing Documentation as a Retrieval Pipeline
Document support also required more than uploading files.
The implementation includes a document-ingestion path that moves uploaded material through storage and a queue, an ingestion worker, passage creation and embeddings, and an Azure AI Search index.
Withdrawal is part of that lifecycle. A document marked as withdrawn must also have its passages removed from retrieval; changing a database status alone does not prevent stale indexed content from being used.
The ingestion worker records why a document did not produce indexed passages. That helps distinguish an ingestion failure from a genuine gap in the available knowledge.
Results
Question-Answering Success Increased from Approximately 60% to 90%+
The primary result was measured by comparing both assistants against the same set of test questions drawn from actual questions asked in production.
| Evaluation measure | Previous chatbot | Rebuilt assistant |
|---|---|---|
| Question-answering success on the shared test set | Approximately 60% | 90%+ |
Using the same questions kept the comparison focused on how well each implementation handled the same information needs. The result describes performance on that evaluation set.
The improvement followed a broader redevelopment of the assistant, including model use, routing, tool access, retrieval, and troubleshooting visibility. It should not be attributed to a model upgrade alone.
Existing Business Services Remained the Source of Truth
The assistant uses the platform's APIs for live information and supported calculations rather than duplicating business logic in its prompts or maintaining a competing operational data store.
Answer Failures Became Investigable
Conversation and tool-call traces provide evidence about routing, parameters, responses, and missing information. This gives the engineering team a more targeted way to investigate unsuccessful answers.
Read-Only Boundaries Limited the Scope of Agent Actions
The assistant focuses on retrieving and explaining information. It does not allow the model to publish, edit, or delete application records through its tool catalog.
Key Technical Decisions
Why Rebuild the Assistant Instead of the Entire Application?
The platform already had useful APIs, search capabilities, and identity infrastructure. Reusing those assets allowed the follow-on project to focus on the underperforming AI layer rather than unnecessarily replacing working application services.
Why Use Separate Routing and Reasoning Models?
Classification and answer generation serve different purposes. A routing model identifies the appropriate path, while the reasoning model selects tools and composes the answer. Keeping those roles separate makes each stage easier to inspect and tune.
Why Limit Tools by Capability?
Not every question needs every tool. Capability-based selection narrows the tools available for a particular request. However, that selection must be tested: excluding a necessary capability can make an answerable question appear unsupported.
Why Use Read-Only Agentic Behavior?
Useful agentic behavior does not require permission to modify records. The assistant can select tools, retrieve information, and answer multi-step questions while leaving changes to established application workflows. Adding write actions would require its own interaction and control design.
Technologies and Methods Used
Application and cloud technologies: Azure OpenAI, .NET / ASP.NET, React, Auth0, Azure AI Search, SQL Server, Azure App Service, and Azure Key Vault.
AI and engineering methods: agentic tool calling, capability-based routing, document retrieval, vector embeddings, server-derived tenant context, API integration, conversation tracing, and document-ingestion workflows.
Lessons for Organizations Modernizing Enterprise AI Assistants
Evaluate with Questions Customers Actually Ask
A test set drawn from real customer questions gives an evaluation a direct connection to the information users seek. Reusing the same questions makes changes in measured performance easier to interpret. Keep test-set results distinct from adoption, customer satisfaction, or other outcomes that need their own measurement.
Investigate the Entire Answer Path
A stronger model cannot retrieve data from a tool it was never allowed to see. Review routing, capability selection, arguments, downstream responses, and document coverage alongside model behavior.
Use the Right Source for Each Question
Keep live business information in authoritative application systems. Use document retrieval for product knowledge and procedures. Do not expect one retrieval mechanism to solve every question type.
Keep Business Calculations Outside the Language Model
When an answer depends on a business calculation, use the application's supported result. The assistant's role is to communicate that result, not replace the underlying business logic.
Build Traceability into the Implementation
The final answer does not explain why a failure occurred. Recording the route, selected capabilities, tool calls, and responses creates a practical feedback loop for engineering improvements.
Make Scope Limits Explicit
Declining an unsupported request is different from failing to answer a supported question. Product-help, live-data, out-of-scope, and professional-judgment requests need distinct handling.
Frequently Asked Questions
What makes this an agentic AI assistant rather than a conventional chatbot?
The reasoning model can choose among supported application tools and use the returned information to answer a request. It is not limited to selecting a predefined response for every recognized question. Its actions remain bounded by the available capabilities, tool definitions, and read-only scope.
Does agentic AI need permission to change business data?
No. In this implementation, the assistant performs multi-step information retrieval without changing application records. Read-only tools provide useful assistance while leaving data changes in the application's established workflows.
Why combine live API tools with document retrieval?
They answer different kinds of questions. APIs provide live application information and supported calculations. Indexed documents explain software features and procedures. The assistant routes each request to the appropriate source rather than treating every question as a knowledge-base lookup.
How does the assistant preserve access controls for live data?
The backend validates the user's Auth0 token, resolves the relevant user and organization context on the server, and forwards the user's bearer token to downstream APIs. Model-generated tool arguments cannot act as the authority for tenant identity.
How does the team investigate an incorrect or missing answer?
The implementation records routing decisions, selected capabilities, tool arguments, and API response information. The team can use those records to distinguish routing errors, missing parameters, empty responses, and gaps in the available documentation.
What improvement did the redevelopment achieve?
Question-answering success improved from approximately 60% to 90%+ when both assistants were evaluated against the same test set of questions drawn from actual production conversations. The redevelopment addressed the assistant's architecture as well as its model use, including routing, tool integration, retrieval, and traceability.
Discuss Your AI Assistant
If your enterprise chatbot struggles to answer questions reliably, the next step may be to examine how it reaches your data, selects application tools, and handles unsupported requests - not simply to change the model.
Talk to Stackular about designing an AI assistant around your application, business rules, and users.
Related Services
Related Case Studies
Extranet Modernization for a Global Hospitality Organization
A global hospitality organization relied on a long-running extranet to provide customers and business partners with access to operational content, training, supplier information, menus, recipes, alerts, and other digital services.
