When "chatting to your data" is doomed to fail

There is considerable misunderstanding about the feasibility of implementing AI-powered question-answering capabilities. Ultimately, it comes down to details that are difficult to grasp unless you are the one developing the solution.

From an outside perspective, all use cases for knowledge-based question answering may appear similar. However, we have identified at least three distinct scenarios, each requiring different levels of implementation effort and initial data requirements:

1. “ChatGPT for Your Internal Documents”

2. High-Confidence Information Retrieval (such as for legal or medical purposes)

3. Customer Service Agent

If any of these scenarios align with your business needs, keep reading—we will explain the differences and estimate the necessary workload to initiate a Proof of Concept (PoC).

One of the most popular use cases for generative AI in companies is searching for answers within a vast array of unstructured documents or other sources of knowledge presented in a "messy" format. You have likely seen numerous promising implementations and are considering streamlining your operations with this generative AI solution.

You might assume that having documents serve as a source of knowledge means you are ready to proceed.

However, from my discussions with various business stakeholders, I have noticed that the understanding of the term "knowledge" varies.

Simply having access to a large number of documents and internal company information may not be sufficient to effectively answer questions.

Let me break down and reveal what might be hidden beneath the general idea of “using internal data to answer questions.”

Scenario 1: “ChatGPT for your internal documents”

This scenario is the most promising and is likely the one you see implemented successfully most often.

Generative AI excels when you need an assistant to help speed up your processes, especially if you are not expecting absolute accuracy or precise final answers.

Consider Perplexity.ai, one of my favorite tools for everyday work. I use it to achieve better search results and take advantage of its summarization capabilities. As a result, instead of visiting multiple webpages, I have everything ready in one place. However, I always double-check the links to ensure that the conclusions presented to me are correct.

In a company environment, this scenario can be understood as a “ChatGPT for your internal documents.” I have definitely seen these solutions within large organizations, and they serve as a valuable addition to search bars in platforms like Confluence, SharePoint, or internal wikis. Remember to manage the expectations of end users correctly,