The Necessity of Overfitting LLM Applications

In this post I will explain:

1. How to quickly automate parts of customer service traffic with minimal investment.

2. Which technology to use for achieving a high ROI on your first automation solution.

3. The challenges encountered when using AI for the seemingly simple task of email classification.

To create an effective automated customer support solution, it’s essential to view it as a system of interconnected components working seamlessly together.

Previously, we’ve discussed RAG (Retrieval Augmented Generation) and why it is often the preferred choice for automating customer service.

However, a significant challenge remains: many companies lack a well-structured knowledge base. As a result, applying the standard RAG approach—which relies on high-quality data and organized knowledge—becomes difficult.

That said, even without an optimal dataset, the capabilities of modern LLMs (Large Language Models) offer an opportunity to take a crucial first step and achieve two key objectives simultaneously:

1. Deeply understand your data and evaluate the technical feasibility of automating 20%, 40%, 60%, or even 90% of customer requests.

2. Implement an initial automation solution that delivers immediate, measurable business impact.

I firmly believe in the “low-hanging fruit” approach. While it may sound like a cliché, my experience shows it builds trust and helps uncover unknowns. You can’t fully anticipate the challenges or benefits until you begin testing and iterating.

For many companies, a significant portion of incoming customer requests are repetitive. While they may not appear so at first glance—since each issue is described differently and reflects the unique perspective of the customer—the underlying problem is often the same.

In my experience, a considerable percentage of these inquiries involve recurring questions that human customer support routinely addresses.

This process, which often requires agents to send standardized responses, is both monotonous and unnecessary given current technological advancements.

Before the advent of LLMs, automating such workflows required training a classification model. Each email would be categorized into a specific topic, and once classified, a predefined template would be used to respond to the customer. This process, while functional, was far more labor-intensive and less adaptable than modern solutions.

This approach has a relatively low ROI because it requires extensive data collection and annotation. For instance, if you introduce a new feature and customers frequently ask specific questions about it, updating the system is impossible without retraining. Moreover, since this solution is not generative, it remains imperfect and inflexible.

In contrast, LLMs do not have