Critical Analysis of Real Use Cases of LLMs: Between Promise and Reality

As the debate over the actual value of large language models (LLMs) continues, it is essential to critically examine real use cases implemented by companies. This analysis aims to examine concrete applications of LLMs in different industries, critically assessing their actual value, limitations and potential.

‍

E-commerce and Retail: Targeted Optimization or Overengineering?

In the retail and e-commerce sectors, LLMs are used for a variety of tasks:

• Internal assistants and workflow improvement: Instacart has developed an AI assistant called Ava to support teams in writing, reviewing, and debugging code, improving communications, and building internal tools. Although promising, one wonders if these assistants offer substantially more value than more traditional and less complex collaboration tools.
• Content Moderation and Security: Whatnot uses LLM to improve multimodal content moderation, fraud protection, and bid irregularity detection. Zillow employs LLM to identify discriminatory content in real estate listings. These cases represent specific applications where LLMs can provide real value, but require careful verification systems to avoid false positives and negatives.
• Information extraction and classification: OLX created Prosus AI Assistant to identify job roles in ads, while Walmart developed a system to extract product attributes from PDFs. These cases demonstrate the usefulness of LLMs in automating repetitive tasks that would otherwise require significant manual labor.
• Creative content generation: StitchFix combines algorithmically generated text with human oversight to simplify the creation of advertising headlines and product descriptions. Instacart generates images of food products. These applications raise questions about the originality of generated content and the potential homogenization of advertising language.
• Search improvement: Leboncoin, Mercado Libre, and Faire use LLMs to improve search relevance, while Amazon employs LLMs to understand common-sense relationships and provide more relevant product recommendations. These cases represent an area where the added value of LLMs is potentially significant, but the computational complexity and associated energy costs may not justify the incremental improvement over existing search algorithms.

Fintech and Banking: Navigating Between Value and Regulatory Risks

‍

In the financial sector, LLMs are applied with caution, given the sensitive nature of the data and stringent regulatory requirements:

• Data classification and tagging: Grab uses LLM for data governance, classifying entities, identifying sensitive information, and assigning appropriate tags. This use case is particularly interesting because it addresses a critical challenge for financial institutions, but requires strict control mechanisms to avoid misclassification.
• Financial crime report generation: SumUp generates structured narratives for reports on financial fraud and money laundering. This application, while promising for reducing manual workload, raises concerns about the ability of LLMs to properly handle legally sensitive topics without human supervision.
• Supporting financial queries: Digits suggests queries related to banking transactions. This use case shows how LLMs can assist professionals without replacing them, a potentially more sustainable approach than full automation.

Technology: Automation and Technical Assistance

‍

In the technology sector, LLMs are widely used to improve internal workflows and user experience:

• Incident management and security: According to reports on security.googleblog.com, Google uses LLM to provide security and privacy incident summaries for various audiences, including executives, managers, and partner teams. This approach saves managers' time and improves the quality of incident summaries. Microsoft employs LLM to diagnose production incidents, while Meta has developed an AI-assisted root cause analysis system. Incident.io generates software incident summaries. These cases demonstrate the value of LLMs in accelerating critical processes, but raise questions about their reliability in high-stakes situations.
• Programming assistance: GitHub Copilot offers code suggestions and automatic completions, while Replit has developed LLM for code repair. NVIDIA uses LLM to detect software vulnerabilities. These tools increase developer productivity, but could also propagate inefficient or insecure code patterns if used uncritically.
• Data queries and internal search: Honeycomb helps users write queries on data; Pinterest turns user queries into SQL queries. These cases show how LLMs can democratize access to data, but could also lead to misinterpretation or inefficiency without a thorough understanding of the underlying data structures.
• Classification and management of support requests: GoDaddy classifies support requests to improve the customer experience. Dropbox summarizes and answers file questions. These cases show the potential of LLMs in improving customer service, but raise concerns about the quality and accuracy of the responses generated.

Deliveries and Mobility: Operational Efficiency and Personalization

‍

In the delivery and mobility industry, LLMs are used to improve operational efficiency and user experience:

• Testing and technical support: Uber uses LLM to test mobile applications with DragonCrawl and has built Genie, an AI co-pilot to answer support questions. These tools can significantly reduce the time spent on testing and support, but they may not capture complex problems or edge cases the way a human tester would.
• Extracting and matching product information: DoorDash extracts product details from SKU data and Delivery Hero matches its inventory with competitors' products. These cases show how LLMs can automate complex data matching processes, but could introduce bias or misinterpretation without adequate controls.
• Conversational search and relevance: Picnic improves search relevance for product listings, while Swiggy implemented neural search to help users discover food and groceries conversationally. These cases illustrate how LLMs can make search interfaces more intuitive, but could also create "filter bubbles" that limit new product discovery.
• Support automation: DoorDash has built an LLM-based support chatbot that retrieves information from the knowledge base to generate responses that quickly resolve problems. This approach can improve response times, but it requires solid guardrails to handle complex or emotionally charged situations.

Social, Media and B2C: Personalized Content and Interactions

‍

In social media and B2C services, LLMs are used to create personalized content and improve interactions:

• Content analysis and moderation: Yelp has updated its content moderation system with LLM to detect threats, harassment, obscenity, personal attacks, or hate speech. LinkedIn analyzes various content on the platform to extract information about skills. These cases show the potential of LLMs in improving the quality of content, but raise concerns about censorship and the potential restriction of freedom of expression.
• Educational content generation and marketing: Duolingo uses LLM to help designers generate relevant exercises, while Nextdoor employs LLM to create eye-catching email objects. These applications can increase efficiency, but they can also lead to over-standardization of content.
• Multilingual translation and communication: Roblox leverages a customized multilingual model to enable users to communicate seamlessly using their own language. This application shows the potential of LLMs in overcoming language barriers, but may introduce cultural nuances that are incorrect in translations.
• Interaction with media content: Vimeo allows users to converse with videos through a RAG-based question-and-answer system that can summarize video content, link to key moments, and suggest additional questions. This application shows how LLMs can transform the way we interact with multimedia content, but raises questions about the fidelity of the interpretations generated.

Critical Evaluation: Real Value vs. Following the Trend

As Chitra Sundaram, director of the data management practice at Cleartelligence, Inc. points out, "LLMs are resource devourers. Training and running these models requires enormous computing power, leading to a significant carbon footprint. Sustainable IT is about optimizing resource use, minimizing waste and choosing the right size solution." This observation is particularly relevant when analyzing the use cases presented.

‍

In analyzing these use cases, several critical considerations emerge:

1. Incremental Value vs. Complexity

‍

Many applications of LLMs offer incremental improvements over existing solutions, but with significantly higher computational, energy and implementation costs. As Chitra Sundaram states, "Using an LLM to compute a simple average is like using a bazooka to hit a fly" (paste-2.txt). It is critical to consider whether the value added justifies this complexity, especially considering:

• The need for robust monitoring systems
• Energy costs and environmental impact
• The complexity of maintenance and updating
• The requirements for specialized skills

2. Dependence on Human Supervision.

‍

Most successful use cases maintain a "human-in-the-loop" approach, where LLMs assist rather than completely replace human intervention. This suggests that:

• Full automation through LLM remains problematic
• The main value is in increasing human capabilities, not replacing them
• Effectiveness depends on the quality of human-machine interaction

3. Domain Specificity vs. Generic Applications.

‍

The most compelling use cases are those in which LLMs have been adapted and optimized for specific domains, with domain knowledge embedded through:

• Fine-tuning on industry-specific data
• Integration with existing systems and knowledge sources
• Guardrails and context-specific constraints

4. Integration with Existing Technologies

The most effective cases do not use LLMs in isolation, but supplement them with:

• Data recovery and archiving systems (RAG)
• Specialized algorithms and existing workflows
• Verification and control mechanisms

As Google's use case highlights, integrating LLMs into security and privacy incident workflows enables "accelerated incident response using generative AI," with generated summaries being tailored to various audiences, ensuring that relevant information reaches the right people in the most useful format.

‍

Conclusion: A Pragmatic Approach to LLMs.

Chitra Sundaram offers an enlightening perspective when he says, "The path to sustainable analytics is about choosing the right tool for the job, not just chasing the latest trend. It is about investing in skilled analysts and sound data governance. It's about making sustainability a key priority."

‍

Analysis of these real-world use cases confirms that LLMs are not a miracle solution, but powerful tools that, when applied strategically to specific problems, can offer significant value. Organizations should:

1. Identify specific problems where natural language processing offers a substantial advantage over traditional approaches
2. Start with pilot projects that can demonstrate value quickly and measurably
3. Integrating LLMs with existing systems rather than completely replacing workflows
4. Maintain human supervision mechanisms, especially for critical applications
5. Systematically assess cost-effectiveness, considering not only performance improvements but also energy, maintenance, and upgrade costs

Companies that thrive in the era of LLMs are not necessarily those that adopt them more widely, but those that apply them more strategically, balancing innovation and pragmatism, and keeping a critical eye on the real value generated beyond the hype.

‍