Generative AI Faces ROI Challenges: Key Insights and Solutions

While executives and managers are enthusiastic about applying generative artificial intelligence (AI) and large language models (LLMs) to their operations, understanding and realizing the returns on investment (ROI) remain significant challenges. This area requires new approaches and skillsets distinct from previous technology waves.

The ROI Challenge

Despite the impressive proofs of concept that AI can deliver, monetizing these innovations is difficult. Steve Jones, executive VP at Capgemini, highlighted this challenge at the recent Databricks conference in San Francisco, noting that "Proving the ROI is the biggest challenge of putting 20, 30, 40 GenAI solutions into production."

Essential investments include testing and monitoring LLMs in production. Testing, in particular, is crucial to maintain accuracy and reliability. Jones advised being rigorous and even intentionally "poisoning" models during testing to assess their robustness against erroneous information.

Testing and Validation

Jones shared an example where he prompted a business model with a fictional scenario involving dragons for long-distance haulage. The model responded with detailed but fictional information, highlighting the need for rigorous testing to prevent such errors in real-world applications.

Generative AI, according to Jones, is a technology prone to being poorly integrated into existing systems, adding superficial features while posing security and risk challenges in production. He predicts that generative AI will take two to five years to reach mainstream adoption, a relatively rapid timeline.

Market Competition and Variation

The generative AI market is expected to see intense competition among vendors and platforms. Jones emphasized the need for businesses to focus on efficient and cost-effective use of LLMs, avoiding over-reliance on a single, costly model for all tasks.

Optimizing AI Deployment

Businesses should look for cheaper and more efficient ways to leverage LLMs. Jones suggested being prepared to decommission solutions as quickly as they are commissioned and ensuring all related artifacts are managed in step with the models.

He also recommended using multiple models to measure performance and quality, capturing metrics to compare different models' effectiveness. For instance, comparing responses from GPT-4 Turbo against Llama can reveal more cost-effective options.

The Need for Guardrails

Generative AI can produce unexpected results, such as generating irrelevant content from a simple query. Implementing guardrails is essential to prevent such errors and ensure AI solutions deliver meaningful and accurate outputs.

Conclusion

Understanding and proving the ROI of generative AI is a complex task requiring new strategies and rigorous testing. Businesses must focus on efficient deployment, competitive analysis, and robust validation processes to realize the full potential of generative AI technologies.