ElevenLabs Launches Tools for Custom Conversational AI Agents

ElevenLabs, known for its advanced AI voice cloning and text-to-speech services, announced the launch of new tools for building conversational AI agents. The update allows developers to create agents with custom features, including tone of voice, response length, and integration with user-specific knowledge bases, directly through the ElevenLabs developer platform.

How the New Tools Work

Developers can now log into their ElevenLabs accounts and use templates or start from scratch to create conversational AI agents. Key features include:

Customizable Agent Settings:

• Persona Creation: Set the agent’s primary language, first message, and system prompt to define its personality.

• Response Controls: Adjust tone of voice, response creativity (temperature), and token usage limits.

Knowledge Base Integration:

• Developers can upload files, URLs, or text blocks to build a unique knowledge base for their AI agents.

• Custom large language models (LLMs) can also be integrated.

Technical Adjustments:

• Options to fine-tune voice latency, stability, and maximum conversation length.

• SDK compatibility with Python, JavaScript, React, and Swift, as well as a WebSocket API for deeper customization.

Data Collection and Evaluation:

Companies can collect user data, like names and emails, and define natural language evaluation criteria to measure interaction success.

A Fully Integrated Pipeline

Sam Sklar, ElevenLabs’ head of growth, explained that many clients were already using its tools to create conversational AI agents but faced challenges with integrating knowledge bases and managing interruptions. This new product aims to address those pain points by providing an end-to-end solution.

ElevenLabs leverages its existing pipeline for text-to-speech (TTS) functionality while developing new speech-to-text (STT) capabilities for its conversational AI product. Although the company doesn’t yet offer a standalone speech-to-text API, it may do so in the future, positioning it as a competitor to major players like Google, Microsoft, Amazon, and OpenAI.

Competition in the Conversational AI Market

ElevenLabs’ move places it in direct competition with both emerging startups and established tech giants:

• AI Voice Startups: Companies like Vapi and Retell are also building conversational AI tools.

• Real-Time APIs: ElevenLabs will rival OpenAI’s real-time conversational API, but it believes its ability to switch models and offer extensive customization gives it an edge.

• Speech-to-Text Providers: If ElevenLabs launches its STT API, it will compete with specialized solutions like OpenAI’s Whisper, AssemblyAI, and Deepgram.

Looking Ahead

With its new conversational AI tools, ElevenLabs aims to cater to businesses looking for highly customizable agents that integrate seamlessly with their specific needs. By combining flexibility with state-of-the-art text-to-speech and speech-to-text pipelines, the company seeks to differentiate itself in an increasingly competitive market.

As ElevenLabs continues to expand its offerings and develop new capabilities, it positions itself not only as a leader in voice AI but also as a formidable player in the conversational AI space. Its focus on customization could appeal to businesses seeking tailored solutions, setting it apart from more generalized platforms.

Editor’s Note: This article was created by Alicia Shapiro, CMO of AiNews.com, with writing, image, and idea-generation support from ChatGPT, an AI assistant. However, the final perspective and editorial choices are solely Alicia Shapiro’s. Special thanks to ChatGPT for assistance with research and editorial support in crafting this article.