Podium Enhances AI Agent Efficiency, Cuts Engineering Needs by 90% with LangSmith
Podium, a communication platform designed to help small businesses connect with customers, has significantly optimized its AI agent behavior and reduced engineering intervention by 90% using LangSmith, according to LangChain Blog.
About Podium
Podium assists small businesses, such as automotive dealers and jewelers, in responding promptly to customer inquiries via phone, text, email, and social media. Their mission is to convert leads into sales by ensuring quick responses to customer inquiries. Data from Podium reveals that responding within five minutes can result in a 46% higher lead conversion rate compared to responses made within an hour.
To enhance lead capture, Podium launched its AI Employee, an agentic application that engages with local business customers, schedules appointments, and closes sales. Initially, Podium utilized the LangChain framework for single-turn interactions. However, as their use cases expanded, they turned to LangSmith for comprehensive testing and monitoring of their large language model (LLM) interactions.
Testing Across the Agentic Development Lifecycle
LangSmith enabled Podium to establish feedback loops crucial for the development lifecycle of their AI Employee. The testing approach includes:
- Baseline Dataset Curation: Creating an initial dataset to represent basic use cases and requirements.
- Baseline Offline Evaluation: Conducting initial tests with the curated dataset to assess performance before production.
- Collecting Feedback: Gathering user-provided feedback and conducting online evaluations to monitor response quality in real-time.
- Optimization: Refining prompts, tuning retrieval mechanisms, and fine-tuning the model based on traced data.
- Ongoing Evaluation: Continuously updating the dataset with new scenarios and conducting offline evaluations to identify optimization opportunities.
Dataset Curation and Fine-Tuning Agents with LangSmith
Before adopting LangSmith, Podium engineers had to make numerous LLM calls per interaction to understand and resolve customer inquiries. LangSmith streamlined this process by logging and viewing traces to aggregate insights.
One challenge was the AI Employee's difficulty in recognizing when a conversation had ended, leading to awkward repeated goodbyes. Podium addressed this by creating a dataset with various conversation scenarios, including different ways conversations might conclude. They then upgraded to a larger model and curated outputs into a smaller model using model distillation. This approach allowed Podium to compare the fine-tuned model against the original, larger model, improving the agent’s ability to detect natural conversation endings.
The fine-tuned model showed a significant improvement in F1 scores, increasing from 91.7% to 98.6%, exceeding Podium's quality threshold of 98%.
High-Quality Customer Support for AI Platform Without Engineering Intervention
Since launching their AI Employee in January, Podium's Technical Product Specialists (TPS) team has been essential in troubleshooting real-time user issues. By using LangSmith, the TPS team could quickly identify and resolve issues without needing engineering intervention.
LangSmith provided clarity on whether issues were due to application bugs, incomplete context, misaligned instructions, or LLM issues. This capability significantly reduced the need for engineering intervention, allowing engineers to focus on development tasks rather than support.
What’s Next for Podium
By integrating LangSmith and LangChain, Podium has enhanced its customer experience tools. The company is also incorporating LangGraph to streamline agent orchestration and improve controllability over agent conversations. These advancements enable Podium to concentrate on their core mission of helping small businesses capture leads more effectively while efficiently designing, testing, and monitoring their LLM applications.