Manager – AI Performance & Analytics

About the Role

We are looking for a Manager – AI Performance & Analytics to lead a team responsible for analyzing conversational AI performance and driving continuous improvements to Netomi’s AI platform.

This role sits at the intersection of data analytics, conversational AI, and LLM evaluation. The team analyzes customer conversations at scale, identifies quality gaps and automation opportunities, and partners with AI and product teams to improve model performance.

You will help define how we measure and improve the performance of AI systems that power customer support automation across millions of interactions.

About the Role

We are looking for a Manager – AI Performance & Analytics to lead a team responsible for analyzing conversational AI performance and driving continuous improvements to Netomi’s AI platform.

This role sits at the intersection of data analytics, conversational AI, and LLM evaluation. The team analyzes customer conversations at scale, identifies quality gaps and automation opportunities, and partners with AI and product teams to improve model performance.

You will help define how we measure and improve the performance of AI systems that power customer support automation across millions of interactions.

AI Performance Analytics

• Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities.

• Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction.

• Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau.

LLM Evaluation & Quality Insights

• Design frameworks to evaluate the response quality of LLM responses across dimensions such as:

  • factuality and groundedness

  • Relevance to user intent

  • Instruction Adherence

  • tone and customer experience

  • safety and compliance

  • Build scalable processes for LLM-based automated evaluation and human review workflows.

  • Identify systematic failure patterns in AI responses and recommend improvements.

  Model Improvement & Experimentation

  • Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities.

  • Design and analyze experiments to measure the impact of model improvements.

  • Generate insights that inform model training data, prompts, and product features.

  Conversational Insights

  • Identify trends in customer interactions to uncover:

    • new AI automation opportunities

    • gaps in intent coverage

    • common customer pain points

    • Translate analytics insights into actionable recommendations for product and AI teams.

    Team Leadership

    • Lead and mentor a team of data analysts focused on AI performance and conversational analytics.

    • Establish best practices for analytics, experimentation, and AI evaluation.

    • Collaborate closely with product, AI research, engineering, and customer success teams.

Experience

• 7–10 years of experience in data analytics, data science, or AI analytics.

• Experience managing or mentoring analytics teams.

• Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.

Technical Skills

• Strong SQL and data analysis skills.

• Experience with BI tools such as Tableau or similar platforms.

• Experience using Python for data analysis or experimentation.

• Familiarity with machine learning or large language models (LLMs).




Preferred Experience

• Experience evaluating AI or machine learning model performance.

• Experience working with conversational AI platforms or chatbots.

• Experience with experimentation frameworks and A/B testing.

Netomi is an equal opportunity employer committed to diversity in the workplace. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, disability, veteran status, and other protected characteristics.