Senior Operations Manager, Evaluations

About Us:

Hippocratic AI is building safety-focused large language model (LLM) for the healthcare industry. Our team comprised of ex-researchers from Microsoft, Meta, Nvidia, Apple, Stanford, John Hopkins and HuggingFace are reinventing the next generation of foundation model training and alignment to create AI-powered conversational agents for real time patient-AI interactions.

As Senior Operations Manager, Evaluations, you will be the engine behind our clinical-safety evaluation platform—the cornerstone that proves our AI agents are safe, effective, and ready for real-world care. Thousands of licensed clinicians have used our app to score conversations, flag safety issues, and suggest improvements. Internal researchers rely on the resulting data to iterate fast.

Your mission: keep the app humming, the clinicians happy, and the research teams unblocked—all while driving down turnaround time and cost per evaluation. You will partner with Product, Engineering, Finance, and Clinical Safety to improve work flows and build processes to make evaluation of our AI agents as effective and seamless as possible.

Responsibilities

• Run the shop: Own day-to-day operations for all clinical, safety, and user-experience evaluations—scheduling, SLAs, escalation paths, and reporting.

• Build & scale teams: Recruit, and onboard a global bench of licensed nurses and doctors; design performance-management loops that keep quality high

• Automate everything: Drive adoption of auto-evaluation tooling; partner with ML engineers to expand test coverage, add new rubrics, and tune pass-/fail thresholds.

• Process improvement: Map current workflows, identify bottlenecks, and ship iterative fixes—reducing turnaround time, cost per eval, and defect rate quarter over quarter.

• Data & dashboards: Define KPIs (e.g., evaluation cycle time, false-negative/positive rates, reviewer utilization); build real-time dashboards and monthly business reviews.

• Compliance & audit readiness: Ensure every evaluation artifact (prompts, ratings, nurse notes) is stored, traceable, and audit-ready for regulators and enterprise customers.
  

Required Skills & Qualifications

• 5 + years in operations, program management, or business process ownership—ideally in health-tech, clinical trials, digital health, or AI/ML.

• Demonstrated success scaling high-volume, high-accuracy processes (millions of transactions or thousands of clinical cases per month).

• Metrics obsession—comfortable defining KPIs, running A/B or DOE experiments, and presenting insights to executives.

• Outstanding written & verbal communication; able to translate clinical or technical nuance into clear SOPs and exec summaries.
  

Preferred

• Experience running eval or annotation pipelines for LLMs or ML products.

• Prior work in a regulated environment (healthcare, med-device, pharma, finance).

• Lean Six Sigma, PMP, or similar process-improvement certification.

• Clinical background (RN/PA/MD) or formal training in quality management.

• Fluency with process-automation tools and data dashboards (Airtable/Asana/Jira, SQL, Looker/Mode, Python or similar).
  

Why Join Our Team:

• Innovative Mission: We are developing a safe, healthcare-focused large language model (LLM) designed to revolutionize health outcomes on a global scale.

• Visionary Leadership: Hippocratic AI was co-founded by CEO Munjal Shah, alongside a group of physicians, hospital administrators, healthcare professionals, and artificial intelligence researchers from leading institutions, including El Camino Health, Johns Hopkins, Stanford, Microsoft, Google, and NVIDIA.

• Strategic Investors: We have raised a total of $278 million in funding, backed by top investors such as Andreessen Horowitz, General Catalyst, Kleiner Perkins, NVIDIA’s NVentures, Premji Invest, SV Angel, and six health systems.

• World-Class Team: Our team is composed of leading experts in healthcare and artificial intelligence, ensuring our technology is safe, effective, and capable of delivering meaningful improvements to healthcare delivery and outcomes.

For more information, visit www.HippocraticAI.com.

We value in-person teamwork and believe the best ideas happen together. Our team is expected to be in the office five days a week in Palo Alto, CA.

Why You’ll Love Working Here: At Hippocratic AI, we are revolutionizing the healthcare landscape through cutting-edge technology. We want talented individuals who thrive at the intersection of innovation and impact. You’ll work alongside some of the brightest minds in healthcare and AI to shape the future of healthcare accessibility. This is your chance to get in on the ground floor of something big, with a direct line to transforming healthcare with innovative AI solutions.