Software Engineer – Annotation & Evaluation (AI Platform)

At Datadog, we are scaling our  AI platform that enables the safe and effective development of AI features across the whole of Datadog. The Annotation & Evaluation team is a key player in this mission. We build infrastructure to help teams evaluate their models’ behaviour through labeled datasets, automated testing environments, and benchmark tracking tools.

Our users—applied scientists, AI engineers, and product teams—depend on our systems to iterate safely and ship models with confidence. We work across the stack to integrate human and automated evaluation workflows and to ensure model results are explainable and trustworthy.

We’re looking for a Software Engineer to contribute to our model evaluation infrastructure and support large-scale annotation and benchmarking workflows. This is a great opportunity for someone interested in the intersection of software engineering, AI, and data quality.

At Datadog, we place value in our office culture - the relationships that it builds, the creativity it brings to the table, and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them.

What You’ll Do:

• Contribute to evaluation infrastructure that enables model comparison, performance tracking, and trust & safety scoring.
• Build services and pipelines to support annotation workflows, benchmarking dashboards, and result aggregation.
• Collaborate with scientists and engineers to build and scale tools for evaluating models in controlled and production-like environments.
• Help maintain integrations with data labeling platforms and model evaluation libraries.
• Write clean, testable code to ensure reliability of evaluation tooling and data pipelines.

Who You Are:

• You have 3+ years of experience as a backend or platform engineer, ideally with exposure to ML systems or data engineering.
• You’re comfortable working in Python, Go, or similar languages and have built or maintained backend services or pipelines.
• You’re excited by the challenge of evaluating ML/LLM models in real-world contexts.
• You understand the importance of data quality, safety checks, and explainability in AI systems.
• Bonus points: experience with annotation platforms, telemetry replay, LLM evaluation tools, or metrics dashboards.

Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you’re passionate about technology and want to grow your skills, we encourage you to apply.

Benefits and Growth:

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training, and career pathing
• Intradepartmental mentor and buddy program for in-house networking
• An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks, our internal panel discussions
• Free, global mental health benefits for employees and dependents age 6+
• Competitive global benefits

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.