Senior Software Engineer – Training & Registry (AI Platform)

At Datadog, we are building a next-generation AI platform that enables seamless training, tracking, and deployment of ML and LLM models at scale. The Training & Registry team is responsible for the infrastructure and tooling that allows applied scientists to iterate rapidly and reliably—managing training jobs, tracking experimentation, and versioning model artifacts across distributed systems.

Our work is foundational to AI development at Datadog, powering everything from classic ML workflows to large-scale LLM fine-tuning and embedding generation. We build deeply technical infrastructure: distributed systems for job orchestration, model lifecycle management, and training observability. This is a high-impact team working on problems critical to Datadog’s AI evolution.

We’re looking for a Senior Software Engineer to design and build robust backend and platform systems that drive model experimentation and registry workflows. In this role, you'll collaborate with platform teams, applied scientists, and infra stakeholders to shape the future of AI infrastructure at Datadog.

At Datadog, we place value in our office culture - the relationships that it builds, the creativity it brings to the table, and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them.

What You’ll Do:

• Design and implement scalable, reliable systems for training orchestration, artifact tracking, and model registration across multiple data centers and cloud regions.
• Improve and streamline ML experimentation workflows by integrating tooling like Ray, Airflow, and interactive notebooks.
• Develop APIs and services that enable applied scientists to seamlessly launch, debug, and track training jobs.
• Ensure reproducibility and traceability by building robust version control and metadata systems for model artifacts.
• Collaborate with AI infra teams (LLMObs, Compute, etc.) to deliver consistent user experiences and integrated telemetry.
• Mentor engineers and help drive architectural decisions and technical standards.

Who You Are:

• You have 6+ years of experience in backend, distributed systems, or platform engineering roles.
• You have worked on ML platforms or infrastructure, ideally supporting real-world training or model lifecycle workflows.
• You’re comfortable designing APIs, managing data at scale, and architecting systems for reliability and observability.
• You’re fluent in Python or Go and have experience with cloud-native tools (e.g., Kubernetes, object stores, queueing systems).
• You’re comfortable navigating cross-functional environments and translating scientific requirements into reliable systems.
• Bonus points: experience with model registries, experiment tracking tools (e.g., MLflow, Weights & Biases), or distributed training frameworks.

Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you’re passionate about technology and want to grow your skills, we encourage you to apply.

Benefits and Growth:

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP)
• Continuous professional development, product training, and career pathing
• Intradepartmental mentor and buddy program for in-house networking
• An inclusive company culture, ability to join our Community Guilds (Datadog employee resource groups)
• Access to Inclusion Talks, our internal panel discussions
• Free, global mental health benefits for employees and dependents age 6+
• Competitive global benefits

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.