Overview As Staff Applied Scientist at Datadog, you will define and guarantee the quality of an AI-powered Dashboards system at scale. You will shape evaluation strategies that cover offline/online metrics, cost, and end-to-end agent performance, driving improvements across retrieval, tool selection, and context efficiency. You'll lead with a product mindset, mentor teams, and collaborate across engineering to catch quality regressions before customers experience them. This role offers a chance to influence a central, data-driven observability platform with broad cross-team impact.
Compensation / Benefits- New hire stock equity (RSUs)
- Employee stock purchase plan (ESPP)
- Continuous professional development
- Inclusive culture and Employee Resource Groups
- Global benefits including health, dental, parental planning, mental health
- 401(k) plan and match
Responsibilities- Define and own the evaluation strategy for Dashboards and related teams
- Specify metrics for offline/online, quality, cost, single-turn and multi-turn scenarios
- Build evaluation datasets, golden traces, and regression harnesses for quality monitoring
- Improve retrieval relevance, tool-selection accuracy, and context efficiency in collaboration with engineers
- Provide technical leadership through design reviews, working groups, and mentorship
Key requirements- BS/MS/PhD in a scientific field, or equivalent experience
- 10+ years of engineering or applied science experience, including technical lead experience
- Proven track record leading ML or GenAI initiatives from research to production
- Significant experience with evaluation, experimentation, or measurement of ML systems at scale
- Strong product mindset and ability to drive cross-functional initiatives
- Ability to thrive in ambiguity and make technical decisions when unclear
- strong collaboration across cross-functional teams
- mentorship and technical leadership
- ambition and comfort with ambiguity
- ML/evaluation at scale
- GenAI initiatives in product environments
- measurement of end-to-end ML systems
