Overview In this role, you will help advance AGI work by building robust quality and auditing frameworks for data used to train and evaluate LLMs and multimodal systems. You will collaborate with core scientists to enhance Nova model performance through rigorous data quality controls and expert-level audits. You will design judge architectures and automated evaluation tools that scale quality feedback to engineers and stakeholders. This role offers impact through shaping training data standards, evaluation benchmarks, and customer-facing AI quality. You will work in a fast paced, cross-functional environment focused on high quality data and state of the art AI systems.
Compensation / Benefits- health insurance
- 401(k) matching
- paid time off
- parential leave
- RSUs
- sign-on payments
Responsibilities- Lead development of quality strategies and auditing frameworks for data collection workflows
- Design auditing SOPs, quality metrics, and sampling methodologies to improve Nova benchmarks
- Perform expert manual audits and meta-audits to assess auditor performance
- Develop and maintain LLM-as-a-Judge systems, including judge architectures and evaluation rubrics
- Build ML models for automated quality assessment and communicate feedback to stakeholders
- Configure data collection workflows and support cross-functional collaboration to implement quality practices
- Contribute to high-quality training and evaluation data for state-of-the-art LLM products
Key requirements- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field
- Experience programming in Java, C++, Python or related language
- Experience with SQL and an RDBMS or Data Warehouse
- Ability to perform root cause analysis and research auditing methodologies
- collaboration
- coaching
- stakeholder communication
- quality assurance frameworks
- data auditing methodologies
- LLM evaluation and judge systems
