Overview In this role you will strengthen data quality for state-of-the-art LLMs and multimodal systems by developing comprehensive auditing frameworks. You'll work with the AGI team to improve Nova models through rigorous quality strategies, meta-audits, and automated evaluation systems. You will design judge architectures and evaluation rubrics, and coach others to elevate overall quality. You'll drive impactful data collection workflows and communicate quality feedback to stakeholders, directly shaping customer experiences through better training and evaluation data.
Compensation / Benefits- health insurance (medical, dental, vision)
- RSUs and sign-on incentives
- 401(k) matching
- paid time off
- par medical leave
- EAP and mental health support
Responsibilities- Lead development of quality strategies and auditing frameworks for data collection workflows
- Design SOPs, quality metrics, and sampling methodologies to improve Nova benchmark performance
- Perform expert manual audits and meta-audits to assess auditor performance and provide coaching
- Develop and maintain LLM-as-a-Judge systems, including judge architectures and evaluation rubrics
- Build ML models for automated quality assessment and set up data collection workflow configurations
- Communicate quality feedback to stakeholders and influence training/evaluation data quality
- Collaborate with engineers, domain experts, and vendors to implement quality strategies and automated judging systems
- Support quality solution design, root cause analysis, and research of new auditing methodologies
Key requirements- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field
- Experience programming in Java, C++, Python
- Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse
- coaching and mentoring
- cross-functional collaboration
- strong communication of complex concepts
- machine learning
- statistics
- quality assurance
