AI Evaluation Analyst at Toptal

Toptal is seeking an AI Evaluation Analyst to assess the quality, performance, and reliability of AI systems. You’ll play a key role in improving AI products by analyzing outputs, identifying patterns, and delivering structured feedback to technical and cross-functional teams.

What You’ll Do

• Evaluate AI-generated outputs for accuracy, relevance, safety, and consistency
• Apply defined evaluation frameworks and quality standards
• Analyze model performance (qualitative and quantitative)
• Identify edge cases and failure patterns
• Document findings and provide actionable recommendations
• Collaborate with data scientists, engineers, and product teams

• Support testing, experiments, and benchmarking initiatives

What You Bring

• Strong analytical and critical thinking skills
• Excellent written communication and documentation abilities
• Experience with datasets, spreadsheets, and reporting tools
• Ability to apply structured guidelines consistently

• High attention to detail and ability to manage repetitive review tasks

Nice to Have: Background in QA, data analysis, research, content review, or exposure to AI/ML, NLP, or LLM evaluation workflows.

If you’re detail-oriented and passionate about improving AI systems, apply now.