Model Evaluator (DevOps)
- Company
- Benture
- Location
- Remote, South Africa
- Salary
- 25 $/h
- Posted
- 2026-11-08
Skills: ChatGPT, Docker, Git, GitHub, Java, JavaScript, PHP, Python, Automated Testing, TypeScript
Model Evaluator - DevOps | $25/task | Remote (Asia/Africa/LATAM) | 1-month contractor role
Turing is seeking experienced software engineers to evaluate AI coding agents for a frontier AI training and evaluation initiative. You will review and compare AI-generated coding solutions on real-world software engineering tasks and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.
What you'll do
• Review and compare three AI coding agents working on the same real-world developer request. • Analyze each agent's complete execution, including shell commands, files modified, code changes, test results and final responses. • Evaluate each agent across 10 quality dimensions, including instruction following, code correctness, efficiency, safety, honesty and communication. • Assign each agent an overall quality score from 1 to 5 and rank all three solutions. • Identify coding errors, incomplete implementations, inefficient approaches, security concerns and incorrect assumptions. • Write concise, structured, evidence-based rationales explaining your scores and rankings.
Requirements
• 3+ years of hands-on software development experience in at least one major language such as Python, JavaScript/TypeScript, Java, Go or PHP. • Strong understanding of code review, debugging, testing and development best practices. • Practical experience with shell commands, Git, Docker, build tools and automated test outputs. • Ability to independently…
Full text and application on the employer's site.