Job Radar › All jobs

Model Evaluator (DevOps)

Company
Benture
Location
Remote, South Africa
Salary
25 $/h
Posted
2026-11-08

Skills: ChatGPT, Docker, Git, GitHub, Java, JavaScript, PHP, Python, Automated Testing, TypeScript

Model Evaluator - DevOps | $25/task | Remote (Asia/Africa/LATAM) | 1-month contractor role

Turing is seeking experienced software engineers to evaluate AI coding agents for a frontier AI training and evaluation initiative. You will review and compare AI-generated coding solutions on real-world software engineering tasks and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What you'll do

• Review and compare three AI coding agents working on the same real-world developer request. • Analyze each agent's complete execution, including shell commands, files modified, code changes, test results and final responses. • Evaluate each agent across 10 quality dimensions, including instruction following, code correctness, efficiency, safety, honesty and communication. • Assign each agent an overall quality score from 1 to 5 and rank all three solutions. • Identify coding errors, incomplete implementations, inefficient approaches, security concerns and incorrect assumptions. • Write concise, structured, evidence-based rationales explaining your scores and rankings.

Requirements

• 3+ years of hands-on software development experience in at least one major language such as Python, JavaScript/TypeScript, Java, Go or PHP. • Strong understanding of code review, debugging, testing and development best practices. • Practical experience with shell commands, Git, Docker, build tools and automated test outputs. • Ability to independently…

Full text and application on the employer's site.

Apply