Software Engineering Evaluation Specialist

Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. About The Role You'll design coding tasks that challenge frontier AI coding agents. Each task is a self-contained Docker environment with a broken piece of software; an AI agent attempts the fix; automated tests verify the outcome. Your deliverable is the full task package: broken code, tests, instructions, and a reference solution proving the task is solvable. Responsibilities: Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problemBuild a reproducible Docker environment with pinned dependenciesWrite a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fixWrite an instruction.md that reads like a Jira ticket a developer would receiveWrite a reference solve.sh proving the task is solvableCalibrate difficulty so current state-of-the-art agents solve the task 20-60% of the timeIterate based on feedback from expert QA reviewersLater: review other authors' tasks as a QA reviewer Not in scope Data labeling, prompt engineeringProduction code to ship — you design problems and verification for AI agentsLeetcode puzzles — scenarios must look like real developer workNot every candidate task ships — quality over quantity Requirements 3+ years of production software development in one backend stack — Python, Go, Node.js, Java, or Rust. Depth in one stack beats breadthPython + pytest fluency — required regardless of primary stack. The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest.pyDocker authoring — reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root userLinux & Bash — comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefailAI coding agent experience — Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught itEnglish — B2+ written Not a fit Data Science, ML, or Computer Vision engineers without backend-engineering outputManual QA testers without automation or test authoringFrontend-only, low-code / no-code, IT Support, or Business AnalystsEngineers who have never written pytest from scratchJunior, intern, or assistant as the most recent role Preferred Qualifications Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internalsModern Python tooling (uv, poetry, pyproject.toml)Coverage tooling (pytest-cov, coverage.py, gcov, llvm-cov, kcov)Fuzzing or property-based testing (Hypothesis)Prior contribution to agent-evaluation benchmarks or related frameworks Process Apply → Pass qualification (90-minute sample-task screen + short behavioral interview) → Join a project → Complete tasks → Get paid. Time commitment Onboarding: :10 hours per first taskSteady state: :5 hours per task, 2-4 parallel tasks per authorRealistic weekly load: 8-20 hours. Higher volume available for top performersYou choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria Compensation: Paid contributions, rates up to $35/hour*Task-based compensation equivalent to hourly rate, depending on performance and volumeSome projects include incentive paymentsRates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be provided to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project Apply Submit your CV via the Mindrift platform. Indicate your English level, note this role (Software Engineering Evaluation Specialist — Terminal Bench), and include a GitHub profile link if available.