O

Python Software Engineering LLM Evaluator

OpenTrain AI · Anywhere

Full-timeJuniorPythonDocker

🔥11 people viewed this job

About the Role

About OpenTrainOpenTrain is the #1 platform for finding and building careers in AI training and data labeling. It helps contributors discover projects, build a professional profile, and apply to opportunities that match their skills. Creating an OpenTrain account is free. Build a lasting portfolio of AI training and evaluation experience.Find flexible contractor work connected to the rapidly growing AI industry.About AI Training and LLM EvaluationAI training is the human side of building artificial intelligence. For language models, skilled contributors review outputs, design realistic tasks, and assess whether systems can produce accurate, useful code. Your software engineering judgment can help improve how models handle real development challenges. Work with realistic programming and bug-fixing scenarios.Evaluate model behavior against real codebases and software engineering expectations.Contribute to cutting-edge AI development through practical technical review.The RoleOpenTrain is recruiting Python software engineers for a three-month contractor assignment focused on LLM evaluation and training. You will work with public GitHub repositories and realistic software engineering tasks to assess how language models handle real code and bug-fixing scenarios. This is a fully remote, part-time assignment requiring 20+ hours per week. The role is available to candidates based in India, Pakistan, Nigeria, Egypt, Ghana, Bangladesh, Turkey, or Mexico, and pays $100 per task. Role type: Remote contractor, part timeAssignment length: Three monthsTime requirement: 20+ hours per weekPayment: $100 per taskWorking language: EnglishWhat You'll DoAnalyze and triage GitHub issues across open-source libraries.Configure repositories and Docker-based development environments.Run, modify, and test real codebases locally.Evaluate unit-test coverage and test quality.Assess LLM performance on software engineering and bug-fixing tasks.Collaborate with researchers to identify repositories and issues that provide meaningful challenges for LLMs.RequirementsThe structured role level is entry level, while the assignment specifically requires at least three years of software engineering experience. Candidates should be comfortable working independently with complex public codebases and evaluating both software quality and LLM performance. At least three years of software engineering experience.Strong Python software engineering experience.Proficiency with Git, Docker, and basic software pipeline setup.Ability to understand and navigate complex codebases.Experience running, modifying, and testing real-world projects locally.Judgment in evaluating unit-test coverage, test quality, and LLM bug-fixing performance.English proficiency for technical evaluation work.Preferred ExperienceExperience contributing to or evaluating open-source projects.Previous LLM research or evaluation experience.Why Build an AI Training Career With OpenTrainAI training and data-labeling work gives people a way to contribute directly to state-of-the-art AI systems. Many projects are remote and flexible, while specialized technical assignments let experienced professionals apply their existing expertise to emerging model-development work. Through OpenTrain, you can build a profile that showcases your AI training experience, discover projects aligned with your skills, and develop a longer-term portfolio in this fast-growing field. Work remotely with a flexible weekly commitment.Apply software engineering expertise to advanced AI evaluation.Grow experience in LLM testing, code assessment, and AI training.How to ApplyCreate or use your free OpenTrain account, review the assignment details, and submit your application. Make sure your profile reflects your Python, Git, Docker, software engineering, and code evaluation experience. Confirm that you are based in an eligible country.Highlight experience with complex codebases and real-world testing.Apply through OpenTrain in minutes.

OpenTrain AI has 1 open position on Remote Vibe Coding Jobs.

💬 Developer Questions

Ask the team a question — answers show up here

🎯

What does the interview process look like?

🤖

What AI/vibe coding tools does the team use daily?

👥

How big is the engineering team?

⏰

Is the team fully async or are there required meetings?

🚀

What does onboarding look like for remote hires?

🔧

Can you share more about the tech stack and architecture?

📈

What does career growth look like in this role?

📅

What does a typical day look like?

💰

Is there a salary range you can share?

📊

Is equity or stock options part of the package?

🌍

Are there timezone requirements or preferences?

🛂

Do you sponsor work visas?

🏢 Is this your listing? Claim it to answer questions

Similar Jobs

Helpful resources

Hiring for a similar role? Post your job here — it's free →