Remote | Senior Software Engineer – LLM Evaluation & Repository Validation

We are sharing a specialised part-time consulting opportunity for experienced Senior Software Engineers with tech-lead-level expertise to contribute to advanced LLM evaluation, repository validation, and realistic software-engineering benchmark development. Selected professionals will work with high-quality public repositories to develop and validate verifiable software-engineering tasks, configure reproducible development environments, analyse open-source issues, assess test quality, and evaluate how advanced language models perform on realistic bug-fixing and code-modification scenarios. Key Responsibilities Repository Analysis & Issue Triage Analyse issues across well-maintained public software repositoriesIdentify technically meaningful problems suitable for LLM evaluationTriage issues by complexity, reproducibility, and engineering relevanceNavigate repository history to understand implementation contextPrioritise problems that require substantive software-engineering reasoning Repository Setup & Environment Validation Configure complex repositories for reliable local executionBuild reproducible development and testing environmentsContainerise projects using Docker where appropriateResolve dependencies and environment-specific configuration issuesValidate that repositories can be consistently executed and tested Code Modification & LLM Evaluation Modify and run real-world codebases locallyEvaluate model performance on bug-fixing and implementation tasksAssess whether generated solutions correctly address repository issuesIdentify failure patterns across different programming languages and task typesCompare generated solutions against professional engineering expectations Testing & Verification Quality Evaluate unit-test coverage and overall test qualityDetermine whether existing tests adequately validate intended behaviourIdentify missing edge cases and weak verification mechanismsDevelop or refine tests where stronger validation is requiredEnsure evaluation tasks have objective and reproducible outcomes Research Collaboration & Technical Leadership Collaborate with research teams on repository and task selectionIdentify software-engineering problems that remain challenging for LLMsContribute insight into benchmark difficulty and dataset coverageSupport expansion across programming languages and task complexityProvide technical leadership or guidance to junior engineers where required Ideal Profile Senior or tech-lead-level software-engineering experienceStrong expertise in at least one of Python, JavaScript, Java, Go, Rust, C, C++, C#, or RubyComfortable working across unfamiliar and complex codebasesStrong proficiency with Git and repository-based development workflowsPractical Docker and environment-setup experienceAbility to run, modify, debug, and test production-quality software locallyStrong understanding of software testing and unit-test designAbility to evaluate code quality, test coverage, and implementation correctnessExperience working with high-quality public repositoriesFamiliarity with widely used repositories with 500+ starsExperience contributing to or evaluating open-source software is advantageousStrong debugging and issue-triage capabilitiesAbility to identify technically challenging software-engineering tasksClear written communication and technical reasoning skillsComfortable collaborating remotely with research and engineering teamsPrevious LLM research or evaluation experience is advantageous but not requiredExperience with developer tools or software-automation agents is also beneficial Engagement Details Part-time independent contractor engagementFully remoteCommitment: approximately 20 hours per weekSome overlap with PST working hours is requiredInitial contract duration is approximately 3 monthsExpected project start is the following weekNo medical or paid-leave benefits are included under the contractor arrangementWork will involve public repository analysis, issue triage, environment setup, testing, and LLM evaluationAssignments may span Python, JavaScript, Java, Go, Rust, C/C++, C#, Ruby, and related technologiesProject work may include development-environment automation and repository validationContributors may have opportunities to lead or support junior engineersWork must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party About the Platform This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy