About the Role
Note: The job is a remote job and is open to candidates in USA. LeoTech is passionate about building software that solves real-world problems in the Public Safety sector. They are seeking an AI/LLM Evaluation & Alignment Engineer to ensure their AI solutions are accurate, safe, and aligned with public safety workflows, focusing on evaluation frameworks and responsible AI deployment.
Responsibilities
Build and maintain evaluation frameworks for LLMs and generative AI systems tailored to public safety and intelligence use casesDesign guardrails and alignment strategies to minimize bias, toxicity, hallucinations, and other ethical risks in production workflowsPartner with AI engineers and data scientists to define online and offline evaluation metrics (e.g., model drifts, data drifts, factual accuracy, consistency, safety, interpretability)Implement continuous evaluation pipelines for AI models, integrated into CI/CD and production monitoring systemsCollaborate with stakeholders to stress test models against edge cases, adversarial prompts, and sensitive data scenariosResearch and integrate third-party evaluation frameworks and solutions; adapt them to our regulated, high-stakes environmentWork with product and customer-facing teams to ensure explainability, transparency, and auditability of AI outputsProvide technical leadership in responsible AI practices, influencing standards across the organizationContribute to DevOps/MLOps workflows for deployment, monitoring, and scaling of AI evaluation and guardrail systems (experience with Kubernetes is a plus)Document best practices and findings, and share knowledge across teams to foster a culture of responsible AI innovation
Skills
Bachelor's or Master's in Computer Science, Artificial Intelligence, Data Science, or related field3–5+ years of hands-on experience in ML/AI engineering, with at least 2 years working directly on LLM evaluation, QA, or safetyStrong familiarity with evaluation techniques for generative AI: human-in-the-loop evaluation, automated metrics, adversarial testing, red-teamingExperience with bias detection, fairness approaches, and responsible AI designKnowledge of LLM observability, monitoring, and guardrail frameworks e.g Langfuse, LangsmithProficiency with Python and modern AI/ML/LLM/Agentic AI libraries (LangGraph, Strands Agents, Pydantic AI, LangChain, HuggingFace, PyTorch, LlamaIndex)Experience integrating evaluations into DevOps/MLOps pipelines, preferably with Kubernetes, Terraform, ArgoCD, or GitHub ActionsUnderstanding of cloud AI platforms (AWS, Azure) and deployment best practicesStrong problem-solving skills, with the ability to design practical evaluation systems for real-world, high-stakes scenariosExcellent communication skills to translate technical risks and evaluation results into insights for both technical and non-technical stakeholdersExperience with Kubernetes is a plus
Benefits
3 weeks of paid vacation – out the gate!!Generous medical, dental, and vision plans.Sick, and paid holidays are offered.
Company Overview
LeoTech is leading the effort to assist public safety efforts around the nation. It was founded in 2018, and is headquartered in Los Angeles, California, USA, with a workforce of 51-200 employees. Its website is https://leotechnologies.com.