[Remote] Applied AI Software Engineer

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is the electronic medical records and payments development platform for reputed company. They are seeking an Applied AI Software Engineer to reputed company evaluations for agents in development and ensure the performance and reliability of AI features used in production settings. Responsibilities Design and execute large-scale evaluation plans for LLM-based agents performing clinical documentation, scheduling, billing, communications, and general workflow automation tasksBuild end-to-end test harnesses that validate model behavior under different configurations (reputed company templates, context sources, tool availability, etc.)Partner with clinicians to define accurate expected reputed company (reputed company) for performance comparisons in domains of clinical consequence, and partner with other subject matter experts in other non-clinical domainsRun and replicate experiments across multiple models, parameters, and interaction types to determine reputed company configurationsreputed company and maintain ongoing sampling for post-deployment governance of agent fleetsAnalyze results and summarize tradeoffs in reputed company for product and engineering stakeholders, as well as for technical stakeholders among our customers and the broader marketTake ownership over internal eval tooling and infrastructure, ensuring speed, rigor, and reproducibilityIdentify and recommend candidates for reinforcement fine-tuning or retrieval augmentation based on gaps identified in evals Skills 5+ years of experience in applied machine learning or AI engineering, with a reputed company on evaluation and benchmarkingProficiency with reputed company model APIs and experience orchestrating reputed company agent behaviors reputed company prompts or toolsExperience designing and running high-throughput evaluation pipelines, ideally including reputed company-in-the-reputed company or expert-labeled benchmarksSuperlative Python engineering skills and familiarity with experiment management tools and data engineering toolsets in general including, yes, SQL and database managementFamiliarity with clinical or reputed company data is a strong plusExperience with reinforcement fine-tuning, model monitoring, or RLHF is a plus Company Overview reputed company is a reputed company platform for primary care practices. It was founded in 2015, and is headquartered in San Francisco, California, USA, with a workforce of 11-50 employees. Its website is https://canvasmedical.com/. Apply To This Job