About the Role
Staff AI Engineer, Agents & EvaluationCompensation: $250,000 - $275,000 base + equityLocation: New York City, Onsite
We're partnering with an ambitious, venture-backed technology startup developing sophisticated AI systems to automate complex, data-intensive workflows.
The company is building a next-generation AI platform that combines large language models, autonomous agents, and advanced data infrastructure to solve challenging real-world problems. With strong early commercial traction and backing from established investors, the business is expanding its core engineering team as it scales its technology and customer base.
The OpportunityThis is a hands-on, high-impact engineering role at the intersection of LLM agents, backend engineering, and AI infrastructure.
You'll help define how intelligent agents are designed, evaluated, deployed, and continuously improved in production. The focus is on building technically sophisticated, reliable AI systems that deliver measurable outcomes, rather than experimentation or proof-of-concept development.
What You'll Be Doing
• Design, develop, and deploy production-grade LLM agents, including tool calling, orchestration, context management, and guardrails.
• Build evaluation infrastructure, including golden datasets, automated testing frameworks, LLM-as-judge systems, and regression testing.
• Develop observability and monitoring capabilities to track agent performance, accuracy, latency, and cost.
• Identify and resolve complex production failure modes to improve overall agent reliability.
• Benchmark LLMs, agent architectures, and emerging technologies to inform technical decisions.
• Collaborate closely with engineers and product stakeholders to develop scalable AI-powered workflows.
• Influence technical architecture and establish best practices for building and deploying reliable agentic systems.
What We're Looking For
• Strong software engineering background with extensive hands-on experience in Python and production backend development.
• Demonstrated experience building and deploying LLM-powered agents in production environments.
• Practical experience developing evaluation frameworks, automated testing infrastructure, or agent reliability systems.
• Deep understanding of agent orchestration, tool use, context management, and failure handling.
• Familiarity with emerging agent technologies and protocols such as MCP, A2A, OpenHands, or similar frameworks.
• Experience with production observability, performance monitoring, and model benchmarking.
• Strong architectural judgment, technical ownership, and the ability to operate independently.
• Experience working in fast-paced, high-growth engineering environments.
Why Join?
• Technical ownership: Influence the architecture and development of sophisticated AI systems from an early stage.
• Production AI: Work on complex agentic systems solving meaningful real-world problems.
• Early-stage impact: Join a growing engineering organization where your technical decisions will directly shape the product.
• Strong momentum: Join a well-funded business with meaningful early commercial traction.
• Collaborative engineering culture: Work alongside a highly technical team focused on building ambitious AI products.