Principal AI Engineer

Loaction New York, NY Job Type Full-time Experience 5 - 10 Remote work policy On-site Visa sponsorship No Preferred Time Zone ET About The Role This role owns the design and implementation of AI agents that power intelligent workflows across our platform, from reasoning and automation to evidence-driven RCA, customer-facing copilots, internal operations, and product intelligence. You're a strong Python engineer who understands modern agentic AI architecture and can turn ambiguous product needs into production-grade systems. You are comfortable building agents that use tools, retrieve context, reason over structured and unstructured data, execute workflows, and integrate deeply with backend services. This is not a prompt-only role. We're looking for someone who can architect, build, test, deploy, and operate AI-powered features end to end. What You'll Build Agentic AI systems • Production-grade AI agents for different product and internal applications: • site-level connectivity analysis • evidence-first root cause analysis workflows • customer-facing assistant and copilot experiences • internal automation for operations, support, and engineering workflows • reasoning workflows over telemetry, incidents, site data, and knowledge sources • Modern agentic architecture: • tool/function calling • planning and execution loops • state and memory management • retrieval-augmented generation • structured outputs • workflow orchestration • multi-agent patterns where appropriate • guardrails, permissions, and safety constraints • observability and evaluation frameworks • AI-native product primitives: • agent task models • context assembly pipelines • tool registries • prompt/version management • human-in-the-loop review flows • agent execution traces • eval datasets and regression testing Python backend + product engineering • Backend services and APIs in Python: • FastAPI/Django/Flask-style services • workers and async jobs • integrations with internal systems and external APIs • data models supporting AI workflows • event-driven and workflow-driven architectures • Retrieval and knowledge systems: • vector search • hybrid search • document ingestion • chunking and indexing strategies • metadata filtering • grounding and citation workflows • Production AI infrastructure: • LLM provider integration • model routing • cost and latency optimization • Caching • rate limits and retries • monitoring and debugging • failure handling and fallback behavior • Feature delivery end to end: • product scoping • Architecture • Implementation • Testing • Deployment • Observability • iteration based on user feedback Responsibilities • Architect and implement production-grade AI agents that solve real business and product problems. • Build agent workflows that can reason over Eino's data, tools, telemetry, site models, incidents, and knowledge sources. • Own agent architecture patterns across planning, memory, retrieval, tool execution, structured outputs, evals, and observability. • Build and operate core Python backend services that support AI-powered product features. • Work closely with product, engineering, and leadership to identify high-value agentic AI use cases. • Move quickly from prototype to production while maintaining reliability, security, and maintainability. • Establish testing and evaluation discipline for AI systems: • unit/integration tests • prompt and workflow regression tests • agent evals • golden datasets • trace review • failure analysis • Drive practical AI engineering standards: • correctness over demos • grounded outputs • measurable quality • clear contracts between agents, tools, and backend services Required Qualifications • Strong experience building production Python systems, including services, APIs, workers, and backend infrastructure. • Hands-on experience building AI agents, LLM-powered applications, RAG systems, workflow automation, or tool-using AI systems. • Deep familiarity with modern agentic AI architectures, including tool/function calling, planning and execution loops, state and memory management, retrieval, structured outputs, guardrails, observability, and evaluation frameworks. • Ability to take features from ambiguous requirements to production deployment. • Strong software engineering fundamentals: system design, testing, debugging, performance, reliability, and maintainability. • Experience integrating AI systems with real products, databases, APIs, and operational workflows. • Strong product sense and ability to identify where AI agents create practical customer or business value. • Startup mindset: high ownership, bias to shipping, comfort with ambiguity, and ability to operate with limited direction. • Experience working in a fast-moving Seed, Series A, or similarly early-stage startup environment. Preferred / Nice-to-have • Experience with agent frameworks such as LangGraph, LangChain, LlamaIndex, CrewAI, AutoGen, or similar. • Experience with LLM APIs such as