[Remote] AI Application Engineer

Note: The job is a remote job and is open to candidates in USA. Software Guidance & Assistance, Inc. (SGA) is seeking an AI Application Engineer for a contract assignment with an infrastructure services client. The role designs, tests, optimizes, and supports AI-powered agent experiences across customer and internal business workflows, including prompt engineering, agent orchestration, quality assurance, evaluation frameworks, and AI observability. ResponsibilitiesDesign, develop, and iterate on AI agent logic for business use casesCreate, test, and optimize prompts, system instructions, tools, and workflows to improve agent performanceConduct prompt experiments and A/B testing to measure effectivenessAnalyze agent responses and refine prompting strategies for accuracy, consistency, groundedness, and user satisfactionDocument prompt patterns, reasoning frameworks, reusable templates, and agent design decisionsCollaborate with engineering teams to deploy prompt and agent updates through established SDLC processesDesign and execute comprehensive test plans covering AI agents, web applications, APIs, and backend servicesDevelop evaluation frameworks and scoring methodologies for AI output qualityCreate test cases covering: Accuracy; Hallucination detection; Safety; Tone and style adherence; Instruction following; Context retention; Response consistencyPerform regression testing for prompt and model changesTrack, prioritize, validate, and retest defects using Azure DevOpsUtilize Arize or similar observability platforms to monitor model and agent performanceAnalyze user interactions and evaluation results to identify trends and improvement opportunitiesDevelop quality metrics and dashboards for: Accuracy; Relevance; Latency; User satisfaction; Defect rates; Prompt performanceInvestigate production issues and recommend prompt, workflow, or architectural improvementsSupport integration of AI capabilities into enterprise applications and workflowsCollaborate with developers to validate API integrations, retrieval pipelines, and agent workflowsAssist in CI/CD deployment and release validation activities using Azure DevOpsCreate and maintain automated testing assets where appropriateMaintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurationsParticipate in sprint planning, backlog refinement, and Agile ceremoniesCommunicate findings and recommendations to technical and non-technical stakeholdersShare best practices and lessons learned across the teamSkills Bachelor's degree in Computer Science, Engineering, Information Systems, or related field5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or related technical roles1+ years of hands-on experience with Generative AI, LLMs, prompt engineering, or conversational AI solutionsExperience working with Azure DevOps or similar Agile delivery platformsExperience designing and executing structured test plansStrong analytical and troubleshooting skillsExperience working with APIs and JSON data structuresExcellent written and verbal communication skillsPrompt EngineeringAgent DesignAI Evaluation FrameworksRAG ConceptsLLM TestingAI Quality AssuranceAI ObservabilityPrompt OptimizationArize AIAzure DevOpsAzure OpenAIGitPostmanPythonSQLREST APIsJSONTest Automation FrameworksExperience with Arize AI or comparable AI observability toolsExperience with Azure OpenAI, OpenAI, Anthropic, Gemini, or other LLM platformsKnowledge of Retrieval-Augmented Generation (RAG) architecturesExperience evaluating AI systems using human and automated scoring methodsFamiliarity with vector databases and semantic searchExperience creating automated test frameworksUnderstanding of responsible AI principles and AI governanceExperience with Python, SQL, JavaScript, or C#Knowledge of prompt versioning and experimentation practicesJira (optional)Power BI (optional)BenefitsThis role will be remote.Company Overview For 45 years, SGA has delivered technology and resource solutions that Fortune 500 organizations, mid-size companies, state and local governments, and higher education institutions rely on to achieve their goals/missions. It was founded in 1981, and is headquartered in Tarrytown, New York, USA, with a workforce of 201-500 employees. Its website is https://www.sgainc.com. Com