> Markdown version of [/jobs/ext/2696151-ai-application-engineer](https://www.wearedevelopers.com/jobs/ext/2696151-ai-application-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Application Engineer - **Company:** SGA Inc. - **Location:** Phoenix, AZ, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** JavaScript (Programming Language), A/B Testing, Application Programming Interfaces (APIs), Agile Methodology, Artificial Intelligence, JIRA, Automation of Tests, Microsoft Azure, Business Systems, C Sharp (Programming Language), Information Systems, Continuous Integration, JSON, Python (Programming Language), Scrum Methodology, Systems Development Life Cycle, Regression Testing, Power BI, Search Technologies, Software Engineering, SQL Databases, Software Testing Automation Framework, Strategies of Testing, Management of Software Versions, Web Applications, Software Organization, Enterprise Software Applications, Chatbots, Postman, Large Language Models, Grafana, Multi-Agent Systems, Prompt Engineering, Generative AI, Backend, Git, Information Technology, Low Latency, Virtual Agents, Api Design, Restful APIs, Api Management - **Published:** September 3, 2026 - **Apply:** https://www.dice.com/job-detail/3a13b746-ca4e-4af5-9580-f7ae65e05e4b ## About the Role * Bachelor's degree in Computer Science, Engineering, Information Systems, or related field. * 5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or related technical roles. * 1+ years of hands-on experience with Generative AI, LLMs, prompt engineering, or conversational AI solutions. * Experience working with Azure DevOps or similar Agile delivery platforms. * Experience designing and executing structured test plans. * Strong analytical and troubleshooting skills. * Experience working with APIs and JSON data structures. * Excellent written and verbal communication skills. * Technical Skills + AI & GenAI o Prompt Engineering o Agent Design o AI Evaluation Frameworks o RAG Concepts o LLM Testing o AI Quality Assurance o AI Observability o Prompt Optimization + Tools o Arize AI o Azure DevOps o Azure OpenAI o Git o Postman o Jira (optional) o Power BI (optional) + Development o Python o SQL o REST APIs o JSON o Test Automation Frameworks Preferred Skills : * Experience with Arize AI or comparable AI observability tools. * Experience with Azure OpenAI, OpenAI, Anthropic, Gemini, or other LLM platforms. * Knowledge of Retrieval-Augmented Generation (RAG) architectures. * Experience evaluating AI systems using human and automated scoring methods. * Familiarity with vector databases and semantic search. * Experience creating automated test frameworks. * Understanding of responsible AI principles and AI governance. * Experience with Python, SQL, JavaScript, or C#. * Knowledge of prompt versioning and experimentation practices. ## Description Software Guidance & Assistance, Inc., (SGA), is searching for a Sr. AI Application Engineer for a contract/right to hire/direct placement assignment with one of our premier Infrastructure services clients in Phoenix, AZ . We are seeking an AI Engineer to help design, test, optimize, and support AI-powered agent experiences across customer and internal business workflows. This role combines prompt engineering, agent orchestration, quality assurance, evaluation framework development, and AI observability. The ideal candidate is curious, analytical, and hands-on, with experience working with Large Language Models (LLMs), prompt design, testing methodologies, and modern software development practices. You will partner closely with product managers, software engineers, architects, and business stakeholders to improve agent behavior, reliability, accuracy, and user experience. Responsibilities : * Prompt Engineering & Agent Design + Design, develop, and iterate on AI agent logic for business use cases. + Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance. + Conduct prompt experiments and A/B testing to measure effectiveness. + Analyze agent responses and refine prompting strategies for accuracy, consistency, groundedness, and user satisfaction. + Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions. + Collaborate with engineering teams to deploy prompt and agent updates through established SDLC processes. * AI Quality Assurance & Testing + Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services. + Develop evaluation frameworks and scoring methodologies for AI output quality. + Create test cases covering: o Accuracy o Hallucination detection o Safety o Tone and style adherence o Instruction following o Context retention o Response consistency + Perform regression testing for prompt and model changes. + Track, prioritize, validate, and retest defects using Azure DevOps. * AI Evaluation & Observability + Utilize Arize or similar observability platforms to monitor model and agent performance. + Analyze user interactions and evaluation results to identify trends and improvement opportunities. + Develop quality metrics and dashboards for: o Accuracy o Relevance o Latency o User satisfaction o Defect rates o Prompt performance + Investigate production issues and recommend prompt, workflow, or architectural improvements. * Software Engineering & Integration + Support integration of AI capabilities into enterprise applications and workflows. + Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows. + Assist in CI/CD deployment and release validation activities using Azure DevOps. + Create and maintain automated testing assets where appropriate. * Documentation & Collaboration + Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations. + Participate in sprint planning, backlog refinement, and Agile ceremonies. + Communicate findings and recommendations to technical and non-technical stakeholders. + Share best practices and lessons learned across the team. ## Related Videos - [Should we build Generative AI into our existing software?](https://www.wearedevelopers.com/videos/1129-should-we-build-generative-ai-into-our-existing-software) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [Tips and Tricks for Working with JSON](https://www.wearedevelopers.com/videos/1229-tips-and-tricks-for-working-with-json) - [Bringing the power of AI to your application.](https://www.wearedevelopers.com/videos/1010-bringing-the-power-of-ai-to-your-application) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) ## Related Articles - [The Prompt Engineer ✍️](https://www.wearedevelopers.com/magazine/216-the-prompt-engineer) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Prompt Engineering is a Job of the Past](https://www.wearedevelopers.com/magazine/342-prompt-engineering-is-a-job-of-the-past) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)