Senior Application Engineer

The Senior AI Engineer will design, test, optimize, and support AI-powered agent experiences across customer-facing and internal business workflows. This senior-level role combines application engineering, prompt engineering, agent orchestration, AI quality assurance, evaluation framework development, and AI observability to improve agent reliability, accuracy, consistency, and user experience. The ideal candidate has 7+ years of relevant technical experience, a strong application development background, hands-on experience with Generative AI and Large Language Models (LLMs), and the ability to quickly understand complex business challenges and deliver effective solutions. The Senior AI Engineer will work closely with product managers, software engineers, architects, and business stakeholders while contributing independently with minimal ramp-up time., * Design, develop, and iterate on AI agent logic and workflows for complex business use cases.

• Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance.
• Conduct prompt experiments and A/B testing to measure effectiveness and identify improvement opportunities.
• Analyze agent responses and refine prompting strategies to improve accuracy, consistency, groundedness, and user satisfaction.
• Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions.
• Collaborate with engineering teams to deploy prompt and agent updates through established software development lifecycle processes.
• Design and execute comprehensive test plans for AI agents, web applications, APIs, and backend services.
• Develop evaluation frameworks and scoring methodologies for AI output quality.
• Create and maintain test cases covering accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention, and response consistency.
• Perform regression testing for prompt, model, application, and workflow changes.
• Track, prioritize, validate, and retest defects using Azure DevOps or similar platforms.
• Utilize AI observability platforms to monitor model and agent performance.
• Analyze user interactions and evaluation results to identify trends and improvement opportunities.
• Develop and monitor quality metrics related to accuracy, relevance, latency, user satisfaction, defect rates, and prompt performance.
• Investigate production issues and recommend prompt, workflow, application, or architectural improvements.
• Support the integration of AI capabilities into enterprise applications and business workflows.
• Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows.
• Support CI/CD deployment, release validation, and automated testing activities.
• Create and maintain automated testing assets and frameworks where appropriate.
• Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations.
• Participate in sprint planning, backlog refinement, and other Agile ceremonies.
• Communicate technical findings, recommendations, and solutions to technical and non-technical stakeholders.
• Share best practices, technical knowledge, and lessons learned across the team.

• Minimum of 7 years of relevant experience in application development, software engineering, QA engineering, automation testing, business systems analysis, or a related technical role.
• At least 1 year of hands-on experience with Generative AI, LLMs, prompt engineering, conversational AI, or AI-powered agent solutions.
• Strong application development and software engineering background with demonstrated experience solving complex business and technical challenges.
• Experience designing, developing, testing, and supporting enterprise applications.
• Experience designing and executing structured test plans and testing methodologies.
• Experience working with AI agents and agent-based workflows.
• Strong experience with Azure technologies and Azure DevOps or similar Agile delivery platforms.
• Experience working with APIs and JSON data structures.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent written and verbal communication skills.
• Ability to work independently, quickly understand complex requirements, and contribute effectively with minimal ramp-up time.
• Ability to collaborate effectively with cross-functional technical and business teams.
• No certifications are required for this position.

Education

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field is required.

Technical Skills

• Prompt engineering and prompt optimization.
• AI agent design and orchestration.
• AI evaluation frameworks and quality assurance.
• LLM testing and evaluation.
• Retrieval-Augmented Generation (RAG) concepts.
• AI observability and performance monitoring.
• Application development and software engineering practices.
• REST APIs and JSON.
• Python.
• SQL.
• Test automation frameworks.
• Git.
• Azure DevOps.
• Azure AI and LLM platforms.
• API testing tools.

Preferred Qualifications

• Experience with AI observability platforms such as Arize AI or similar tools.
• Experience with Azure OpenAI or other LLM platforms.
• Knowledge of Retrieval-Augmented Generation (RAG) architectures.
• Experience evaluating AI systems using human and automated scoring methods.
• Familiarity with vector databases and semantic search.
• Experience creating automated test frameworks.
• Understanding of responsible AI principles and AI governance.
• Experience with Python, SQL, JavaScript, or C#.
• Knowledge of prompt versioning and experimentation practices.
• Salesforce application development, integration, or platform experience is preferred for one available opening., We are committed to providing equal employment opportunities to all employees and applicants without regard to race, religion, color sex, national origin, citizenship status, age, sexual orientation, gender identity, marital status, uniform service member status, genetic information, disability, or any other protected status, in accordance with all applicable federal, state, and local laws. This practice extends to all aspects of our employment practices, including but not limited to, recruiting, hiring, firing, promoting, transferring, compensation, benefits, training, leaves of absence, and other terms and conditions of employment.

• This is a 100% remote position.
• Standard business hours are generally 8:00 AM to 5:00 PM local time, with some flexibility.
• This is a senior-level position and is not intended for entry-level candidates.

About Equiliem

Equiliem believes in empowering success. It’s our job to cultivate relationships that connect people and employers in a way that is inclusive, intelligent, and allows both to thrive.

Across the U.S., leading companies in healthcare, government, engineering, manufacturing, professional services, and energy rely on us for their workforce solutions. Our recruiting and HR services include contract and direct hire staffing, Payrolling/EOR, Independent Contractor Compliance, and Managed Services.

For almost 30 years, we’ve helped shape our industry. Today, we continue to research, ask questions, and continuously enhance the candidate journey and client experience., We offer a comprehensive benefits package to our employees, which includes:

• Medical Insurance
• Vision & Dental Insurance
• Life Insurance
• Matching 401(k)
• Commuter Benefits
• Employee Discounts & Rewards
• Payroll Payment Options