> Markdown version of [/jobs/ext/2634381-ai-solutions-architect](https://www.wearedevelopers.com/jobs/ext/2634381-ai-solutions-architect). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Solutions Architect - **Company:** TESTLIO INC. - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $140,000.0 - $170,000.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Data Analysis, Software as a Service, Machine Learning, Software Engineering, Technical Data Management Systems, Test Execution Engine, Strategies of Testing, Large Language Models, Multi-Agent Systems, SailPoint, Data Pipelines - **Published:** August 24, 2026 - **Apply:** https://job-boards.greenhouse.io/testlio/jobs/8121713 ## About the Role * Hands-on Data Science Background: Extensive practical experience developing, training, or fine-tuning machine learning models, NLP structures, or data pipelines. * AI Evaluation Expertise: Deep familiarity and hands-on experience with modern LLM evaluation frameworks (e.g., DeepEval, RAGAs, etc) and metric-driven validation ecosystems. * Agentic Framework Proficiency: Direct experience working with, deploying, or testing autonomous agent architectures, planning patterns, or multi-agent swarms (e.g., LangGraph, AutoGen, CrewAI). * Strong Software Engineering Foundations: Strong software engineering foundations, including proficiency in dealing with complex non-deterministic and deterministic systems. Human Skills * Proven Client-Facing Experience: Solid track record successfully leading discovery calls, managing stakeholder workshops, and consulting on real-world enterprise architectures. * Commercial & Strategic Acumen: Experience designing service offerings, understanding pricing dynamics, and formulating go-to-market strategies within professional services or SaaS. * Consultative Communication: Exceptional ability to bridge technical data science outputs with clear, strategic customer business value, speaking confidently with both engineers and executives. * Comfort with Ambiguity: High adaptability to navigate fast-evolving client layouts, unvetted sources, and the non-deterministic characteristics of AI application lifecycles. * Mentorship Mindset: A passion for continuous learning, documentation, and sharing architectural best practices to uplift internal team members. ## Description * Define and maintain the AI service catalog: offerings, tiers, packaging, and delivery methodology. Track, analyze, and present key metrics to stakeholder leadership as proof of quality, efficiency, and ROI. * Vision Activation & Marketing: Imagine, drive, evangelize, and activate a viewpoint on the future of AI Testing. Produce white papers, articles, posts, speeches, and webinars. * Sales Collaboration: Collaborate with sales and engagement teams to support pre-deal discovery, scoping, and articulating Testlio's AI testing value proposition. * Offering Lifecycle Management: Use market and competitive intelligence to refine offerings and supply Product Marketing with details for accurate positioning. Collaboration on Hands-on Delivery * System & Architectural Deconstruction: Execute comprehensive system-level breakdowns of client AI stack components-including autonomous agents, system prompts, foundation models, and RAG architectures-to formulate a tailored AI testing approach, inform test strategy, design evaluations, and provide technical advisory. * Delivery Expertise: Act as an escalation point and subject-matter expert for Delivery, stepping in hands-on from scoping to review during complex or new client engagements. * Technical Problem Solving: Solve technical roadblocks, stand up environments, and work on solutions that make AI testing and evaluation scalable and repeatable in strategic engagements. * Framework Advancement: Advance Testlio's proprietary AI testing framework by combining deterministic code graders, LLM-as-a-judge, and human-in-the-loop validation. * Empower Internal Teams: Develop templates, curated training datasets, and comprehensive documentation to drive seamless, high-quality test execution., Artificial Intelligence * Big Data * Cloud * Machine Learning * Software * Business Intelligence * Data Privacy The role involves designing cloud infrastructure, managing production Kubernetes clusters, optimizing CI/CD pipelines, enhancing developer experience, and ensuring reliable AI workloads. Candidates should have extensive experience in infrastructure and distributed systems engineering with strong coding skills and cloud expertise. Top Skills: AWSAzureDatadogDockerElkGCPGoGrafanaJavaKubernetesPrometheusPythonTerraform Civic Roundtable ## Related Videos - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) - [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) - [User 1st! Technology 2nd! Stop building AI nobody uses - start delivering real business outcomes](https://www.wearedevelopers.com/videos/100337-user-1st-technology-2nd-stop-building-ai-nobody-uses-start-delivering-real-business-outcomes) - [Python-Based Data Streaming Pipelines Within Minutes](https://www.wearedevelopers.com/videos/1233-python-based-data-streaming-pipelines-within-minutes) - [Data Science on Software Data](https://www.wearedevelopers.com/videos/162-data-science-on-software-data) ## Related Articles - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere) - [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts)