AI Solutions Architect

• Define and maintain the AI service catalog: offerings, tiers, packaging, and delivery methodology. Track, analyze, and present key metrics to stakeholder leadership as proof of quality, efficiency, and ROI.
• Vision Activation & Marketing: Imagine, drive, evangelize, and activate a viewpoint on the future of AI Testing. Produce white papers, articles, posts, speeches, and webinars.
• Sales Collaboration: Collaborate with sales and engagement teams to support pre-deal discovery, scoping, and articulating Testlio’s AI testing value proposition.
• Offering Lifecycle Management: Use market and competitive intelligence to refine offerings and supply Product Marketing with details for accurate positioning.

Collaboration on Hands-on Delivery

• System & Architectural Deconstruction: Execute comprehensive system-level breakdowns of client AI stack components-including autonomous agents, system prompts, foundation models, and RAG architectures-to formulate a tailored AI testing approach, inform test strategy, design evaluations, and provide technical advisory.
• Delivery Expertise: Act as an escalation point and subject-matter expert for Delivery, stepping in hands-on from scoping to review during complex or new client engagements.
• Technical Problem Solving: Solve technical roadblocks, stand up environments, and work on solutions that make AI testing and evaluation scalable and repeatable in strategic engagements.
• Framework Advancement: Advance Testlio’s proprietary AI testing framework by combining deterministic code graders, LLM-as-a-judge, and human-in-the-loop validation.
• Empower Internal Teams: Develop templates, curated training datasets, and comprehensive documentation to drive seamless, high-quality test execution., Artificial Intelligence * Big Data * Cloud * Machine Learning * Software * Business Intelligence * Data Privacy The role involves designing cloud infrastructure, managing production Kubernetes clusters, optimizing CI/CD pipelines, enhancing developer experience, and ensuring reliable AI workloads. Candidates should have extensive experience in infrastructure and distributed systems engineering with strong coding skills and cloud expertise. Top Skills: AWSAzureDatadogDockerElkGCPGoGrafanaJavaKubernetesPrometheusPythonTerraform Civic Roundtable

• Hands-on Data Science Background: Extensive practical experience developing, training, or fine-tuning machine learning models, NLP structures, or data pipelines.
• AI Evaluation Expertise: Deep familiarity and hands-on experience with modern LLM evaluation frameworks (e.g., DeepEval, RAGAs, etc) and metric-driven validation ecosystems.
• Agentic Framework Proficiency: Direct experience working with, deploying, or testing autonomous agent architectures, planning patterns, or multi-agent swarms (e.g., LangGraph, AutoGen, CrewAI).
• Strong Software Engineering Foundations: Strong software engineering foundations, including proficiency in dealing with complex non-deterministic and deterministic systems.

Human Skills

• Proven Client-Facing Experience: Solid track record successfully leading discovery calls, managing stakeholder workshops, and consulting on real-world enterprise architectures.
• Commercial & Strategic Acumen: Experience designing service offerings, understanding pricing dynamics, and formulating go-to-market strategies within professional services or SaaS.
• Consultative Communication: Exceptional ability to bridge technical data science outputs with clear, strategic customer business value, speaking confidently with both engineers and executives.
• Comfort with Ambiguity: High adaptability to navigate fast-evolving client layouts, unvetted sources, and the non-deterministic characteristics of AI application lifecycles.
• Mentorship Mindset: A passion for continuous learning, documentation, and sharing architectural best practices to uplift internal team members.

Be an Early Applicant Remote Hiring Remotely in United States Expert/Leader Remote Hiring Remotely in United States Expert/Leader Owns the strategy, packaging, architecture, and delivery methodology for AI testing services. Designs evaluation approaches for LLMs, RAG systems, autonomous agents, and multi-agent architectures using automated graders, LLM judges, and human validation. Supports enterprise discovery, sales, delivery, and technical escalations; advances internal frameworks, develops training datasets and documentation, and defines service offerings, pricing, metrics, and go-to-market content. The summary above was generated by AI, An Hour Ago In-Office or Remote 140K-170K Annually Senior level 140K-170K Annually Senior level Social Impact * Software Owns core product areas across the full lifecycle, from discovery and strategy through execution, launch, and iteration. Conducts customer research, workflow analysis, usability testing, competitive research, and data analysis. Partners closely with engineering and design to deliver scalable user experiences, defines success metrics and instrumentation, and supports launches through messaging and cross-functional readiness. The role also requires hands-on AI experience and operates in a high-ownership, low-structure B2B SaaS environment. Top Skills: AICRM SailPoint

Customer Success Manager

An Hour Ago Remote or Hybrid United States 60K-101K Annually Mid level 60K-101K Annually Mid level Artificial Intelligence * Cloud * Sales * Security * Software * Cybersecurity * Data Privacy Manage assigned client accounts to ensure satisfaction and renewals. Coach clients on SailPoint/IdentityIQ identity and access solutions, monitor usage and risks, provide strategic updates, identify expansion opportunities, and drive resolutions to customer issues. Top Skills: IdentityiqSailpoint

What you need to know about the Colorado Tech Scene

With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.

Key Facts About Colorado Tech

• Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
• Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
• Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
• Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
• Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
• Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute

Testlio’s fully managed crowdsourced testing platform, powered by our proprietary intelligence technology - LeoCore , integrates expert, on-demand testers directly into your release process. Ship faster and more confidently with global coverage across 600,000+ devices, 800+ payment methods, 150+ countries, and 100+ languages. To learn more, visit testlio.com.

We are hiring a visionary and hands-on AI Solutions Architect to own the definition, packaging, technical design, and delivery methodology for our AI testing offerings. This role brings together deep practitioner expertise with a strong business perspective. You will design the testing approach for complex AI-powered applications (including agentic systems), and you will advance Testlio’s proprietary framework combining automated and human-in-the-loop evaluation while defining what Testlio sells as a service-setting the catalog, staffing models, and pricing structure. As the technical authority, you will decompose architectures into structured testing approaches and collaborate with sales and delivery to drive our AI business forward. Why will you love this job?

• Evaluate at a Scale That Exists Nowhere Else: Every vendor has LLM-as-a-judge. Only Testlio can put thousands of vetted experts across 150 countries and 100+ languages behind a grading rubric. You will design evaluation systems that combine model graders and human experts.
• Innovate at the Frontier of QA: Shape the industry standards and operational playbooks for validating agentic AI, large language models, and the data pipelines behind them.
• High-Impact Consultative Visibility: Collaborate with global market leaders across diverse verticals. Interface directly with data scientists, machine learning engineers, and executive sponsors at market-leading global brands to architect their AI quality blueprints.
• Decompose Complex Ecosystems: Turn “unknown unknowns” into structured testing criteria: what the system must do, how it can fail, and what a failure costs.
• Drive company influence: Help build the pipeline, join sales calls and guide delivery on AI Testing opportunities, bringing your expertise to high-impact business decisions.

Why will you love being a part of Testlio?

• Great Culture: Testlio is a female-founded company, and half of our team identifies as women. We’re proud of our inclusive, purpose-driven culture where people genuinely enjoy collaborating. As part of our team (we call ourselves TestLions), you’ll help create exceptional digital experiences for our customers, while also contributing to our freelance network.
• Remote Work: Our culture is built around remote work. We’ve created systems to allow us to successfully work together asynchronously as a fully remote and globally distributed team. Testlio provides the tools and guidance for everyone to succeed in their careers in a fully remote setting. Our working style encourages everyone to make decisions, communicate effectively, and work at a sustainable pace.
• Investment in You: Your growth and well-being matter to us. You’ll have flexible paid time off-including national holidays, personal days, and sick days-plus stock options so you can grow with Testlio. We also provide a $300 annual learning stipend to support your personal and professional development.
• Winning Business: Testlio is growing, profitable, and cash-strong. We are leading our industry with exceptional clients who provide us with a high NPS score and a 4.7 rating on G2. Our business model is global, enterprise, and subscription-based. Several of our largest clients have been with us for 7+ years, and many spend $500K+/year with Testlio., Testlio is an equal-opportunity employer deeply committed to creating an inclusive environment for people of all backgrounds and identities. We are female-founded, and half of our team members identify as women. For more information, see the DEI section of our website.