AI Solutions Architect

InnoVet Health is seeking an AI Architect to lead technical architecture and solution design for national AI initiatives across federal healthcare, with a primary focus on the Department of Veterans Affairs. This is a full-performance-level role: candidates must arrive with the architectural judgment, security fluency, and self-sufficiency to make defensible design decisions inside federal environments from day one.

You will own the technical architecture for prioritized AI use cases, taking them from concept through operational pilot in authorized government cloud environments. Work includes designing secure and compliant solution patterns, defining evaluation and monitoring strategy, establishing model lifecycle management, and working hands-on with engineering, infrastructure, cloud, and security partners to validate that solutions are deployable, performant, and scalable. The role is approximately half hands-on technical work and half architecture, documentation, and stakeholder coordination.

This role offers remote flexibility, competitive benefits, and the opportunity to shape the technical foundation for responsible AI in federal healthcare.

ResponsibilitiesArchitecture & Solution Design

• Lead AI technical architecture and solution design for prioritized use cases, covering model and configuration approach, data flows, integration patterns, environment and infrastructure dependencies, and technical readiness for alpha testing.
• Design solutions that align with VA data architecture, cloud, security, privacy, interoperability, and enterprise technology standards, including deployment across Azure Government, AWS GovCloud, VAEC, and other approved VA hosting environments.
• Select platform and serving patterns appropriate to each use case, with clear rationale for tradeoffs across latency, cost, scalability, and operational complexity.

Security, Compliance & Data Protection

• Design for control inheritance and minimize authorization burden, working with ISSOs, Privacy Officers, and cloud teams to keep solutions inside existing authorized boundaries.
• Ensure solutions meet NIST 800-53, FedRAMP, and VA privacy and PHI handling requirements, and support security authorization (ATO) activities.
• Apply secure architecture patterns for PHI workloads, including network isolation, private endpoints, managed identities and service principals, and secrets management.

Evaluation, Monitoring & Model Lifecycle

• Define the evaluation strategy for each use case, including latency and throughput targets, task-appropriate accuracy metrics, and operating point and threshold selection under real-world prevalence.
• Design structured evaluations for LLM-based and agentic approaches, including groundedness, hallucination rate, robustness, and shadow-mode validation prior to user exposure.
• Define bias and subgroup performance testing, explainability needs, and human-in-the-loop safeguards proportionate to use case risk.
• Define model lifecycle management patterns including versioning, retraining triggers, drift monitoring, rollback, and model registry integration.

Deployment & Hands-On Implementation

• Work hands-on with AI engineers, data scientists, data engineers, IT infrastructure partners, cloud teams, and security stakeholders on solution installation, configuration, and initial validation.
• Confirm compatibility, performance, scalability, and deployment feasibility across approved VA hosting environments.
• Provide architecture support for pilot planning and execution, including technical success criteria, monitoring requirements, pilot data collection approach, and deployment approach needed to move solutions from controlled testing into operational pilots.

Stakeholder Engagement & Workflow Integration

• Partner with Product Owners, field users, value management leads, and scaling partners to assess whether AI solutions are technically viable for broader adoption.
• Ensure AI solutions integrate into existing clinical and operational workflows, minimizing burden and maximizing adoption.
• Support handoff to permanent solution owners or enterprise scaling teams, including documentation of architecture decisions and lessons learned.

Deliverables & Federal Contract Execution

• Develop and maintain VA-specific architecture artifacts, including current-state and future-state workflow models, system context diagrams, data flow diagrams, integration and solution architecture views, deployment diagrams, and pilot-readiness documentation.
• Prepare formal federal deliverables including architecture decision records, technical memos, and pilot-readiness documentation suitable for audit, external review, and transition into federal environments.
• Ensure user-facing components of AI solutions meet Section 508 accessibility requirements.

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related technical field; Master’s preferred.
• 5+ years of hands-on experience in solution architecture, ML engineering, or applied AI, including 3+ years delivering systems in federal or otherwise regulated environments.
• Demonstrated experience architecting and deploying AI/ML systems in secure government cloud environments (Azure Government, AWS GovCloud, VAEC), including working knowledge of service and model availability gaps relative to commercial regions.
• Hands-on Databricks platform architecture experience, including workspace and cluster design, cluster policies, Delta, job orchestration, and model serving patterns.
• Proficiency in Python, SQL, and PySpark.
• Experience with production AI/ML deployment patterns, including containerized deployment, CI/CD pipelines, real-time endpoint versus batch serving tradeoffs, and model registry integration (MLflow or equivalent).
• Demonstrated experience designing evaluation and monitoring approaches for ML and LLM systems, including metric selection, threshold and operating point analysis, and drift detection.
• Working knowledge of federal security authorization, including the ATO process, NIST 800-53, FedRAMP, and control inheritance.
• Ability to produce architecture documentation including system context, data flow, integration, and deployment diagrams using Visio, Lucidchart, or Draw.io.
• Ability to clearly communicate architecture decisions and tradeoffs to technical, clinical, and executive audiences.
• Ability to obtain and maintain VA suitability and a federal PIV badge.
• U.S. Citizen or Green Card holder.
• No 1099, corp-to-corp, or international outsourcing.

Preferred

• Direct experience working within the Department of Veterans Affairs on architecture, data science, or adjacent projects.
• Familiarity with Unity Catalog or equivalent data governance, lineage, and access control tooling.
• Familiarity with the VA data landscape, including CDW, Millennium and Cerner data, and VistA-era source systems.
• Experience with Azure OpenAI, Hugging Face, and LangChain or similar orchestration frameworks.
• Experience with Terraform, Kubernetes, and Azure DevOps, GitHub, or GitLab.
• Familiarity with monitoring and operations tooling such as Azure Monitor, CloudWatch, Grafana, Splunk, Dynatrace, or Datadog.
• Familiarity with the NIST AI Risk Management Framework.
• Experience with point-of-care delivery patterns including FHIR APIs, CDS Hooks, and SMART on FHIR., * This position works with government contracts. Under Order 11935, either U.S. Citizenship or valid permanent residency is required. Answer 2 if you are a US citizen, 1 if you have a permanent resident card.
• Please provide the link to your LinkedIn account.
• Please provide the link to your GitHub account.
• How many years of experience do you have working on federal contracts?

Education:

• Bachelor’s (Required)

Experience:

• Python: 5 years (Required)
• SQL: 3 years (Required)
• AI technical architecture: 5 years (Required)
• Model evaluation: 5 years (Required)
• Retrieval-Augmented Generation: 5 years (Required)
• healthcare data : 5 years (Required)

Pulled from the full job description

• Referral program
• 401(k)
• Health insurance
• 401(k) matching
• Paid time off
• Vision insurance
• Dental insurance, * 401(k)
• 401(k) matching
• Dental insurance
• Health insurance
• Paid time off
• Referral program
• Vision insurance