Senior AI/ML Software Engineer
Role details
Job location
Tech stack
Job description
Peraton Labs is seeking a Senior AI/ML Software Engineer to join the Labs Agentic AI team, you'll design, build, and ship AI-powered systems a compliance-ready, low-code platform for dynamically generating and orchestrating AI agentic workflows. You'll work across the full product lifecycle: from architecting multi-step agentic pipelines backed by Temporal.io, to building the plugin system, APIs, and interfaces that bring them to life, from within federal-grade security and accreditation constraints.
This is a role for someone who thinks deeply about how AI agents should behave in high-trust environments, cares about reliability and auditability, and can move fluidly between distributed orchestration, backend systems, and product-facing features.
Your responsibilities may include: Design and implement agentic AI capabilities using Python-based frameworks (LangChain, LangGraph, DeepAgents) and orchestrated workflows
- Build and maintain integrations with LLM APIs (Anthropic/Claude, OpenAI, AWS Bedrock, Ollama) to power intelligent, multi-step automations
- Develop full-stack product features (FastAPI + React) that surface AI capabilities to users - from REST APIs and streaming interfaces to workflow builders and dashboards
- Instrument agent pipelines with OpenTelemetry tracing, provenance audit trails, and observability tooling for debugging and performance evaluation
- Write clear, well-tested, maintainable code that passes strict pre-commit validation, and contribute to engineering standards in a compliance-driven environment
- Evaluate agent performance, debug distributed workflows, and continuously improve reliability and output quality
Requirements
- Minimum of a BS degree with 12 years of experience, MS degree with 10 years, or PhD with 7 years with meaningful exposure to AI/ML systems or LLM-based products
- Hands-on experience building agentic systems using multi-step reasoning, tool use, RAG pipelines, or autonomous task execution
- Strong Python skills (3.12+); comfort with async/await patterns, type hints, and modern Python tooling
- Experience with workflow or task orchestration systems (Airflow, Prefect, Celery, or similar distributed execution frameworks)
- Familiarity with agentic frameworks and an understanding of the underlying concepts (chains, tool calling, agent loops) that transfer across tools
- Experience working with LLM APIs (OpenAI, Anthropic, AWS Bedrock, or similar)
- Comfort working across the stack: FastAPI/Python backends, React frontends, Docker containerization, and PostgreSQL
- A product mindset: you think about the end user, not just the technical implementation
- Comfort operating with some ambiguity in a fast-moving environment
- US Citizenship is a requirement for this position
Desired Additional Experience:
- Experience with workflow orchestration frameworks for workflow/activity patterns, task queues, worker lifecycle management
- Familiarity with federal compliance environments: FedRAMP, FIPS 140-2/3, IronBank container hardening, OPA policy enforcement, or Section 508 accessibility
- Experience building plugin or extension systems: dynamic code loading, container isolation, API mixin patterns
- Exposure to orchestration patterns: supervisor agents, parallel tool calls, human-in-the-loop flows, DAG-based pipeline execution
- Experience with observability tooling: OpenTelemetry, Jaeger, Prometheus, Grafana, or similar distributed tracing/metrics stacks
- Familiarity with prompt engineering, evaluation frameworks, or agent observability
- Experience with container orchestration (Docker SDK, Kubernetes) and distributed storage (S3, MinIO, JuiceFS)
- Prior work building internal tooling, enterprise automation products, or platforms for government customers