> Markdown version of [/jobs/ext/2651264-ai-ml-ops-engineer-data-reporting](https://www.wearedevelopers.com/jobs/ext/2651264-ai-ml-ops-engineer-data-reporting). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI/ML Ops Engineer - Data & Reporting - **Company:** Landmarkit Llc - **Location:** Austin, TX, United States - **Experience:** Expert - **Contract:** Temporary contract - **Skills:** A/B Testing, Data Analysis, Continuous Integration, Software Debugging, Distributed Systems, Github, Graph Database, Statistical Hypothesis Testing, Python (Programming Language), Machine Learning, Prometheus, Software Engineering, TypeScript, Datadog, Delivery Pipeline, Large Language Models, Grafana, Multi-Agent Systems, Build Management, Gitlab-ci, Machine Learning Operations, Virtual Agents, Artificial Intelligence Markup Language (AIML), Jenkins - **Published:** August 15, 2026 - **Apply:** https://www.dice.com/job-detail/8ac13ad1-2efb-46c7-b9d5-5c1b2d14b5b0 ## About the Role 5+ years of experience in ML engineering, MLOps, platform engineering, or SRE, including 2+ years working hands-on with LLMs or LLM-powered applications in production. Demonstrated experience building evaluation systems for ML or LLM applications: test harnesses, benchmark datasets, automated scoring (including LLM-as-judge approaches), and regression detection. Strong software engineering skills in Python (and ideally TypeScript), with a track record of building reliable, well-tested internal platforms and tooling. Deep familiarity with CI/CD systems (e.g., GitHub Actions, GitLab CI, Jenkins, Buildkite) and experience embedding automated quality gates into deployment pipelines. Experience with observability and monitoring stacks (e.g., OpenTelemetry, Datadog, Grafana/Prometheus) and, ideally, LLM-specific observability tools (e.g., LangSmith, Langfuse, Arize Phoenix, Braintrust, W&B Weave). Proven ability to debug complex distributed systems under pressure, including production incident response, root-cause analysis, and blameless postmortems. Excellent cross-functional communication: able to translate evaluation results into clear findings and recommendations for both engineers and non-technical stakeholders. Comfort with ambiguity and a builder s mindset: this role starts with a blank page and ends with the evaluation platform the whole organization relies on. Experience with agentic frameworks and orchestration patterns (e.g., multi-agent systems, tool use, RAG pipelines) and their distinct failure modes. Experience with prompt management, model routing, or fine-tuning workflows and evaluating changes across model versions and providers. Background in statistics or experimentation (A/B testing, significance testing, sampling strategies for human review). Design and build reusable AI agent skills, plugins and maintain internal marketplace infrastructure to extend and scale Data, AIML capabilities across the organization. Expertise in causal inference and measurement strategy including causal graphs, ontologies, and knowledge graphs to drive rigorous, decision grade data analysis. Experience operating in regulated or high-stakes domains where agent errors carry real business or customer impact. A production-grade evaluation framework is in place, with automated evaluation suites covering every critical agent. CI/CD quality gates catch agent regressions before release, with clear pass/fail criteria trusted by agent teams. Production agents have real-time quality monitoring and alerting, with documented runbooks and severity-based escalation paths. ## Related Videos - [Beyond the Benchmark: How to Evaluate AI Agents in the Real World](https://www.wearedevelopers.com/videos/100269-beyond-the-benchmark-how-to-evaluate-ai-agents-in-the-real-world) - [5 steps for running a Kubernetes environment at scale](https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale) - [The Road to MLOps: How Verivox Transitioned to AWS](https://www.wearedevelopers.com/videos/1050-the-road-to-mlops-how-verivox-transitioned-to-aws) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Building a Multi-Agent Orchestration Engine That Actually Follows the Rules](https://www.wearedevelopers.com/videos/100159-building-a-multi-agent-orchestration-engine-that-actually-follows-the-rules) - [All your telemetry data from any source in one place](https://www.wearedevelopers.com/videos/57-all-your-telemetry-data-from-any-source-in-one-place) ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it) - [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)