> Markdown version of [/jobs/ext/2156113-ai-observability-engineer](https://www.wearedevelopers.com/jobs/ext/2156113-ai-observability-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Observability Engineer - **Company:** American International Group, Inc. - **Location:** Atlanta, GA, United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Fiddler (Software), Machine Learning, Software Product Management, Large Language Models, Grafana, Model Validation, Generative AI, Virtual Agents, Data Pipelines, Dynatrace - **Published:** August 20, 2026 - **Apply:** https://www.juju.com/job/00000000gnydg4 ## About the Role + Minimum 3 years' experience and a Bachelor's degree is required. + Machine Learning, Large Language Models knowledge of various models, algorithms, and evaluation metrics is crucial for effective monitoring. + Data pipelines and analysis skills with the ability to handle, clean, and transform data for monitoring and analysis. + Excited about the problem of scaling human oversight of AI systems and familiar with LLMs development + Comfort with ambiguity and curiosity as our team is scaling with strong ownership + You bring experience and passion in Responsible AI or AI ethics, with an understanding of AI technologies and frameworks such as Generative AI + You have a strong ability to collaborate with cross-functional teams and drive the implementation of Responsible AI practices, ideally within financial services or other regulated industries + You have a working knowledge of effective policies and processes around the usage of emerging AI tools and products, bringing best practices from the industry + You have a proven track record of solving complex problems, developing mitigation strategies, and influencing others on key AI initiatives + You are passionate about staying ahead of emerging global AI regulations and translating these frameworks into practical, actionable updates Nice to have + Prior experience building production LLM, RAG or model inference monitoring pipelines + Instrumenting applications using OpenTelemetry distributed tracing or structured logs. + Hands on experience with AI Observability tools such as Fiddler, Arize, LangSmith or similar tools. + Ability to build dashboards and alerts tied to AI model behavior beyond system uptime. ## Description At AIG, we're reshaping how the world manages risk, and we're inviting you to be a key part of that transformation. As an AI Observability Engineer, you will have the opportunity to make a meaningful impact, leveraging and further developing your skills to guide groundbreaking AI initiatives. If you're looking for a place to grow your career and where your contributions will shape the future, AIG is where you belong. How you will create impact The AI Observability Engineer plays a critical role in interaction with our AI product teams to ensure the AI solutions are aligned with the enterprise Responsible AI principles and standards. Additionally, the AI Observability Analyst supports the design and execution of the overall AI Governance program as well as the communication and awareness initiatives associated with Responsible AI principles and methods through the following activities: + Model Performance Monitoring: Tracking ML,LLM, Agentic AI key performance indicators (KPIs) like context relevance, accuracy, precision, recall, and F1 scores to ensure model effectiveness + Data Quality and Bias Detection: Monitoring data inputs for anomalies, bias, and drift, which can impact the fairness and accuracy of AI systems + Utilize observability tools to produce structured, trustworthy insights from detailed data sets on AI system activities. + Manage on-going operations of our AI model management platform and pipelines including evolving our KPIs and model observability capabilities + Review AI solutions across the business to ensure alignment with AIG's Organizational Policies, AI Governance Framework and Responsible AI principles + Provide support to AI product teams, conduct independent reviews, responding to ad-hoc queries, and participating in workshops + Collaborate with legal, compliance, security, and other teams to coordinate the review of Responsible AI issues, ensuring a cohesive approach across the organization + Create executive-level reports on Responsible AI and provide strategic recommendations to senior leadership + Create and update RAI training content for technical and non-technical audiences at all levels of seniority + Contribute to the development and improvement of RAI assets: methods, technical tutorials, tools + Monitor and remain up to date on emerging global AI regulatory frameworks, standards and policies by participating in workshops, reading publications, networking with the goal of helping to translate into RAI framework updates ## Related Videos - [Unlocking the AI Black Box: Building Trust in the Era of Agentic Production](https://www.wearedevelopers.com/videos/100086-unlocking-the-ai-black-box-building-trust-in-the-era-of-agentic-production) - [Your imaginations is (no longer) the limit: how Generative AI empowers people to be creative](https://www.wearedevelopers.com/videos/741-your-imaginations-is-no-longer-the-limit-how-generative-ai-empowers-people-to-be-creative) - [The Power of Purpose: Unlocking Potential and Innovation](https://www.wearedevelopers.com/videos/1110-the-power-of-purpose-unlocking-potential-and-innovation) - [5 steps for running a Kubernetes environment at scale](https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale) - [The shadows that follow the AI generative models](https://www.wearedevelopers.com/videos/624-the-shadows-that-follow-the-ai-generative-models) - [It's Not Vibe Coding If You Know What You're Doing](https://www.wearedevelopers.com/videos/100119-it-s-not-vibe-coding-if-you-know-what-you-re-doing) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Trustworthy AI Starts at Deployment: 5 Checks Before You Ship](https://www.wearedevelopers.com/magazine/753-trustworthy-ai-starts-at-deployment-5-checks-before-you-ship) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)