AI Test and Ethics Manager, IT Business Services

We are seeking a highly skilled and motivated AI Test and Ethics Manager to join AI CoE. The ideal candidate will be responsible for ensuring that our AI systems are developed and deployed ethically and meet high-quality standards. This role involves collaborating with cross-functional teams to implement ethical guidelines, conduct thorough quality assurance tests, and continuously monitor AI systems for fairness, accountability, and transparency. You won't just "check boxes" for compliance; you will design the safety architecture for our next generation of Agentic Workflows, and Generative AI solutions. You will be the bridge between cutting-edge LLM innovation and robust, human-centric safety standards. You will also have the opportunity to develop your career and learn new skills in a supportive and inclusive environment. The role will be reporting to Head of Intelligent Automation & Quality., * Develop and implement "Policy-as-Code" frameworks, operationalizing the EU AI Act, NIST AI RMF, ISO 42001 and other Deloitte standards directly into AI development and production AI solutions.

• Orchestrate adversarial testing campaigns to stress-test GenAI applications against jailbreaks, prompt injection, PII leakage, hallucination, etc.
• Define and monitor GenAI Quality Metrics beyond simple accuracy, focusing on groundedness, context recall, toxicity, faithfulness and other relevant metrics using automated evaluation frameworks (e.g., Ragas, DeepEval, LangSmith or others.).
• Design Assessment-as-a-Service for all production AI Solutions.
• Integrate automated governance into LLMOps/MLOps CI/CD pipelines, ensuring no solution reaches production without passing automated safety gates.
• Monitor compliance with ethical standards and regulatory requirements.
• Conduct ethical impact assessments and audits on AI systems.
• Perform end-to-end testing and QA on AI solutions.
• Identify potential ethical and quality risks associated with AI systems.
• Monitor AI systems for ongoing compliance with ethical standards and quality benchmarks.
• Design, implement, and maintain comprehensive testing strategies for AI models and systems.
• Ensure that AI systems meet quality standards, including performance, reliability, and security.
• Identify and document defects, and work with development teams to resolve issues.
• Work closely with AI engineers, data scientists, software engineers, product managers, and other stakeholders to integrate ethical considerations into AI development.
• Provide training and guidance on ethical AI practices to internal teams.
• Communicate findings and recommendations to senior management and other stakeholders.
• Stay updated on the latest trends, research, and best practices in ethical AI and quality assurance. Stay ahead of the curve on Futuristic AI Safety topics.
• Propose and implement improvements to existing processes and methodologies.
• Evaluate and integrate new tools and technologies to enhance AI ethics and quality assurance capabilities.
• Supervise & mentor junior QA analysts, guide them in complex ethical impact assessments and technical debugging of safety failures.

• Experience with regulatory compliance related to AI and data privacy.
• Certification in ethical AI and quality assurance.
• Experience with AI ethics frameworks and guidelines, such as those from IEEE, EU, or other organizations.
• Security Focus Knowledge of OWASP Top 10 for LLMs.
• Experience in designing and implementing AI incident management.
• Experience in prompt engineering and security guardrails.
• Knowledge on (Deep Agents, LangChain, LangGraph or other AI Frameworks.
• Familiarity with LLM Evaluators.
• Knowledge of Python and LLMOps platforms.
• Strong understanding of AI technologies, including machine learning, neural networks, and natural language processing.
• Practical knowledge of ethical considerations in AI, such as bias, fairness, transparency, and accountability.
• Proven experience in quality assurance methodologies, tools, and best practices.
• Excellent problem-solving skills and attention to detail.
• Strong communication and interpersonal skills.
• Ability to work effectively in a collaborative, fast-paced environment.

Deloitte drives progress. Our firms around the world help our clients become market leaders wherever they compete. Deloitte invests in outstanding people with diverse talents and backgrounds, empowering them to achieve more than they can elsewhere. Our work combines consulting with action and integrity. We believe that when our clients and society are stronger, so are we.