AI Engineer

As an AI Engineer - RAG Platform Optimization within banks central AI capability, you take ownership of continuously improving the accuracy, retrieval quality, and response effectiveness of the bank’s enterprise Retrieval-Augmented Generation (RAG) platform. Operating at the convergence of Information Retrieval, Data Science, and Software Engineering, your core mission is failure analysis, hypothesis-driven experimentation, and evaluation framework optimization. Rather than managing people or building standard web UI applications, you focus strictly on measurable model quality-analyzing production telemetry in Databricks, running chunking/embedding experiments, building continuous feedback loops, and ensuring every release raises the performance benchmark for thousands of internal users., * Continuous RAG Model Optimization: Perform failure analysis on low-quality responses, identify underlying retrieval or grounding flaws, and run large-scale experiments on chunking, embedding models, prompt optimization, and reranking strategies.

• GenAI Evaluation & Measurement Frameworks: Build and enhance automated evaluation pipelines to measure retrieval precision/recall, response correctness, hallucination rates, and user satisfaction metrics.
• Data Analysis & Telemetry Insights: Analyze user interaction data, production telemetry, and evaluation datasets within Databricks to detect performance regressions and uncover quality improvement opportunities.
• Production Pipeline Implementation: Translate data-driven experimentation results into scalable, maintainable Python production code and automated CI/CD data pipelines on Azure.
• Cross-Functional Technical Alignment: Partner with Software Engineers, Data Scientists, Platform Engineers, and Product Managers to embed evaluation metrics into automated release pipelines.

Your Deliverables:

• Automated GenAI evaluation suites and synthetic benchmark datasets tracking retrieval and response quality.
• Optimized chunking, embedding, reranking, and prompt strategies deployed directly into production pipelines.
• Production telemetry dashboards in Databricks tracking model performance metrics, error categories, and user trust indicators.
• CI/CD-integrated testing scripts that prevent quality regressions prior to production deployments.

• Education (Hard Requirement): Master’s degree (WO) or equivalent academic level in Data Science, Artificial Intelligence, Computer Science, Mathematics, Statistics, or a related quantitative field.
• Experience & Track Record (Hard Requirements):
• Minimum of 3+ years of professional experience in machine learning, applied AI, or data science.
• Demonstrable track record of improving production-grade RAG systems or advanced retrieval-based AI solutions.
• Strong hands-on proficiency in Python, Databricks, LLM architecture, and Azure cloud environments.
• Proven experience designing evaluation methodologies, quality metrics, and continuous feedback loops for GenAI systems.
• Proficiency in written and spoken English (corporate business language).
• Nice-to-Have Technical Experience:
• Experience with fine-tuning language models and reranking algorithms.
• Hands-on experience with AI observability tools (e.g., MLflow, TruLens, Ragas, LangSmith).
• Knowledge of synthetic evaluation dataset creation and benchmark curation.