Senior Scientist - Genai Evaluation

Experteer Overview In this role you shape how Johnson u**amp; Johnson validates generative AI in pharma Ru**amp; D, building measurable, repeatable evaluation methods for AI systems.You’ll work within the EQS function to define rubrics, curate datasets, and deliver quality readouts guiding release decisions.You partner with clinical and regulatory groups to ensure scientific defensibility and risk-aware AI use.This is an opportunity to influence AI readiness at a top pharma company and advance rigorous, data-driven decision making.Compensaciones / Beneficios - Design, build, and maintain automated evaluation pipelines for LLM quality, RAG performance, agent reliability, safety, and scientific accuracy - Author evaluation rubrics, curate golden and synthetic datasets with domain experts, and maintain registry of reusable evaluation assets - Validate AI judges against human expert agreement and run model, prompt, retriever, and agent benchmarks producing standardized quality readouts - Analyze failure patterns (hallucination, unsupported claims, weak traceability) and turn findings into actionable recommendations - Develop therapeutic-area-specific evaluation criteria with scientific, clinical, and regulatory partners, refining based on real-world feedback - Design evaluation methods for scientific reasoning, evidence synthesis, and hypothesis quality where generic benchmarks fall short - Build evaluation tooling and reusable patterns enabling other teams to self-serveResponsabilidades - 6+ years of hands-on AI/ML evaluation or data science experience (Master’s) or 3+ years with PhD - Experience designing and running evaluation frameworks, scientific benchmarks, or quality assessments for AI/ML systems - Hands-on work with generative AI (LLMs, retrieval-augmented generation, agentic frameworks, prompt engineering) - Strong proficiency in Python and modern AI/ML tooling (evaluation harnesses, embedding models, vector databases, LLM APIs) - Ability to translate expert scientific judgment into measurable criteria, rubrics, and reproducible protocols - Collaborative, self-driven approach across multidisciplinary teamsRequisitos principales - annual bonus - vacation days - parenteral leave - bereavement leave - well-being reimbursement - insurance plans

Build evaluation tooling and reusable patterns enabling other teams to self-serveResponsabilidades - 6+ years of hands-on AI/ML evaluation or data science experience (Master’s) or 3+ years with PhD - Experience designing and running evaluation frameworks, scientific benchmarks, or quality assessments for AI/ML systems - Hands-on work with generative AI (LLMs, retrieval-augmented generation, agentic frameworks, prompt engineering) - Strong proficiency in Python and modern AI/ML tooling (evaluation harnesses, embedding models, vector databases, LLM APIs) - Ability to translate expert scientific judgment into measurable criteria, rubrics, and reproducible protocols - Collaborative, self-driven approach across multidisciplinary teamsRequisitos principales - annual bonus - vacation days - parenteral leave - bereavement leave - well-being reimbursement - insurance plans