> Markdown version of [/jobs/ext/4704974-senior-director-nlp-market-research-data-engineering-lead](https://www.wearedevelopers.com/jobs/ext/4704974-senior-director-nlp-market-research-data-engineering-lead). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Director, NLP & Market Research Data Engineering Lead - **Company:** Eli Lilly and Company - **Location:** Southport, IN, United States - **Experience:** Expert - **Salary:** $159,000 - $233,200 - **Contract:** Permanent contract - **Skills:** Training Data, Artificial Intelligence, Microsoft Azure, Encodings, Continuous Integration, Information Engineering, Data Governance, Data Infrastructure, Enterprise Content Management, Python (Programming Language), Natural Language Processing, Parsing, Data Streaming, Unstructured Data, Policy as Code, Sql Optimization, Great Expectations (Foster Youth College-readiness and Support Program), Retrieval-Augmented Generation, Large Language Models, Topic Modeling, Agentic-AI, Git, Pytest, Pyspark, Information Technology, Text Analysis, Terraform, Data Pipelines, Vectorization, Databricks - **Published:** November 13, 2026 - **Apply:** https://www.careerbuilder.com/job-details/senior-director-nlp-market-research-data-engineering-lead-indianapolis-in--e697a2c7-b2b4-4644-bd10-8e678a9ef31e ## About the Role Market Research, Natural Language Processing (NLP), Medical Products, Community Support, Market Research, Natural Language Processing (NLP), Customer/Client Research, International Health, Volunteer Experience, Disease Prevention and Control, Bachelor's degree in Computer Science, Data Engineering, or a related technical discipline. 8+ years in data engineering, including building and operating production NLP, text-processing pipelines at enterprise scale. Additional Skills / Preferences Demonstrated experience designing lakehouse / medallion architectures (Databricks, Unity Catalog, Delta). Hands-on NLP and text engineering - parsing and normalization, entity and topic extraction, embeddings / vectorization, and retrieval / RAG. Advanced SQL and Python (PySpark), with data-quality and testing practices for production pipelines. Working knowledge of data governance, classification, and access control (row- and column-level security), including the handling of sensitive and unstructured data. Ability to influence and align technical and business partners without direct authority in an ambiguous, matrixed environment. Experience with vector databases and search, LLM grounding, and agentic / RAG architectures on enterprise content. Familiarity with market-research data - social listening, survey and verbatim data, qualitative transcripts, and secondary / medical-scientific text. Data-contract frameworks (ODCS), policy-as-code (OPA / Rego), and Git-based CI/CD (Azure DevOps or equivalent); transformation- and infrastructure-as-code (dbt, Terraform). Experience in regulated or healthcare data environments; awareness of HIPAA, consent, and MLR / content-governance obligations. Readiness to build and lead a small engineering team as the capability scales; experience mentoring or leading engineers, formally or in a matrix, preferred. Comfort building a capability that does not exist today, and translating clearly for research, engineering, and governance audiences. ## Description The NLP & Market Research Data Engineering Lead builds and runs the engineered data foundation for unstructured and research data - the non-tabular sources that carry the consumer, patient, and HCP "why": social-listening streams, survey verbatims, qualitative transcripts, call and advisory notes, secondary reports, and medical and scientific text. As Lilly extends its agentic platform beyond structured data, this is the engineering role that delivers the unstructured-data half of the AI-ready foundation for Market Research - the NLP and text pipelines, the enrichment and embedding layer, and the retrieval (RAG) foundation on which research agents and MR analytics depend. The role sits inside CIA's central Data Engineering function, reporting to the Head of Data Engineering & Risk Mitigation, and is a joint priority of Data Engineering and Agentic Intelligence & Products. It builds the capability centrally, once, and makes it reusable across the business - partnering with the embedded Research & Unstructured Data Product Owners, the Agent Factory, Research Science (which owns social listening and MR methods as central capabilities), the Strategic Intelligence Market Research teams, and the governance and stewardship functions. This role builds and runs the pipelines, the unstructured-data products, and the retrieval foundation; it does not frame or interpret the research, and it does not own the agents or Skills. Where the Agent Factory builds the agent and the Market Research teams frame the question, this role builds the governed, AI-ready data foundation they both depend on. Key Objectives Build & run the NLP / unstructured-data pipelines. Design and operate production NLP and text pipelines on Databricks (Unity Catalog, Delta) that ingest, parse, structure, and enrich Market Research's unstructured sources across the medallion (bronze silver gold). Stand up the enrichment & embedding layer. Entity and theme extraction, topic modeling, sentiment, summarization, and embedding / vectorization - turning raw text into structured, queryable, feature-ready signal. Own the retrieval / RAG data foundation. Chunking, embedding, vector indexing, and grounding so research agents reason over the full MR evidence base, in partnership with the Agent Factory and Head of AI. Make unstructured data AI-ready & governed. Author and steward data contracts (ODCS); ensure PII, consent, classification, and provenance travel with the data into agentic consumption; partner with the Policy-as-Code function to render governance intent as enforced policy (OPA / Rego), with no leakage. Establish data quality & evaluation for NLP. Accuracy, coverage, drift, and grounding / hallucination checks (pytest, DLT / Great Expectations), with monitoring and alerting so text data stays trustworthy and auditable. Serve Market Research at scale. Engineer the pipelines behind social listening, brand-health, and secondary-MR text analysis, partnering with Research Science and the shared LBI Market Research pool that runs the analysis; establish reference architecture, reusable patterns, and self-service. Set technical direction & standards. Own the standards for NLP and unstructured-data engineering across CIA; provide architectural review and gatekeeping for new data flows, integrations, and agent-consumption paths; provide technical direction and mentorship to engineers and the vendor / LBI bench. Other key initiatives and special projects as needed. Leadership While there are no direct reports today, this person will work through Lilly's teams in India and across teams in CIA and Tech@Lilly to build out the Market Research data products (bronze and silver) needed to power agentic solutions for the teams. Leadership is exercised through technical direction, standards, mentorship, and influence across this matrixed set of partners rather than direct people management. The capability this role builds is new and expected to grow - as unstructured-data and Market Research engineering demand scales, this role is positioned to build and lead a small engineering team, and the successful candidate should be able to grow into people leadership over time. Lead through influence today. Set technical standards and provide architectural review across CIA's unstructured-data and NLP engineering; direct and quality-assure the vendor / LBI engineering bench that helps deliver the work. Mentor and develop. Coach engineers and analysts on NLP and unstructured-data engineering practice, and raise the technical bar across the extended team. Build for a future team. Establish the standards, playbooks, reusable architecture, and onboarding patterns a future in-house team will inherit, so the capability can scale without re-engineering. Grow into people leadership. As the function matures, take on direct reports - hiring, developing, and retaining engineering talent and shaping the operating model for the Market Research / unstructured-data engineering squad. Team Lilly leadership. Lead through change, translate strategy into clear technical goals, remove barriers, and model an inclusive, high-trust team environment. Key Relationships Head of Data Engineering & Risk Mitigation - direct manager; sets the central engineering agenda and standards this role operates within. Head of AI / Agentic Intelligence & Products and the Agent Factory - joint owners of the agentic roadmap and primary consumers of the retrieval / RAG foundation. Research Science - owns social listening and Market Research methods as central capabilities; defines what "good" looks like for the signal this role engineers. Strategic Intelligence Market Research teams (by therapeutic area) - frame the business and research questions the data foundation must serve. Research & Unstructured Data Product Owners - translate research demand into the product backlog this role builds against. Governance, Stewardship & Policy-as-Code - partner to ensure PII, consent, classification, and provenance are enforced in code. LBI / vendor engineering bench - extended delivery capacity this role directs and quality-assures.