> Markdown version of [/jobs/ext/4087464-senior-data-engineer-consultant-ai-forward-pharmacy-data-databricks-kafka](https://www.wearedevelopers.com/jobs/ext/4087464-senior-data-engineer-consultant-ai-forward-pharmacy-data-databricks-kafka). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer Consultant (AI-Forward) - Pharmacy Data, Databricks + Kafka - **Company:** Proactive Logic Consulting Inc - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $160,555.0 - $228,800.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Microsoft Azure, Code Review, Continuous Integration, Data Validation, Information Engineering, Cursor, Github, Python (Programming Language), Microsoft SQL Server, Operational Databases, Reference Data, Microsoft Copilot, Standard Sql, DataOps, SQL Databases, Data Streaming, Test Data, GitHub Copilot, Delivery Pipeline, Claude Code, Apache Spark, Change Data Capture, AI Coding Agents, Agentic-AI, Gitlab, Pytest, Data Lakes, Pyspark, Debezium, Deployment Automation, Apache Kafka, OpenAI Codex, Key Vault, Databricks - **Published:** October 13, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=2001242b0904932d ## About the Role * 10+ Years in Data Engineering * Building and operating production streaming and batch pipelines and lakehouse architectures. * Strong data modeling and testing; you leave behind clean, well-tested, documented tables. * Databricks & Spark (2+ years) * Azure Databricks, PySpark, Delta Lake, declarative pipelines (Delta Live Tables / Lakeflow), Structured Streaming. * Databricks CI/CD and deployment automation; job orchestration, monitoring and alerting. * Kafka & Change Data Capture * Production Kafka experience; CDC ingestion from operational databases. * Idempotent, replayable pipelines with clear failure handling. * SQL & Python * Strong SQL, including SQL Server. * Python packaging and testing (pyproject, wheels, pytest). * Pharmacy Data (Required, 2+ years) * NCPDP claims, NDCs and drug reference data, prescriptions and fills, prescriber identifiers (NPI, DEA). * Good PHI habits: minimum-necessary access, no PHI in logs or test data, HIPAA-aligned practices. ## Description We need a Senior Data Engineer Consultant (AI-Forward) to build and run the pipelines behind a modern pharmacy platform. You'll bring together data from pharmacy systems, claims and e-prescribing sources and make it reliable for operations and reporting. You know pharmacy data well enough to spot when something looks wrong. This is a corp-to-corp, 1099 engagement for an independent consultant who works autonomously and uses AI tools daily to multiply their impact., * Daily practitioner with AI coding tools and agents: Claude Code, OpenAI Codex CLI, GitHub Copilot or Cursor. * Prompt and instruction design: turn data requirements into structured prompts, tools and context. * Agentic workflows for data operations: pipeline triage, schema-change diffing, test generation and code review with a human in the loop. * Safety and compliance: PHI redaction, synthetic test data, secrets in Key Vault, nothing sensitive in prompts or logs. * Consulting & High EQ * Excellent client-facing communication; comfortable with engineers, analysts and pharmacy operations staff alike. * Clear trade-off framing and requirements discovery. * Entrepreneurial Mindset * Think like an owner; deliver measurable value. We welcome consultants building their own ventures. * Business Structure * Must have an active LLC or S-Corp; 1099 only (Corp-to-Corp). 2+ years running your own consulting company strongly preferred. What You'll Do * Build Pipelines: Streaming and batch pipelines on Azure Databricks with PySpark, Delta Lake and declarative pipelines. * Ingest Change Data: Bring operational database changes in through CDC and Kafka. * Model Pharmacy Data: Turn prescription, fill, claim, drug and prescriber data into clean, well-tested tables. * Publish Curated Data: Publish curated datasets for downstream applications and reporting. * Ship Safely: Automated deployments through CI/CD, with monitoring, alerting and data quality checks. * Use AI Well: Apply AI tools to speed up delivery, documentation and review without letting PHI leave controlled boundaries. Preferred Tech & Tooling * Azure Databricks, PySpark, Delta Lake, Lakeflow / Delta Live Tables, Unity Catalog * Kafka, CDC (e.g. Debezium), SQL Server * Python (pyproject, wheels, pytest); CI/CD (GitLab, GitHub or Azure DevOps) * Agentic IDEs: Claude Code, OpenAI Codex CLI, Copilot/Cursor Nice to Have * Pharmacy management system data, or e-prescribing data (NCPDP SCRIPT, medication history). * Databricks certification. * Former pharmacy technician or pharmacy-system analyst. Why Join Proactive Logic Consulting? * Flexible Engagements: Shape your schedule around your life. * Top-Tier Peers: Collaborate with senior experts across cloud, data and AI. * Impactful Projects: Healthcare and pharmacy work with measurable outcomes. * AI-Forward Culture: Agentic software delivery and practical wins over hype. * Entrepreneurial Community: Many of our consultants build their own products too., * Your resume or consultant profile highlighting Databricks, Kafka and pharmacy data work. * A short note on your LLC or S-Corp (name, state). * Links to relevant work (pipelines, write-ups, demos or repos). * 3-5 lines on how you use AI tools (Claude Code, Codex CLI, etc.) to accelerate delivery while keeping PHI safe. Optional (helps us fast-track you): answer in 1-2 sentences each. * Walk us through a pharmacy claim from submission to paid or rejected. * A drug's NDC changes between reference-data updates. What can break in your pipeline, and how do you guard against it?