> Markdown version of [/jobs/ext/1469475-lead-principal-data-engineer](https://www.wearedevelopers.com/jobs/ext/1469475-lead-principal-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead/Principal Data Engineer - **Company:** Fitch Group, Inc. - **Location:** Chicago, IL, United States - **Experience:** Expert - **Salary:** $140,000.0 - $160,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Amazon Web Services, Amazon S3, Data Analysis, Confluence, JIRA, Automation of Tests, Big Data, Software Quality, Databases, Continuous Integration, Data Validation, Information Engineering, Data Infrastructure, Data Systems, Data Warehousing, Relational Databases, Shard (Database Architecture), Programming Tools, Github, Data Intelligence, PostgreSQL, MongoDB, NoSQL, Octopus Deploy, Operational Databases, Cloud Services, Tensorflow, Data Streaming, Enterprise Data Management, Datadog, GitHub Copilot, Large Language Models, Snowflake, Grafana, Multi-Agent Systems, Prompt Engineering, Spring-boot, Data Lakes, Git Flow, Kubernetes, Deployment Automation, Playwright, Atlassian Tools, AWS Data Analytics, Apache Kafka, Data Management, Virtual Agents, Data Pipelines, Serverless Computing, AWS EKS, Docker, Databricks, Vulnerability Analysis - **Published:** July 28, 2026 - **Apply:** https://www.dice.com/job-detail/f6283bab-53a4-47bd-8a6b-dae0a219bd0a ## About the Role * You have 8+ years of data engineering experience, including 3+ years in a lead role architecting large-scale data platforms. * Expert-level proficiency in Java, Springboot for building cloud-native data processing solutions running on Docker/Kubernetes. * Deep hands-on experience with Apache Airflow, Snowflake (data warehousing, modeling, optimization), and Databricks. * Strong AWS expertise including S3, Lambda, Glue, EMR, Kinesis, EKS, RDS etc. * Production database experience with PostgreSQL (design, optimization, replication) and MongoDB (document modeling, sharding, replica sets). * You have proven CI/CD and GitOps experience using GitHub, GitHub Actions, and ArgoCD for automated deployments and multi-environment management. * You are proficient with agile tools such as JIRA for sprint management and Confluence for technical documentation and knowledge sharing. * You have excellent analytical, problem-solving, and communication skills, with the ability to explain complex concepts to non-technical stakeholders and drive initiatives in complex environments. * You have working knowledge of AI/ML frameworks (LangChain, LlamaIndex, AutoGen, etc.) and understand how Agentic AI can enhance data engineering workflows through automated data validation, intelligent orchestration, and self-healing pipelines. * You have practical understanding of AI integration patterns in data platforms, including prompt engineering, RAG architectures, and vector database implementations. * You are familiar with Model Context Protocol (MCP) or similar frameworks for enabling AI agents to interact securely and efficiently with data sources, APIs, and tools. * You have experience with AI-powered development tools such as GitHub Copilot and Amazon Q. What Would Make You Stand Out: * Experience with code quality metrics and shift-left principles. * Experience testing container resiliency (Docker/Kubernetes). * Experience designing large end-to-end performance scenarios. * Experience building large and high-performing data pipelines. * Exposure to Playwright and BDD for automated testing. * Exposure to the financial industry and data platforms (data warehouses, data lakes). * Experience with modern data stack tools, data mesh/fabric architectures, and streaming platforms (Kafka, Kinesis). * Proficiency with observability tools (Datadog) and data quality/governance frameworks. * Understanding of data security and compliance standards (GDPR, SOC 2, CCPA) and contributions to open-source data projects. * Relevant certifications (AWS Data Analytics/Solutions Architect, Databricks/Snowflake Data Engineer, CKA). * Hands-on experience building production Agentic AI systems that operate on data platforms, including multi-agent orchestration and intelligent pipeline optimization. * Deep expertise with Model Context Protocol (MCP) implementation, including building custom MCP servers or integration patterns for enterprise data platforms. ## Description * Lead the design and architecture of end-to-end data pipelines and solutions on modern cloud-based platforms, including Snowflake, Databricks, and AWS. * Lead the design and architecture of end-to-end data pipelines and solutions on platforms including Java, Springboot, Docker/Kubernetes, Snowflake, Databricks, and AWS. * Design and implement data solutions using PostgreSQL for relational data and MongoDB for NoSQL requirements, ensuring optimal performance and scalability. * Architect and deploy containerized data applications using Docker, Kubernetes, and AWS EKS, incorporating GitHub Actions for automated deployments. * Design and implement CI/CD pipelines using GitHub Actions, establish branching strategies, and ensure automated testing, code quality checks, and security scanning. * Collaborate with cross-functional teams-including Data Scientists, Analytics teams, and business stakeholders-to translate requirements into scalable technical solutions. * Mentor and guide data engineers by promoting technical excellence, establishing coding standards, and conducting architecture reviews. * Drive data platform modernization initiatives and ensure data quality, reliability, and governance across all data systems. * Design and implement AI-enhanced data pipelines that leverage LLMs and Agentic AI frameworks to automate data quality checks, anomaly detection, and intelligent data transformation workflows. * Architect data infrastructure to support AI/ML workloads, including feature stores, vector databases, and real-time inference pipelines integrated with cloud-native services. * Leverage established standards and best practices to integrate AI agents into data engineering workflows, including context management protocols (MCP) for seamless AI-to-data-platform communication. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Inside Bitpanda's Tech Stack: Scaling a European Fintech Leader - Markus Dorner](https://www.wearedevelopers.com/videos/1979-inside-bitpanda-s-tech-stack-scaling-a-european-fintech-leader-markus-dorner) - [Collaboration Quantified: Lessons from Open Source Developer Networks](https://www.wearedevelopers.com/videos/1422-collaboration-quantified-lessons-from-open-source-developer-networks) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)