Sr. Data Engineer

A leading Consulting firm is seeking a Sr. Data Engineer for a 6+ month contract to hire role. This position is full remote, but client is based in Chicago, IL. Mandatory skills: Databricks, ADF, Big Data, Hive, Hadoop, HDFS, Python, Pyspark Duties: * Design, develop, and maintain scalable Big Data applications, ETL/ELT pipelines, and enterprise data integration solutions. * Develop large-scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hadoop/HDFS, Hive, and Azure Data Factory (ADF). * Build and optimize Databricks notebooks, Spark jobs, Delta Lake tables, and Lakehouse data processing workflows. * Design and develop data ingestion, transformation, and integration pipelines using Azure Data Factory (ADF), including pipelines, activities, triggers, and dependencies. * Develop complex SQL queries, stored procedures, data transformations, and source-to-target mapping logic. * Build and maintain ETL workflows using SSIS and modern cloud-based data integration technologies. * Develop reusable Python, PySpark, and Unix/Linux Shell Scripts for data processing, automation, validation, and operational support. * Integrate data from relational databases, APIs, files, cloud storage, and enterprise applications. * Develop and manage batch processing, job scheduling, dependencies, and workflow orchestration using Control-M, ADF, and Databricks Workflows. * Implement data quality checks, reconciliation, exception handling, logging, and error recovery mechanisms. * Perform performance tuning and optimization of Spark jobs, Databricks workloads, SQL queries, and data pipelines. * Collaborate with data architects, business analysts, QA teams, and engineering teams to deliver scalable enterprise data solutions. * Participate in Agile/Scrum activities, including sprint planning, refinement, daily stand-ups, reviews, and retrospectives. * Support production deployments, troubleshooting, incident resolution, and post-deployment verification while ensuring established performance and quality standards are met. Requirements: Essential Qualifications * 8+ years of experience in Big Data Development, Data Engineering, ETL/ELT development, and enterprise data integration. * Strong hands-on development experience with Python, PySpark, Apache Spark, SQL, and Databricks. * Experience designing and developing large-scale data processing applications and ETL/ELT pipelines. * Hands-on experience with Azure Data Factory (ADF) for data ingestion, transformation, integration, and workflow orchestration. * Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive. * Strong SQL development skills, including complex queries, stored procedures, transformations, and performance optimization. * Experience developing and maintaining ETL workflows using SSIS. * Experience with Unix/Linux Shell Scripting and enterprise job scheduling tools such as Control-M. * Strong understanding of Delta Lake, Databricks Workflows, data warehousing, data modeling, and source-to-target data processing. * Experience in data validation, reconciliation, error handling, production support, and performance tuning. * Experience working in Agile/Scrum environments with strong analytical, troubleshooting, communication, and collaboration skills. Bachelors degree required. Desired Qualifications * Experience with Azure cloud services, including ADLS Gen2, Azure Synapse Analytics, and Azure DevOps. * Experience with AWS services, including S3, Glue, EMR, Lambda, and Redshift. * Knowledge of Kafka, Airflow, Java/Scala, Git, Jenkins, and CI/CD pipelines. * Experience with Power BI, SSRS, Tableau, or other enterprise reporting and analytics tools. * Prior experience supporting large-scale enterprise data platforms in healthcare, insurance, or other regulated industries. U.S. Medicare Experience Big Data Development & ETL/ELT 6+ years Python, PySpark & Apache Spark 6+ years Databricks / Apache Spark / PySpark /Delta lakes 6+ years SQL & Data Transformation 6+ years Azure Data Factory (ADF) 5+ years Hadoop/HDFS & Hive 5+ Unix/Linux Shell Scripting & Control-M 5+

while ensuring established performance and quality standards are met. Requirements: Essential Qualifications * 8+ years of experience in Big Data Development, Data Engineering, ETL/ELT development, and enterprise data integration. * Strong hands-on development experience with Python, PySpark, Apache Spark, SQL, and Databricks. * Experience designing and developing large-scale data processing applications and ETL/ELT pipelines. * Hands-on experience with Azure Data Factory (ADF) for data ingestion, transformation, integration, and workflow orchestration. * Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive. * Strong SQL development skills, including complex queries, stored procedures, transformations, and performance optimization. * Experience developing and maintaining ETL workflows using SSIS. * Experience with Unix/Linux Shell Scripting and enterprise job scheduling tools such as Control-M. * Strong understanding of Delta Lake, Databricks Workflows, data warehousing, data modeling, and source-to-target data processing. * Experience in data validation, reconciliation, error handling, production support, and performance tuning. * Experience working in Agile/Scrum environments with strong analytical, troubleshooting, communication, and collaboration skills. Bachelors degree required. Desired Qualifications * Experience with Azure cloud services, including ADLS Gen2, Azure Synapse Analytics, and Azure DevOps. * Experience with AWS services, including S3, Glue, EMR, Lambda, and Redshift. * Knowledge of Kafka, Airflow, Java/Scala, Git, Jenkins, and CI/CD pipelines. * Experience with Power BI, SSRS, Tableau, or other enterprise reporting and analytics tools. * Prior experience supporting large-scale enterprise data platforms in healthcare, insurance, or other regulated industries. U.S. Medicare Experience Big Data Development & ETL/ELT 6+ years Python, PySpark & Apache Spark 6+ years Databricks / Apache Spark / PySpark /Delta lakes 6+ years SQL & Data Transformation 6+ years Azure Data Factory (ADF) 5+ years Hadoop/HDFS & Hive 5+ Unix/Linux Shell Scripting & Control-M 5+