Sr. Data Engineer with Python and Azure

Planet Group
Chicago, IL, United States
14 days ago
Apply on www.thejobnetwork.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Compensation
$145,600 - $166,400
Working hours
Regular working hours

Tech stack

Data Analysis Microsoft Azure Big Data Data as a Services Information Engineering Extract Transform Load (ETL) Data Transformation Database Queries Python (Programming Language) Operational Databases Performance Tuning Power BI
+7 more
Standard Sql Data Processing Azure Data Factory Sql Optimization Apache Spark Data Pipelines Databricks

Job description

You will collaborate with technical and business partners to translate data requirements into efficient workflows, maintain data quality, and improve pipeline performance.

Requirements

This hands-on role is ideal for someone with strong Python, Apache Spark, and SQL skills who enjoys solving complex data challenges and delivering reliable production solutions., * 5+ years of hands-on experience developing and maintaining ETL pipelines in production environments. \n

  • Strong Python programming skills for data processing, transformation, and automation. \n

  • Practical experience using Apache Spark to process large datasets. \n

  • Advanced SQL skills, including complex queries and data transformations. \n

  • Solid understanding of data engineering principles, including scalability, reliability, data quality, and performance optimization. \n

  • Experience troubleshooting production data workflows and resolving technical issues. \n

  • Strong communication, problem-solving, and collaboration skills. \n, n \n

  • Experience with Microsoft Azure data services, particularly Azure Data Factory. \n

  • Experience developing or running Spark workloads in Databricks. \n

  • Familiarity with Power BI and how data pipelines support reporting and analytics.

Benefits & conditions

n \n

  • Design, build, and maintain production ETL pipelines that extract, transform, and integrate large datasets. \n

  • Develop data processing workflows using Python and Apache Spark. \n

  • Write and optimize complex SQL queries to support data transformation, validation, and analysis. \n

  • Partner with cross-functional teams to understand source data, define requirements, and deliver usable datasets. \n

  • Implement data quality checks to improve accuracy and consistency across pipelines. \n

  • Investigate pipeline failures, troubleshoot processing issues, and resolve performance bottlenecks. \n

  • Improve the scalability, reliability, and maintainability of existing data solutions. \n

  • Follow engineering best practices for version control, code documentation, and collaborative development. \n

  • Participate in an agile team environment and communicate progress, issues, and technical decisions clearly. \n

\n

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.thejobnetwork.com
Prepare application

Good distractions

Loading talks and stories from around this role…