Data Engineer

SN Cloud Solutions LLC
Chicago, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Shift work
Languages
English
Experience level
Intermediate
Compensation
$ 50K

Job location

Chicago, United States of America

Tech stack

API
Airflow
Amazon Web Services (AWS)
Azure
Big Data
Google BigQuery
Cloud Computing
Cloud Storage
Computer Programming
Databases
Continuous Integration
ETL
Data Warehousing
Database Design
Database Schema
Python
NoSQL
Scrum
SQL Stored Procedures
SQL Databases
Unstructured Data
Workflow Management Systems
Azure
Snowflake
Spark
GIT
Data Lake
PySpark
Kafka
Data Management
Video Streaming
Data Pipelines
Jenkins
Redshift
Databricks

Job description

We are seeking an experienced Data Engineer with years of experience in designing, developing, and maintaining scalable data pipelines and data platforms. The ideal candidate should have strong expertise in Python, PySpark, SQL, cloud technologies, and modern ETL/ELT frameworks to support enterprise analytics and business intelligence initiatives., * Design, develop, and optimize ETL/ELT data pipelines for large-scale data processing.

  • Build scalable data solutions using Python, PySpark, and Apache Spark.
  • Develop and maintain data ingestion frameworks for structured and unstructured data.
  • Create and optimize SQL queries, stored procedures, and database objects.
  • Integrate data from multiple sources, including APIs, databases, and cloud storage.
  • Implement data quality, validation, and monitoring processes.
  • Work with cloud platforms such as AWS, Azure, or GCP.
  • Build and maintain data lakes and data warehouses.
  • Collaborate with data scientists, analysts, and business stakeholders.
  • Automate workflows using orchestration tools such as Apache Airflow.
  • Troubleshoot production issues and optimize pipeline performance.
  • Follow Agile/Scrum methodologies and CI/CD best practices.

Requirements

  • years of experience as a Data Engineer.
  • Strong experience with Python and PySpark.
  • Hands-on experience with Apache Spark.
  • Strong SQL programming skills.
  • Experience with ETL/ELT development.
  • Experience with cloud platforms (AWS, Azure, or GCP).
  • Knowledge of data warehousing concepts.
  • Experience with Apache Airflow or similar orchestration tools.
  • Experience with Git, Jenkins, or Azure DevOps.
  • Strong understanding of data modeling and database design.
  • Experience with relational and NoSQL databases.
  • Excellent problem-solving and communication skills.

Preferred Skills

  • Experience with Databricks.
  • Experience with Kafka or other streaming technologies.
  • Knowledge of Delta Lake.
  • Experience with Snowflake, Redshift, BigQuery, or Synapse., * Azure Data factory: 7 years (Required)
  • SQL: 3 years (Required)
  • Data warehouse: 4 years (Required)

Work Location: Hybrid remote in Chicago, IL 60618

Benefits & conditions

Pulled from the full job description

  • Flexible schedule

Apply for this position