Data Engineer

SN Cloud Solutions LLC

Chicago, IL, United States

19 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Experience required

3 years minimum

Compensation

$50,000.0

Working hours

Shift work

Job source

www.indeed.com

Tech stack

Application Programming Interfaces (APIs) Airflow Amazon Web Services Microsoft Azure Big Data BigQuery Cloud Computing Cloud Storage Computer Programming Databases Continuous Integration Extract Transform Load (ETL)

+23 more

Data Warehousing Database Design Database Schema Python (Programming Language) NoSQL Scrum Methodology SQL Stored Procedures SQL Databases Unstructured Data Workflow Management Systems Azure Data Factory Snowflake Apache Spark Git Data Lakes Pyspark Apache Kafka Data Management Video Streaming Data Pipelines Jenkins Amazon Redshift Databricks

Job description

We are seeking an experienced Data Engineer with years of experience in designing, developing, and maintaining scalable data pipelines and data platforms. The ideal candidate should have strong expertise in Python, PySpark, SQL, cloud technologies, and modern ETL/ELT frameworks to support enterprise analytics and business intelligence initiatives., * Design, develop, and optimize ETL/ELT data pipelines for large-scale data processing.

Build scalable data solutions using Python, PySpark, and Apache Spark.
Develop and maintain data ingestion frameworks for structured and unstructured data.
Create and optimize SQL queries, stored procedures, and database objects.
Integrate data from multiple sources, including APIs, databases, and cloud storage.
Implement data quality, validation, and monitoring processes.
Work with cloud platforms such as AWS, Azure, or GCP.
Build and maintain data lakes and data warehouses.
Collaborate with data scientists, analysts, and business stakeholders.
Automate workflows using orchestration tools such as Apache Airflow.
Troubleshoot production issues and optimize pipeline performance.
Follow Agile/Scrum methodologies and CI/CD best practices.

Requirements

years of experience as a Data Engineer.
Strong experience with Python and PySpark.
Hands-on experience with Apache Spark.
Strong SQL programming skills.
Experience with ETL/ELT development.
Experience with cloud platforms (AWS, Azure, or GCP).
Knowledge of data warehousing concepts.
Experience with Apache Airflow or similar orchestration tools.
Experience with Git, Jenkins, or Azure DevOps.
Strong understanding of data modeling and database design.
Experience with relational and NoSQL databases.
Excellent problem-solving and communication skills.

Preferred Skills

Experience with Databricks.
Experience with Kafka or other streaming technologies.
Knowledge of Delta Lake.
Experience with Snowflake, Redshift, BigQuery, or Synapse., * Azure Data factory: 7 years (Required)
SQL: 3 years (Required)
Data warehouse: 4 years (Required)

Work Location: Hybrid remote in Chicago, IL 60618