Big Data Engineer Spark & Python
Job Cloud Inc.
Charlotte, United States of America
yesterday
Role details
Contract type
Permanent contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
SeniorJob location
Charlotte, United States of America
Tech stack
Airflow
Amazon Web Services (AWS)
Azure
Big Data
Cloud Computing
Continuous Integration
Information Engineering
ETL
Data Transformation
Software Debugging
DevOps
Distributed Data Store
Distributed Systems
Memory Management
Hadoop
Hive
Python
Operational Databases
Performance Tuning
SQL Databases
Data Streaming
Unstructured Data
Data Processing
Spark
Event Driven Architecture
Containerization
Data Lake
Information Technology
Kafka
Non-relational Database
Stream Processing
Software Version Control
Data Pipelines
Databricks
Job description
We are seeking a skilled Big Data Engineer with strong hands-on experience in Spark and Python to design, build, and optimize scalable data processing solutions. The ideal candidate will develop high-performance pipelines, work with large datasets, and support analytics and reporting needs across the enterprise., * Design, develop, and maintain scalable big data pipelines using Spark and Python.
- Build efficient batch and streaming data processing solutions for large-volume datasets.
- Work with structured, semi-structured, and unstructured data from multiple sources.
- Optimize Spark jobs for performance, memory usage, and cost efficiency.
- Develop reusable Python modules and data transformation scripts.
- Collaborate with data architects, analysts, and business teams to translate requirements into technical solutions.
- Ensure data quality, validation, and error handling across pipelines.
- Work with cloud or distributed data platforms as needed.
- Monitor, troubleshoot, and support production data workflows.
- Follow best practices for coding, version control, testing, and documentation.
Requirements
- Strong experience in Apache Spark and Python.
- Hands-on knowledge of big data processing frameworks and distributed computing.
- Experience with SQL and relational/non-relational databases.
- Understanding of data engineering concepts such as ETL/ELT, data modelling, and pipeline orchestration.
- Experience handling large datasets in batch and/or streaming environments.
- Strong problem-solving and debugging skills.
- Experience with cloud platforms such as AWS, Azure, or Databricks is a plus.
- Familiarity with tools like Hadoop, Hive, Kafka, Airflow, or Delta Lake is preferred., * 5+ years of experience in data engineering or big data development.
- Experience in performance tuning Spark applications.
- Exposure to real-time data processing and event-driven architecture.
- Knowledge of CI/CD, DevOps, and containerized deployments.
- Bachelor s degree in computer science, Engineering, or related field.
