Senior Data Engineer - Python, PySpark & AWS
Sci Solutions Global
Plano, TX, United States
1 day ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
Java (Programming Language)
Airflow
Amazon Web Services
Amazon S3
Big Data
Cloud Computing
Data Integration
Extract Transform Load (ETL)
Distributed Computing Environment
Python (Programming Language)
Machine Learning
Operational Databases
+10 more
Data Storage Management
Pyspark
Infrastructure Automation Frameworks
Cassandra
Apache Kafka
Restful APIs
Terraform
Data Pipelines
Databricks
Control M
Job description
We are seeking a hands-on Senior Data Engineer with strong Python, PySpark, and AWS experience to develop and maintain scalable data pipelines and distributed data processing solutions. The ideal candidate will have experience supporting production data engineering workloads, cloud infrastructure, and data orchestration., * Develop, test, and maintain ETL/ELT data pipelines using Python and PySpark.
- Build and support big-data processing solutions on AWS using S3, EMR, and Lambda.
- Implement infrastructure automation using Terraform.
- Develop data integrations using REST APIs, Kafka, and MQ messaging.
- Support data storage and retrieval using Cassandra.
- Orchestrate and monitor workflows using Airflow, Control-M, and native schedulers.
- Troubleshoot pipeline failures and optimize performance, reliability, and data quality.
Requirements
- Strong hands-on Python and PySpark experience.
- Experience with big-data processing and production data pipelines.
- AWS experience with S3, EMR, and Lambda.
- Working knowledge of Terraform.
- Familiarity with Kafka, MQ messaging, REST APIs, Java, and Cassandra.
- Experience with Airflow, Control-M, or other workflow scheduling tools.
Preferred: Databricks experience and exposure to machine-learning workflows.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦