Data Engineer
SFE PARTNERS LLC
Atlanta, GA, United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerjet.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Job source
Tech stack
Agile Methodology
Amazon Web Services
Amazon S3
Data Analysis
Big Data
BigQuery
Cloud Computing
Databases
Information Engineering
Data Governance
Data Infrastructure
Extract Transform Load (ETL)
+28 more
Data Warehousing
IBM DB2
IBM InfoSphere DataStage
Apache Hadoop
Hadoop Distributed File System
Apache Hive
Python (Programming Language)
Machine Learning
Mainframes
Scrum Methodology
Power BI
DataOps
Sqoop
Data Streaming
Tableau (Software)
Teradata SQL
Unstructured Data
Data Ingestion
Apache Spark
Git
Data Lakes
Pyspark
Apache Kafka
Spark Streaming
Spotfire
Data Pipelines
TIBCO (Software)
Databricks
Job description
- Design, develop, and maintain scalable data pipelines and ETL solutions.
- Build and optimize data ingestion frameworks for structured and unstructured data.
- Implement data quality, validation, standardization, and governance processes.
- Develop data infrastructure integrating multiple heterogeneous data sources.
- Support analytics, reporting, machine learning, and business intelligence initiatives.
- Work with Data Scientists and BI teams to prepare and curate data for analytics.
- Develop scalable solutions for Big Data processing and streaming workloads.
- Analyze data and provide actionable insights and recommendations to business stakeholders.
- Gather requirements from business teams and translate them into technical solutions.
- Lead and mentor junior data engineers.
- Collaborate across multiple projects as the Data Engineering SME.
- Drive adoption of Data Engineering best practices and standards.
Requirements
- Databricks (Notebooks, Delta Lake, Delta Live Tables, Unity Catalog)
- Python and PySpark
- Spark SQL and Apache Spark
- ETL/Data Pipeline Development
- Big Data technologies: Hadoop, Spark, Kafka, Hive, HDFS, Sqoop
- Strong SQL querying and data modeling
- Data ingestion, transformation, cleansing, and validation
- Cloud platforms (AWS/GCP)
- Data warehouse and large-scale data processing experience
- Experience with streaming data using Kafka and Spark Streaming
- Agile methodologies (Scrum, Kanban, SAFe)
Preferred Skills:
- Scala / Scala-Spark
- IBM DataStage, Tibco, or similar ETL tools
- AWS services: S3, Glue, Lambda, Kinesis/MSK
- BI Tools: Tableau, Power BI, Spotfire, OAC
- Databases: Teradata, DB2, BigQuery, Mainframe
- Git version control
- Data governance and retention modeling
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerjet.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦