Data Engineer
Merican Inc
Fairfax County, VA, United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Temporary to permanent
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source
Tech stack
Application Programming Interfaces (APIs)
Airflow
Amazon Web Services
Data Analysis
Big Data
Computer Programming
Information Engineering
Data Governance
Extract Transform Load (ETL)
Data Transformation
Database Queries
Python (Programming Language)
+10 more
SQL Databases
Transaction Data
Data Processing
Freeform SQL
Data Storage Technologies
Snowflake
Apache Spark
AWS Glue
Data Pipelines
Databricks
Job description
- Design, develop, and maintain scalable batch data pipelines for transaction data processing and integration.
- Build, enhance, and support data pipelines using Python, Spark, AWS Glue, and Databricks.
- Develop complex SQL queries to extract, transform, validate, and analyze data.
- Work with Snowflake for data storage, transformation, and analytical processing.
- Use Apache Airflow for workflow orchestration, scheduling, and monitoring of data pipelines.
- Troubleshoot pipeline failures, investigate data quality issues, and optimize performance.
- Support the integration of one client’s data into current client’s existing data ecosystem.
- Collaborate with engineers across multiple initiatives within a small, agile team structure.
- For API-focused initiatives, develop and maintain APIs to serve data to downstream applications and systems.
- Follow established engineering standards, data governance practices, and best practices for reliable data processing.
Requirements
- Data Engineering: 5-7 years of hands-on experience building and supporting data pipelines.
- Programming: Strong Python programming skills.
- Big Data: Hands-on experience with Apache Spark.
- AWS: Experience with AWS Glue and related data engineering services.
- SQL: Strong SQL skills, including complex queries, joins, aggregations, and data analysis.
- Databricks: Experience developing and processing data using Databricks.
- Snowflake: Experience working with Snowflake for data processing and warehousing.
- Orchestration: Experience with Apache Airflow for scheduling and managing data workflows.
- Data Processing: Experience with batch pipelines, ETL/ELT processes, data transformation, and pipeline support.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…