Data Engineer
Spectraforce
McLean, VA, United States
3 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on leoforce.us
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
Amazon Web Services
Amazon S3
Big Data
Databases
Continuous Integration
Data Integrity
Relational Databases
DevOps
Python (Programming Language)
PostgreSQL
MySQL
Real Time Systems
+6 more
Pyspark
Data Lineage
Apache Kafka
Data Delivery
Data Pipelines
Amazon Elastic Mapreduce (EMR)
Job description
- Designing and implementing high-performance data pipelines that handle large volumes of data
- Ensuring data integrity through advanced quality checks and lineage tracking
Requirements
- Core Languages: Expert-level proficiency in Python and PySpark.
- AWS Ecosystem: Deep hands-on experience with AWS Glue and Amazon EMR.
- Big Data Architecture: Proven track record of building scalable pipelines for large-scale data (Terabyte-scale preferred).
- Data Integrity: Demonstrated experience implementing frameworks for Data Lineage and Data Quality.
- Database Knowledge: Strong experience working with S3-based lakes and relational databases (RDS/PostgreSQL/MySQL).
Preferred Experience
- Industry: Experience within the Financial Services industry is a huge plus.
- Real-time Processing: Experience with Kinesis or Kafka integration for low-latency data delivery.
- DevOps: Familiarity with CI/CD for data pipelines and Infrastructure as Code.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on leoforce.us
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦