Cloud Data Engineer - PySpark
Zachary Piper
United States
7 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.clearancejobs.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
10 years minimum
Compensation
$110,000.0 - $140,000.0
Working hours
Regular working hours
Job source
Tech stack
Amazon Web Services
Microsoft Azure
Big Data
Cloud Computing
Cloud Computing Security
Cloud Database
Continuous Delivery
Extract Transform Load (ETL)
Data Warehousing
Relational Databases
Apache Hive
JSON
+18 more
Python (Programming Language)
Role-Based Access Control
SQL Databases
Unstructured Data
Data Logging
Data Ingestion
Azure Data Factory
Change Data Capture
SC Clearance
Data Lakes
Pyspark
Deployment Automation
AWS Glue
AWS Data Analytics
Restful APIs
Azure Synapse Analytics
Data Pipelines
Databricks
Job description
- Design and maintain scalable ETL/ELT data pipelines using Azure Synapse, Databricks, AWS Glue, and related cloud technologies.
- Develop large-scale data transformations using PySpark, Spark SQL, Python, and SQL.
- Build data ingestion frameworks for structured and unstructured data, including relational databases, CSV, JSON, and REST APIs.
- Implement full and incremental data loads, Change Data Capture (CDC), and rerunnable pipeline architectures.
- Design and support cloud-based data lakes, data warehouses, and analytics-ready datasets.
- Implement data quality and governance controls including schema validation, RBAC, lineage, cataloging, and credential management.
- Monitor and troubleshoot data pipelines for performance, failures, latency, logging, and operational reliability.
- Implement CI/CD pipelines supporting automated deployments and environment promotion.
- Partner with cloud and cybersecurity teams to maintain compliance with RMF, STIG, FedRAMP, and DoD security requirements.
- Support cloud data migration, integration, and modernization initiatives across Azure and AWS environments.
Requirements
- 10+ years of experience designing and supporting production-grade cloud data pipelines.
- Strong experience with Azure Synapse, Databricks, Azure Data Factory, and/or AWS data services.
- Advanced experience with PySpark, Spark SQL, Python, and SQL.
- Experience with ETL/ELT, CDC, data lakes/warehouses, data quality, and governance.
- Experience implementing CI/CD and automated deployments for cloud data environments.
- Bachelor’s degree in a related technical field.
- Active Secret clearance required.
Benefits & conditions
- Salary Range: $110,000 - $140,000 depending on experience.
- Comprehensive Benefits: Medical, Dental, Vision, 401(k), PTO, and Paid Holidays.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.clearancejobs.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…