Data Engineer - SC Cleared
Initialize IT
2 days ago
Role details
Contract type
Temporary contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
EnglishJob location
Remote
Tech stack
Agile Methodologies
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Azure
Big Data
Continuous Integration
Information Engineering
Data Governance
ETL
Data Systems
Data Warehousing
DevOps
Distributed Data Store
Github
Python
Meta-Data Management
Performance Tuning
SQL Databases
Data Processing
Data Ingestion
Azure
Informatica Powercenter
Gitlab
PySpark
Core Data
Amazon Web Services (AWS)
Software Version Control
Data Pipelines
Job description
- Develop, maintain, and optimize ETL pipelines using tools such as Informatica, Azure Data Factory, AWS Glue or Azure Data Factory
- Build and manage cloud based data pipelines leveraging AWS services (eg, EMR, S3, Lambda, Glue).
- Implement scalable data processing workflows using PySpark, Python, and SQL.
- Design and support data ingestion, transformation, and integration processes across structured and unstructured data sources.
- Collaborate with data architects, analysts, and business stakeholders to understand requirements and deliver reliable data solutions.
- Monitor pipeline performance, troubleshoot issues, and ensure data quality and reliability.
- Contribute to best practices for data engineering, including version control, CI/CD, and automation.
Requirements
- Strong hands on experience with ETL development and orchestration (Informatica, Azure, or AWS).
- Solid AWS cloud experience, including working with core data services.
- Expertise in building distributed data pipelines using EMR, PySpark, or similar technologies.
- Strong data processing and transformation experience across large datasets.
- Proficiency in PySpark, Python, and SQL for data manipulation and automation.
- Understanding of data modelling, data warehousing concepts, and performance optimization.
- Familiarity with CI/CD tools (DevOps, GitHub, GitLab).
- Exposure to data governance, metadata management, and data quality frameworks.
- Experience working in Agile environments is a plus.
