Data Engineer
Role details
Job location
Tech stack
Job description
Client is seeking an AWS Data Engineer to design, build, and optimize large-scale data pipelines and analytics solutions on Amazon Web Services (AWS). The role will design and build the infrastructure and pipelines that enable organizations to collect, store, process, and analyze large volumes of structured and unstructured data efficiently and securely. A Data Engineer is responsible for the end-to-end data lifecycle, from ingestion and transformation to storage and delivery for analytics, machine learning, and operational systems. They ensure data is reliable, high-quality, scalable, and accessible for business and technical stakeholders., * Design, build, and optimize ETL/ELT workflows to ingest data from multiple sources. (e.g., S3, Redshift, Lake Formation, Glue, lambda).
- Implement data cleansing, enrichment, and standardization processes.
- Automate batch and streaming data pipelines for real-time analytics.Build solutions for both streaming (Kinesis, MSK, Lambda) and batch processing (Glue, EMR, Step Functions).
- Ensure pipelines are optimized for scalability, performance, and fault tolerance.
- Optimize SQL queries, data models, and pipeline performance.
- Ensure efficient use of cloud-native resources (compute, storage, networking).
- Design and implement data architecture across data lakes, data warehouses, and lakehouses.
- Optimize data storage strategies (partitioning, indexing, schema design).
- Implement data integration from diverse sources (databases, APIs, IoT, third-party systems).
- Work with Data Scientists, Analysts, and BI developers to deliver clean, well-structured data.
- Document data assets and processes for discoverability.
- Training of existing core staff who will maintain infrastructure and pipelines.
Requirements
The ideal candidate will have strong expertise in cloud-based data engineering, hands-on experience with AWS native services, and a solid understanding of data lake, data warehouse, and real-time streaming architectures., * Bachelor's degree in Computer Science, Data Engineering, or related field.
- 5+ years of experience in data engineering roles
- Proficiency in SQL, Python, or Scala for data transformation and processing.
- Experience in the utility industry data including meter data, customer data, grid/asset data, work management, outage data.
- Familiarity with IEC CIM standards and utility integration frameworks.
- Working knowledge of Databricks on AWS
- Working knowledge of DevOps and CI/CD
Preferred Qualifications:
- Experience with IaC tools (e.g. Terraform) is a plus
