AWS Data Engineer
Library & Archives Commission, Texas State
Foster City, United States of America
6 days ago
Role details
Contract type
Permanent contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
SeniorJob location
Foster City, United States of America
Tech stack
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Big Data
Continuous Integration
Data Architecture
ETL
Data Structures
DevOps
Distributed Systems
Hadoop
Hive
Identity and Access Management
Python
Sql Optimization
Spark
Amazon Web Services (AWS)
Data Lake
Information Technology
Data Lineage
Functional Programming
Data Pipelines
Requirements
- This position requires a Bachelor's Degree in Computer Science or a related technical field, and 5+ years of relevant employment experience
- 10+ years of work experience with ETL, Data Modeling, and Data Architecture
- Expert-level skills in writing and optimizing SQL
- Expertise in ETL optimization, designing, coding, and tuning big data processes using Apache Spark or similar technologies
- Experience with Big Data technologies such as Hadoop/Hive/Spark
- Experience operating very large data warehouses or data lakes
- Experience of delivering CI/CD and DevOps capabilities in a data environment
- Experience with building data pipelines and applications to stream and process datasets
- Experience in Python
- Show efficiency in handling data - tracking data lineage, ensuring data quality, and improving discoverability of data
- Sound knowledge of distributed systems and data architecture (lambda)- design and implement batch and stream data processing pipelines, knows how to optimize the distribution, partitioning, and MPP of high-level data structures
- Can clearly articulate Enterprise Standards & Operational Governance mechanisms that can be built-in for each layer of AWS Cloud data and analytics (Ingestion, Transformation, Consumption, Governance, Business touchpoints)
- Technical knowledge on EC2, IAM, S3, VPC, Lambda, Glue
- AWS Certified Solution Architect - Professional OR AWS Certified Solution Architect - Associate
- Good knowledge and experience in Lifesciences (Commercial, Managed Markets domains)
- Excellent communication skills