Data Engineer

• Collaborate with Lead Developers (Data Engineer, Software Engineer, Data Scientist, Technical Test Lead) to understand requirements and use cases, outline technical scope, and deliver technical solutions
• Collaborate with Data and Solution architects on key technical decisions
• Develop data pipelines with focus on long-term reliability and maintaining high data quality
• Design data lake and warehousing solutions with the end-user in mind, ensuring ease of use without compromising on performance
• Manage and resolve issues in production data warehouse environments on AWS

Core Experience and Abilities:

• Perform hands-on development and peer review for certain components and tech stack
• Set up development instances and migration paths with required security, access, and roles
• Develop components and related processes (e.g., data pipelines, ETL processes, workflows)
• Build new data pipelines, identify existing data gaps, and provide automated solutions to deliver analytical capabilities and enriched data to applications
• Implement data pipelines with attentiveness to durability and data quality
• Implement data warehousing products with focus on end-user experience (ease of use with appropriate performance)
• Implement data quality frameworks and validation rules (e.g., schema validation, null checks, referential integrity, deduplication)
• Design and implement automated data tests for ETL pipelines using Python, PySpark, and SQL
• Write unit, integration, and regression tests for data pipelines (e.g., pytest-based testing for transformations and business rules)
• Demonstrate familiarity with data observability and monitoring concepts, including freshness, volume, and anomaly detection
• Understand data reconciliation and source-to-target validation techniques to ensure business logic accuracy Embed data quality checks into CI/CD pipelines to prevent defective data from reaching downstream consumers *

• 2+ years of AWS experience
• AWS services: S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, Step Functions, Redshift
• Experience with Kafka, preferably Confluent Kafka
• Experience with Lake Formation, Amazon Redshift and Amazon Athena
• Strong SQL and data modeling skills, executing ETL processes tailored for data warehousing
• Competence in developing and refining data pipelines within AWS
• Extensive understanding of database management fundamentals
• Tools and Languages: Python (good experience in PySpark), SQL
• Infrastructure as Code technology: Terraform
• DevOps pipeline (CI/CD): GitHub
• Deep knowledge of IAM roles and policies
• Experience with AWS workflow orchestration tools like Airflow or Step Functions