Sr. Data Engineer # 26-27370

• Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions
• Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors
• Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations
• Build serverless data processing workflows using AWS Lambda
• Design and optimize queries in AWS Athena for analytical workloads
• Manage data warehouse solutions in AWS Redshift including data modeling and performance tuning
• Implement monitoring and alerting using CloudWatch for data pipeline health and performance
• Provision and manage AWS infrastructure using Terraform
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Implement data governance and security best practices using Lake Formation, IAM, and KMS

• Python: Advanced proficiency in Python 3.8+ for data processing, scripting, and automation
• PySpark: Extensive experience with PySpark for large-scale data processing and transformations on EMR clusters
• AWS Lambda: Hands-on experience building serverless functions for data processing and orchestration
• AWS Glue: Expertise in AWS Glue for ETL jobs, Data Catalog management.
• AWS Athena: Experience with Athena for SQL-based data analysis and querying data lakes
• AWS EMR: Strong experience with EMR for running Spark jobs, EMR Notebooks, and cluster management
• AWSRedshift: Knowledge of Redshift for data warehousing, query optimization, and integration with data lakes
• Kafka: Experience with Confluent Kafka for real-time data streaming, including Schema Registry, source/sink connectors, and producers/consumers
• AWS S3: Deep understanding of S3 for data lake architecture, bucket policies, and data organization (raw, curated layers)
• CloudWatch: Proficiency in CloudWatch for logging, monitoring, and alerting on data pipelines
• Terraform: Experience with Infrastructure as Code using Terraform for provisioning AWS resources

Additional Technical Experience

• Knowledge of AWS EventBridge and Step Functions for workflow orchestration
• Familiarity with EMR runtime roles and execution roles for secure data access
• Experience with DynamoDB for NoSQL data storage
• Knowledge of API Gateway for building data APIs, * Bachelor’s degree in Computer Science, Data Engineering, Information Technology, or related field
• Strong understanding of distributed computing and big data processing frameworks
• Experience with Agile development methodologies
• Excellent problem-solving and analytical skills
• Strong communication and collaboration abilities

Development and Data Tools

• EMR Notebooks or Jupyter for interactive development
• VS Code or similar IDE for Python development
• Git for version control
• AWS CLI and boto3 for AWS service interaction
• Terraform for infrastructure provisioning
• DBeaver, Data Studio for database management