Big Data Engineer

• Design, develop, and maintain scalable big data pipelines and ETL workflows using AWS Glue and Python.
• Build and optimize Spark-based Glue jobs for efficient data processing and transformation.
• Develop and manage real-time data streaming solutions using Apache Kafka.
• Create and support event-driven serverless applications using AWS Lambda.
• Design, optimize, and maintain Client data warehouse solutions, including schema design, query optimization, and performance tuning.
• Write advanced SQL queries for large-scale data transformation, reporting, and analytics.
• Manage and integrate AWS cloud services such as S3, CloudWatch, SES, and IAM.
• Implement CI/CD pipelines and infrastructure automation using Terraform and Git Actions.
• Monitor data workflows and troubleshoot issues related to data quality, system performance, and scalability.
• Collaborate with cross-functional teams including developers, analysts, and business stakeholders to deliver data-driven solutions.
• Ensure data security, reliability, and operational efficiency across cloud-based data platforms.
• Participate in code reviews, technical discussions, and continuous improvement initiatives.
• Work effectively in a fast-paced hybrid environment with minimal supervision.

• Strong Python experience (hands-on coding required)
• AWS Glue experience with Spark-based Glue jobs
• Apache Kafka experience for real-time streaming
• Experience with Client data warehousing
• Advanced SQL skills
• Experience with AWS Lambda
• Experience with Terraform
• Experience with Git Actions / CI-CD
• AWS Glue: Proficiency in developing and managing Spark-based Glue jobs.
• Client: Strong experience in Client data warehousing, including schema design, query optimization, and performance tuning.
• Kafka: Hands-on experience with Kafka for real-time data streaming and integration.
• AWS Lambda: Experience in building serverless functions for event-driven workflows.Programming: Proficiency in Python - SQL: Advanced SQL skills for data transformation and analysis.
• Cloud Infrastructure: Familiarity with AWS services such as S3, CloudWatch, SES and IAM.
• CI/CD: Familiarity with Terraform and Git actions.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Ability to work in a fast-paced, dynamic environment with minimal supervision.
• Spark optimization experience
• Real-time data pipeline architecture
• Event-driven/serverless architecture
• AWS certification
• Experience in enterprise-scale data engineering
• Strong communication/client-facing experience
• Bachelor s degree in Computer Science, Information Technology, Engineering, or a related field is preferred

Pay Range: $100000 - $110000/annum. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Our client is a leading IT Industry and we are currently interviewing to fill this and other similar fulltime positions. If you are interested in this position, please apply online for immediate consideration. Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. By applying to our jobs you agree to receive calls, AI-generated calls, text messages, or emails from Pyramid Consulting, Inc. and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy .