Software Engineer II - Python, ETL, AWS, Kubernetes
Role details
Job location
Tech stack
Job description
- Design, develop, and maintain scalable data pipelines and ETL processes to support data integration and analytics.
- Frequently utilizes SQL and understands NoSQL databases and their niche in the marketplace
- Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards.
- Contribute to data modernization efforts by leveraging cloud solutions and optimizing data processing workflows
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation
- Perform data extraction and implement complex data transformation logic to meet business requirements
- Ensure data availability and accuracy for analytical purposes, implement best practices for data engineering, ensuring data quality, reliability, and performance, monitor and executes data quality checks.
- Identify opportunities for process automation within data engineering workflows
- Deploy and manage containerized applications using Kubernetes (EKS) and Amazon ECS.
- Implement data orchestration and workflow automation using AWS step , Event Bridge
- Use Terraform for infrastructure provisioning and management, ensuring a robust and scalable data infrastructure.
Requirements
- Formal training or certification on Data Engineering concepts and 3+ years applied experience
- Experience across the data lifecycle
- Experience working with modern Lakehouse : Databricks )
- Proficient in SQL coding (e.g., joins and aggregations) including RDBMS skills
- Experience in Micro service based component using ECS or EKS
- Working understanding of NoSQL databases
- Experience in building and optimizing data pipelines, architectures, and data sets ( Glue or Data bricks ETL)
- Proficient in object-oriented and object function scripting languages (Python etc.)
- Experience in developing ETL process and workflows for streaming data from heterogeneous data sources
- Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, testing, troubleshooting, or documentation) with demonstrated ability to critically evaluate and validate AI-generated outputs.
- Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations., * Strong analytical and problem-solving skills, with attention to detail.
- Experience building Pipeline on AWS using Terraform and using CI/CD pipelines
- Knowledge of RDBMS like Aurora
- Experience with data pipeline and workflow management tools (Airflow, etc.)
Benefits & conditions
We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission-based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.