AWS Lead Software Engineer-Python/PySpark
Role details
Job location
Tech stack
Job description
- Executes creative software solutions, design, development, and technical troubleshooting with ability to think beyond routine or conventional approaches to build solutions or break down technical problems
- Develops secure high-quality production code, and reviews and debugs code written by others
- Identifies opportunities to eliminate or automate remediation of recurring issues to improve overall operational stability of software applications and systems
- Leads evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture
- Leads communities of practice across Software Engineering to drive awareness and use of new and leading-edge technologies
- Design reusable data processing and data quality frameworks, writing production-ready Python/PySpark with testing, performance tuning, and maintainable patterns
- Build and continuously improve reliable batch and streaming data pipelines, enhancing scalability, security, and operational excellence for critical data systems
- Develop data models and transformations using SQL and dbt to support analytics, BI, and reporting use cases
- Create and operate workflow orchestration (e.g., Airflow) to schedule, monitor, and troubleshoot data jobs, leveraging infrastructure-as-code (e.g., Terraform) to provision and manage platform infrastructure
- Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
Requirements
- Formal training or certification on software engineering concepts and 5+ years applied experience
- Hands-on experience delivering end-to-end software solutions across system design, application development, testing, and operational stability; proficient in all aspects of the SDLC
- Advanced programming skills, with strong Python expertise (including unit and integration testing) and advanced PySpark for building and maintaining data processing solutions
- Proficiency with automation, CI/CD, and continuous delivery practices
- Hands-on experience building and operating cloud-native solutions on AWS (e.g., EKS/ECS, Lambda, API Gateway, VPC, IAM, S3, RDS/DynamoDB, SQS/SNS, CloudWatch/CloudTrail)
- Experience building and running cloud data platforms on AWS, Google Cloud, or Azure
- Experience with large-scale distributed data processing, performance tuning, and optimization
- Strong SQL/Spark SQL skills, including data modeling, query optimization, and execution plan analysis
- Experience with modern warehouse/lakehouse ecosystems (e.g., Redshift, BigQuery, Snowflake; Spark/Flink/Trino; Iceberg/Hudi) and using approved AI-assisted development tools with standards to validate correctness, performance, and security
- Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices
Preferred qualifications, capabilities, and skills
- Experience in financial services, ideally supporting home lending products and processes
- Familiarity with modern front-end technologies and patterns for building user-facing experiences
- Strong data modeling experience for analytics and reporting use cases
- Knowledge of data platform security, risk, compliance, and governance practices
- Experience building delivery automation for data/platform services, including CI/CD and containerized deployments (Docker, Kubernetes)
- Expertise in modern data/streaming platforms and patterns (Kafka topic design and operations; Spark Structured Streaming and streaming ETL)
- Ability to coach and mentor teammates, contribute to a collaborative and inclusive culture, and use AI-assisted engineering tools in an enterprise-safe way (spec-driven work, refactoring, code review); plus experience with Delta Lake and how it compares to Iceberg
Benefits & conditions
We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission-based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.