Data Engineer III - Python, Databricks
Role details
Job location
Tech stack
Job description
- Develop workflows and ELT pipelines using Python and Databricks.
- Support review of controls to ensure sufficient protection of enterprise data.
- Implement data security using entitlements frameworks.
- Update logical or physical data models based on new use cases.
- Use SQL frequently and understand NoSQL databases
- Uses enterprise-authorized AI capabilities within the work environment to accelerate data pipeline/design analysis and documentation, validating outputs and handling data according to sensitivity and security requirements.
- Applies reuse-first, AI-assisted practices to strengthen SDLC-quality routines for data pipelines (e.g., test generation and control validation), ensuring traceability/auditability and alignment to resiliency and security expectations.
Requirements
- Formal training or certification on software engineering concepts and 3 years applied experience.
- Good working knowledge of AWS, Databricks, and Python, Experience across the data lifecycle.
- Advanced at SQL, including joins and aggregations, Working understanding of NoSQL databases.
- Significant experience with statistical data analysis and ability to determine appropriate tools and data patterns for analysis.
- Utilize AWS Cloud Services for developing, deploying, and managing applications at scale.
- Proficiency in AI Coding Assistants, Daily use of tools like Cursor, GitHub Copilot, and Claude to accelerate code generation, documentation, and refactoring.
- Effective Prompt Engineering, Providing AI models with context, clear goals, relevant source material, and - defined output expectations to generate accurate, usable code.
- Critical Evaluation & Validation, Ability to identify hallucination patterns, security vulnerabilities, and logic errors in AI-generated code, ensuring safety before production deployment.
- Demonstrated experience using enterprise-authorized AI capabilities within the work environment to support data engineering workflows with strong validation habits and awareness of data sensitivity.
- Ability to review and validate AI-assisted outputs (e.g., query suggestions, test ideas, or model change summaries) before use, escalating when uncertain and following data handling requirements., * Familiarity with the Standardized data layer practices (Medallion architecture)
- Exposure to Aurora Postgres and MongoDB
- Experience developing and supporting AWS GLUE Jobs, Federated Data Lake
- Skills in designing efficient data models including normalization, denormalization, and schema design and an understanding around relational and star schemas.
- Augmented Development Workflow: Integrating tools into CI/CD pipelines, containerization (e.g., Docker), and leveraging AI to quickly bridge language gaps (e.g., transitioning between Python, JavaScript, or Java).
Benefits & conditions
We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission-based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on-site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more. Additional details about total compensation and benefits will be provided during the hiring process.