Sr. Data Engineer

We are seeking a Senior Data Engineer who will play a pivotal role in shaping our data platform team. This position offers the opportunity to work with cutting-edge technology while making a meaningful impact on our data-driven initiatives. Responsibilities

• Design and implement robust data pipelines utilizing Databricks and AWS services for both batch and real-time processing.
• Architect and maintain a scalable lakehouse infrastructure, ensuring data is reliable and accessible for analytics and AI applications.
• Collaborate with data scientists to develop feature store architecture and support predictive modeling efforts.
• Build and manage ETL/ELT orchestration and stream processing pipelines, integrating diverse data sources.
• Ensure compliance with HIPAA regulations across all data handling processes.
• Mentor junior engineers and enforce best engineering practices within the team.
• Participate in design discussions and contribute to special projects as needed.

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
• 7-10 years of professional experience in data engineering, with a strong focus on cloud-based architectures.
• Proficiency in Apache Spark and hands-on experience with Databricks.
• Expertise in AWS data services including S3, Redshift, and Glue.
• Strong programming skills in Python, with additional knowledge of SQL.

Preferred Skills

• Experience with ETL orchestration tools such as Airflow and dbt.
• Familiarity with container orchestration using Docker and Kubernetes.
• Knowledge of vector databases and LLM use cases.
• Excellent communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

We are committed to fostering a diverse, equitable, and inclusive workplace where all individuals feel valued and respected. We encourage candidates from all backgrounds to apply and contribute to our mission.