Senior Data Engineer
Role details
Job location
Tech stack
Job description
General
- Create data architecture, pipelines, and analytical solutions to meet software and data science requirements for various PG-MX Healthcare Products.
- Identify, evaluate, select, and prove out new technologies and toolsets.
- Create and execute Proofs of Concept and Proofs of Technology.
- Lead and direct the work of others in data dependent projects.
Data Storage Design
- Collaborate with software development, business teams, analysts, and data scientists to establish data storage, pipeline, and structure requirements.
- Design, develop, and maintain ETL/ELT pipelines using Databricks (PySpark, Delta Lake, SQL).
- Implement Data Lakehouse architecture leveraging Databrick Unity Catalog, Delta Live Tables, and Workflows.
- Build and optimize data ingestion frameworks for structured and unstructured data from diverse sources.
- Identify and plan for data storage performance requirements.
- Optimize Databricks clusters, jobs, and queries for performance and cost efficiency.
Application Interface and Data Storage Implementation
- Collaborate with software development, business teams, and data scientists to create and execute implementations.
- Identify impact of implementation on other applications and databases.
- Lead and mentor data engineers on data projects.
- Implement CI/CD for data pipelines using Git, Databricks Repos, and DevOps tools.
- Ensure Data quality, reliability, security compliances across environments.
Master Data Management and Data Governance
- Build and evolve Trusted Record systems to manage entities across the enterprise.
- Design, implement, and evolve solutions around person identity management.
- Develop and enforce data governance, lineage, and cataloging standards.
Mentors Data Engineers
- Identify areas of development and need.
- Provide targeted training and exploration for team members.
- Train and mentor data engineers on standards and best practices.
Requirements
Does analytical data flow through your veins? Are you fluent across cloud, on-premises, platforms, and technologies? Do you have deep healthcare experience? Do you love understanding strategic direction, creating design and direction, then getting fingers on the keyboard? This role is for a Senior Data Engineer with strong data architecture experience. This person will design, develop, and support solutions to transport, store, and analyze our analytical data. They will be key in evolving and maturing our enterprise data strategy and capabilities., * Hands on experience of Azure data technologies (Databricks, Data Factory, Stream Analytics, Data Lake Storage, Synapse), on-premises Microsoft tools (SQL DB and SSIS), and familiar with AWS data technologies.
- Proficiency in Python, SQL and distributed data processing frameworks (Spark), and familiar with C#, PowerShell, and APIs.
- Significant experience with analytical solutions in relational databases such as MS SQL Server, Oracle, and DB2 as well as experience with NoSQL databases and solutions such as data lakes, document-oriented databases, and graph databases.
- Strong understanding of data modeling, schema design, and ETL best practices
- Experience with data lineage, cataloging, and metadata management in Databricks and Unity Catalog.
- Skill in data modeling and experience with tools like ER/Studio or Erwin.
- Familiarity with version control (Git) and DevOps/CI-CD practices.
- Familiarity with SQL performance tuning and Spark optimization techniques.
- Excellent problem-solving and communication skills.
Minimum Qualifications
- Minimum of 5 years Data Engineering experience in an enterprise environment.
- Bachelor's degree in technology or like field required.
Benefits & conditions
The expected base salary for this position ranges from $100,000 to $120,000. It is not typical for offers to be made at or near the top of the range. Salary offers are based on a wide range of factors including relevant skills, training, experience, education, and, where applicable, licensure or certifications obtained. Market and organizational factors are also considered. In addition to base salary and a competitive benefits package, successful candidates are eligible to receive a discretionary bonus or commission tied to achieved results.