DATA ENGINEER I
Role details
Job location
Tech stack
Job description
Data Engineer I provides daily support in developing, deploying and maintaining Data Warehouse and operational data stores that delivers accessible data to power research and operational needs. Data Engineer I will be supporting senior engineers in developing data pipelines in AWS and on-prem.
This is a hands-on data engineering role that builds, tests and deploys large scale data processing pipelines that enables majority functions in data world with capabilities & features that will support re-usability, perform at scale, be supportable, and be extensible. Works closely with business analysts, researchers, data scientists and IT staff to ensure their data as well as reporting requirements are met. This role combines with technical aspects of data engineering with the business understanding of the systems and processes involved.
Qualifications - High Level
- Design and implement Snowflake-based data solutions aligned with business requirements and industry best practices
- Engineering tasks including data analysis, data modeling, reverse engineering, and performance optimization
- Collaborate closely with other data engineers to design and optimize ELT/ETL pipelines using tools such as Matillion and Snowflake's native features
- Design, develop, and deploy data pipelines to create or enhance data warehouses/marts
- Building required infrastructure for optimal extraction, transformation, and loading of data from various data sources using AWS and Snowflake database technologies
- Stay current with and best practices, with a focus on Snowflake's capabilities and platform advancements
- Document technical specifications, data models, process flows, standards, processes, and procedures
- Identifying, designing, and implementing internal process improvements including redesigning infrastructure for greater scalability, optimizing data delivery, and automating manual processes
- Provide on-call production support on a rotational basis, including monitoring and facilitating resolution of support for all data engineering related services
- Sets design, test, documentation, and implementation standards in accordance with HIPPA and other patient data management policies and ensures that data management systems adhere to them
Qualifications - Tech Specifics
- Matillion - design, build, and maintain data pipelines with Matillion ETL platform
- Snowflake - hands on development with Snowflake including but not limited to -
- Snowflake object development & deployment
- Snowpipe development
- Query profiling and performance tuning
- Streamlit
- AWS - hands on use with AWS including but not limited to the following -
- S3
- EC2
- SQS/SNS
- Lambda
- Secret Manager
- Glue
- Airflow
- SQL development for Snowflake, Oracle, SQL Server, etc.
- Python - hands on development with python for data manipulation, conversion, and orchestration
- Scripting - bash, Linux, etc.
- Ingesting multiple datatypes and source format including but not limited to -
- JDBC/ODBC
- API
- Semi-structured file formats (XML, JSON, etc.)
- Flat file formats (TXT, CSV, etc.)
Requirements
- Snowflake AISQL - processing discreet and non-discreet data using Cortex/AISQL functions within Snowflake; Snowflake CoCo
- Snowflake Streamlit - application development with Streamlit
- AWS Glue - pipeline development with AWS Glue creating reusable & scalable jobs
- Amazon Managed Workflows for Apache Airflow (MWAA) setup, configuration, and DAG development for orchestrating data pipelines
- DBT - usage of the DBT platform with Snowflake
- Data modeling concepts - conceptual, logical, physical, etc.
Minimum Education Level: Bachelor's Degree Minimum Field of Study: IS, Computer Science, Business Administration, or similar Training Required: Ability to collaborate and work with cross functional teams, organizational communication and change management, strong analytical and problem-solving skills.