DATA ENGINEERS
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
Job description
Design and implement automation frameworks underpinning the migration. Build reusable transformation component libraries covering parameterized ingestion, SCD processing, data quality assertion, and reconciliation reporting. Construct end-to-end column-level data lineage and data flow diagrams. Identify and eliminate data processing redundancies and perform pipeline inventory analysis. Implement Terraform IaC managing all Databricks and AWS resources. Execute Ab Initio legacy pipelines in parallel with new PySpark/Databricks equivalents. Track lineage from Oracle/Snowflake source systems through Ab Initio transformation layers and PySpark pipelines to Delta Lake Gold consumption tables.
Requirements
Bachelor’s degree in Science, Technology, or Engineering (any) with 5 years of experience in the job offered or related occupation is required. Work location: Piscataway, NJ and various unanticipated locations throughout the U.S.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…