Databricks Data Engineer

NewGen Technologies

United States

4 days ago

Apply on www.clearancejobs.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.clearancejobs.com

Tech stack

Agile Methodology Amazon Web Services Cloud Computing Information Engineering Extract Transform Load (ETL) Data Warehousing Apache Hive SQL Databases Data Ingestion IT Architecture SC Clearance Data Lakes

+2 more

Pyspark Databricks

Job description

Build and maintain Databricks pipelines (60% of the time)
Gather and analyze requirements (20% of the time)
Monitor and troubleshoot data jobs (10% of the time)
Document processes and workflows (10% of the time)
Perform additional assigned tasks

Requirements

US Citizenship
Secret Clearance
Excellent interpersonal skills and ability in overseeing/prioritizing work requests
Excellent written and oral communication skills
Strong data engineering, ETL/ELT development, or data warehousing
Strong PySpark and/or Spark SQL skills; solid SQL fundamentals
Solid understanding of the Medallion Architecture (Bronze/Silver/Gold layering)
Proficiency with Delta Lake (ACID transactions, schema evolution, time travel, MERGE/Upsert patterns)
Experience designing both batch and incremental/streaming ingestion pipelines
Familiarity with the Agile process

Desired Skills

Databricks Certification/experience/familiarity preferred
Knowledge of and experience with the DoD
Familiar with Customer and/or DoD IT architectures
Federal experience
Experience in post go-live production support and break/fix
Focus on customer service and responsiveness, with sound customer handling skills
Experience working within AWS Gov Cloud
Experience working migrating code from non-accredited to accredited environments within AWS