Azure Data Engineer

First Call Trading Corporation

Apex, NC, United States

2 days ago

Apply on www.disabledperson.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

www.disabledperson.com

Tech stack

Microsoft Azure Big Data Cloud Database Continuous Integration Data Validation Information Engineering Data Governance Extract Transform Load (ETL) Data Migration Github Python (Programming Language) PostgreSQL

+24 more

Metadata Microsoft SQL Server SQL Azure Oracle (Applications) Scrum Methodology Microsoft Copilot Cloudera Azure Data Lake SQL Databases Data Streaming Apex Code Azure Data Factory Informatica Powercenter Apache Spark Generative AI Build Management Data Lakes Pyspark Integration Frameworks Apache Kafka Cosmos DB Data Management Databricks Web Api

Job description

Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
Implement and support Data Lake / Lakehouse architectures
Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
Expertise in handling Data migration using API’s.
Apply data quality checks and support data governance basics (metadata, lineage, access controls)
Optimize Spark jobs for performance, reliability, and cost
Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Requirements

8+ years of experience in data engineering in azure using Azure Databricks.
Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
Strong Data Engineering experience in Big Data (Cloudera)
Hands-on expertise in SQL, Python and PySpark or Scala
Proven experience with batch & real-time ingestion
Experience with Kafka and big data processing

Added good to have experience with:

Informatica PowerCenter
Informatica IDMC or equivalent cloud data integration tools
SQL Server, Oracle, Azure SQL
Cosmos DB
Unix/Linux
Maestro (or similar enterprise schedulers)
Data tools such as Precisely or equivalent

Added Advantage

Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
Practical usage of Copilot for data engineering and development productivity
Exposure to Generative AI / GenAI use cases in data platforms and analytics