Streaming Data

OpenKyber LLC

United States

3 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Compensation

$124,800.0 - $145,600.0

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Airflow Microsoft Azure Cloud Storage Continuous Integration Data Architecture Information Engineering Data Governance Data Security DevOps Distributed Systems Apache Hadoop Python (Programming Language)

+22 more

Performance Tuning Power BI DataOps Salesforce.Com SAP (Applications) Scala (Programming Language) SQL Databases Data Streaming Tableau (Software) Workflow Management Systems Enterprise Data Management Azure Data Factory Snowflake Apache Spark Microsoft Fabric Data Lakes Pyspark Information Technology Apache Kafka Machine Learning Operations Data Pipelines Databricks

Job description

Location: Houston TX- Hybrid Exp: 10+ Years Job Description: Azure Databricks Engineer Salary: $60 /140K Rate: $60-70/hr on C2C Core Responsibilities Design and implement scalable data platforms and pipelines using Azure Databricks, Apache Spark, Pyspark, Delta Lake, and MLflow.

Design and implement data pipelines using Databricks, Spark, and Delta Lake for batch and streaming data.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Develop and maintain Lakehouse architecture leveraging Databricks and cloud storage solutions.

Lead the migration from legacy platforms to Lakehouse architecture.

Develop batch and streaming data pipelines for ingestion, transformation, and analytics.

Establish standards for data governance, quality, and security.

Collaborate with stakeholders to align architecture with business goals.

Mentor data engineers and developers on Databricks best practices.

Integrate Databricks with tools like Power BI, Tableau, Kafka, Snowflake, and Azure Data Factory.

Requirements

Required Skills & Experience 8 10+ years in data engineering or architecture roles. 8+ years of hands-on experience with Databricks and Azure. Strong command of SQL, Python, Scala, and Spark.

Experience with CI/CD pipelines, DevOps, and orchestration tools like Airflow or Data Factory.

Familiarity with Azure cloud platforms.

Deep understanding of distributed computing, performance tuning, and data security.

Preferred Qualifications Bachelor’s or Master’s in Computer Science, Data Science, or related field. Certifications such as Azure Solution Architect, Azure Data Engineer, or Databricks Certified Data Engineer or Solutions Architect.

Experience with data mesh, data fabric, or enterprise data architectures.

Domain knowledge in finance, healthcare, or public sector is a plus.

Hadoop & Scala experience Additional Insights from Internal Communications Emphasis on DataOps, data orchestration, and data modeling.

Strong ERP knowledge (e.g., SAP, Salesforce) is valued.

Candidates should be hands-on, capable of self-exploration, and able to drive analytics adoption across business teams.

A structured questionnaire is often used to assess candidates on areas like Unity Catalog, Spark optimization, partitioning techniques, and data security.