Data Engineer

TechDigital Corporation
Fort Worth, TX, United States
1 day ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours

Tech stack

Unity 3d Adobe InDesign Airflow Data Analysis Microsoft Azure Cloud Computing Code Review Extract Transform Load (ETL) Data Warehousing Software Design Patterns Github Job Scheduling
+24 more
Python (Programming Language) Mainframes SQL Azure Performance Tuning Azure Active Directory SQL Databases Unstructured Data Data Processing Scripting Apache Cassandra Data Ingestion Azure Data Factory Marklogic Apache Spark Event Driven Architecture Data Lakes Pyspark Cassandra Apache Kafka Data Management Data Lakehouse Data Pipelines Cisco Databricks

Job description

  • Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
  • Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
  • Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
  • Implement and maintain ETL workflows for scalability cost effective and operational efficiency
  • Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
  • Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.

Requirements

SQL (Structured Query Language), Microsoft Windows Azure, Cloud Computing, Cisco Unity, Python Programming/Scripting Language, Apache Cassandra, Unstructured Data, Mainframe Computer, Structured Data, Schedule Development, * 5+ years’ experience in SQL (Expert)

  • 4+ years of experience in Azure Databricks with PySpark
  • 4+ years of experience in Azure Cloud platform
  • 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
  • 2+ years of experience in Databricks workflow & Unity catalog
  • 2+ years of experience in Python programming & package builds
  • Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
  • Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
  • Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
  • Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
  • Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
  • Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
  • Good to have Azure Entra/AD skills and GitHub Actions
  • Good to have experience working on event-driven architectures using Kafka, Azure Event Hub

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…