Data Engineer
TechDigital Corporation
Fort Worth, TX, United States
1 day ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours
Job source
Tech stack
Unity 3d
Adobe InDesign
Airflow
Data Analysis
Microsoft Azure
Cloud Computing
Code Review
Extract Transform Load (ETL)
Data Warehousing
Software Design Patterns
Github
Job Scheduling
+24 more
Python (Programming Language)
Mainframes
SQL Azure
Performance Tuning
Azure Active Directory
SQL Databases
Unstructured Data
Data Processing
Scripting
Apache Cassandra
Data Ingestion
Azure Data Factory
Marklogic
Apache Spark
Event Driven Architecture
Data Lakes
Pyspark
Cassandra
Apache Kafka
Data Management
Data Lakehouse
Data Pipelines
Cisco
Databricks
Job description
- Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
- Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
- Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
- Implement and maintain ETL workflows for scalability cost effective and operational efficiency
- Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
- Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.
Requirements
SQL (Structured Query Language), Microsoft Windows Azure, Cloud Computing, Cisco Unity, Python Programming/Scripting Language, Apache Cassandra, Unstructured Data, Mainframe Computer, Structured Data, Schedule Development, * 5+ yearsβ experience in SQL (Expert)
- 4+ years of experience in Azure Databricks with PySpark
- 4+ years of experience in Azure Cloud platform
- 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
- 2+ years of experience in Databricks workflow & Unity catalog
- 2+ years of experience in Python programming & package builds
- Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
- Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
- Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
- Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
- Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
- Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
- Good to have Azure Entra/AD skills and GitHub Actions
- Good to have experience working on event-driven architectures using Kafka, Azure Event Hub
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦