Data Engineer with Databricks

The candidate will work closely with data scientists, analysts, business stakeholders, and cross-functional teams to ensure data is reliable, secure, high-quality, and readily available., * Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark.

• Build and optimize batch and real-time/streaming data pipelines for large volumes of structured and unstructured data.
• Develop data engineering solutions using Databricks, Spark, and related data technologies.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Monitor, troubleshoot, and optimize data processing workflows to ensure reliability and performance.
• Optimize Databricks clusters for performance, scalability, and cost efficiency.
• Implement data quality, governance, security, and compliance standards across data pipelines.
• Perform data validation and troubleshooting to identify and resolve pipeline issues.
• Maintain technical documentation related to data architecture, data pipelines, workflows, and processes.
• Support continuous improvement of data engineering processes and platform capabilities.
• Explore and integrate AI/Agentic AI capabilities into data engineering workflows where applicable.

Apache Spark, Data Management, Scalable System Development, Data Science, Cross-Functional, Data Analysis, Database Extract Transform and Load (ETL), Unstructured Data, Structured Data, Data Processing, We are seeking an experienced Data Engineer with 4 6 years of experience and strong hands-on expertise in Databricks and Apache Spark. The ideal candidate will design, develop, and optimize scalable data pipelines to support analytics and data-driven business initiatives., * 4 6 years of experience in Data Engineering.

• Strong hands-on experience with Databricks.
• Strong experience with Apache Spark.
• Experience developing ETL/ELT pipelines.
• Experience with both batch and streaming data processing.
• Experience working with large volumes of structured and unstructured data.
• Strong understanding of data engineering and data processing concepts.
• Experience with data pipeline monitoring, troubleshooting, and performance optimization.
• Knowledge of data governance, security, and compliance.
• Experience optimizing Databricks clusters for cost and performance.
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.

Preferred Skills

• Experience with AI Agents / Agentic AI.
• Experience integrating AI capabilities into data platforms or data workflows.
• Knowledge of cloud data platforms such as AWS, Azure, or GCP.
• Experience with streaming technologies such as Kafka.
• Knowledge of data lake/lakehouse architecture.