Azure Databricks Engineer

Note : Candidate should have hands-on coding experience in Apache Spark, strong proficiency in Databricks, and proven expertise in designing, developing, and maintaining scalable ETL pipelines.", We are looking for a skilled Databricks Engineer with strong hands-on experience in Apache Spark, Azure Databricks, PySpark, and SQL. The ideal candidate will have proven expertise in designing, developing, and maintaining scalable ETL pipelines and working with large-scale data processing systems., * Design, develop, and optimize ETL pipelines using Azure Databricks, PySpark, and SQL

• Perform hands-on coding in Apache Spark for large-scale data processing and transformation
• Implement end-to-end data engineering solutions on Azure platform
• Work with structured and unstructured data across multiple sources
• Optimize performance and troubleshoot issues in Spark jobs and Databricks workflows
• Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders
• Ensure data quality, integrity, and governance across pipelines
• Develop reusable and scalable data frameworks and best practices

• Extensive hands-on coding experience in Apache Spark (PySpark)
• Strong expertise in Azure Databricks implementation
• Proficiency in SQL and data transformation techniques
• Experience in building and maintaining scalable ETL pipelines
• Good understanding of data warehousing concepts and data modeling
• Experience working with Azure Data Services (e.g., ADLS, ADF is a plus)
• Strong problem-solving and analytical skills

Preferred Skills:

• Experience with CI/CD pipelines and DevOps practices
• Knowledge of Delta Lake and data lake architecture
• Familiarity with performance tuning and optimization in Spark
• Experience working in Agile environments