Databricks Data Engineer/Architect

1. Design and develop scalable data pipelines using Databricks and Apache Spark.
2. Build and maintain ETL/ELT processes for ingesting, transforming, and loading large volumes of data.
3. Architect modern data platforms using the Databricks Lakehouse architecture.
4. Implement data modeling solutions for data lakes, data warehouses, and analytics platforms.
5. Optimize Spark jobs, clusters, and workloads for performance and cost efficiency.
6. Integrate data from multiple sources, including cloud storage, databases, APIs, and streaming platforms.
7. Work with cloud platforms such as AWS, Azure, or Google Cloud Platform, leveraging native data services.
8. Establish data governance, security, data quality, and compliance standards.
9. Develop and manage Delta Lake solutions, including data versioning and optimization.
10. Collaborate with business stakeholders, data scientists, analysts, and engineering teams to define data requirements.
11. Provide technical leadership, architecture guidance, and best practices for data engineering initiatives.
12. Support CI/CD, automation, monitoring, and production deployment of data solutions.

Key Technical Skills Typically Required:

• Databricks
• Apache Spark (PySpark/Scala)
• SQL
• Delta Lake
• Data Lakehouse Architecture
• Azure Data Factory / AWS Glue (depending on cloud platform)
• Python
• Cloud Platforms (Azure, AWS, or Google Cloud Platform)
• ETL/ELT Development
• Data Modeling
• CI/CD and DevOps Practices

• Strong hands-on experience with Databricks and Spark.
• Experience designing enterprise-scale data architectures.
• Expertise in cloud-based data engineering solutions.
• Ability to lead technical discussions and mentor engineering teams.
• Strong understanding of performance tuning, security, and data governance.