Senior Databricks Data Engineer - Remote / Telecommute

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark.
• Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured datasets to support enterprise reporting, analytics, and regulatory requirements.
• Develop reusable frameworks and components for data ingestion, cleansing, transformation, validation, reconciliation, enrichment, and business rule implementation.
• Implement robust data quality checks, monitoring, and governance controls to ensure data accuracy and reliability.
• Perform performance tuning and optimization of Spark applications and Databricks workloads to improve scalability and processing efficiency.
• Collaborate with business stakeholders, architects, and analytics teams to understand data requirements and deliver scalable solutions.
• Troubleshoot and resolve production data pipeline issues while ensuring high availability and reliability.
• Support advanced analytics, customer segmentation, campaign analytics, AML/regulatory reporting, and data validation initiatives.

Our client is seeking an experienced Senior Databricks Data Engineer with strong expertise in designing, developing, and optimizing large-scale data engineering solutions on the Databricks platform. The candidate will have extensive hands-on experience building enterprise-grade data pipelines, processing high-volume datasets, and delivering scalable analytics solutions using PySpark, Spark SQL, and AWS cloud technologies., * 7+ years of hands-on experience with Databricks and modern data engineering platforms.

• Strong expertise in PySpark, Spark SQL, Apache Spark, and Databricks Lakehouse Architecture.
• Experience building and managing ETL/ELT pipelines on cloud platforms.
• Strong experience with AWS S3, Snowflake, Apache Airflow, and AWS Glue.
• Knowledge of data modeling, data warehousing, and big data processing techniques.
• Experience working with structured and semi-structured data formats such as JSON, Parquet, Avro, and CSV.
• Hands-on experience implementing data quality frameworks and monitoring solutions.
• Strong SQL and data analysis skills.

Preferred Skills:

• Experience with regulatory reporting, AML, compliance, or financial data domains.
• Exposure to CI/CD and DevOps practices for data engineering.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Knowledge of Agile/Scrum delivery methodologies.

Key Experience Areas: Candidates should have demonstrated experience in one or more of the following:

• Customer Segmentation Analytics.
• Campaign Analytics.
• AML and Regulatory Reporting.
• Enterprise Data Platforms.
• Production Data Pipeline Support.
• Data Validation and Reconciliation.
• Large-Scale Data Migration and Modernization Programs.