Data Engineer

Client Corporation is seeking a Senior Data Engineer to design, build, and operate scalable data solutions within a Databricks-based Lakehouse and streaming analytics environment. This role partners with business, BI, data science, and data modeling teams to translate requirements into production-ready data pipelines and curated datasets for reporting, analytics, and machine learning., * Partner with business, BI, data science, and data modeling teams to define requirements and deliver scalable data solutions.

• Design, develop, and operate production-grade data ingestion, transformation, and integration pipelines.
• Build and maintain Databricks, Spark, Delta Lake, and Delta Live Tables (DLT) pipelines supporting batch and streaming workloads.
• Integrate and transform structured and unstructured data using Databricks, SQL, Python, Spark, AWS, and Kafka.
• Implement incremental processing, merges/upserts, schema evolution, data quality checks, and other Lakehouse best practices.
• Apply data governance, security, access, retention, and sensitive-data handling practices in accordance with enterprise standards.
• Establish testing, data quality, observability, and operational readiness practices.
• Troubleshoot and optimize pipelines for performance, reliability, scalability, cost, and maintainability.
• Contribute to data architecture and engineering standards and support initiatives from requirements through deployment and ongoing operations.
• Participate in Agile delivery, including backlog refinement, iterative development, and dependency coordination.

• 5+ years of professional data engineering experience working with large datasets in production.
• Hands-on Databricks experience, including Jobs/Workflows, notebooks, and production pipelines.
• Hands-on Delta Lake experience, including incremental processing, merges/upserts, schema evolution, and performance optimization.
• Hands-on experience designing and operating Delta Live Tables (DLT) pipelines.
• Practical Unity Catalog or comparable Databricks governance experience.
• 4+ years of Apache Spark engineering experience using Spark SQL and/or PySpark.
• 3+ years of Apache Kafka or managed Kafka experience, including high-volume event processing, scaling, lag, and replay.
• 3+ years of AWS experience supporting data and analytics platforms, including S3, IAM, Glue, Lambda, or MSK.
• Strong SQL skills, including intermediate-to-advanced query development and optimization.
• Experience building and operating ETL/ELT pipelines in a Databricks Lakehouse environment.
• Experience working in an Agile environment such as Scrum, Kanban, or SAFe.

Preferred Experience

• Snowflake or other large-scale analytical databases.
• NoSQL databases such as Cassandra.
• Enterprise messaging technologies such as TIBCO EMS or IBM MQ.
• Integrating operational system feeds into cloud-based analytics and Lakehouse environments.
• Supporting BI, reporting, analytics, or machine learning initiatives.

Education

• Bachelor’s degree in Information Systems, Computer Science, Computer Information Systems, or a related field preferred.
• Equivalent practical experience may be considered.