AWS Data & Lakehouse Architect

Client is looking for an experienced AWS Data & Lakehouse Architect to support enterprise data modernization initiatives involving migration of legacy/on-premises data platforms to a modern AWS-based Data Lakehouse architecture using Databricks and Apache Spark., 1. AWS Data & Cloud Architecture

• Design scalable, resilient, secure, and cost-effective enterprise data platforms on AWS.
• Define reference architectures for migration from on-premises data platforms to AWS Data Lake/Lakehouse.

• Architect AWS solutions leveraging services such as:

• Amazon S3
• IAM
• AWS Glue
• AWS Lake Formation
• Amazon Redshift
• Amazon RDS/Aurora
• AWS Lambda
• Amazon EC2
• Amazon VPC
• AWS KMS
• CloudWatch
• CloudTrail
• Secrets Manager
• AWS Step Functions
• Amazon EventBridge

Define appropriate patterns for networking, security, data ingestion, storage, processing, consumption, and monitoring.

1. Databricks & Lakehouse Architecture * Design and implement enterprise Databricks Lakehouse solutions on AWS. * Strong hands-on experience with Databricks, Apache Spark, PySpark, SQL and Delta Lake. * Design Bronze, Silver and Gold data-layer patterns based on workload requirements. * Define scalable ingestion, transformation, orchestration, and data-serving patterns. * Implement and optimize Delta tables, partitioning strategies, Spark workloads, cluster configurations, and large-scale ETL/ELT pipelines. * Experience with Databricks Unity Catalog, governance, access controls, lineage, and workspace architecture is highly desirable.

2. On-Premises to AWS Modernization * Assess existing on-premises data applications, ETL workloads, databases, file systems, and batch processing environments. * Define migration and modernization approaches based on rehost, replatform, refactor, or retire decisions. * Modernize legacy ETL/data-processing workloads into cloud-native and Lakehouse-based architectures. * Identify application and data dependencies, migration risks, sequencing, and integration requirements. * Support coexistence and transition patterns between legacy/on-premises and AWS platforms during migration.

3. Data Engineering

The architect must be capable of going hands-on when required, including:

• Developing and reviewing PySpark/Spark and SQL workloads.
• Designing complex ETL/ELT pipelines.
• Debugging Spark jobs and data-processing failures.
• Reviewing code and engineering standards.
• Optimizing Spark transformations, joins, partitioning, caching, and processing patterns.
• Troubleshooting performance bottlenecks across data pipelines and platform components.
• Designing reusable data-engineering frameworks and patterns.

1. Security, Governance & Compliance * Design security using IAM roles/policies, KMS encryption, network isolation, secrets management, and least-privilege principles. * Establish data access patterns across Databricks, S3, databases, and downstream applications. * Work with enterprise security and infrastructure teams on secure connectivity between on-premises platforms and AWS. * Incorporate data governance, metadata management, lineage, auditing, and access controls into the architecture. * Strong understanding of enterprise data protection and security standards.

2. Performance, Scalability & Cost Optimization * Architect for large-scale enterprise data workloads. * Optimize Spark/Databricks workloads for performance and scalability. * Evaluate compute, storage, cluster sizing, and workload patterns to improve cloud cost efficiency. * Establish monitoring, logging, alerting, and operational support patterns. * Identify opportunities to reduce unnecessary cloud consumption while maintaining required performance and SLAs.

The ideal candidate should combine strong architecture capabilities with hands-on engineering experience. This is not purely an advisory or high-level solution architect role. The candidate should be comfortable designing the target architecture as well as working directly with AWS services, Databricks, Spark, data pipelines, security configurations, performance optimization, and migration patterns., 7. DevOps / Infrastructure Automation

• Experience implementing CI/CD for data and Databricks workloads.
• Familiarity with Infrastructure as Code, preferably Terraform and AWS-native automation.
• Experience integrating Git-based development, automated deployments, environment promotion, and release-management processes.
• Establish standards for DEV, QA/UAT and Production environments.

Mandatory Technical Skills

Vendors should prioritize candidates with strong hands-on depth across the following:

Area

Required Expertise

AWS

Strong hands-on AWS architecture and implementation

Databricks

Strong architecture + hands-on Databricks experience

Apache Spark

Advanced Spark/PySpark development and optimization

Data Lakehouse

Strong Data Lake/Lakehouse architecture patterns

AWS Storage

Amazon S3 and related data-lake architecture

Data Engineering

ETL/ELT, batch and scalable data pipelines

SQL

Advanced SQL and query optimization

Delta Lake

Strong experience with Delta-based architectures

Migration

On-premises to AWS data migration/modernization

Security

IAM, KMS, secrets, encryption, network/security patterns

Governance

Unity Catalog/Lake Formation or equivalent concepts

DevOps

CI/CD, Git and deployment automation

IaC

Terraform preferred

Monitoring

CloudWatch and Databricks/platform monitoring

Programming

Python/PySpark

Preferred Qualifications

• AWS Certified Solutions Architect - Professional or Associate.
• Databricks Data Engineer/Databricks Architect certifications are desirable.
• Experience modernizing large enterprise data platforms onto AWS.
• Experience with Redshift, Glue, Lake Formation and AWS data services.
• Experience integrating Databricks with enterprise AWS environments.
• Exposure to streaming/event-driven data processing such as Kafka or Amazon MSK/Kinesis.
• Experience with enterprise data governance and metadata solutions.
• Automotive industry experience, particularly large-scale manufacturing, sales, vehicle, supply-chain, or connected-vehicle datasets, is an advantage but not mandatory.