AWS Engineer ( Senior 10+ years)

Summary: You will design, build, and operate secure, reliable, and cost, Ingestion and CDC: Implement pipelines using AWS Lambda, EMR for batch/streaming( future need); manage schema evolution and backfills.

Orchestration: Build workflows with Step Functions and/or MWAA (Airflow), including retries, alerts, and SLAs.

Lakehouse and warehouse: Model datasets on S3 (Parquet/Iceberg), Glue Data Catalog, Athena; partitioning, compaction, and performance tuning.

Data quality and testing: Add automated tests (Great Expectations/Deequ, PySpark unit tests), data contracts, and CI gates.

Security and governance: Implement IAM, KMS, VPC endpoints, Lake Formation policies, PII handling, audit trails (CloudTrail), and RBAC/RLS.

Observability and FinOps: CloudWatch metrics/alerts, lineage, usage dashboards; cost optimisation (S3 lifecycle, compression, job sizing).

CI/CD and IaC: Provision with Terraform/CloudFormation/CDK; build pipelines with GitHub Actions; environment promotion and rollbacks.

Documentation and runbooks: Maintain pipeline diagrams, SLAs/SLOs, and incident playbooks.

Adoption: be able to adopt capabilities from other teams instead of building duplicate capabilities and be able to contribute back to community.

Outcomes (first 60-90 days)

Productionise one end to end pipeline with tests, monitoring, and alerting.

Establish governance baseline (Lake Formation, tagging/classification, encryption) and data contracts for two key sources.

Stand up CI/CD and IaC for data services; reduce at least one cost driver via storage/compute optimisation.

10+ years building data platforms on AWS; strong Python (incl. PySpark) and SQL.

Hands on with DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA, S3/Parquet/Iceberg, Glue Catalog, Athena, Redshift/Serverless.

IaC (Terraform) and CI/CD (GitHub Actions).

Data quality frameworks (Great Expectations/Deequ), schema evolution, backfills.