Senior Data Platform Engineer

Marici Solutions
New York, NY, United States
2 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source

Tech stack

Java (Programming Language) Airflow Amazon Web Services Amazon S3 Apache HTTP Server BigQuery Cloud Computing Continuous Integration Data Files Data Governance Data Infrastructure Data Migration
+21 more
Federated Identity Management Identity and Access Management Python (Programming Language) Performance Tuning Runbook SQL Databases Data Streaming Workflow Management Systems Parquet Data Processing Google Cloud Autoscaling Apache Spark AWS Lambda Core Data Kubernetes Avro Apache Kafka Spark Streaming Terraform Data Pipelines

Requirements

Experience level - This Engineer should have 8+ years of total experience, with 5+ years of hands-on Cloud experience in either AWS or Google Cloud Platform and infrastructure ownership. Please screen out profiles that only design and do not implement the code. Key Responsibilities: Must have Β· Data Processing : Hands-on experience in Spark Jobs, Streaming ingestion, Data Migration and performance tuning (skew, partitioning, autoscaling, cost). Β· Job Orchestration : Manage and optimize core data platform infrastructure, workflow orchestration (Airflow / Cloud Composer), and automated data quality, observability, and validation frameworks. Β· Terraform or similar IaC : Infrastructure Design, implementation and CI/CD Process. Β· On-Call : Own platform SLAs, automated alerting and compute/storage cost optimization. Β· Data Governance : Dataset access control, policy tags , column-level security, row-level security, cost and quota management. Β· Hands-on experience in AWS S3, Trino & GCS. Β· Strong Skills in Python/Java and SQL. Β· Production ownership: troubleshooting, root-cause analysis and on-call for shared platform services. Added advantage Β· Experience with the data migration from AWS to Google Cloud Platform. This includes access between the two clouds, for example Workload Identity Federation and BigQuery Omni. Β· Data formats: Apache Iceberg, Avro, and Parquet. Β· Streaming: Kafka, Spark Structured Streaming. Β· Data governance: PII masking, Cloud DLP, PII Scanner, Apache Ranger. Β· Kubernetes basics, AWS Lambda, Google Cloud Platform Cloud Run and dbt. Β· Experience building self-service internal platforms tools for the data users Β· Google Cloud Platform Professional Cloud Architect or Data Engineer certification. Collaboration and communication (must have) Β· Coordinate daily with an offshore engineering team: break down work, write clear tickets and hand-offs, review their code and unblock them across time zones. Β· Communicate with stakeholders: explain platform decisions, trade-offs and timelines to engineering leads, analytics teams and product partners. Β· Write clear design docs, runbooks and guides. Prioritization - We need a senior data platform engineer. This engineer must have hands-on experience in Cloud & Data Platform. We prefer experience with Google Cloud Platform and AWS. This engineer builds and operates the platform infrastructure for the data, ML, and analytics teams. The scope includes data governance, security, data pipeline design, CI/CD, and related tools. Platform skills are the first priority: Terraform or a similar IaC tool, IAM and security policies, Spark compute platforms, BigQuery, Trino, job orchestration, and networking. Pipeline engineering and analytics engineering skills are the second priority.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…