Lead Data Engineer

Ascii Group, LLC
Malvern, United States of America
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
$ 131K

Job location

Malvern, United States of America

Tech stack

Query Performance
API
Airflow
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Google BigQuery
Cloud Computing
Code Review
Computer Programming
Databases
Continuous Integration
Directed Acyclic Graph (Directed Graphs)
Data Governance
Data Masking
DevOps
Distributed Systems
Amazon DynamoDB
Monitoring of Systems
Python
NoSQL
Role-Based Access Control
Software Engineering
SQL Databases
Data Streaming
Workflow Management Systems
Data Logging
Data Processing
Scripting (Bash/Python/Go/Ruby)
Data Storage Technologies
Large Language Models
Snowflake
Spark
AWS Lambda
Event Driven Architecture
Kubernetes
Apache Flink
Cassandra
Kafka
Software Coding
Terraform
Data Pipelines
Docker
Redshift

Job description

· A Tech Lead is primarily responsible for the overall platform vision and ensuring systems do not break under scale.

Requirements

· Distributed Computing: Mastery of frameworks like Apache Spark or Ray for massive-scale parallel data processing.

· Streaming & Event-Driven Architecture: Deep understanding of real-time pipeline design using Kafka, Kinesis, or Flink.

· Cloud Infrastructure: Expertise in at least one major public cloud (AWS), specifically understanding storage/compute decoupling and cost optimization.

· Core Programming & Database Management

· Leads set coding standards and review code, requiring complete fluency in the fundamentals. [1]

· SQL: Advanced mastery for metrics computation, window functions, and query performance tuning across relational and columnar databases (e.g., Snowflake, Redshift, BigQuery).

· Scripting Languages: High proficiency in Python or Scala for writing reusable pipeline code and interacting with APIs.

· Data Storage: Deep familiarity with both columnar/analytical stores and NoSQL databases (e.g., DynamoDb, Cassandra).

· Pipeline Orchestration & DevOps

· Ensuring pipelines run smoothly, idempotently, and securely in production. [1, 2]

· Workflow Orchestration: Ability to architect Directed Acyclic Graphs (DAGs) in tools like Apache Airflow or Prefect.

· CI/CD & Infrastructure as Code (IaC): Applying software engineering principles to data by using Docker, Kubernetes, and Terraform.

· Data Governance & Security: Implementing Role-Based Access Control (RBAC), data masking, and compliance frameworks.

· Leadership & Soft Skills

· Tech leads also mentor junior engineers, estimate project timelines, and translate ambiguous business needs into concrete technical specifications.

· Mentorship & Code Review: Fostering a collaborative development environment and enforcing style guidelines.

· System Observability: Building logging, monitoring, and alerting mechanisms so the team knows exactly when and why pipelines fail.

Must have skills:

· Lead Data Engineer

· Python, AWS- S3, Lambdas

· Glue, Gen-Ai, LLMs, SQL

Apply for this position