DevOps/Data Engineer

Pivotal Solutions Inc

United States

15 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Experience required

2 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Amazon Web Services Amazon Cloudfront Amazon Elastic Compute Cloud Amazon S3 Big Data DevOps Identity and Access Management IP Routing Python (Programming Language) Aws Command Line Interface (CLI) Amazon Simple Notification Service (SNS) Software Deployment

+25 more

Datadog Circleci Autoscaling Apache Spark Amazon Virtual Private Cloud (VPC) Git Debezium Kubernetes Deployment Automation Complex Event Processing Tenable Nessus Apache Kafka Non-relational Database Build Tools Machine Learning Operations Sumo Logic (Software) Route53 Cloudwatch Api Gateway Amazon Simple Queue Service (SQS) Terraform Data Pipelines Serverless Computing Docker Databricks

Job description

Drive the infrastructure strategy alongside the rest of the DevOps team
Be responsible for building, managing, and automating our AWS infrastructure
Oversee the maintenance and growth of our data lakehouse infrastructure and implementation
Champion security by adopting necessary tools and processes
Provide leadership to the entire engineering team on various infrastructure and big data topics
Identify and adopt high-level technologies, processes, and services that impact the organization on a system-wide scope
Participate in the L2 rotation for engineering support

Requirements

8+ years of experience as a DevOps Engineer
5+ years of experience with the AWS cloud platform, including proficiency with AWS SDK, AWS CLI/UI, and services such as VPC, Route53, Private and Public subnets, route tables, IGW, EC2 Instances, CloudFront, API Gateway, IAM, ELB, Autoscaling, CloudWatch, EFS, NFS, EBS, S3, RDS, Lambda, SQS/SNS, Kafka, Security groups, etc.
5+ years of experience with Terraform/Terragrunt
5+ years of experience with event processing infrastructure
2+ years of experience as a Data Engineer with Databricks, including data pipelines, Debezium, Spark, and data modeling
Expert proficiency with Python
Model serving experience
Expert proficiency in deploying, maintaining, and scaling applications
Proficiency with Docker, Kubernetes, and Serverless functions
Expert proficiency in setting up CI/CD pipelines (tools/processes/governance)
Experience creating and maintaining fully automated CI/CD pipelines for code deployment using tools such as Git, CircleCI, AWS Code build, and Code Deploy / Code pipeline
Proficient with both Relational and Non-Relational databases
Experience with Security vulnerability scanning tools (Guard Duty, Inspector)
Experience building Monitoring dashboards and leveraging tools such as Cloudwatch, Datadog, and Sumologic