Lead Cloudera/Hadoop Migration Engineer
Shree Narayani Networking Solutions Llc
United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source
Tech stack
Airflow
Amazon Web Services
Bash Shell
Big Data
Cloud Database
Information Systems
Data Transmissions
Data Security
DevOps
Gradle
Apache Hadoop
Hadoop Distributed File System
+23 more
Apache Hive
JSON
Python (Programming Language)
Octopus Deploy
Apache Oozie
Software Maintenance
Migration Manager
Cloudera
Shell Script
Software Engineering
Extensible Markup Language (XML)
Data Logging
Data Import/Export
Data Processing
Data Ingestion
Apache Spark
Software Troubleshooting
Information Technology
Bitbucket
Terraform
Data Pipelines
Amazon Elastic Mapreduce (EMR)
Artifactory
Job description
- Hands-on experience with Cloudera Hadoop stack (HDFS, Hive, Spark, Oozie, Python) Proficiency in writing, maintaining, and troubleshooting shell scripts (Bash or similar) for automation and operational support.
- Migration to AWS will need - Analyze existing Cloudera Hadoop architecture, workflows, and data pipelines.
- Develop a comprehensive migration strategy to transfer data, processing logic, and applications to AWS services.
- Design target architecture using AWS big data services such as Amazon EMR, Airflow, Glue, Lambda.
- Evaluate and select appropriate AWS tools for data ingestion, storage, processing, and analytics.
- Oversee the migration process, including data transfer, application re-platforming, and validation.
- Ensure the new architecture meets performance, scalability, reliability, and security requirements.
- Collaborate with developers, data engineers, DevOps, and stakeholders to ensure a seamless migration.
- Create detailed technical documentation and migration runbooks.
- Mentor team members on AWS big data best practices.
- Strong understanding of data security, compliance, and governance in the cloud.
- Proven track record of migrating or modernizing big data workloads to AWS.
- Experience with cloud automation and Infrastructure as Code (e.g. Terraform).
- AWS certifications (e.g., AWS Certified Solutions Architect, AWS Certified Big Data - Specialty) are highly desirable., * Design and develop scalable, maintainable software and data-processing solutions.
- Analyze existing Cloudera Hadoop architecture, workflows, and data pipelines.
- Develop and execute migration strategies from on-premises Hadoop to AWS.
- Design AWS architectures using EMR, Glue, Airflow, Lambda, and related services.
- Oversee data transfer, application re-platforming, testing, and validation.
- Ensure solutions meet performance, scalability, security, and reliability requirements.
- Troubleshoot software, platform, and data-pipeline issues.
- Create technical documentation, migration plans, and operational runbooks.
- Collaborate with engineering, DevOps, QA, product, and business stakeholders.
- Mentor team members on software engineering and AWS big data practices.
Requirements
- Bachelorβs Degree in Computer Science, Information Technology, Information Systems, or related field.
- 10+ years of experience with enterprise and application design patterns that are applied on Hadoop and Big data platforms in aws.
- The ideal candidate will possess deep expertise in both on-premises Hadoop ecosystems and modern cloud-native big data solutions on AWS., * Familiarity with xml/json data formats Knowledge of monitoring and logging tools
- 5+ years of experience with platform engineering tools such as Bitbucket, Gradle, Octopus, Artifactory.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦