Big Data Engineer
Optomi LLC
Vienna, United States of America
7 days ago
Role details
Contract type
Permanent contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
EnglishJob location
Vienna, United States of America
Tech stack
Artificial Intelligence
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Big Data
Data Integration
ETL
Hadoop
Hive
Operational Databases
SQL Databases
GitHub Copilot
Istio
Spark
Kubernetes
GPT
Data Pipelines
Amazon Web Services (AWS)
Docker
Job description
- We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms.
Requirements
- Experience with Big Data technologies such as Hadoop, Spark, Hive & Trino
- Strong experience with Kubernetes architecture
- Hands-on experience with Amazon EMR on EKS
- Deep understanding of Spark's core architecture
- Experience with AWS services like S3, EMR, EMR on EKS, Glue, Lambda, Athena
- Proficiency with SQL window functions, multi-table joins, and aggregations
- Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)
Job Nice to Haves:
- Experience with managing production data pipelines/ETL systems
- Experience with CI/CD pipelines
- Experience with Infrastructure as Code
- Experience with Docker and container image optimization
- Knowledge of service mesh technologies
- AWS certifications