Scala Sparks Developer
Ztek Consulting
New York, NY, United States
4 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Job source
Tech stack
Amazon Web Services
Microsoft Azure
Big Data
Cloud Computing
Code Review
Data Governance
Extract Transform Load (ETL)
Data Transformation
Data Warehousing
Distributed Systems
Apache Hadoop
Apache HBase
+14 more
Apache Hive
Python (Programming Language)
Data Streaming
Systems Integration
Unstructured Data
Data Processing
Scripting
Google Cloud
Apache Spark
Caching
Apache Kafka
Spark Streaming
Machine Learning Operations
Data Pipelines
Job description
- Develop, test, and maintain scalable data processing pipelines using Scala and Apache Spark.
- Implement data transformation and ETL workflows to handle structured and unstructured data.
- Utilize Python for data processing, scripting, and integrating Spark workflows.
- Optimize performance of Spark applications through tuning, partitioning, and caching strategies.
- Participate in code reviews, design discussions, and ensure adherence to best practices.
- Document workflows, architecture, and solutions for internal knowledge sharing.
Requirements
- Strong experience in Scala and Apache Spark (Spark Core, Spark SQL, DataFrames, RDDs).
- Proficiency in Python for data manipulation, scripting, and automation.
- Proven experience with distributed computing concepts and large-scale data processing.
- Knowledge of ETL development, data warehousing, and data modeling techniques.
- Familiarity with Big Data ecosystems such as Hadoop, Hive, Kafka, or HBase is a plus.
- Excellent communication and collaboration skills.
Preferred Qualifications
- Experience with cloud platforms such as AWS, Google Cloud Platform, or Azure (particularly with Spark services).
- Familiarity with streaming data processing using Spark Streaming or Structured Streaming.
- Knowledge of data governance, security, and compliance practices.
- Basic understanding of machine learning workflows is advantageous.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦