> Markdown version of [/jobs/ext/3588496-big-data-engineer](https://www.wearedevelopers.com/jobs/ext/3588496-big-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Big Data Engineer

- **Company:** Indotronix Avani Group
- **Location:** Costa Mesa, CA, United States (Remote available)
- **Experience:** Experienced
- **Contract:** Temporary contract
- **Skills:** Application Programming Interfaces (APIs), Airflow, Amazon Web Services, Amazon Elastic Compute Cloud, Amazon S3, Batch Processing, Big Data, Continuous Integration, Information Engineering, Linux, Github, Gradle, Apache Hadoop, Hadoop Distributed File System, MapReduce, Integrated Development Environments, IntelliJ IDEA, Python (Programming Language), Apache Maven, Performance Tuning, Shell Script, DevOps Tools - Open-source, Snowflake, Apache Spark, SOAPAPI, Gitlab, Cloudformation, Bitbucket, Data Management, Api Gateway, Data Pipelines, Jenkins, Artifactory
- **Published:** October 5, 2026
- **Apply:** https://candidateportal.ceipal.com/job-details/fgZRKY-N4dYoWKVfKi8pvZF157tGpXxrP3_RGH0T9hE

## About the Role

7+ years designing, developing, and testing Big Data solutions (batch and APIs)

 - Deep hands-on AWS experience: EMR, EC2, ECS, S3, Step Functions, API Gateway, Airflow
 - Proficiency in Scala, PySpark, Java (REST APIs, SOAP web services)
 - Strong knowledge of Hadoop, HDFS, Spark, MapReduce, and Cassandra
 - Experience with performance tuning in Spark, Hadoop, and EMR environments
 - Solid Linux administration skills with shell scripting and Python automation
 - Proven track record building data pipelines and transformations in Snowflake using Spark

Preferred Skills

 - Familiarity with DevOps tooling: GitHub, GitLab, Bitbucket, IntelliJ, PyCharm
 - Experience with CI/CD: Maven, Gradle, Jenkins, Artifactory, AWS CodeCommit, CloudFormation, 7+ years of experience building Big Data applications (batch & API)Strong AWS experience (EMR EC2 ECS S3 Step Functions Airflow API Gateway)Proficiency in Scala Pyspark and Java (REST APIs & SOAP web services)Experience with Big Data technologies: Hadoop HDFS Spark MapReduceHands-on experience with CassandraStrong batch processing & performance tuning (Spark Hadoop EMR)Solid Linux experience (shell scripting and/or Python)Experience building data pipelines in Snowflake (ingestion & transformation using Spark)

## Description

Join Experian as a Big Data Engineer and drive the development, optimization, and support of scalable, high-performance data platforms in the AWS Cloud. Leverage your expertise to build robust data pipelines and API-driven microservices powering mission-critical systems. This is a fully remote, long-term contract offering you autonomy and technical growth on innovative projects., Design, build, and maintain large-scale batch and real-time data pipelines using Spark, Hadoop, and AWS EMR

 - Develop, deploy, and optimize API-based microservices leveraging Java, Scala, and AWS API Gateway
 - Implement data ingestion, transformation, and automation workflows with PySpark, Snowflake, and Airflow
 - Tune and scale data processing systems for maximum performance and reliability
 - Collaborate cross-functionally to deliver high-quality, production-ready solutions in a cloud-first environment
 - Troubleshoot and resolve complex issues in Linux-based environments using shell scripting and Python
 - Ensure best practices in code versioning and CI/CD processes for continuous integration and delivery


## Related Videos

- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Docker network without Docker](https://www.wearedevelopers.com/videos/1418-docker-network-without-docker)
- [GitLab CI pipelines for a whole company](https://www.wearedevelopers.com/videos/143-gitlab-ci-pipelines-for-a-whole-company)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Docker exec without Docker](https://www.wearedevelopers.com/videos/1094-docker-exec-without-docker)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)