Big Data Engineer

Join Experian as a Big Data Engineer and drive the development, optimization, and support of scalable, high-performance data platforms in the AWS Cloud. Leverage your expertise to build robust data pipelines and API-driven microservices powering mission-critical systems. This is a fully remote, long-term contract offering you autonomy and technical growth on innovative projects., Design, build, and maintain large-scale batch and real-time data pipelines using Spark, Hadoop, and AWS EMR

• Develop, deploy, and optimize API-based microservices leveraging Java, Scala, and AWS API Gateway
• Implement data ingestion, transformation, and automation workflows with PySpark, Snowflake, and Airflow
• Tune and scale data processing systems for maximum performance and reliability
• Collaborate cross-functionally to deliver high-quality, production-ready solutions in a cloud-first environment
• Troubleshoot and resolve complex issues in Linux-based environments using shell scripting and Python
• Ensure best practices in code versioning and CI/CD processes for continuous integration and delivery

7+ years designing, developing, and testing Big Data solutions (batch and APIs)

• Deep hands-on AWS experience: EMR, EC2, ECS, S3, Step Functions, API Gateway, Airflow
• Proficiency in Scala, PySpark, Java (REST APIs, SOAP web services)
• Strong knowledge of Hadoop, HDFS, Spark, MapReduce, and Cassandra
• Experience with performance tuning in Spark, Hadoop, and EMR environments
• Solid Linux administration skills with shell scripting and Python automation
• Proven track record building data pipelines and transformations in Snowflake using Spark

Preferred Skills

• Familiarity with DevOps tooling: GitHub, GitLab, Bitbucket, IntelliJ, PyCharm
• Experience with CI/CD: Maven, Gradle, Jenkins, Artifactory, AWS CodeCommit, CloudFormation, 7+ years of experience building Big Data applications (batch & API)Strong AWS experience (EMR EC2 ECS S3 Step Functions Airflow API Gateway)Proficiency in Scala Pyspark and Java (REST APIs & SOAP web services)Experience with Big Data technologies: Hadoop HDFS Spark MapReduceHands-on experience with CassandraStrong batch processing & performance tuning (Spark Hadoop EMR)Solid Linux experience (shell scripting and/or Python)Experience building data pipelines in Snowflake (ingestion & transformation using Spark)

Fully remote work environment - collaborate from anywhere in the U.S.

• Work with cutting-edge AWS and Big Data tools
• Opportunity to own challenging, high-impact projects
• Long-term contract with potential for future engagement
• Expand your expertise with a global leader in data-driven solutions