> Markdown version of [/jobs/ext/1429327-big-data-engineer-ii](https://www.wearedevelopers.com/jobs/ext/1429327-big-data-engineer-ii). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Big Data Engineer II

- **Company:** IntraEdge, Inc.
- **Location:** United States
- **Experience:** Experienced
- **Contract:** Permanent contract
- **Skills:** Computer-Aided Design, Amazon Web Services, Software Applications, Application Performance Management, Computer Programming, Couchbase Servers, Data Visualization, Data Warehousing, Github, Apache Hadoop, MapReduce, Apache HBase, Apache Hive, Python (Programming Language), Unix Shell, MongoDB, NoSQL, Scrum Methodology, Software Engineering, SQL Databases, Tableau (Software), Apache Spark, Pyspark, Apache Kafka, Stream Processing, Looker Analytics, Data Pipelines, Microservices
- **Published:** July 24, 2026
- **Apply:** https://jobs.intraedge.com/jobdetails.asp?jo_num=17438&apply=yes&

## About the Role

* 5+ years of software development experience and leading teams of engineers and scrum teams
 * 3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and PySpark)
 * Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes), optimizing joins while processing huge amount of data
 * Experience in UNIX shell scripting

Additional Good to have requirements:

 * Solid Datawarehousing concepts
 * Knowledge of Financial reporting ecosystem will be a plus
 * Experience with Data Visualization tools like Tableau, SiSense, Looker
 * Expert on Distributed ecosystem
 * Hands-on experience with programming using Python/Scala
 * Expert on Hadoop and Spark Architecture and its working principle
 * Ability to design and develop optimized Data pipelines for batch and real time data processing
 * Should have experience in analysis, design, development, testing, and implementation of system applications
 * Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows
 * Aptitude for learning and applying programming concepts.
 * Ability to effectively communicate with internal and external business partners. Preferred Qualifications:
 * Knowledge of cloud platforms like GCP/AWS, building Microservices and scalable solutions, will be preferred
 * 2+ years of experience in designing and building solutions using Kafka streams or queues
 * Experience with GitHub and leveraging CI/CD pipelines
 * Experience with NoSQL i.e., HBase, Couchbase, MongoDB

## Description

Strong HIVE, SPARK, SQL, UNIX

 * Develop and design software applications, translating user needs into system architecture. Assess and validate application performance and integration of component systems and provide process flow diagrams. Test the engineering resilience of software and automation tools.
 * Be part of an enthusiastic, high performing technology team developing solutions to drive engagement and loyalty within our existing cardmember base and attract new customers to the Amex brand.
 * The position will also play a critical role partnering with other development teams, testing and quality, and production support, to meet implementation dates and allow smooth transition throughout the development life-cycle.
 * The successful candidate will be focused on building and executing against a strategy and roadmap focused on moving from monolithic, tightly coupled, batch-based legacy platforms to a loosely coupled, event-driven, microservices-based architecture to meet our long-term business goals.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [7 Cloud Computing Trends Coming in 2025 for Developers](https://www.wearedevelopers.com/magazine/412-7-cloud-computing-trends-coming-in-2025-for-developers)
- [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)