> Markdown version of [/jobs/ext/2708501-big-data-engineer](https://www.wearedevelopers.com/jobs/ext/2708501-big-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Big Data Engineer - **Company:** Eliassen Group - **Location:** Vienna, VA, United States - **Salary:** $135,200.0 - $145,600.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Agile Methodology, Artificial Intelligence, Amazon Web Services, Amazon S3, Build Automation, Automation of Tests, Big Data, Configuration Management, Information Systems, Databases, Continuous Integration, Data Architecture, Data Integration, Extract Transform Load (ETL), Distributed Systems, Apache Hadoop, Apache Hive, Python (Programming Language), Object-Oriented Software Development, Operational Databases, Scrum Methodology, E2e Testing, Standard Sql, Scala (Programming Language), Simple Data Format, Software Engineering, System Testing, Test Case, Data Ingestion, GitHub Copilot, Concurrency, Prompt Engineering, Apache Spark, Caching, Information Technology, Functional Programming, GPT, Data Pipelines - **Published:** September 4, 2026 - **Apply:** https://www.techcareers.com/job.asp?id=3377259277&tx=KR5151FFL&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54 ## About the Role * Bachelor's degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience. Master's degree and Financial Services experience preferred. * Demonstrated expertise in object-oriented and database technologies leading to enterprise-quality solutions. * Experience delivering enterprise solutions in iterative or Agile environments. * Knowledge of software engineering approaches including test automation, build automation, and configuration management frameworks. * Strong written and verbal technical communication skills. * Ability to build effective working relationships and improve work product quality. * Strong organization, thoroughness, and ability to handle competing priorities. * Ability to maintain focus and develop proficiency in new skills rapidly. * Experience working in a fast paced environment. * Experience with Java, Scala, or Python. * Essential technical skills include AI tool proficiency (e.g., GitHub Copilot, Q Developer, ChatGPT, Claude), strong software development background, and extensive experience with Scrum, Kanban, and continuous improvement. * Big Data technologies: Hadoop, Spark, Hive, and Trino, with understanding of data skew, PB-scale data, and resource-related job failures. * Good to have: managing production data pipelines and ETL systems, CI/CD, writing test cases, and AWS certifications. Education Requirements: * Bachelor's degree in Computer Science, Information Systems, or related discipline required. Master's degree preferred. ## Description Our client is seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark., * Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala. * Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable. * Stay current with industry trends and emerging Big Data technologies to improve the data architecture. * Collaborate with cross-functional teams to translate business requirements into technical solutions. * Optimize and enhance existing data pipelines for performance, scalability, and reliability. * Develop automated testing frameworks and implement continuous testing for data quality assurance. * Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines. * Partner with data scientists and analysts to support data-driven decision-making. * Write and maintain automated unit, integration, and end-to-end tests. * Monitor and troubleshoot data pipelines in production to identify and resolve issues. * Apply SQL skills including window functions, joins, aggregations, and handling of NULLs, duplicates, and ordering. * Design and tune Apache Spark jobs, including partitioning, caching, broadcast joins, and troubleshooting DAG, stages, tasks, and executor performance. * Leverage AWS services such as S3, EMR, Glue, Lambda, and Athena, including Spark with S3 file formats and consistency considerations. * Write clean, modular, and performant Python or Scala code using functional programming concepts, and manage collections, concurrency, and memory. * Employ AI tools and prompt engineering to enhance development workflows, analysis, and continuous improvement. ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster) - [ Evaluating AI models for code comprehension](https://www.wearedevelopers.com/videos/1462-evaluating-ai-models-for-code-comprehension) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Event based cache invalidation in GraphQL](https://www.wearedevelopers.com/videos/433-event-based-cache-invalidation-in-graphql) - [Streaming AI Responses in Real-Time with SSE in Next.js & NestJS](https://www.wearedevelopers.com/videos/1630-streaming-ai-responses-in-real-time-with-sse-in-next-js-nestjs) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)