Big Data Engineer

Our client is seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark., * Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala.

• Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
• Stay current with industry trends and emerging Big Data technologies to improve the data architecture.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
• Partner with data scientists and analysts to support data-driven decision-making.
• Write and maintain automated unit, integration, and end-to-end tests.
• Monitor and troubleshoot data pipelines in production to identify and resolve issues.
• Apply SQL skills including window functions, joins, aggregations, and handling of NULLs, duplicates, and ordering.
• Design and tune Apache Spark jobs, including partitioning, caching, broadcast joins, and troubleshooting DAG, stages, tasks, and executor performance.
• Leverage AWS services such as S3, EMR, Glue, Lambda, and Athena, including Spark with S3 file formats and consistency considerations.
• Write clean, modular, and performant Python or Scala code using functional programming concepts, and manage collections, concurrency, and memory.
• Employ AI tools and prompt engineering to enhance development workflows, analysis, and continuous improvement.

• Bachelor’s degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience. Master’s degree and Financial Services experience preferred.
• Demonstrated expertise in object-oriented and database technologies leading to enterprise-quality solutions.
• Experience delivering enterprise solutions in iterative or Agile environments.
• Knowledge of software engineering approaches including test automation, build automation, and configuration management frameworks.
• Strong written and verbal technical communication skills.
• Ability to build effective working relationships and improve work product quality.
• Strong organization, thoroughness, and ability to handle competing priorities.
• Ability to maintain focus and develop proficiency in new skills rapidly.
• Experience working in a fast paced environment.
• Experience with Java, Scala, or Python.
• Essential technical skills include AI tool proficiency (e.g., GitHub Copilot, Q Developer, ChatGPT, Claude), strong software development background, and extensive experience with Scrum, Kanban, and continuous improvement.
• Big Data technologies: Hadoop, Spark, Hive, and Trino, with understanding of data skew, PB-scale data, and resource-related job failures.
• Good to have: managing production data pipelines and ETL systems, CI/CD, writing test cases, and AWS certifications.

Education Requirements:

• Bachelor’s degree in Computer Science, Information Systems, or related discipline required. Master’s degree preferred.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Rate: $65.00 to $70.00/hr. w2, Skills, experience, and other compensable factors will be considered when determining pay rate. The pay range provided in this posting reflects a W2 hourly rate; other employment options may be available that may result in pay outside of the provided range.

W2 employees of Eliassen Group who are regularly scheduled to work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), dental, vision, pre-tax accounts, other voluntary benefits including life and disability insurance, 401(k) with match, and sick time if required by law in the worked-in state/locality.

If anyone reaches out to you about an open position connected with Eliassen Group, please ensure that you are working directly with us by confirming the following:

· When you work with Eliassen Group, all email communication will come from an Eliassen.com address, never Gmail, Yahoo, etc.

Eliassen Group is a strategic consulting firm that helps organizations reach further and achieve more through our technology, business advisory, and life sciences solutions. For nearly 40 years, we have combined exceptional people, deep domain expertise, and intelligent capabilities to expand our clients’ capacity and accelerate meaningful outcomes. We are driven by a purpose to positively impact the lives of our employees, clients, consultants, and the communities we serve.