Big Data/Spark Engineer

Seeking a Big Data Engineer to design, develop, and optimize large-scale data processing systems. Will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. Ideally will have deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark etc.\n \n, * Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).

• Bachelor’s degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master’s degree and past Financial Services industry experience preferred. \n

• Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions. \n

• Past experience with developing enterprise quality solutions in an iterative or Agile environment. \n

• Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks. \n

• Experience with object oriented programming languages such as Java, Scala or Python. \n, * AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.). \n

• Technical Background: Strong software development background with ability to contribute to technical discussions. \n

• Agile Methodology: Extensive experience with Scrum, Kanban, and continuous improvement practices. \n

n

• Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable. \n

• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture. \n

• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. \n

• Optimize and enhance existing data pipelines for performance, scalability, and reliability. \n

• Develop automated testing frameworks and implement continuous testing for data quality assurance. \n

• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines. \n

• Work with data scientists and analysts to support data-driven decision-making across the organization. \n

• Ability to write and maintain automated unit, integration, and end-to-end tests. \n

• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues. \n, AI Skills\n \n \n \n

• Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools. \n

• AI Workflow Design: Experience redesigning development processes to leverage AI capabilities. \n

• Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements. \n

• Change Management: Experience leading teams through AI adoption and workflow transformation. \n

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data/Spark Engineer for a contract assignment with one of our premier Regulatory clients. Must be local to one of these office locations: Rockville MD or Tysons VA., SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let’s work better together, we mean it. You’ll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at https://sgainc.com/ .