> Markdown version of [/jobs/ext/584927-lead-software-engineer-big-data](https://www.wearedevelopers.com/jobs/ext/584927-lead-software-engineer-big-data). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Software Engineer - Big Data

- **Company:** Financial Industry Regulatory Authority, Inc.
- **Location:** Washington, DC, United States
- **Experience:** Expert
- **Salary:** $114,200.0 - $248,700.0
- **Contract:** Permanent contract
- **Skills:** Agile Methodology, Artificial Intelligence, Algorithm Design, Amazon S3, Data Analysis, Automation of Tests, Unit Testing, Big Data, Cloud Computing, Cloud Database, Information Systems, Computer Programming, Databases, Continuous Delivery, Continuous Integration, Data Structures, Software Design Patterns, DevOps, Distributed Computing Environment, Memory Management, Apache Hadoop, Apache Hive, Integrated Development Environments, Python (Programming Language), NoSQL, Object-Oriented Software Development, Performance Tuning, Scrum Methodology, Query Optimization, Big O, Secure Coding, Software Construction, Software Engineering, SQL Databases, Workflow Management Systems, Data Processing, Enterprise Software Applications, Data Storage Technologies, GitHub Copilot, Concurrency, Prompt Engineering, Apache Spark, Software Security, Caching, Information Technology, Data Analytics, Functional Programming, GPT, Serverless Computing
- **Published:** June 18, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=496570ebf4fc78f4

## About the Role

Do you have experience in Workflow management (operations management method)?, Do you have a Bachelor's degree?, Big Data Technologies: Extensive hands-on experience with Hadoop, Spark, Hive, and Trino for processing petabyte-scale datasets. Proven ability to diagnose and resolve data skew, resource limitations, scalability challenges, and job failures in production environments.

Apache Spark Expertise: Deep understanding of Spark architecture including executors, tasks, stages, and DAG execution. Demonstrated proficiency in performance tuning techniques such as partitioning strategies, caching, broadcast joins, and optimization of large-scale data processing jobs.

SQL Mastery: Advanced proficiency in complex SQL including window functions, multi-table joins, aggregations, and query optimization. Experience handling edge cases involving NULLs, duplicates, and ordering in production environments.

Cloud Technologies: Strong experience with AWS services including S3, EMR, Glue, Lambda, Athena, EKS, and serverless architectures. Practical knowledge of file format optimization, consistency management, and cloud-based data processing workflows.

Programming (Python/Scala): Proficiency in writing clean, modular, and performant code using Python or Scala. Strong understanding of functional programming concepts (immutability, higher-order functions), collections, concurrency, and memory management for scalable data processing.

AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.) and AI workflow design. Proven ability in prompt engineering and leveraging AI coding assistants for enhanced productivity.

Agile Methodology & Change Management: Extensive experience with Scrum, Kanban, and continuous improvement practices. Demonstrated experience leading teams through AI adoption, technology transformation, and workflow optimization initiatives.

System Design & Architecture: Strong system design experience with ability to architect scalable, distributed data processing solutions and enterprise-level applications.

Object-Oriented Development: Strong experience in object-oriented programming principles, design patterns, and software engineering best practices.

Data Storage Technologies: Strong experience with modern data storage solutions, database technologies, and data modeling approaches for both relational and NoSQL systems.

Performance Tuning & Optimization: Demonstrated expertise in identifying bottlenecks and optimizing application and data processing performance across the full stack.

DevOps & CI/CD: Strong experience with DevOps practices, continuous integration, continuous deployment pipelines, and infrastructure as code.

Software Security: Strong understanding of security best practices, secure coding principles, and application security across all layers.

Data Analysis & Insights: Ability to interpret AI-generated insights and data-driven metrics, translating them into actionable technical improvements and measurable business value., * Bachelor's degree in Computer Science, Information Systems or related discipline with at least 7 years of related experience, or equivalent training and/or work experience.

 * Strong system design experience
 * Strong experience in object-oriented development
 * Strong experience with cloud technologies
 * Strong experience in data storage technologies
 * Strong experience in performance tuning and optimization
 * Strong experience in DevOps and CI\CD technologies
 * Strong experience test automation and unit testing
 * Strong experience software security

## Description

Working independently, the Lead Software Engineer owns development of software products and works on improving the overall quality of the product throughout the software development life cycle and mentors other Software Engineers. Reports directly to a Director or Senior Director., * Seasoned contributor that works on complex projects requiring extensive knowledge of specialty area and industry practices

 * Manages processes, projects or products to improve efficiency within org or product area
 * Influences outcomes in cross-functional settings
 * Demonstrates solid understanding of systems and processes within job area and recommends improvements
 * Designs and develops industry-leading solutions that meet defined quality standards, reliability specifications, and help achieve key strategic objectives
 * Ensures comparison of tasks and quality of work, supporting peers to ensure the team delivers consistently
 * Excellent understanding in computer science fundamentals - Algorithm design, Problem solving, Complexity analysis and data structures.

Engineering/Operations Excellence:

 * Helps identify gaps between system components and designs and delivers solutions
 * Implements methods for managing and maintaining the organization's software development environments to ensure software engineers are equipped with the tools and resources necessary to create and validate builds
 * Takes broad perspective to problems and exercises independent judgement in techniques and evaluation criteria to obtain results
 * Develops actionable insights from analyzing software development trends, software engineering best practices, and innovative test environment management tools and communicates integration recommendations to management
 * Identifies ways to overcome challenging situations that jeopardize the achievement of business goals

Collaboration:

 * Coaches and trains less experienced colleagues in techniques, processes, and responsibilities
 * Integrates the culture of intelligence, integrity, and initiative into team goals and decisions
 * Partners within and across teams to remove barriers and identify shared goals and priorities
 * Openly shares progress and priorities with key stakeholders, communicating complex or technical information at the right level.
 * Demonstration of FINRA's values.
 * Collaboration, both in-person and virtually, in furtherance of FINRA's mission of investor protection and market integrity., FINRA's Code of Conduct imposes restrictions on employees' investments and requires financial disclosures that are uniquely related to our role as a securities regulator. FINRA employees are required to disclose to FINRA all brokerage accounts that they maintain, and those in which they control trading or have a financial interest (including any trust account of which they are a trustee or beneficiary and all accounts of a spouse, domestic partner or minor child who lives with the employee) and to authorize their broker-dealers to provide FINRA with duplicate statements for all of those accounts. All of those accounts are subject to the Code's investment and securities account restrictions, and new employees must comply with those investment restrictions-including disposing of any security issued by a company on FINRA's Prohibited Company List or obtaining a written waiver from their Executive Vice President-by the date they begin employment with FINRA.
Employees may only maintain securities accounts that must be disclosed to FINRA at one or more securities firms that provide an electronic feed (e-feed) of data to FINRA, and must move securities accounts from other securities firms to a firm that provides an e-feed within three months of beginning employment.

You can read more about these restrictions here.

As standard practice, employees must also execute FINRA's Employee Confidentiality and Invention Assignment Agreement without qualification or modification and comply with the company's policy on nepotism.

Search Firm Representatives


## Related Videos

- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis)
- [ Evaluating AI models for code comprehension](https://www.wearedevelopers.com/videos/1462-evaluating-ai-models-for-code-comprehension)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Tomorrow's cloud data platforms - fully managed database-as-a-service (DBaaS)](https://www.wearedevelopers.com/videos/254-tomorrow-s-cloud-data-platforms-fully-managed-database-as-a-service-dbaas)
- [Streaming AI Responses in Real-Time with SSE in Next.js & NestJS](https://www.wearedevelopers.com/videos/1630-streaming-ai-responses-in-real-time-with-sse-in-next-js-nestjs)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)