Lead Data Engineer, Advanced

• Execute creative software solutions, including design, development, and technical troubleshooting for complex data challenges.
• Drive team adoption of enterprise-authorized AI-assisted engineering practices and establish validation standards including secure coding, peer review, and automated testing.
• Apply SDLC toolchain and automation, including AI-assisted development capabilities, to increase delivery value.
• Lead architecture and hands-on delivery of large-scale ETL/ELT pipelines across ingestion, transformation, validation, reconciliation, and publishing.
• Design and operationalize data lake patterns including partitioning, data quality controls, lineage, governance, and reusable datasets.
• Build and optimize distributed processing workloads using Apache Spark and modern formats such as Parquet and Avro.
• Drive performance tuning across compute and storage, including Spark tuning for shuffle, joins, caching, and skew handling.
• Implement engineering best practices including code quality, automated testing, CI/CD, observability, security-by-design, and operational readiness.
• Build and maintain Java-based services supporting data workflows, including ingestion services, orchestration helpers, APIs, and data access layers.
• Develop REST APIs and integration components to enable downstream consumption and platform interoperability.
• Apply modern application practices including clean architecture, SOLID principles, test automation, and secure coding.

We are seeking a Lead Data Engineer to join our team. The ideal candidate will have strong experience in Java, Python, AWS, Apache Spark, and large-scale ETL/ELT data engineering and a proven ability to design and deliver high-quality data platforms, optimize performance, and lead AI-assisted engineering adoption., * Formal training or certification in software engineering and 5+ years of applied experience.

• 10+ years of software engineering experience with depth in data engineering and big data platforms.
• Strong hands-on development experience in Java and proficiency in Python.
• Proven experience designing and building robust ETL/ELT pipelines and data integration frameworks.
• Strong experience with Apache Spark, distributed processing concepts, and performance tuning.
• Knowledge of data storage serialization and formats such as Parquet and Avro.
• Solid understanding of data lake and lakehouse concepts, including data quality, metadata, and governance.
• Experience with cloud data warehouses such as Snowflake or equivalent.
• Demonstrated experience leading responsible use of AI-assisted development tools and setting validation standards for correctness, performance, and security.
• Ability to lead technical discussions, communicate across stakeholders, and drive delivery in cross-functional environments.

Preferred Skills:

• Experience with orchestration and workflow scheduling tools such as Airflow, Dagster, or Control-M.
• Exposure to streaming and event-driven processing, including Kafka and Spark Structured Streaming, and incremental processing patterns such as CDC and upserts.
• Experience with table formats and ACID layers such as Delta Lake, Apache Iceberg, Hudi.
• Strong production operations mindset covering on-call, SLAs/SLOs, observability, and incident response.
• Experience in regulated environments including risk, audit, compliance, privacy, and PII handling.
• Experience with cost optimization and FinOps practices for data platforms.
• Experience working with Agentic AI and Gen AI solutions and capabilities.

Must Have Summary:

• Python and Java.
• ETL pipeline design and development.
• AWS.
• AI tool experience in engineering workflows.

• Competitive compensation and benefits.
• Opportunities for growth with global clients.
• A supportive, inclusive culture that values innovation and people.
• Exposure to cutting-edge technologies and projects.

BCforward is a leading global IT consulting and workforce solutions firm providing services and support to Fortune 500 and government clients. Founded in 1998, BCforward has grown with our customers needs into a full-service business solutions provider. With delivery centers and offices across North America and India, we take pride in building long-term relationships and delivering excellence through innovation, collaboration, and integrity.