Sr. Data Engineer

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable, high-performance data platforms that support analytics and machine learning initiatives. This role requires strong hands-on expertise in modern data engineering technologies and the ability to take end-to-end ownership of data services in production environments.

The ideal candidate will collaborate closely with cross-functional teams, contribute to architectural decisions, and ensure the reliability, security, and quality of data across multiple platforms.

Mandatory Technical Skills

• Java
• Apache Spark
• AWS
• RESTful Web Services / APIs
• Apache Kafka
• Data Engineering (end-to-end pipeline design and implementation)

Key Responsibilities

Software Development & Data Engineering

• Design, build, and maintain scalable data pipelines and software applications using modern engineering practices.
• Write clean, reusable, and maintainable code by applying standard design patterns and libraries.
• Refactor and optimize existing codebases to improve performance, scalability, and maintainability.
• Ensure application quality by applying appropriate testing strategies and adhering to defined test standards.
• Maintain data security, integrity, and quality in line with established standards and best practices.

End-to-End System Ownership

• Own data services and applications end-to-end, including monitoring application health, performance, and key metrics.
• Proactively identify and mitigate business continuity risks by implementing best practices and maintaining documentation such as runbooks and operational guides.
• Apply continuous delivery and experimentation frameworks to reduce risk and incorporate feedback.
• Independently manage deployment, operations, and production support for data applications.

Technical Incident Management

• Respond to and resolve live production incidents within defined SLAs, minimizing customer and business impact.
• Perform root cause analysis and implement long-term solutions to improve system reliability.
• Contribute to post-incident reviews, postmortems, and incident documentation.

Architectural Guidance

• Provide technical guidance to product and engineering teams, ensuring solutions meet functional, non-functional, and architectural requirements.
• Challenge design decisions constructively and provide context within the broader enterprise architecture.
• Contribute to defining and evolving target architecture and technical capabilities for data platforms.

Software Systems Design

• Evaluate and recommend architecture solutions considering cost, business needs, technology constraints, and emerging trends.
• Assess the impact of system changes or new integrations with a high-level understanding of infrastructure and platform architecture.
• Support business growth and development velocity through prototyping, technical spikes, and technology evaluations.
• Design solutions that meet current requirements while remaining flexible for future enhancements.

Critical Thinking & Problem Solving

• Identify patterns and root causes in complex technical challenges using analytical and logical thinking.
• Evaluate ideas, plans, and solutions objectively, incorporating external insights and defining measurable improvements.
• Clearly articulate technical decisions and their rationale to stakeholders.

Continuous Improvement & Quality

• Identify opportunities for process, system, and performance improvements by reviewing existing workflows and standards.
• Design and implement improved processes, practices, and standards to enhance business and engineering performance.
• Guide and mentor junior team members on data quality, security, and engineering best practices.

Communication & Collaboration

• Communicate technical information clearly and effectively to both technical and non-technical audiences.
• Collaborate with stakeholders to achieve mutually agreeable solutions through clear communication and adaptability.
• Demonstrate active listening by asking relevant questions and engaging constructively in discussions.

Do you have experience in Spark?, Mandatory Skills - Java, spark, AWS, Web Services Rest API, Kafka, Data Engineering