Data Engineer

Sr Engineer, Data is responsible for designing and developing scalable data architectures and ingestion pipelines across on-premises, cloud, and hybrid platforms to support enterprise data and analytics needs. This role collaborates closely with data engineers to analyze, architect, design, and deliver data warehouse and business analytics solutions. The Sr Engineer designs and implements complex data pipelines, optimizes data delivery, and automates manual processes using Azure Data Factory, Databricks, ADLS, Snowflake, Oracle, Python, SQL, and Azure. The role includes mentoring team members to strengthen data engineering capabilities and drive technical excellence. Success is measured by the effectiveness of data engineering solutions, team skill development, and contributions to data architecture innovation that enable data-driven decision-making and developing privacy compliance solutions at scale.

What You’ll Do Design and build scalable data pipelines using Azure Data Factory for seamless data integration across diverse sources. Mastery of SQL for efficient data querying and manipulation. Analyze complex datasets to identify data anomalies, trends and actionable insights. Strong data modeling skills to design and optimize data structures. Implement data ingestion and transformations using Snowflake features like Stored Procedures, Streams, Tasks, Snowpipe, Iceberg Tables, Dynamic Tables, Storage Integrations and Views. Manage and optimize Azure Data Lake Storage (ADLS) for secure, scalable and high-performance data storage and retrieval. Design and build scalable big data processing workflows using Azure Databricks with Unity Catalog to ensure secure, governed data access. Leverage Apache Spark for advanced data transformation and optimize cluster configurations for performance and cost efficiency. Familiarity with the Azure cloud platform to leverage its services and tools. Diagnose and resolve issues within data pipelines to ensure smooth data flow. Tune SQL queries and pipelines for performance and cost efficiency. Optimize data pipelines to improve data delivery, reliability, and performance Ensure data quality by implementing validation, cleansing and monitoring mechanisms. Ensure data governance and compliance with privacy regulations. Effective communication and collaboration skills to work seamlessly with cross-functional teams. Adaptability to work in a fast-paced and evolving environment to meet dynamic business needs. Daily use of AI productivity tools (current stack: Claude, and Cursor or similar IDE) is required for core data engineering activities including pipeline development, SQL authoring, code generation, code review, debugging, testing, and documentation. Design, build, and operate AI agents to automate privacy data engineering solutions. Apply foundation models, prompt engineering, and retrieval-augmented generation (RAG) patterns to privacy data engineering use cases. Implement audit logging, observability, and human-in-the-loop controls for AI agents and AI-assisted workflows in privacy data engineering.

Bachelor’s degree in computer science, Computer Engineering, or a related field 5 7 years of hands-on experience designing, building, and supporting data engineering and ETL solutions Ability to write complex, highly performable, scalable, large volume handling SQLs, stored procedures. Strong experience developing and migrating data solutions in Azure Strong experience in building data engineering solutions using Azure Flink, dbt and any open-source frameworks to build vendor agnostic technology solutions. Demonstrated experience in data pipelines and ETL development using Azure Data Factory, Databricks, ADLS, Oracle, Snowflake, and Python Strong experience in developing CI/CD solutions using Azure DevOps or gitlab Strong analytical and problem-solving skills applied to complex data ingestion and integration challenges Ability to manage multiple workstreams with strong organizational and prioritization skills Passion for learning and applying new data engineering technologies and design patterns. Familiarity with foundation models, prompt engineering, retrieval-augmented generation (RAG), and AI agent development applied to data engineering use cases, including privacy and enterprise reusable solutions

Must Have Skills Advanced experience designing, building, and optimizing complex data pipelines using Azure Data Factory, Databricks, ADLS, Oracle, Snowflake, and Python Hands-on experience with cloud-native data platforms and services, including Azure, Databricks, and Snowflake Strong experience with SQL, NoSQL, and relational database design and development Working knowledge of message queuing, stream processing using Kafka. Hands-on experience with AI productivity tools (Claude and Cursor or similar IDE) and working knowledge of foundation models, prompt engineering, retrieval-augmented generation (RAG), and AI agent development.

Nice to Have Cloud or data platform certifications such as Microsoft Azure Data Engineer (DP-203), Snowflake, or Databricks Experience developing reports using Power BI.