Data Engineer - CDP

Design and develop scalable ETL and data ingestion pipelines using Azure Data Factory (ADF). Build and optimize big data processing workflows using Azure Databricks and Apache Spark. Implement and manage data storage solutions using Azure Data Lake Storage (ADLS). Develop and optimize data models, complex SQL queries, stored procedures, and large-scale data transformations. Design and implement Snowflake solutions leveraging Streams, Tasks, Snowpipe, Dynamic Tables, Iceberg Tables, Storage Integrations, and Views. Integrate data from multiple sources, including Oracle and cloud-based platforms. Ensure data quality, governance, privacy compliance, and observability across data pipelines. Optimize pipeline performance, scalability, reliability, and cost efficiency. Develop CI/CD pipelines and deployment automation using Azure DevOps or GitLab. Implement stream processing and messaging solutions using Kafka. Collaborate with cross-functional teams to deliver data-driven solutions and mentor junior engineers. Utilize AI productivity tools such as Claude, Cursor, or similar IDEs for code generation, debugging, testing, and documentation. Design and implement AI-driven solutions using foundation models, prompt engineering, Retrieval-Augmented Generation (RAG), and AI agents to automate data engineering and privacy-related workflows.

Bachelor’s degree in Computer Science, Computer Engineering, Information Systems, or a related field. 5 7+ years of hands-on experience in data engineering and ETL development. Strong expertise in Azure Data Factory, Azure Databricks, ADLS, Snowflake, Oracle, and Python. Advanced SQL skills with experience writing highly scalable and high-performance queries and stored procedures. Hands-on experience with Azure cloud services and cloud-native data platforms. Experience with SQL, NoSQL, and relational database design. Strong understanding of Kafka and stream processing architectures. Experience implementing CI/CD pipelines using Azure DevOps or GitLab. Excellent analytical, troubleshooting, and problem-solving skills. Ability to manage multiple priorities in a fast-paced environment. Strong communication and collaboration skills. Preferred Qualifications Experience with dbt and Azure Flink. Experience developing reports and dashboards using Power BI. Knowledge of data governance, privacy, and compliance frameworks. Hands-on experience with AI productivity tools and AI-driven development practices. Microsoft Azure Data Engineer (DP-203), Snowflake, or Databricks certifications.