Data Engineer

Our client is building a next generation SaaS application that ingests data from various sources, processes through a data pipeline and executes machine learning models to provide predictions on Project Management data. The solution will leverage cutting edge technologies hosted in Microsoft Azure including Databricks, Apache Airflow, Fabric database mirroring, Fabric Lakehouse, Semantic Models, Materialized Lake Views and Power BI. The app users will have the ability to access Machine Learning predictions, create custom reports and combine data from various sources.

Do you have experience in Version control?, * Python (expert)

• Spark/pySpark (advanced)
• T-SQL (advanced)
• ETL/ELT and creation of data pipelines (expert)
• Databricks (advanced)
• Data modeling (advanced)
• Experience with scheduling and orchestration tools, specifically Apache Airflow (advanced)
• Azure services including compute, storage, databases and developer tools (advanced)
• Data warehousing concepts (expert)
• Version control tools such as Azure DevOps Git (advanced)
• Performance monitoring and optimization of code and tooling (advanced)
• Security including row level and object level
• Invoking RESTful API’s
• Strong aptitude and ability to work independently
• Medallion architecture

Desired Skills

• Experience with Microsoft Fabric and implementing data pipelines using Fabric tooling
• Knowledge of AI concepts and experience executing ML models
• Cloud cost optimization
• Knowledge of OneLake security
• Sharing of Power BI and Semantic Models in Microsoft Fabric
• Understanding of cross tenant data sharing in Microsoft Fabric
• Team leadership experience
• Observability tooling, * Knowledge of Project Management and Financial concepts including budgets, tasks, revenue, profit and earned value
• Certification in Microsoft Azure and/or Python, SQL, Databricks