Cloud Data & AI Platform Engineer

The Cloud Data & AI Platform Engineer is a hands-on technical role responsible for designing, building, and operating advanced data and AI orchestration capabilities within an Azure Databricks-based lakehouse environment. This role focuses on delivering reliable, governed, and auditable automation of data and analytics workflows using Azure Databricks and related Azure services.

The position supports enterprise data and analytics platforms by developing orchestration frameworks, AI-enabled processing pipelines, and secure integration services that extend beyond traditional ETL workflows, while meeting security, regulatory, and cost-management expectations common in highly regulated environments. This role partners closely with data engineering, analytics, and application teams to ensure AI-enabled solutions are production-ready, maintainable, and aligned with enterprise architecture standards., Data & AI Orchestration Engineering

• Design and implement reusable Python-based orchestration frameworks to manage multi-step analytics workflows, data quality checks, and AI-assisted processes.
• Develop controlled, task-specialized components to support data validation, metadata enrichment, code generation assistance, and operational diagnostics.
• Ensure orchestration logic is deterministic, testable, and suitable for regulated production environments.

Azure Databricks Platform Integration

• Deploy and operate orchestration and AI-enabled workloads within Azure Databricks, leveraging:

• Delta Lake and Medallion Architecture (Bronze/Silver/Gold)
• Databricks Workflows and Jobs
• Unity Catalog for data governance and access control

• Align solutions with enterprise platform and architectural standards.

System & API Integration

• Design and implement secure integration patterns with internal applications and approved external vendor systems.
• Ensure integrations comply with enterprise security, identity management, auditability, and least-privilege access standards.

Performance, Cost, and Reliability Management

• Monitor and optimize Spark workloads, orchestration processes, and AI service calls for efficient resource utilization.
• Apply cost-awareness and cloud financial management best practices.
• Build scalable, resilient solutions capable of handling variable workload demands without manual intervention.

Technical Leadership & Standards

• Contribute to architecture guidance, design reviews, and technical standards for data and AI platforms.
• Ensure solutions are modular, maintainable, and aligned with long-term platform strategy.
• Provide clear documentation and handoff materials to support ongoing operations.

• Advanced proficiency in Python, including object-oriented design and asynchronous or event-driven patterns.
• Strong experience with PySpark, Delta Lake, and enterprise data lake architectures.
• Practical experience with Azure services such as Azure Databricks, Azure Functions and/or Logic Apps, and container-based services.
• Experience implementing AI-assisted or LLM-enabled workflows using structured orchestration patterns.

Platform & DevOps

• Experience with CI/CD pipelines using GitHub Actions or Azure DevOps.
• Familiarity with infrastructure-as-code or environment configuration management in Azure.
• Strong understanding of secure software development practices.

Professional Experience

• 5+ years of experience in software engineering, cloud platform engineering, or data engineering roles.
• Demonstrated success delivering production-grade data or analytics solutions in an enterprise environment.
• Experience in energy, utilities, nuclear, or other highly regulated industries preferred.
• Experience handling sensitive operational or telemetry data is a plus.

Performance & Operating Expectations

• Solutions must be accurate, auditable, and operationally reliable.
• Designs should scale efficiently while remaining resilient and stable.
• Code and documentation must adhere to enterprise development and quality standards, with comprehensive README files and architectural documentation.

Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRated's Best of Staffing in Talent Satisfaction in the United States and Great Place to Work in the United Kingdom and Mexico. Everforth Apex uses a virtual recruiter as part of the application process. Click for more details.