Data Engineer

Sigma Software
Huelva, Spain
1 day ago
Apply on www.buscojobs.com.es
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience required
2 years minimum
Working hours
Regular working hours
Languages
English

Tech stack

Application Programming Interfaces (APIs) Artificial Intelligence Amazon Web Services Amazon S3 Data Analysis Automation of Tests Microsoft Azure Continuous Integration Information Engineering Data Governance Data Infrastructure Data Structures
+25 more
Distributed Computing Environment Distributed Data Store Identity and Access Management Python (Programming Language) OpenID Cloud Services SQL Databases Value Engineering Data Processing Feature Engineering Great Expectations (Foster Youth College-readiness and Support Program) Apache Spark Data Lakes Pyspark Git Flow Scikit Learn Data Lineage Data Management Machine Learning Operations Model Registry Api Gateway Terraform Data Pipelines Docker Databricks

Job description

Join a project shaping the future of sustainable energy through cutting?edge data engineering and scientific innovation. We are looking for a Data Engineer to help build scalable cloud-native data platforms for a deep?tech Customer working on one of the most ambitious fusion energy initiatives in the world.You will work with modern distributed data technologies, including Databricks, Apache Spark, AWS, and Azure, while solving complex engineering challenges in a research?intensive environment.We at Sigma Software value engineering excellence, long?term partnerships, and continuous learning. This is an opportunity to contribute to breakthrough scientific initiatives while working with modern technologies and highly skilled global teams.PROJECTThe project focuses on building and evolving a modern cloud?based data platform supporting scientific research, analytics, and operational workflows in the fusion energy domain. The team develops scalable distributed data processing solutions while ensuring high standards of data quality, lineage, governance, reproducibility, and operational reliability.You will work closely with scientists and engineers to support experimentation, analytics, and machine learning workflows in a highly innovative R&D environment.Job DescriptioDesign, develop, and maintain scalable and reliable data platforms and pipelines for analytics and operational use caseBuild distributed data processing workflows using Apache Spark, pySpark, and DatabricDevelop and evolve cloud-native data solutions leveraging Azure and AWS servicesModel and optimize data structures for analytics, reporting, and downstream consumptionIntegrate data from multiple heterogeneous sources while ensuring data quality, consistency, and traceabilityImplement data quality controls, schema validation, lineage tracking, and monitoring processesOptimize performance, scalability, reliability, and cost?efficiency of data pipelines and infrastructureCollaborate closely with scientists, analysts, and engineering teams to translate business and research requirements into technical solutionsContribute to CI/CD processes, infrastructure automation, and engineering best practicesResearch and introduce new technologies and approaches within the data engineering ecosystemecosystemQual3+ years of commercial experience in Data Engineering2+ years of hands?on production experience with DatabricksStrong expertise with Delta Lake, including medallion architecture and table optimizationAdvanced proficiency in Python, pySpark, and SQLDeep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs / Workflows, and job optimizationHands?on experience with AWS services including S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API GatewayExperience designing event?driven and asynchronous data pipelinesStrong knowledge of Infrastructure as Code and CI/CD practicesPractical experience with Terraform and Databricks Asset BundlesExperience working with Git workflows, pull request reviews, and automated testingUnderstanding of secure authentication approaches such as OIDC and service principalsPractical experience with MLflow, experiment tracking, model registry, and reproducibility practicesStrong understanding of data quality, lineage, schema contracts, and provenance conceptsExperience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQXExperience monitoring and optimizing data platform reliability and cloud costsUpper-Intermediate or higher English levelWILL BE A PLUS:Experience packaging workloads using Docker and running batch workloads in AWSFamiliarity with ECR and containerized data processing solutionsExperience with experiment optimization libraries such as scikit?optimize, BoTorch, Ax, or scipyUnderstanding of ML feature engineering and leakage prevention techniquesExperience working in regulated or high?security environmentsKnowledge of data governance, compliance, and data residency requirementsExperience disabling or restricting AI?assisted tooling/ features in secure environments#J-*****-Ljbffr

Requirements

3+ years of commercial experience in Data Engineering 2+ years of hands?on production experience with Databricks Strong expertise with Delta Lake, including medallion architecture and table optimization Advanced proficiency in Python, pySpark, and SQL Deep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs / Workflows, and job optimization Hands?on experience with AWS services including S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway Experience designing event?driven and asynchronous data pipelines Strong knowledge of Infrastructure as Code and CI/CD practices Practical experience with Terraform and Databricks Asset Bundles Experience working with Git workflows, pull request reviews, and automated testing Understanding of secure authentication approaches such as OIDC and service principals Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices Strong understanding of data quality, lineage, schema contracts, and provenance concepts Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQXExperience monitoring and optimizing data platform reliability and cloud costs Upper-Intermediate or higher English level WILL BE A PLUS: Experience packaging workloads using Docker and running batch workloads in AWS Familiarity with ECR and containerized data processing solutions Experience with experiment optimization libraries such as scikit?optimize, BoTorch, Ax, or scipy Understanding of ML feature engineering and leakage prevention techniques Experience working in regulated or high?security environments Knowledge of data governance, compliance, and data residency requirements Experience disabling or restricting AI?assisted tooling/ features in secure environments #J-*****-Ljbffr

About the company

Join a project shaping the future of sustainable energy through cutting?edge data engineering and scientific innovation. We are looking for a Data Engineer to help build scalable cloud-native data platforms for a deep?tech Customer working on one of the most ambitious fusion energy initiatives in the world.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.buscojobs.com.es
Prepare application

Good distractions

Loading talks and stories from around this role…