Data Engineer
Role details
Job location
Tech stack
Job description
Our client is the best technological consultant company, specialist in Microsoft, as well as Dynamics 365 FO, Dynamics 365, AX, Power BI, CRM, EPR, Office 365 and Azure. With offices in Madrid, Barcelona, Valencia and Castellon with specialized professional experts, the company has aided in the development of small businesses through automatization and implantation of Microsoft solutions. THE ROLE As a Data Engineer/Big Data Developer you will develop and enhance data systems and pipelines, support the process of optimization in extraction, transformation and loading of large data sets, and design solutions with strategic focus on code quality, reliability, reusability, maintainability and dataops, all of which will have an immediate impact on the day-to-day functioning of the company.
- Data reporting and visualization to support business operations and decisions.
- Design data structures and write code to combine and transform datasets to meet business needs.
- Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
- Perform statistical analyses on a wide range of datasets.
- Convert SAS based pipelines into languages like PySpark, Scala to execute on Hadoop and non-Hadoop ecosystems
- Build new datasets proactively; seek out, develop, and share new algorithms and best practices.
- Build and maintain the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python and SQL (Snowflake & SQL Server) in both on premise and cloud environments
- Manage and improve the user facing server infrastructure (web + api) including authorization and load balancing
- Work with Cloud Computing Platforms (GCP/AWS), Luigi, Kafka and other open-source technologies
- Implement both batch and streaming workflows that combine, transform and enrich data utilizing Microsoft Azure
Requirements
- Bachelor/Master´s in Computer Sc./App or Information Technology
- Experience of more 3+ years in a similar role
- Microsoft certifications are desirable
- Desirable experience on version control (Git), Data Modelling, Data Pipelines, Solution Development, Product Industrialization, CI/CD (Jenkins or Git hub)
- Experience using analytics programming languages (Python, Spark, Scala, Java…) to manipulate data and draw insights from large data sets.
- Knowledge in machine learning techniques.
- Experience in cloud infrastructure such as MS Azure
- Experience in agile software development.
- Experience with Spark, SQL, NoSQL, Databricks
- Knowledge in distributed systems and frameworks such as Hadoop, HBase, NoSQL. Spark SQL, Spark Streaming, HDFS, Hive.
- Minimum 2-year experience in: Spark with Scala, SQL, Azure, Power BI, Agile Working
- Ability to work individually without much direction while also working as part of a team towards a common goal.
- Good written and oral communication skills.
- Results-oriented, self-directed ability to drive change in an unstructured environment.
- International experience and multicultural skills are highly value
- 100% remote work
- English level: Fluent reading/writing/conversation
- Travel availability: Yes, 10% maximum of total working time
