3329-A Senior Data Engineer

The NATO Communications and Information Agency (NCIA) has been established with a view to meeting the collective requirements of some or all NATO nations in the fields of capability delivery and service provision related to Consultation, Command & Control as well as Communications, Information and Cyber Defense functions, thereby also facilitating the integration of Intelligence, Surveillance, Reconnaissance, Target Acquisition functions and their associated information exchange.

NCIA is dedicated to acquiring, deploying, and defending communication systems for NATO's political decision-makers and Commands. It operates on the frontlines against cyber-attacks, collaborating closely with governments and industry to prevent future debilitating attacks. NCIA plays a crucial role in maintaining NATO's technological edge and ensuring the collective defence and crisis management capabilities of the Alliance. In pursuit of our mission, we require specialized advisory services to enhance our interim workforce capacity.

Introduction In response to the North Atlantic Council (NAC) decision to establish NATO Security Assistance and Training for Ukraine (NSATU), a Deployable Communications and Information System (DCIS) asset was delivered to Wiesbaden DEU to enable personnel to plan and conduct security assistance and training activities in support of Ukraine.

Within the NCIA Chief Technology Office, the Exploiting Data Science and Artificial Intelligence (EDS&AI) Team has as primary goal to ensure that Big Data and AI can be used to make better and faster decisions. In leveraging the volume of big data, EDS&AI team aims to turn data overload into decision dominance.

Inside NATO, leaders in different functions need support for their decision making. EDS&AI brings the expertise to exploit the vast and varied data available for extracting insights, and we achieve that by using both cloud computing and the Agency's high performance computing sandbox.

The EDS&AI team provide the core data science skills and technology needed for big data analysis and AI.

The EDS&AI team applies innovative technology to data whenever it is not possible to extract value with conventional approaches.

Problem Statement

• NSATU lacks the proper tools to collect, collate, analyze, and portray NSATU and stakeholder data that are necessary to enable data-informed decision-making in a timely manner.
• Although KOROVAI is one of the main pillars for data extraction, NSATU is struggling to collect data from the multiple data sources (excel sheets, SkyBlue, EDMS, Common Drives, etc.) to analyze information (movement logistics, training, LENs operations, FLOe's activities, ammo stockpiling, equipment ETRs, nation pledges, etc.) to establish the minimum required level of visualization and dashboards to enable informed decisions.
• Although GBR is assisting NSATU with in-kind national solutions (Palantir), the provided solution is still limited in functionalities (e.g. limited to charts presentation mode only) due to network domain difference and accessibility.
• NSATU lacks the analytics tools to transition from its current "crisis mode" reporting that is solely focused on feeding the Cooperation and Interoperability Groups (CIGs) with fragmented data into a "campaign mode" that will focus on establishing mid-term forecasting and projections. The transition will allow NSATU to not limit its analytics to descriptive products that can be interpreted differently, but also to exploit data to service further analytical requirements.

SCOPE OF WORK The Senior Data Engineer that will provide comprehensive project support services for the NSATU Data Fusion Cell team located at Wiesbaden, Germany under the direction / guidance of the NCIA The Hague Exploiting Data Science & AI team and NSATU Data Analytics Project Management Team. The resources will support the following activities:

Data engineering

• Designs and implements data pipelines and data stores to acquire and prepare data.
• Applies data engineering standards and tools to create and maintain data pipelines and extract, transform and load data.
• Carries out routine data quality checks and remediation Programming/software development
• Designs, codes, verifies, tests, documents, amends and refactors moderately complex programs/scripts.
• Applies agreed standards and tools to achieve a well-engineered result.
• Proposes practical solutions to resolve issues.
• Collaborates in reviews of work with others as appropriate.

Systems integration and build

• Defines the software modules needed for an integration build and produces a build definition for each generation of the software.

Data Management

• Assesses the integrity of data from multiple sources.
• Provides advice on the transformation of data from one format/medium to another.
• Enables the availability, integrity and search ability of information through the application of formal data and metadata structures and protection measures.

Data modelling and design

• Applies standard data modelling and design techniques based upon a detailed understanding of requirements.
• Establishes, modifies and maintains data structures and associated components.
• Communicates the details of data structures and associated components to others using the data structures and associated components.

Do you have experience in Unit testing?, The Contractor shall fulfill the following Data Engineer Requirements:

• Security Clearance: The Contractor shall possess valid NATO-Secret Personal Security Clearances for the duration of the Contract performance.
• Language Proficiency: English: C1

Qualifications:

• Ability to design, build and manage data pipelines for data structures encompassing data transformation, data models, schemas, metadata and workload management in an on premise environment.
• At least four years' experience planning and maintaining data lakes, databases and pipeline processes, both batch-wise and (near) real-time.
• Experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using data integration technologies on premise.
• Experience in using of data engineering tools to support data science, data analytics and data visualization (KNIME, SQL Server, Power BI is the preferred toolset).
• Experience with state-of-the-art data engineering, analytics, data integration and pipeline tools/programming language (Python).
• Solid understanding of data security best practices, including data encryption, access controls, authentication/authorization mechanisms, and compliance requirements.
• Experience with monitoring and logging tools to ensure the reliability and performance of data pipelines.
• Familiarity with containerization and orchestration tools such as Docker and Kubernetes is a plus.
• Knowledge of best practice of software development. E.g. CI/CD pipelines, unit/functional testing, following styling guides is a plus.
• Familiarity with KNIME Analytics Platform and Server is preferred.

Desired Qualifications:

• Experience working within NCIA or similar defense-related environments.
• Knowledge of NATO Communication and Information Systems Infrastructure.
• Prior experience working in an international environment comprising both military and civilian elements.