Data Platform Engineer

As a Data Platform Engineer in a highly regulated environment, you will be responsible for designing, building, and maintaining secure and scalable data infrastructure that supports both cloud and on-premises platforms. You will play a key role in ensuring that all data systems comply with industry regulations and security standards while enabling efficient access for analytics and operational teams., * Design, develop, and maintain robust and secure data pipelines using NiFi and related big data technologies.

• Troubleshoot and optimize NiFi flows, including performance tuning, error resolution, and flow control.
• Integrate NiFi with cloud platforms, databases (SQL & NoSQL), APIs, and third-party systems.
• Ensure compliance with regulatory and security requirements across data storage, transfer, and access layers.
• Support data migration efforts and implement disaster recovery protocols.
• Continuously monitor data infrastructure performance and recommend improvements.
• Collaborate with cross-functional teams to align data platform capabilities with business needs and compliance requirements.
• Maintain documentation of data flows and processes, ensuring knowledge sharing and operational transparency.

A strong command of Apache NiFi is essential for this role. You will be expected to design, implement, and maintain data flows using NiFi, ensuring accurate, efficient, and secure data ingestion, transformation, and delivery. You should be adept at identifying and resolving issues within NiFi flows, managing performance bottlenecks, and implementing robust error handling strategies.

You'll work closely with cross-functional teams including data architects, compliance officers, and cybersecurity specialists to integrate data from various systems such as databases, APIs, and cloud platforms. Your work will directly support batch processing, real-time streaming, and event-driven data pipelines across a variety of use cases.

We're looking for candidates with over 3 years of relevant experience in data engineering, platform engineering, or a related field, with demonstrated hands-on expertise in NiFi and data pipeline design in regulated environments., You will have the following skills or proven experience:

Apache NiFi Expertise:

• Deep understanding of core NiFi concepts: FlowFiles, Processors, Controller Services, Schedulers, Web UI.
• Experience designing and optimizing data flows for batch, real-time streaming, and event-driven architectures.
• Ability to identify and resolve flow issues, optimize performance, and implement error-handling strategies.
• Optional scripting skills for creating custom NiFi processors.

Programming & Data Technologies:

• Proficiency in Java and SQL.
• Experience with C# and Scala is a plus.
• Experience with ETL tools and big data platforms.
• Knowledge of data modelling, replication, and query optimization.
• Hands-on experience with SQL and NoSQL databases is desirable.
• Familiarity with data warehousing solutions (e.g., Snowflake, Redshift, BigQuery) would be beneficial.

Data Platform Management:

• Comfortable operating in hybrid environments (cloud and on-prem).
• Experience integrating diverse data sources and systems.
• Understanding of secure data transfer protocols and disaster recovery planning.

CI/CD & DevOps Practices:

• Experience working with CI/CD pipelines to automate deployment of NiFi flows and data infrastructure components.
• Familiarity with tools such as Git, Jenkins, Azure DevOps, or similar.

Ways of Working:

• Comfortable working in Agile/Scrum environments, participating in sprint planning, stand-ups, and retrospectives.
• Experience using Agile collaboration tools (e.g., Jira, Azure DevOps, Confluence).

Communication:

• Ability to clearly explain technical concepts to both technical and non-technical stakeholders.
• Strong written and verbal communication skills.
• Experience creating and maintaining technical documentation.

Collaboration & Relationship Management:

• Proven track record of cross-functional collaboration with product, engineering, security, compliance, and business teams.
• Strong interpersonal skills; able to build trust and manage stakeholder expectations.
• Able to adapt to diverse working styles and team dynamics.

Analytical Thinking & Problem-Solving:

• Strong analytical skills for troubleshooting complex data flow and infrastructure issues.
• Ability to juggle multiple priorities and adapt to dynamic workloads.
• Detail-oriented with a commitment to high-quality delivery.

At Peregrine, we're always seeking Specialist Talent that have the ideal mix of skills, experience, and attitude, to place with our vast array of clients. From Business Analysts in large government organisations to Software Developers in the private sector - we are always in search of the best talent to place, now. How Specialist Talent Works: At Peregrine, we find the best talent for our clients. As a permanent employee of Peregrine, with access to all our standard benefits, you will be deployed across our portfolio of clients as a specialist consultant, working on a wide array of complex projects across multiple industries., At Peregrine, we see beyond the immediate and look to the horizon. We build lasting, meaningful partnerships with our clients, and deliver flexible solutions for every resourcing need, both now and in the future. Together, we help our clients to engage, develop and harness the skills they need to achieve and grow the workforce they want. We have a range of benefits you will receive alongside your salary. Our culture: At Peregrine we embrace fresh ideas, and we love learning fast. Our solutions are trusted and established, so we have the confidence of knowing we have a solid foundation. We rely on openness and honesty, and we're always ready to help each other out. And we believe that our work can benefit society - whether it's finding the digital talent of the future or being a driver for social mobility. Our commitment to diversity: At Peregrine, we're proudly committed to championing diversity and inclusion, with company-wide initiatives to drive greater social mobility and reduce our environmental impact. Our teams represent a huge breadth of cultures, languages, and ethnicities, and over 20 different nationalities. We also employ candidates from a range of educational and socioeconomic backgrounds. Our partnerships with numerous charities ensure that we can stay well-informed and continue to improve our practices for the future. It reflects in the way we recruit for our clients as we assist them in becoming more diverse.