Data Engineer - OESIS Framework

OPSWAT INC

United States

14 days ago

Apply on www.thejobnetwork.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Experience required

3 years minimum

Working hours

Regular working hours

Job source

www.thejobnetwork.com

Tech stack

Application Programming Interfaces (APIs) Airflow Data Deduplication Information Engineering Extract Transform Load (ETL) Data Structures Python (Programming Language) PostgreSQL MySQL NoSQL PCI Data Security Standards Zero Trust Network Access

+11 more

Standard Sql Data Ingestion Microsoft InTune Falcon Platform Patch Management Apache Kafka Casper Suite Restful APIs Pagination Data Pipelines Web Api

Job description

Build scalable ingestion pipelines that query large volumes of device/compliance data from Endpoint Device Management Platforms (such as CrowdStrike, Intune, JAMF) both full bulk syncs and incremental/delta syncs

Design a normalized compliance data schema that maps each vendor's distinct data model (e.g., CrowdStrike host/device telemetry, Intune compliance policy state, JAMF inventory records) into one common internal representation

Build and tune the storage layer (relational, NoSQL, or warehouse) for high-volume, multi-tenant compliance data with efficient read/query patterns for downstream consumers

Implement caching, deduplication, and delta-sync logic that minimizes redundant API calls and respects each vendor's rate limits and query quotas

Build ETL/ELT orchestration for scheduled and event-driven syncs, with monitoring and alerting on pipeline health and sync failures

Partner with the Integration Engineer on the shape of the raw API responses being handed off, and with the Auth Engineer to handle token expiry gracefully mid-query

</ul>, business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws.</p>

Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.

</div>

Requirements

3+ years in data engineering: ETL/ELT pipeline design, data modeling, and large-scale ingestion

Strong SQL plus hands-on experience with both relational (Postgres/MySQL) and NoSQL or columnar stores

Experience with pipeline orchestration tools (Airflow, Dagster, Prefect) and/or streaming platforms (Kafka, Kinesis)

Direct experience handling REST API pagination, bulk/batch queries, and rate-limited data collection at scale

Demonstrated experience normalizing heterogeneous schemas from multiple external data sources into one model

Strong Python (or equivalent) for pipeline development

Understanding of multi-tenant data isolation and relevant compliance requirements (SOC 2, GDPR)

, <li>Prior experience specifically with security or compliance data (device posture and regulatory compliance (PCI-DSS, SOX, HIPPA, etc.), vulnerability, patch management)</li>

Experience with time-series or event-driven data models for compliance state changes

Direct familiarity with CrowdStrike, Intune, or JAMF data structures and cloud based device management frameworks

Familiarity with Zero Trust Network Access (ZTNA) and device posture or compliance

About the company

external

OPSWAT, a global leader in IT, OT, and ICS critical infrastructure cybersecurity, delivers an end-to-end platform that gives public and private sector organizations and enterprises the critical advantage needed to protect their complex networks, secure their devices, and ensure compliance. Over the last 20 years our commitment to innovative technology has earned the trust of more than 1,700 organizations, governments, and institutions globally, solidifying our role in protecting the world's critical infrastructure and securing our way of life.

The Position

At OPSWAT, we're building cutting-edge solutions that protect the world's critical infrastructure. You will play a pivotal role in shaping the future of our OESIS Framework platform with cloud solution integrations. This role owns everything that happens to device compliance data after it's retrieved from a vendor API: bulk and incremental querying at scale, normalizing wildly different vendor data models into one common schema, and storing it in a way that's fast to query and safe for customers.