Data Engineer
TechDigital Corporation
Bellevue, WA, United States
1 day ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Job source
Tech stack
HTML
JavaScript (Programming Language)
Business Analytics Applications
Apache HTTP Server
User Authentication
Cyber Security
Data Validation
Data Governance
Data Integration
Data Loss
Data Transformation
Data Stores
+24 more
Logic Synthesis of Circuits
Groovy
Hypertext Transfer Protocols (HTTP)
Internet Security
Intrusion Detection and Prevention
Intrusion Detection Systems
JSON
Python (Programming Language)
Transport Layer
Log Analysis
Open Source Technology
Data Streaming
Extensible Markup Language (XML)
Azure Service Bus
Data Logging
Dynamic Routing
Scripting
Snowflake
Change Tracking
Data Analytics
Apache Kafka
Apache Nifi
Data Management
Splunk
Job description
- Lead the architecture, design, and implementation of scalable, modular, and reusable data flow pipelines using Cribl, Apache NiFi, Vector, and other open-source platforms, ensuring consistent ingestion strategies across a complex, multi-source telemetry environment.
- Develop platform-agnostic ingestion frameworks and template-driven architectures to enable reusable ingestion patterns, supporting a variety of input types (e.g., syslog, Kafka, HTTP, Event Hubs, Blob Storage) and output destinations (e.g., Snowflake, Splunk, ADX, Log Analytics, Anvilogic).
- Spearhead the creation and adoption of a schema normalization strategy, leveraging the Open Cybersecurity Schema Framework (OCSF), including field mapping, transformation templates, and schema validation logic-designed to be portable across ingestion platforms.
- Design and implement custom data transformations and enrichments using scripting languages such as Groovy, Python, or JavaScript, while enforcing robust governance and security controls (SSL/TLS, client authentication, input validation, logging).
- Ensure full end-to-end traceability and lineage of data across the ingestion, transformation, and storage lifecycle, including metadata tagging, correlation IDs, and change tracking for forensic and audit readiness.
- Collaborate with observability and platform teams to integrate pipeline-level health monitoring, transformation failure logging, and anomaly detection mechanisms.
- Oversee and validate data integration efforts, ensuring high-fidelity delivery into downstream analytics platforms and data stores, with minimal data loss, duplication, or transformation drift.
- Lead technical working sessions to evaluate and recommend best-fit technologies, tools, and practices for managing structured and unstructured security telemetry data at scale.
- Implement data transformation logic including filtering, enrichment, dynamic routing, and format conversions (e.g., JSON * CSV, XML, Logfmt) to prepare data for downstream analytics platforms. (100 plus sources of data)
- Contribute to and maintain a centralized documentation repository, including ingestion patterns, transformation libraries, naming standards, schema definitions, data governance procedures, and platform-specific integration details.
- Coordinate with security, analytics, and platform teams to understand use cases and ensure pipeline logic supports threat detection, compliance, and data analytics requirements.
Requirements
Open Source, Telemetry, Apache, Data Management, Splunk, HTTP (HyperText Transport Protocol), Hubs, Internet Security, Field Mapping, Logic Design
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦