Data Engineer (Clickhouse)

Diverse Lynx LLC
Bellevue, WA, United States
4 days ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Compensation
$114,400
Working hours
Regular working hours

Tech stack

Application Programming Interfaces (APIs) Microsoft Azure BigQuery Cloud Computing Databases Data Files Domain Name System (DNS) Apache Hadoop Hadoop Distributed File System Python (Programming Language) Log Analysis Azure Data Lake
+12 more
Software Engineering SQL Databases Wireless Networks Parquet Scripting Snowflake Data Lakes Kubernetes Vertica Key Vault Amazon Redshift Databricks

Job description

Mandatory Primary Skills: Hadoop, Python, Databricks, Clickhouse DB , ADLS, SQL + Azure cloud technologies above all with Azure (such as ADF, Databricks, Log Analytics, Logic App, Key Vault, Blob Storage, ADLS, etc.)

Secondary Skills:; Snowflake ,

Detailed JD as below.

Data Engineer

We run a near real-time analytics platform for mobile network data, ingesting billions of rows a day from Azure Data Lake/HDFS into ClickHouse and serving subscriber-level queries back to internal APIs in tens of milliseconds. We are expanding the platform footprint to support additional use cases by adding more datasets

Data Engineer would work on: extending the Python ingestion framework that loads from Azure Data Lake/HDFS; table design covering partitioning, sort keys, TTL and materialized views; operational diagnosis of merge behaviour, part counts and replication lag; and integration with Snowflake and Databricks Iceberg alongside the primary feeds.

We need strong analytical SQL and experience with a columnar or MPP database - ClickHouse, BigQuery, Redshift or similar. Python sufficient to own a production ingestion framework rather than write scripts. Comfort at hundreds of billions of rows, since the failure modes differ from smaller systems. Cloud object storage and Parquet. Enough Kubernetes to tell whether a problem is the database or the platform underneath it.

Strongly preferred: direct ClickHouse experience, particularly MergeTree engines and aggregate function states, and telecom data such as DNS , IPFIX, etc. ClickHouse specifically is uncommon, and we would rather have a strong engineer from an adjacent system than a weak one with the exact keyword.

Requirements

Data Lake, Data Sets, Use Cases, HDFS (Hadoop Distributed File System), Microsoft Windows Azure, Application Programming Interface (API), Network Performance/Analysis, Replication and Remote Mirroring, Python Programming/Scripting Language, Software Engineering

About the company

Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…