Senior Big Data Engineer

Data Engineer Owns the data pipelines that move data from operational source systems onto the data platform extraction, ingestion, transformation, orchestration, and the day-to-day operational health of those pipelines. Also plays a role as a hands-on builder of Foundational Data Products (raw record-of-truth) and potentially Derivative Data Products (composed from upstream products). Core skills SQL & Python (table stakes); Scala/Java for high-throughput streaming Pipeline build & operations: design, develop, deploy, monitor, and remediate batch and streaming pipelines that land source data on the platform; manage backfills, replays, late-arriving data, schema drift, and SLA breaches Ingestion patterns: full-load, incremental, change-data-capture (Debezium, Fivetran, Qlik Replicate, GoldenGate), event-driven ingest, API and file-based intake Pipeline frameworks: dbt, Apache Spark, Apache Beam, Airflow, Dagster, Prefect Streaming: Kafka, Kinesis, Flink, Spark Structured Streaming Data product packaging: schema contracts (Avro/Protobuf/JSON Schema), versioning, SLAS/SLOs, data contracts, output-port design (SQL, file, API, event) Storage formats: loeberg, Delta Lake, Hudi (open table formats are now the mesh default) Quality & observability: Great Expectations, Soda, Monte Carlo, dbt tests, data lineage (OpenLineage) CI/CD for data: GitOps pipelines, unit + integration tests, environment promotion Al-adjacent vector store ingestion (pgvector, Pinecone), feature stores (Feast, Tecton), RAG-ready chunking and embedding pipelines Mesh-specific: knowing when to build a derivative product vs. extending an existing one; consuming upstream products through governed input ports rather than reaching into source systems Pi-square technologies is a Michigan (USA) Headquartered Automotive Embedded Engineering Services company, Synergy Partner for major OEMs and Tier 1s and their implementation partners in Automotive Embedded Product Development, Projects, Requirements Analysis, Software Design, Software Implementation, Efficient Build, Release Process, and turnkey software V & V Services. We have more than 20+ years of industry expertise with specialization in the latest cutting-edge automotive technologies such as Infotainment, connected vehicles, Cyber security, OTA, and Advanced Safety/ Body electronics.

Job Category: PD Operations and Quality Degree Level: Bachelor's Degree or equivalent Job Description: We made history and now we work to transform the future - for our custo…

• 2 months ago