> Markdown version of [/jobs/ext/1315948-big-data-engineer](https://www.wearedevelopers.com/jobs/ext/1315948-big-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Big Data Engineer - **Company:** Resourcesys Inc. - **Location:** United States - **Experience:** Expert - **Contract:** Temporary to permanent - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Build Automation, Big Data, Cluster Analysis, Continuous Integration, Data Dictionary, Information Engineering, Data Security, Python (Programming Language), Laboratory Information Management Systems, Machine Learning, SQL Azure, Message Queuing Telemetry Transport (MQTT), Performance Tuning, Role-Based Access Control, Power BI, Standard Sql, OPC Unified Architecture, SAP (Applications), Simple Data Format, SQL Databases, Systems Integration, Management of Software Versions, Data Ingestion, Azure Data Factory, Mttr, Caching, Git, Data Layers, Pyspark, Integration Tests, SAP S/4HANA, Operational Systems, Spark Streaming, Key Vault, Databricks - **Published:** July 17, 2026 - **Apply:** https://www.dice.com/job-detail/4d1e99aa-d8e1-4c94-96f3-68486c0f6ee1 ## About the Role Minimum 5 years' in data engineering building production pipelines at scale (batch/CDC/streaming). Hands-on with Azure data stack: Databricks or Fabric/Synapse, ADF/Pipelines, ADLS/OneLake, Azure SQL/SQL MI, Key Vault. Strong SQL and Python/PySpark; comfort with Spark Structured Streaming and performance tuning. Experience implementing tests/observability (freshness, schema, expectations), and Git-based CI/CD. Familiarity with SAP S/4HANA structures and SAP DataSphere semantic modeling. OT concepts: historians (PHD/PI), OPC UA/MQTT, event/batch frames, ISA-95/99 basics. Understanding of Power BI consumption (semantic models, RLS) and APIs for downstream AI/ML apps/agents. Preferred Qualifications: Time-series/data-quality tooling (e.g., Great Expectations or equivalent patterns), feature/metric stores. MDM concepts (keys, survivorship), lineage/catalog tooling. TMS/WMS, LIMS, Historian, HSE domain exposure; Lean/Six Sigma mindset; FinOps awareness. ## Description Client is seeking a Big Data Engineer to build and operate our enterprise Unified Data Layer (UDL) - spanning IT and OT - to deliver trustworthy, performant data products that power Finance, Operations, Supply Chain & Logistics, HSE, Commercial, and corporate analytics. You'll engineer batch/CDC/streaming pipelines, model curated/semantic layers, and harden run-state with testing, CI/CD, security, and observability. You'll partner closely with the data team and larger IT organization. Mission Design and deliver scalable, secure data pipelines and data models that safely connect operational systems to analytics, ensure trusted and well governed data, and enable repeatable delivery of BI, ML, AI, and automation solutions. Data Engineering & Modeling Build ingestion pipelines (batch, CDC, streaming) from S/4HANA/DataSphere, PHD/historian, LIMS, TMS, HSE, and other sources into landing curated semantic layers. Implement data contracts, schema/versioning, SCD handling, partitioning, and performance tuning (file formats, clustering, caching). Develop dimensional/semantic models that back certified Power BI datasets and APIs for apps/agents. OT/IT Integration & Safety Integrate OT data via OPC UA/MQTT, broker/DMZ patterns, read-only historian feeds, and event/batch frames-no control-net reads. Collaborate with plant controls on change control, signal quality, and downtime windows. Quality, Security & Observability Embed data quality rules, unit/integration tests, and validation checks (freshness, completeness, drift/PSI). Instrument lineage and end-to-end monitoring; build alerting and on-call runbooks to minimize MTTR. Enforce RBAC, secrets management, PII/HSE classifications, and retention aligned to Governance/MDM policies. CI/CD, Cost & Reliability Automate build/test/deploy with Git-based CI/CD (environments, approvals, blue/green). Track and optimize cost/performance (cluster sizing, autoscaling, cache strategy); contribute to FinOps reviews. Collaboration & Documentation Partner with Reporting & BI on semantic model contracts, RLS, and performance SLAs; avoid direct system scraping. Produce "readme" docs, data dictionaries, runbooks, and post-incident reviews; support knowledge transfer with vendors. ## Related Videos - [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster) - [What Developers Get Wrong About Application Quality](https://www.wearedevelopers.com/videos/233-what-developers-get-wrong-about-application-quality) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Event based cache invalidation in GraphQL](https://www.wearedevelopers.com/videos/433-event-based-cache-invalidation-in-graphql) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)