> Markdown version of [/jobs/ext/1077272-data-scala-spark-engineering-specialist](https://www.wearedevelopers.com/jobs/ext/1077272-data-scala-spark-engineering-specialist). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data/Scala/Spark Engineering Specialist

- **Company:** Anagha Techno Soft
- **Location:** New York, NY, United States
- **Experience:** Expert
- **Contract:** Temporary contract
- **Skills:** Application Programming Interfaces (APIs), Airflow, Unit Testing, CA Workload Automation Ae, Microsoft Azure, Continuous Integration, Information Engineering, Extract Transform Load (ETL), Github, Python (Programming Language), Networking Basics, Performance Tuning, Cloud Services, SQL Databases, Teradata SQL, Snowflake, Apache Spark, Star Schema, Serverless Computing, Databricks, Artifactory
- **Published:** June 4, 2026
- **Apply:** https://www.dice.com/job-detail/c19bfb34-4540-4b21-a62d-df0d4e9e5f0b

## About the Role

Scala / Spark production experience writing Spark applications in Scala (not just notebooks); comfortable with the Data Frame API, joins, window functions, partitioning, and performance tuning
Databricks Serverless compute, Unity Catalog, Asset Bundles, Databricks CLI
SQL fluency comfortable writing, analyzing and extracting requirements from complex SQL scripts
Snowflake schema design, performance, Spark-Snowflake connector
Azure ADLS, networking basics, secrets/identity (Entra ID / managed identities)
Orchestration Airflow (DAG authoring, sensors, retries, SLAs)
CI/CD Artifactory, GitHub Actions pipelines: build, sharded test matrices, artifact promotion through dev QA UAT prod
Testing Experience in TDD, writing unit tests (ScalaTest, AnyFlatSpec) and BDD (Concordion or equivalent)
Data quality & reconciliation building automated parity checks against legacy outputs, drift detection, row-level reconciliation tooling
Large-scale migrations proven track record migrating legacy ETL (Autosys/Informatica/etc.) to cloud data platforms, including dependency mapping and cutover planning
Modern data engineering practices medallion architecture (Bronze/Silver/Gold), idempotent pipelines, schema evolution, lineage, observability

Nice-to-have

Financial services / regulatory reporting domain
Python (Databricks utilities, tooling)
Spec-driven development workflows (specs plans tasks implementation)

## Description

We're migrating complex on-prem regulatory reporting pipelines from a legacy ETL + Autosys + SQL + Teradata stack to a modern Databricks + Snowflake platform on Azure. The role is hands-on: design, implement, test, and reconcile production pipelines feeding regulatory reports under strict parity requirements.


## Related Videos

- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [How Cisco embraced a DevOps culture within its network engineering team](https://www.wearedevelopers.com/videos/99-how-cisco-embraced-a-devops-culture-within-its-network-engineering-team)
- [Let's Get Aggregated: Custom UDAFs in Spark  ](https://www.wearedevelopers.com/videos/1649-let-s-get-aggregated-custom-udafs-in-spark)
- [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models)

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)