> Markdown version of [/jobs/ext/1976705-data-engineer-redshift-databricks-intl-india](https://www.wearedevelopers.com/jobs/ext/1976705-data-engineer-redshift-databricks-intl-india). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer (Redshift/Databricks) - INTL India - **Company:** Insight Global - **Location:** Plano, TX, United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Amazon S3, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Extract Transform Load (ETL), Github, Apache Hive, Performance Tuning, SQL Stored Procedures, SQL Databases, Apache Spark, Caching, Data Lakes, Pyspark, AWS Glue, Terraform, Code Restructuring, Databricks - **Published:** August 7, 2026 - **Apply:** https://dejobs.org/x/x/1E08308C400949A5BF865D493491726A/job/ ## About the Role 8+ years of professional data engineering experience, with at least 2+ years in a tech-lead or lead engineer capacity. -5-6 years of experience in each of the following areas: Deep, hands-on experience with Apache Spark (PySpark and Spark SQL), including performance tuning (partitioning, shuffles, skew, caching, file sizing). -Hands-on production experience with Databricks Delta Lake, Unity Catalog, Workflows, and either Delta Live Tables or a comparable declarative pipeline framework. -Strong production experience with AWS Glue (jobs, crawlers, Data Catalog ## Description Insight Global is looking for a Data Engineer to help lead a migration from Glue/Redshift to Databricks What you will do: Own the end-to-end migration roadmap: discovery, assessment, design, build, test, cutover, and decommissioning of legacy AWS Glue jobs, Redshift clusters, and associated tooling. Inventory and prioritize existing pipelines, stored procedures, materialized views, and Redshift workloads; define a wave-based migration plan with clear success criteria for each wave. Architect the target Databricks Lakehouse: workspace and account topology, Unity Catalog design, medallion (bronze/silver/gold) layering, storage layout on S3, cluster and SQL Warehouse strategy, and cost guardrails. Re-platform pipelines from Glue (PySpark / Spark SQL) and Redshift SQL into Databricks using PySpark, Spark SQL, Delta Live Tables, and/or Databricks Workflows; refactor where it improves performance, maintainability, or cost. Define and enforce data parity, reconciliation, and regression-testing strategy between legacy Redshift and the new Lakehouse during dual-run periods. Plan and execute cutover for downstream consumers (BI tools, reverse ETL, ML feature stores, application reads), minimizing disruption. Set technical direction and standards Establish engineering standards for the new platform: repository structure, CI/CD (e.g., GitHub Actions / Azure DevOps with Databricks Asset Bundles or Terraform), environment promotion, code review, and testing (unit, data quality, integration). Define and roll out data quality, observability, and lineage practices (e.g., Great Expectations / DLT expectations, Unity Catalog lineage, monitoring and alerting). Govern access, security, and compliance via Unity Catalog: catalogs, schemas, row/column-level security, and audit. Drive cost optimization: cluster sizing and policies, photon usage, autoscaling, job vs. all-purpose compute decisions, and SQL Warehouse right-sizing. We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)