> Markdown version of [/jobs/ext/4806211-senior-director-software-engineering](https://www.wearedevelopers.com/jobs/ext/4806211-senior-director-software-engineering). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Director, Software Engineering - **Company:** Circana, LLC. - **Location:** New York, NY, United States - **Experience:** Expert - **Salary:** $145,000 - $190,000 - **Contract:** Permanent contract - **Skills:** Clean Code Principles, Artificial Intelligence, Airflow, Amazon Web Services, Automation of Tests, Microsoft Azure, Big Data, Cloud Computing, Cloud Database, Code Generation, Software Quality, Computer Programming, Continuous Integration, Directed Acyclic Graph (Directed Graphs), Information Engineering, Data Warehousing, DevOps, Information Lifecycle Management, Python (Programming Language), Machine Learning, Performance Tuning, Scrum Methodology, Software Engineering, SQL Databases, Workflow Management Systems, Data Processing, System Availability, Prompt Engineering, Apache Spark, Git, Data Lakes, Pyspark, Kubernetes, Terraform, Software Version Control, Data Pipelines, Docker - **Published:** November 20, 2026 - **Apply:** https://www.dice.com/job-detail/5db23926-fcc8-45af-a60c-d182f47d0d25 ## About the Role * 8+ years of experience in data engineering with expertise in Python, PySpark, Spark, and Airflow. * Strong programming skills in Python, SQL with the ability to write efficient and maintainable code * Deep understanding of Spark internals (RDDs, DataFrames, DAG execution, partitioning, etc.) * Experience with Airflow DAGs, scheduling, and dependency management * Knowledge of Git, Docker, Kubernetes, Terraform, and apply best practices of DevOps for CI/CD workflows * Excellent problem-solving skills and ability to optimize large-scale data processing. * Experience in leading teams and working in Agile/Scrum environments * Experience with common data warehouse and/or data lake platform * Knowledge of cloud technologies (Azure/AWS/Google) * Proven experience using generative AI tools to accelerate software development, data engineering, system design, testing, and troubleshooting. * Strong understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows. * Ability to identify opportunities to apply AI solutions that improve team productivity, code quality, and operational efficiency. * Experience of ML Engineering is a plus * Strong communication and collaboration skills, with the ability to effectively work with remote teams ## Description We are seeking a skilled and motivated Data Engineer to join our growing team. In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure cloud platform. You will leverage your expertise in PySpark, Apache Spark, and Apache Airflow to process and orchestrate large-scale data workloads, ensuring data quality, efficiency, and scalability. If you have a passion for data engineering and a desire to make a significant impact, we encourage you to apply!, Data Engineering & Data Pipeline Development * Design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow * Implement real-time and batch data processing using Spark * Enforce best practices for data quality, governance, and security throughout the data lifecycle * Ensure data availability, reliability and performance through monitoring and automation. Cloud Data Engineering: * Manage cloud infrastructure and cost optimization for data processing workloads * Implement CI/CD pipelines for data workflows to ensure smooth and reliable deployments. Big Data & Analytics: * Build and optimize large-scale data processing pipelines using Apache Spark and PySpark * Implement data partitioning, caching, and performance tuning for Spark-based workloads. * Work with diverse data formats (structured and unstructured) to support advanced analytics and machine learning initiatives. Workflow Orchestration (Airflow) * Design and maintain DAGs (Directed Acyclic Graphs) in Airflow to automate complex data workflows * Monitor, troubleshoot, and optimize job execution and dependencies Team Leadership & Collaboration * Lead a team of data engineers, providing technical guidance and mentorship * Foster a collaborative environment and promote best practices for coding standards, version control, and documentation. * Engage directly with clients to understand business needs, communicate technical concepts clearly. * Collaborate with peers across engineering, product, architects to drive scalable and maintainable product * Partner with the Program Management team to align delivery timelines, mitigate risks