Senior Data Engineer (Spark & ETL) (contract)

• Design, develop, and maintain ETL/ELT processes to ingest, transform, and load large-scale data efficiently.
• Build and optimize data pipelines using Apache Spark (PySpark/Scala Spark).
• Develop and maintain end-to-end data orchestration workflows for batch and near real-time processing.
• Configure, monitor, and troubleshoot job scheduling and dependency management using AutoSys or similar enterprise schedulers.
• Develop automation solutions using Shell scripting for operational and deployment activities.
• Write complex and optimized SQL queries, stored procedures, and performance tuning scripts.
• Analyse data quality issues and implement validation frameworks to ensure data integrity.
• Collaborate with business stakeholders, architects, and development teams to understand data requirements and deliver scalable solutions.
• Support production deployments, troubleshoot incidents, and perform root cause analysis.
• Participate in code reviews

In this contingent resource assignment, you may: Consult on complex initiatives with broad impact and large-scale planning for Software Engineering. Review and analyze complex multi-faceted, larger scale or longer-term Software Engineering challenges that require in-depth evaluation of multiple factors including intangibles or unprecedented factors. Contribute to the resolution of complex and multi-faceted situations requiring solid understanding of the function, policies, procedures, and compliance requirements that meet deliverables. Strategically collaborate and consult with client personnel. Required Qualifications: Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work or consulting experience, training, military experience, education., * Applicants must be authorized to work for ANY employer in the U.S. This position is not eligible for visa sponsorship.

• Experience in ETL/Data Engineering development.
• Strong hands-on experience with ETL tools and frameworks.
• Extensive experience in Apache Spark (PySpark/Scala) for large-scale data processing.
• Strong expertise in pipeline orchestration and workflow management.
• Hands-on experience with AutoSys scheduling and job dependency management.
• Strong scripting skills in Unix/Linux Shell scripting.
• Advanced proficiency in SQL, including:

o Complex query development o Performance tuning o Data modeling o Stored procedures and functions

• Strong understanding of data warehousing concepts and dimensional modeling.
• Experience working in Linux/Unix environments.
• Knowledge of source control tools such as Git.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of Hadoop ecosystem technologies.
• Experience with CI/CD pipelines and DevOps practices.
• Familiarity with Snowflake, Databricks, or modern data lake architectures.
• Exposure to Informatica or other ETL tools is a plus.

Pulled from the full job description

• Health insurance
• Life insurance