> Markdown version of [/jobs/ext/1353151-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/1353151-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** CONFIDENTIAL COPY SERVICES - **Location:** Newark, DE, United States - **Experience:** Expert - **Salary:** $140,000.0 - $150,000.0 - **Contract:** Permanent contract - **Skills:** Agile Methodology, Amazon Web Services, Data Analysis, Big Data, Business Software, Software Quality, Information Engineering, Extract Transform Load (ETL), Data Transformation, Data Warehousing, Memory Management, Python (Programming Language), Scrum Methodology, Power BI, DataOps, SQL Databases, Data Streaming, Systems Integration, Strategies of Testing, Data Processing, Sql Optimization, Snowflake, Apache Spark, Pyspark, Semi-structured Data, Information Technology, Optimization Algorithms, Data Analytics, Enterprise Integration, AWS Data Analytics, Apache Kafka, Data Management, Tools for Reporting, Stream Processing, Data Pipelines, Confluent, Databricks - **Published:** July 20, 2026 - **Apply:** https://www.careerjet.com/jobad/us81ecb8286a836e966cc2aa58b3874db1 ## About the Role * Proficient in Databricks Spark: Exceptional skills in Databricks Spark for sophisticated data processing. Proven experience in leveraging Spark for complex ETL tasks, surpassing traditional data processing methods. * ETL Pipeline Mastery: Demonstrated excellence in designing and implementing ETL pipelines specifically within Databricks. Ability to utilize Spark's full capabilities to create efficient, scalable data pipelines. * Data Transformation and Analysis: Expert in data transformation using Databricks Spark, skilled in performing advanced data analytics and processing large datasets with high efficiency. * Optimization Techniques: Deep understanding of optimizing Databricks Spark applications for maximum performance, including fine-tuning Spark configurations, and memory management. Integration with Confluent Kafka: * Kafka Exposure: Solid background in working with Confluent Kafka, particularly in integrating it with Spark-based systems for real-time data streaming and processing. * Efficient Data Pipelines: Proficiency in creating and managing data pipelines that seamlessly integrate Kafka with Databricks Spark, ensuring efficient data flow and processing. * Supporting integrations with business applications, using Kafka DataOps and Agile Methodologies: * DataOps Principles: Strong grasp of DataOps methodologies, with a focus on improving the efficiency and quality of data analytics via automation, collaboration, and process optimization. * Agile Development: Experienced in Agile software development practices, adept at implementing Agile methodologies like Scrum or Kanban in data-centric projects for improved collaboration and rapid delivery. Proficiency in SQL and Platform Integrations: * Advanced SQL Skills: Expertise in SQL, particularly for querying and managing tables/data warehouses within Snowflake. Ability to seamlessly integrate these with Databricks Spark. * Platform Adaptability: Skilled in adapting to and integrating various data platforms and technologies, aligning them with strategic organizational goals. Collaborative and Best Practice-Oriented: * Adherence to High Standards: Committed to maintaining high standards in code quality, documentation, and adhering to DataOps and Agile best practices. * Team Collaboration and Leadership: Ability to work collaboratively in a team, fostering a culture of continuous learning and improvement., * Bachelor's or master's degree in computer science, statistics, or analytics. * Over 8 years of experience in the field of data engineering with capabilities on working through the entire development lifecycle * Ability to work with senior business stakeholders to be able to ascertain data needs out of business opportunities or challenges * Minimum of 5 years of cloud experience on any of the major cloud platforms preferably AWS * Hands-on Knowledge of Data Modeling, Warehousing and Power BI (or equivalent) as a Reporting Tool * Advanced proficiency in SQL with the ability to read and write queries is required * Demonstrable experience with Databricks, Snowflake, Python, and PySpark * Preferably to have experience with Kafka and Power BI ## Description * Actively work with business and other BI stakeholders to understand their needs and solution enhancements to the data warehouse * Be available for addressing issues or bugs in the present implementation * Design, build, and maintain data infrastructure, ETL processes, and data pipelines. * Design, build, and maintain data pipelines using Databricks * Configure and implement data sourcing events on Kafka * Build and optimize python/Pyspark framework * Maintain Snowflake and AWS data infrastructure * Work with Unstructured/Semi-Structured data and build reporting tables * Assist the BI Analyst to manage client reporting requirements * Build Data models and assist the BI Analyst to offer data-driven insights * Design data models and automate manual processes. * Create and execute a test strategy to ensure robustness of data pipelines * Ensure effectiveness in infrastructure consumption by implementing solutions that are optimized and scalable * Foster an environment that emphasizes trust, open communication, creative thinking, and cohesive team effort, across the business, IT and vendor teams ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)