Sr. Data Engineer
Role details
Job location
Tech stack
Job description
As part of the Network Analytics team, the Sr Data Engineer role is crucial in building and maintaining the data infrastructure that supports the clients enterprise networks. This position focuses on constructing robust ETL pipelines using Spark to ensure data availability and quality for analytics tasks. The role requires expertise in data engineering, leveraging AWS for scalable solutions and enhancing operational efficiency. Additionally, the position demands the ability to prioritize efforts, synthesize actionable insights, and effectively communicate findings to audiences with diverse agendas and varying levels of technical expertise., * Design, develop, and maintain scalable ETL pipelines to ensure data quality and availability
-
Implement monitoring and alerting solutions to ensure data pipeline reliability and performance
-
Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions
-
Manage and optimize data storage solutions, including:
-
Distributed file systems
-
Relational databases
-
Flat files
-
External sources accessed via API
Implement data integration solutions to support analytics and reporting needs
Collaborate across teams to ensure data solutions align with existing production architectures and business requirements
Provide technical leadership and guidance on data engineering best practices and methodologies
Document and communicate data engineering processes and standards to business intelligence, data, and analytics professionals with varied backgrounds
Continuously evaluate and improve data engineering tools and approaches to enhance performance and efficiency
Perform other duties as required
Requirements
Top 3 Requirements:
- Advanced SQL skills (ETL)
- Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python
- Experience with BigData tools, particularly Spark, and familiarity with Hadoop ecosystems
- Experience with cloud platforms like AWS, including services such as S3, Lambda, Redshift, and EMR, Skills/Abilities and Knowledge
- Ability to read, write, speak, and understand English
- Strong communication and collaboration skills
- Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python
- Experience with BigData tools, particularly Spark, and familiarity with Hadoop ecosystems
- Proficiency in building and managing ETL pipelines
- Experience with cloud platforms like AWS, including services such as S3, Lambda, Redshift, and EMR
- Strong understanding of relational databases and SQL, and familiarity with NoSQL databases
- Knowledge of data architecture, data warehousing, and data marts
- Experience in implementing monitoring solutions and deployment pipelines for data applications
- Ability to work with distributed systems and manage data storage solutions
- Demonstrated ability and desire to continually expand skill set, and learn from and teach others
PREFERRED QUALIFICATIONS
Skills/Abilities and Knowledge
- Experience with Linux-based operating systems
- Knowledge of other relevant techniques such as text analysis and text mining, Bachelors degree (or relevant experience) in computer science, mathematics, statistics, physics, operations research, or other quantitatively-focused fields