Data Engineer
Role details
Job location
Tech stack
Job description
We are seeking a highly skilled Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in Databricks, DevOps tools (Jenkins/Terraform), and data modeling concepts (3NF, Dimensional, Data Vault). As a Senior Data Engineer, you will play a critical role in designing, implementing, and maintaining our client's data infrastructure while ensuring scalability, reliability, and efficiency., 1. Data Engineering: Design, build, and maintain scalable data pipelines and ETL processes using Databricks and other relevant technologies.
- DevOps Integration: Implement continuous integration and continuous deployment (CI/CD) pipelines using Jenkins and Terraform to automate deployment, monitoring, and scaling of data infrastructure.
- Data Modeling: Develop and implement data models based on business requirements, including 3NF, Dimensional, and Data Vault models. Ensure data models adhere to best practices for efficiency, scalability, and maintainability.
- Performance Optimization: Identify and address performance bottlenecks in data pipelines and queries. Optimize data processing and storage to improve overall system performance.
- Data Quality Assurance: Implement data quality checks and monitoring processes to ensure data accuracy, completeness, and consistency.
- Collaboration: Work closely with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and deliver high-quality solutions.
- Documentation and Best Practices: Document data pipelines, infrastructure configurations, and data models. Define and enforce best practices for data engineering and DevOps processes.
- Training and Mentorship: Provide guidance and mentorship to junior team members. Conduct training sessions to promote knowledge sharing and skill development within the team.
Requirements
- Bachelor's degree in Computer Science, Engineering, or related field. Master's degree preferred.
- Proven experience as a Data Engineer, preferably in a cloud-based environment.
- Strong proficiency in Databricks for data processing and analytics.
- Hands-on experience with DevOps tools such as Jenkins and Terraform for infrastructure automation.
- In-depth knowledge of data modeling concepts including 3NF, Dimensional, and Data Vault.
- Proficiency in SQL and programming languages such as Python or Scala.
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
Applicants for employment in the U.S. must possess work authorization which does not require sponsorship by the employer for a visa., Are you currently legally authorized to work in the US for any employer?* Do you now or will you ever require sponsorship from Infinitive to continue to be authorized to work in the US?* Are you local to the Washington DC/Metro area?* Do you have at least 3 year of experience in data engineering?* What is your experience with SQL?* What is your experience with ETL?* What is your experience with Python?* What is your experience with PySpark?* What is your experience with Apache Spark?* What is your experience with Amazon Redshift?* What is your experience with Databricks?* What is your experience with PostgreSQL?* What is your experience with MongoDB?* What is your experience with Snowflake?* What is your experience with Apache Cassandra?* What is your experience with the ELK stack (Elastic, Logstash, Kibana)?* What is your experience with Azure?*