Data Engineer for Software Engineering Group

We are seeking a skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in building and optimizing data pipelines, handling large datasets, and working within cloud-based and on-premise environments. You will be responsible for designing and implementing efficient, scalable solutions to manage, process, and analyze high volumes of data, ensuring the reliability and performance of our data infrastructure.

What will your job look like:

• Design, build, and maintain scalable data pipelines to process large datasets efficiently
• Develop and implement data models and architectures that support both real-time and batch data processing
• Ensure data integrity, security, and accuracy across all systems
• Collaborate with data scientists, analysts, and other engineers to ensure data availability and quality
• Optimize data retrieval and storage processes to handle large volumes of data seamlessly
• Work with structured, semi-structured, and unstructured data, integrating various data sources
• Troubleshoot and resolve data issues, ensuring continuous operation of the data infrastructure
• Maintain and enhance ETL processes, ensuring scalability and performance in handling large datasets
• Stay up-to-date with industry best practices and emerging technologies related to big data engineering

• Bachelor's degree in Computer Science, Engineering, or a related field
• At least 4 years experience in data engineering, with a focus on large-scale data processing and big data technologies
• Strong proficiency in Python
• Experience with data pipeline and workflow management tools
• Hands-on experience with large-scale data processing frameworks like Apache Spark, Hadoop, or similar
• Familiarity with data modeling, ETL processes, and data warehousing concepts like Table formats. Apache Iceberg, or similar
• Good knowledge of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, MongoDB)
• Experience with AWS-based data lakes, including working with Amazon S3 for storage, querying data using Amazon Athena, and managing datasets stored in Parquet format
• Significant ownership ability

Nice to have:

• Knowledge of machine learning frameworks and integrating data pipelines for model training and deployment
• Experience with version control systems (e.g., Git), CI/CD pipelines, and automation tools
• Experience with containerization (Docker, Kubernetes)