Data Platform Software Engineer
Role details
Job location
Tech stack
Job description
We are seeking a skilled Data Platform Software Engineer to join our Data Platform team in Kansas City, MO. In this role, you will design, code, implement, support, and optimize a cloud-native data stack that leverages best-in-class open-source tools. The ideal candidate will design, build, support, and maintain an opinionated, resilient, and scalable data platform in a private cloud environment-enabling data-driven decision-making, analytics, and machine learning, while providing out-of-the-box deep insights. This role blends data engineering, software development, and infrastructure management, leveraging languages and tools such as Java, Python, Apache Iceberg, Flink, Spark, Kafka, and Trino., * Develop and maintain robust, fault-tolerant data ingestion and transformation pipelines using Java, Python, Apache Flink, Spark, and Trino.
- Define flexible and scalable data schemas using Apache Iceberg.
- Support both batch and real-time data processing, including integration with Apache Kafka.
- Ensure reliability, observability, and integrity of data pipelines.
Metadata Management & Data Governance
- Support metadata management platforms such as DataHub for data cataloging, lineage, and governance use cases.
- Collaborate with data stakeholders to align metadata solutions with organizational needs.
- Enforce governance policies related to data quality, privacy, and compliance (e.g., GDPR, CCPA).
- Implement fine-grained access controls, encryption, and auditing with a focus on regulatory compliance and data traceability.
Automation & CI/CD
- Automate data pipelines, workflows, deployment, and infrastructure provisioning and configuration using tools like ArgoCD, Apache Airflow, Ansible, Salt, and Kubernetes.
- Implement CI/CD pipelines, actions, and workflows on GitHub to deploy updates and enhancements.
Performance Optimization
- Optimize data storage and queries using Apache Iceberg, Flink, and Spark to ensure high performance and low-latency access.
- Identify and address performance bottlenecks; implement partitioning, caching, and indexing strategies.
Monitoring and Alerting
- Monitor data platform health using tools such as Prometheus and Grafana dashboards.
- Configure real-time alerts to proactively detect and resolve pipeline failures or data issues.
- Troubleshoot and resolve platform outages and data incidents promptly.
Collaboration
- Work with other development teams and engineers to understand data needs and deliver performant, scalable solutions.
- Collaborate with cross-functional teams (Cloud Engineering, Network, and DevOps/Solutions Engineering) to troubleshoot and resolve infrastructure issues.
Requirements
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field., * 3+ years of experience in software and data engineering, with a strong focus on cloud-based data platforms.
Technical Skills
- Strong programming skills in Java or Python.
- Knowledge of Apache Iceberg, Flink, Spark, Kafka, and Trino.
- Familiar with metadata management platforms like DataHub or Apache Atlas.
- Familiar with cloud-native infrastructure tools such as Kubernetes, Ansible, Salt, etc.
- Familiar with CI/CD and DevOps operations.
Soft Skills
- Strong analytical and problem-solving skills.
- Effective communication and collaboration with cross-functional teams.
Benefits & conditions
Unless explicitly requested or approached by SS&C Technologies, Inc. or any of its affiliated companies, the company will not accept unsolicited resumes from headhunters, recruitment agencies, or fee-based recruitment services.
SS&C Technologies offers a comprehensive total rewards package designed to support your wellbeing, growth, and future. Our benefits include medical, dental, and vision coverage; a 401(k) plan with company match; paid time off, holidays, and parental leave; and professional development reimbursement opportunity.