Performance Engineer

We are looking for an experienced Data Quality Engineer to ensure the accuracy, integrity, reliability, and performance of enterprise data platforms and pipelines. This role will focus on validating data ingestion, transformation, processing, streaming, and analytics workflows across AWS-based environments. The ideal candidate will have strong expertise in SQL, Python, Apache Kafka, Databricks, and AWS data services, with experience developing automated data validation frameworks and testing distributed data systems. You will collaborate with data engineering, platform, analytics, and DevOps teams to maintain high-quality data and ensure pipeline reliability across the data lifecycle. Key Responsibilities Design and execute comprehensive test strategies for batch processing, streaming pipelines, ETL/ELT workflows, and data analytics solutions. Develop automated data quality checks to identify missing records, duplicate entries, schema changes, inconsistencies, and referential integrity issues. Validate business rules, data transformations, aggregations, and processing logic using SQL and Python. Test Databricks-based data engineering workflows, including Apache Spark applications developed with PySpark or Scala. Verify data accuracy and traceability across source systems, intermediate processing layers, and downstream data consumers. Validate AWS-based data pipelines built using services such as S3, Glue, Lambda, EMR, Step Functions, Redshift, Athena, Kinesis, and DynamoDB. Test Kafka streaming applications to ensure data integrity, event ordering, and appropriate message delivery guarantees. Validate Kafka producers, consumers, serialization formats, topic configurations, partitioning, offsets, retention settings, and schema evolution. Assess pipeline behavior under duplicate events, delayed messages, consumer failures, reprocessing scenarios, backfills, and historical data loads. Develop and maintain reusable Python-based test automation frameworks, validation utilities, test fixtures, and synthetic datasets. Integrate automated data quality tests into CI/CD workflows for pre-merge checks, scheduled executions, and post-deployment verification. Implement automated notifications and reporting mechanisms to identify data quality failures and pipeline anomalies. Evaluate data pipeline scalability and performance under high-volume workloads, including throughput, latency, concurrency, and resource utilization. Test failure recovery, retry mechanisms, idempotency, error handling, regression scenarios, and system resilience. Monitor pipeline health using CloudWatch, Prometheus, Grafana, or comparable monitoring and observability platforms. Validate data security controls, IAM permissions, encryption configurations, storage lifecycle policies, and access restrictions. Collaborate with engineering teams to establish data quality SLAs and SLOs and support incident investigations, root-cause analysis, and post-incident reviews. Required Qualifications

7+ years of experience in Quality Assurance, Software Development Engineer in Test (SDET), Data Quality Engineering, or related testing roles. Strong hands-on expertise in SQL for complex data analysis, reconciliation, and validation. Proficiency in Python for test automation, data processing, and quality assurance. Proven experience testing data pipelines, ETL/ELT processes, and distributed data processing workflows. Practical experience with Apache Kafka or other event-streaming technologies. Strong understanding of AWS data services, including S3, Glue, Redshift, Lambda, Athena, or equivalent platforms. Experience with Databricks and Apache Spark-based data processing is highly desirable. Knowledge of batch and streaming architectures, data transformations, schema validation, and end-to-end data lineage. Experience testing large-scale datasets and distributed systems. Strong troubleshooting, analytical thinking, debugging, and problem-solving capabilities. Ability to collaborate effectively with cross-functional engineering and analytics teams. Technical Environment Programming and Query Languages: Python, SQL, Scala Data Processing: Databricks, Apache Spark, PySpark Streaming Technologies: Apache Kafka AWS Services: S3, Glue, Lambda, EMR, Step Functions, Redshift, Athena, Kinesis, DynamoDB Automation and Delivery: Python-based test frameworks, CI/CD pipelines

Monitoring and Observability: Amazon CloudWatch, Prometheus, Grafana Quality Engineering: Data validation, ETL/ELT testing, reconciliation, regression testing, performance testing Founded in 2014, is a global leader in delivering innovative IT solutions and services. Headquartered in the USA with a presence in the UK, Canada, and India, we specialize in offering industry-leading expertise and cutting-edge products that help our clients maximize their technological investments. Our focus on best-in-class solutions, a highly knowledgeable team, and proactive talent mapping ensure we remain at the forefront of the IT industry. ConglomerateIT is driven by our Center for Excellence and Innovation, an initiative dedicated to keeping us ahead in a rapidly evolving technology landscape. We understand that building strong relationships is key to our success, and this commitment has enabled us to partner with Fortune 500 companies and leading system integrators worldwide. Our ability to provide local talent on a global scale ensures that we can meet the contingent project requirements of our clients efficiently and effectively.

ConglomerateIT LLC