ETL Tester with AI Experience
Addsource, Inc.
Irving, TX, United States
3 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
10 years minimum
Working hours
Regular working hours
Job source
Tech stack
Artificial Intelligence
Automation of Tests
Microsoft Azure
Software Bug Management
Code Coverage
Data Validation
Data Integrity
Extract Transform Load (ETL)
Data Security
Statistical Hypothesis Testing
Python (Programming Language)
Microsoft SQL Server
+25 more
Regression Testing
Standard Sql
Software Engineering
SQL Databases
Software Testing Automation Framework
Test Data
Strategies of Testing
Management of Software Versions
Web Testing
Data Processing
Great Expectations (Foster Youth College-readiness and Support Program)
Large Language Models
Generative AI
Agentic-AI
Pytest
Pyspark
LangSmith
Solid Principles
Machine Learning Operations
Vertica
Epic Caboodle
Domain Driven Design
Multiplatform
Legacy Systems
Databricks
Job description
We are looking for a senior QA professional skilled in ETL, SQL, Azure Databricks, agentic AI evaluation, web testing, and test automation tools. The ideal candidate will drive data integrity, compliance, and scalable validation strategies. Roles and Responsibilities Agentic AI Testing and Evaluation
- Test AI agents that generate extraction queries and transformation logic, along with the workbench through which humans review and approve their outputs.
- Validate agent-generated metadata executed by the clientβs existing data platform.
- Understand and validate the redesigned data processing workflow across raw ingestion, consolidated transformation, and domain-refined layers.
Test Strategy and Reconciliation
- Own the test strategy and reconciliation methodology.
- Define equivalence standards, identifying what must match exactly, what may vary within tolerance, and at which layer each applies.
- Design source-to-target validation across the redesigned data processing workflow.
- Establish risk-based prioritization, as the schedule does not permit exhaustive coverage of hundreds of tables.
- Author the test strategy and test plan for review with the client in Phase 2.
Agent Evaluation
- Build and maintain a golden reference set using existing production extracts, validated with the Epic SME.
- Maintain strict separation between evaluation data and data used to develop or tune the agents.
- Establish the evaluation harness in MLflow, including scoring runs, versioning, and candidate comparison.
- Define and report key metrics, including mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness, and cost per case.
- Operate the regression gate to ensure no prompt, skill, or model change is promoted without passing evaluation.
- Identify and characterize agent failures, including hallucinated tables, incorrect column selection, omission of required fields, and incorrectly selected join grain.
Automated Regression Testing
- Own the automated regression harness.
- Automate comparison of new-path outputs against current production extracts.
- Validate row counts, aggregate reconciliation, field-level comparisons, data types, precision, null and duplicate profiling, and referential integrity.
- Establish repeatable execution so every agent revision can be revalidated efficiently.
Data Security and Control Testing
- Design control-critical test suites for member, health-system, and facility isolation.
- Validate facility mapping changes and ensure one member cannot access another memberβs data.
- Perform contract-scope validation.
- Verify that generated queries comply with data-sharing restrictions at facility, column, and product levels.
- Validate acquisition costs by confirming optimized queries access fewer source tables than the baseline and that facility consolidation works as expected.
Team Leadership and Reporting
- Directly lead two QA engineers.
- Establish testing standards, review test coverage, and manage defect classification.
- Confirm downstream compatibility across Databricks, SQL Server, and Vertica.
- Produce evidence packs suitable for client executive review.
- Defend testing methodology, sample sizes, and statistical findings when required.
Requirements
- Engineering degree BE, ME, BTech, MTech, BSc, or MSc.
- Technical certifications in multiple technologies are desirable.
Mandatory Skills
- 10+ years of experience in data and ETL testing, including test strategy ownership on at least one large migration or re-platforming program.
- Proven experience in reconciliation and equivalence testing, demonstrating that rebuilt pipelines produce outputs equivalent to legacy systems.
- Expert-level SQL skills, including independently writing complex analytical validation queries against unfamiliar schemas.
- Experience with Azure Databricks and PySpark for large-volume data validation.
- Python programming for test automation and test harness development.
- Experience with data quality and test automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
- Experience in test strategy development, coverage models, risk-based prioritization, entry and exit criteria, and defect classification.
- Multi-platform validation experience across at least two of Databricks, SQL Server, Vertica, or comparable platforms.
- Experience testing PHI-scoped healthcare data, including de-identified or synthetic test data.
- Defect management and reporting using Azure Boards.
- Team leadership experience, including setting testing standards and reviewing team membersβ test coverage.
- Strong communication skills and the ability to present and defend quality findings to senior stakeholders., * Experience evaluating LLM or GenAI systems using MLflow, LangSmith, Weights & Biases, or equivalent tools.
- Experience with LLM-as-judge design and understanding its limitations and failure modes.
- Applied statistics, including sampling design, confidence intervals, significance testing, and calibration measurement.
- Epic EHR data experience.
- Experience with Caboodle or Clarity.
- Knowledge of clinical coding systems such as ICD, CPT, SNOMED, and LOINC.
- Experience with multi-tenant data isolation testing.
- Experience testing code-generation or SQL-generation systems.
- Previous experience presenting test evidence directly to client executives.
Software Design and Best Practices
- Apply software design principles such as SOLID and Domain-Driven Design.
- Understand software code patterns and best practices.
- Maintain up-to-date technical skills and industry knowledge.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦