Senior Data Pipeline Engineer

• AWS IAM, encryption, secrets, logging, retries, lifecycle management, and cost-aware operation.
• Large-File Processing and Data Quality
• NDJSON and other high-volume file formats.
• Validation, malformed-record handling and quarantine patterns, with reconciliation and traceable error reporting.
• Change detection and large-dataset matching using deterministic, explainable approaches.
• AI-Forward Delivery (Required)
• Daily use of Claude Code, OpenAI Codex, Cursor, GitHub Copilot, or equivalent coding agents.
• Context and instruction design for decomposing data work into safe, testable implementation steps.
• Critical review of generated code and queries, including correctness tests, representative fixtures, performance checks, and regression protection.
• Responsible use of agents around sensitive data; no uncontrolled PHI or production-data exposure.
• Healthcare and Compliance (Preferred), * Design and build secure data connectors for large inbound and outbound files.
• Implement Python and DuckDB processing jobs running through AWS Batch and S3.
• Create robust validation and quarantine mechanisms, with reliable retry, reconciliation and audit behavior.
• Build implementation-detection workflows using change detection and large-dataset matching.
• Integrate pipeline outputs with Postgres-backed applications and operational processes.
• Establish useful telemetry and performance baselines, along with runbooks and cost controls.
• Use AI coding agents to accelerate implementation while retaining human ownership of correctness, security and production readiness.

We’re hiring a Senior Data Pipeline Engineer Consultant to build reliable data connectors and large-scale implementation-detection workflows for a healthcare technology program. This remote, Corp-to-Corp 1099 engagement is for an independent senior engineer with deep Python experience and practical expertise in DuckDB, Postgres, AWS Batch, and S3.

The work involves large files and imperfect source data. It includes validation, reconciliation and repeatable matching across large datasets. You must be able to own the full path from ingestion and transformation through operational controls, observability and production support.

This is an AI-forward delivery role. You should already use coding agents such as Claude Code, OpenAI Codex, Cursor, or equivalent tools as part of your daily workflow. Rigorous testing and data quality are required, with security and human review throughout.

Key Requirements

• 10+ Years of Data Engineering and Software Delivery
• Production experience building and operating batch data pipelines and data-intensive applications, then improving them over time.
• Strong judgment around reliability, idempotency, retries, partial failure and schema drift. You understand scale, cost and operational support.
• Advanced Python Engineering
• Maintainable Python services and jobs with clear modules, type discipline and tests, plus sound packaging and production diagnostics.
• Performance-aware processing of large files and datasets without defaulting to infrastructure that is more complex than the problem requires.
• DuckDB and Postgres
• Practical DuckDB experience for local or job-oriented analytical processing.
• Strong Postgres experience: relational modeling, bulk loading, query design and indexing, including migrations and performance tuning.
• Ability to design clean boundaries between file processing, analytical work, and operational persistence.
• AWS Batch and S3
• Production use of S3 for secure, observable file-oriented workflows., * Experience with healthcare, pharmacy, HIPAA, PHI, or similarly regulated data.
• Familiarity with data minimization, auditability, least privilege and retention, including redaction and secure operational troubleshooting.
• Consulting Posture and High EQ
• Excellent client-facing communication and requirements discovery.
• Able to explain data-quality findings, constraints and tradeoffs to technical and nontechnical stakeholders.
• Independent and pragmatic. You stay outcome-focused and bring structure to ambiguous data problems.
• Business Structure
• Must have an active LLC or S-Corp.
• 1099 Corp-to-Corp only. No W-2.
• Two or more years operating an independent consulting company is strongly preferred., * Data engineering/software delivery: 10+ years (Required)
• Production Python: 5+ years (Required)
• Postgres: 3+ years (Required)
• DuckDB or comparable embedded analytical database: 1+ years (Required)
• AWS Batch and S3: 2+ years (Required)
• Consulting/client-facing delivery: 5+ years (Required)
• Daily use of AI coding agents: 1+ years (Required)
• Healthcare or pharmacy data: Any (Preferred)

$77.19 - $110.00 an hour - Contract, Agencies: We are not engaging staffing agencies for this role.

Job Type: Contract

Compensation Package:

• 1099 contract

Proactive Logic Consulting INC is a boutique technology consulting firm specializing in:

• Innovation and zero-to-one projects
• Assessments and roadmaps
• Process automation and AI
• Cloud migrations and modernization
• Application modernization, including low-code solutions

We bring together top-tier independent consultants who are experts in their craft and thrive on flexibility and autonomy. Our consultants combine deep technical judgment with high EQ, a customer-obsessed mindset, and an entrepreneurial approach to delivery.