ETL Data Engineer
Role details
Job location
Tech stack
Job description
- Build and maintain ETL/ELT pipelines using Apache Spark, Hive, and Trino across S3-based data lakes.
- Develop and optimize SQL for large-scale surveillance datasets using window functions, joins, and complex aggregations.
- Engineer big data systems on EMR-on-EC2 and EMR-on-EKS and deliver solutions on analytical platforms such as SageMaker, Domino, or Dataiku.
- Participate in data quality monitoring, anomaly detection, and production incident investigation.
- Develop AI agent systems using AWS Bedrock and agent frameworks such as Strands Agents SDK or LangChain/LangGraph.
- Design agent harnesses that combine LLM reasoning with deterministic execution including skill or RAG-based SQL generation and structured output validation.
- Implement agent memory, context management, and tool integration including MCP servers, API connectors, and data catalog lookups.
- Build evaluation frameworks for agent accuracy covering paraphrase robustness, routing precision, and structural consistency.
- Stay informed on advances in LLM frameworks and emerging AI capabilities.
- Write clean, well-tested code and contribute to CI/CD pipelines and infrastructure-as-code on AWS.
- Ensure secure handling of sensitive regulatory data with auditable execution traces.
- Adhere to secure development practices and technology policies.
- Partner across teams, communicate at the appropriate technical level, and maintain documentation on Confluence or Wiki.
- Learn from senior team members and contribute to process improvement.
Requirements
- Experience building data pipelines with Apache Spark (PySpark preferred) and SQL.
- Experience with SQL engines such as Hive or Trino and cloud data platforms including AWS S3, EMR, and Lambda.
- Understanding of data skew, large-volume processing, and troubleshooting job failures due to resources, data quality, and scalability.
- Hands-on debugging and mitigation experience.
- Practical experience building LLM-powered agent systems that use tools and produce structured outputs.
- Experience with agent frameworks such as LangChain, LangGraph, or AWS Strands.
- Knowledge of prompt engineering, RAG architectures, and context or memory management.
- Experience with foundation model APIs such as Anthropic Claude, Amazon Nova, or OpenAI.
- Understanding of agent memory tiers and strategies for persistence, pruning, and retrieval.
- Familiarity with harness patterns including deterministic guardrails, tool routing, and verification loops.
- Hands-on experience with AI development tools such as GitHub Copilot, Q Developer, ChatGPT, or Claude.
- Experience with spec-driven development for AI-assisted code generation and validation.
- Ability to leverage AI pair programming for suggestions, debugging, refactoring, and automated test generation.
- Experience with AWS services including S3, EMR, EMR on EKS, Lambda, Bedrock, and Step Functions.
- Hands-on experience using S3 with Spark and related file format or consistency considerations.
- Familiarity with AWS Bedrock guardrails, knowledge bases, and agent orchestration.
- Exposure to Google Cloud Vertex AI or equivalent managed AI platforms.
- Familiarity with AWS monitoring and logging tools such as CloudWatch and CloudTrail.
- Proficiency in Python with clean, modular, and performant code and understanding of functional concepts.
- Strong understanding of collections, concurrency, and memory management.
- Proficiency with SQL window functions, joins, aggregations, and complex query optimization including edge cases., * Bachelor's degree in Computer Science, Data Science, Information Systems, or related discipline with at least two years of related experience, or equivalent training and work experience. Financial services experience preferred.
- Demonstrated expertise in object-oriented and database technologies resulting in enterprise-quality solutions.
- Knowledge of software engineering approaches including test automation, build automation, and configuration management.
- Strong written and verbal technical communication skills and effective cross-team collaboration.
- Ability to learn new skills rapidly and operate in a fast-paced environment.
Benefits & conditions
Our client seeks an ETL Data Engineer to build and maintain large-scale data pipelines and to design agentic AI systems that support regulatory analytics. The role spans Spark-based ETL on AWS data lake platforms and development of LLM-powered agents with secure, auditable outputs. The engineer will collaborate across teams, uphold secure development practices, and contribute to CI/CD and infrastructure-as-code while monitoring data quality and production performance.
We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.
Rate: $70.00 to $88.00/hr. w2, Skills, experience, and other compensable factors will be considered when determining pay rate. The pay range provided in this posting reflects a W2 hourly rate; other employment options may be available that may result in pay outside of the provided range.
W2 employees of Eliassen Group who are regularly scheduled to work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), dental, vision, pre-tax accounts, other voluntary benefits including life and disability insurance, 401(k) with match, and sick time if required by law in the worked-in state/locality.
If anyone reaches out to you about an open position connected with Eliassen Group, please ensure that you are working directly with us by confirming the following:
· When you work with Eliassen Group, all email communication will come from an Eliassen.com address, never Gmail, Yahoo, etc.