> Markdown version of [/jobs/ext/3090046-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/3090046-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** OPEN QUEUE LLC - **Location:** Atlanta, GA, United States - **Experience:** Expert - **Salary:** $75,000.0 - $100,000.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Airflow, Amazon Web Services, Amazon S3, Business Analytics Applications, Microsoft Azure, Cloud Computing, Continuous Integration, Data Cleansing, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Transformation, Data Warehousing, Identity and Access Management, Python (Programming Language), Operational Databases, SQL Databases, Systems Integration, Enterprise Data Management, Data Processing, Cloud Platform System, Feature Engineering, Sql Optimization, Retrieval-Augmented Generation, Delivery Pipeline, Large Language Models, Snowflake, Prompt Engineering, Apache Spark, Generative AI, AWS Lambda, Data Lakes, Pyspark, Semi-structured Data, AWS Glue, AWS Data Analytics, Apache Kafka, Functional Programming, Cloudwatch, Stream Processing, Data Pipelines, Amazon Elastic Mapreduce (EMR), Amazon Redshift - **Published:** September 26, 2026 - **Apply:** https://www.careerjet.com/jobad/us5cc42afa7242ec5d9ab115d18d9573eb ## About the Role * Advanced Python * Advanced SQL * ETL / ELT * Data Pipelines * Data Lakes * Data Warehousing * Data Modeling * Apache Spark / PySpark * Apache Airflow * Kafka / Kinesis * Snowflake / Redshift * Data Quality & Governance AI / GenAI Skills * Experience integrating Generative AI / LLM solutions with enterprise data platforms. * Experience working with OpenAI, Azure OpenAI, Anthropic Claude, Amazon Bedrock, or Google Gemini . * Knowledge of Prompt Engineering and LLM APIs. * Experience preparing enterprise data for RAG (Retrieval-Augmented Generation) applications. * Experience integrating vector databases and embeddings is a plus. * Understanding of AI/ML data pipelines and model lifecycle. * Ability to use AI-assisted development tools to improve engineering productivity. For applications and inquiries, contact:hirings@openkyber.com ## Description We are seeking a highly experienced Senior AWS Data Engineer with AI/GenAI experience to design, develop, and support scalable cloud-based data platforms and intelligent data solutions. The ideal candidate should have extensive experience in AWS data services, Python, SQL, ETL/ELT pipelines, data lakes, data warehouses, and AI/ML integrations . The candidate should be comfortable working independently, collaborating with architects and business stakeholders, and taking ownership of enterprise-scale data engineering initiatives. Key Responsibilities: * Design and develop scalable data pipelines and ETL/ELT solutions using AWS services. * Build and maintain data ingestion, transformation, and processing frameworks. * Develop solutions using AWS Glue, S3, Lambda, Redshift, Athena, EMR, and Step Functions . * Develop reusable data engineering components using Python . * Write complex and optimized SQL queries for data transformation and analytics. * Design and implement data lake and data warehouse architectures . * Build batch and real-time data processing pipelines. * Implement data quality, validation, reconciliation, and monitoring processes. * Integrate structured and semi-structured data from multiple enterprise sources. * Develop scalable data models for reporting, analytics, and AI applications. * Work with AI/ML and GenAI teams to prepare high-quality datasets for AI use cases. * Integrate AI/LLM capabilities into existing data engineering workflows where applicable. * Develop and maintain AI-enabled data pipelines and analytics solutions. * Support model training, feature engineering, data preparation, and inference pipelines. * Troubleshoot production data pipeline failures and perform root-cause analysis. * Optimize data processing jobs, SQL queries, storage, and AWS infrastructure for performance and cost. * Implement CI/CD practices for data engineering applications. * Collaborate with Data Scientists, ML Engineers, Cloud Architects, Business Analysts, and stakeholders. AWS Technologies * Amazon S3 * AWS Glue * AWS Lambda * Amazon Redshift * Amazon Athena * Amazon EMR * AWS Step Functions * Amazon Kinesis * AWS Lake Formation * AWS IAM * CloudWatch * AWS Secrets Manager ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [How we built an AI-powered code reviewer in 80 hours](https://www.wearedevelopers.com/videos/1511-how-we-built-an-ai-powered-code-reviewer-in-80-hours) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)