Lead AI Data Engineer

Insight Global is seeking a Lead AI Data Engineer to sit hybrid at a Cybersecurity client in Frisco, Texas. You will join their eCommerce Operational Intelligence team, building enterprise-scale data pipelines and analytics foundations (SQL, Spark/PySpark, ETL/ELT) that produce reliable operational insights and measurable business impact.

You will drive the transformation of eCommerce operational analytics and real-time monitoring by building scalable data pipelines, AI-powered insights, and intelligent dashboards. This role leads AI proof-of-concepts and contributes to production-grade solutions that improve platform reliability, accelerate root-cause identification, enhance engineering productivity, and strengthen operational intelligence across their eCommerce ecosystem.

Day to Day:

-Build and operate production ETL/ELT pipelines processing millions of eCommerce events daily and order trends.

-Write and tune complex SQL for operational analytics, KPIs, and reporting.

-Design analytics-ready schemas and data models for performance and scale.

-Troubleshoot pipelines, microservices, and APIs; apply observability to isolate root causes.

-Integrate data across eCommerce, MarTech, and operational systems into unified insights.

10+ years building and architecting large-scale applications and distributed systems.

-5+ years building production data pipelines, ETL/ELT workflows, and analytics platforms.

-Applied AI to operational intelligence (anomaly detection/alerting, forecasting, insights).

-Expert SQL (complex queries, performance tuning)

-Spark/PySpark in production (Spark SQL, optimization)

-Strong Python (testing, packaging, best practices)

-ETL/ELT pipelines (orchestration, monitoring, error handling)

-Databricks & Delta Lake (Jobs, Unity Catalog, Medallion)

-Analytics data modeling (star/snowflake schemas)

-Distributed systems (APIs, microservices, event-driven)

-Production observability & troubleshooting

-AWS (S3, Lambda, Glue, Kinesis, OpenSearch, QuickSight), BI (Power BI, Tableau, Grafana), GenAI (RAG, vector DBs, LangChain, Bedrock)

(This role can pay $200,000 - 220,000 based on years of experience).