> Markdown version of [/jobs/ext/3057437-sr-data-engineer-ii](https://www.wearedevelopers.com/jobs/ext/3057437-sr-data-engineer-ii). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr Data Engineer II - **Company:** American Express Company - **Location:** Palo Alto, CA, United States - **Experience:** Expert - **Salary:** $123,000.0 - $215,250.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Computer Programming, Continuous Integration, Couchbase Servers, Data Architecture, Information Engineering, Data Governance, DevOps, Distributed Computing Environment, Distributed Data Store, Distributed Systems, Elasticsearch, Python (Programming Language), Machine Learning, NoSQL, OpenShift, Redis, Scala (Programming Language), Search Technologies, Data Streaming, Transaction Data, Spring Cloud, Large Language Models, Apache Spark, Model Validation, Generative AI, Event Driven Architecture, Build Management, Containerization, Kubernetes, Infrastructure Automation Frameworks, Cassandra, Apache Kafka, Data Management, Machine Learning Operations, Virtual Agents, Data Pipelines, Docker, Microservices - **Published:** September 24, 2026 - **Apply:** https://www.techcareers.com/job.asp?id=3402958444&tx=HT7468TYV&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54 ## About the Role * Experience designing, implementing, and operating large-scale distributed data platforms and systems . * Strong experience with NoSQL technologies such as Cassandra, Elasticsearch, Couchbase, or Redis . * Experience with large-scale distributed data processing technologies such as Apache Spark . * Strong programming experience with Python, Java, Scala , or similar languages. * Experience designing data pipelines, data models, APIs, and event-driven architectures . * Strong understanding of distributed system concepts, including scalability, reliability, resiliency, security, and performance . * Demonstrated ability to solve complex engineering problems and deliver production-grade solutions., * Experience building AI/ML-ready data platforms supporting machine learning, Generative AI, or intelligent application use cases. * Familiarity with Generative AI, LLMs, RAG, embeddings, vector databases/search, and AI agent architectures . * Experience integrating AI/ML models and services into production environments . * Familiarity with MLOps/LLMOps , including deployment, evaluation, monitoring, observability, governance, and lifecycle management. * Experience with distributed messaging and streaming platforms such as Apache Kafka . * Experience with real-time distributed processing using technologies such as Spark, Kafka, Cassandra, and Elasticsearch . * Experience developing microservices and cloud-native applications . * Experience with Docker, Kubernetes, OpenShift , or similar container platforms. * Experience with CI/CD, DevOps, infrastructure automation, and production observability . * Understanding of Responsible AI, data privacy, security, governance, and model risk considerations . * Experience architecting large-scale systems with an emphasis on availability, scalability, performance, resiliency, and cost efficiency . * Ability to stay current with emerging AI and data engineering technologies and identify opportunities to apply them to real-world business problems. * Strong communication, collaboration, and technical leadership skills, with the ability to mentor engineers and influence technical direction. Depending on factors such as business unit requirements, the nature of the position, cost and applicable laws, American Express may provide visa sponsorship for certain positions. ## Description * Design and build highly scalable, resilient, and low-latency data platforms supporting mission-critical payment workloads. * Develop distributed data processing and real-time streaming pipelines capable of handling high-volume transactional data. * Build AI-ready data architectures and pipelines that enable machine learning, Generative AI, and real-time intelligence use cases. * Explore and implement emerging AI capabilities, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), embeddings, vector search, and AI agents , where appropriate. * Design and develop microservices, APIs, and event-driven architectures using modern engineering practices. * Drive adoption of cloud-native technologies , containerization, orchestration, CI/CD, automation, and observability. * Establish engineering practices for performance, scalability, resiliency, security, data quality, and operational excellence . * Apply Responsible AI and data governance principles , including privacy, security, explainability, model evaluation, and appropriate controls. * Evaluate emerging technologies across AI, data, and distributed computing and translate them into practical solutions for payment platforms. * Provide technical leadership, mentor engineers, conduct design reviews, and contribute to engineering standards and best practices. * Partner with product, architecture, data science, and engineering teams to turn complex business opportunities into scalable technology solutions. ## Related Videos - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Reducing LLM Calls with Vector Search Patterns - Raphael De Lio (Redis)](https://www.wearedevelopers.com/videos/1714-reducing-llm-calls-with-vector-search-patterns-raphael-de-lio-redis) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Accelerating Authentication Architecture: Taking Passwordless to the Next Level](https://www.wearedevelopers.com/videos/733-accelerating-authentication-architecture-taking-passwordless-to-the-next-level) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)