> Markdown version of [/jobs/ext/3984636-staff-machine-learning-engineer](https://www.wearedevelopers.com/jobs/ext/3984636-staff-machine-learning-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Staff Machine Learning Engineer - **Company:** ELLKAY, LLC. - **Location:** Elmwood Park, NJ, United States (Remote available) - **Experience:** Expert - **Salary:** $210,000.0 - $230,000.0 - **Contract:** Permanent contract - **Skills:** LangGraph Framework, Application Programming Interfaces (APIs), Amazon Web Services, Application Integration Architecture, Cloud Computing, Cloud Engineering, Software Quality, Databases, Data Systems, Cursor, DevOps, Programming Tools, Distributed Systems, Amazon DynamoDB, Middleware, Information Retrieval, Python (Programming Language), PostgreSQL, Machine Learning, NoSQL, OAuth, Role-Based Access Control, Microsoft Copilot, Search Technologies, Software Engineering, Data Streaming, TypeScript, AWS Cdk, Data Logging, Enterprise Software Applications, Data Ingestion, Apache Camel, GitHub Copilot, Large Language Models, Claude Code, Multi-Agent Systems, State Machines, Zapier, AWS Lambda, Backend, Agentic-AI, Fastapi, Servicebus, LangSmith, Information Technology, Amazon Bedrock, CrewAI, RAGAS (Retrieval Augmented Generation Assessment), AWS Fargate, AutoGen, Apache Kafka, Invoking Functions, Cloudwatch, Api Gateway, Model Context Protocol, OpenAI Codex, Amazon Simple Queue Service (SQS), Data Pipelines, Dynatrace, Serverless Computing, Docker, Mulesoft, Microservices - **Published:** October 28, 2026 - **Apply:** https://www.dice.com/job-detail/b6f02e1a-cdfe-44be-ac08-76ebf0e4cee4 ## About the Role * 10+ years of progressive experience in software engineering, machine learning engineering, or distributed systems, with significant ownership of production platforms. * Deep Agentic Systems Expertise: Hands-on experience building and deploying stateful agentic workflows, multi-agent frameworks (e.g., LangGraph, AutoGen, CrewAI, Bedrock Agents), dynamic tool/function calling, and agent memory systems. * Context Layer & Security: Proven track record designing enterprise context engines, hybrid retrieval systems (dense/sparse vector search + re-ranking), and strictly enforcing user-level RLS/RBAC access control within vector/retrieval layers. * MCP & Tool Integration: Experience architecting tool gateway solutions (such as MCP), contract/schema validation, authentication delegation, and robust error handling for external tool invocations. * Orchestration & Resiliency: Deep knowledge of state isolation, multi-agent coordination patterns, loop circuit breakers, human-in-the-loop intervention protocols, and automated agent evaluation frameworks. * Core Tech Stack: Advanced proficiency in Python (FastAPI, Pydantic, AsyncIO). Go/Rust a plus. * Cloud & DevOps: Deep expertise with AWS services (Lambda, ECS Fargate, Step Functions, EventBridge, DynamoDB, Aurora), AWS CDK, Docker, and streaming/messaging systems (Kafka, EventBridge, SQS). * AI-Assisted Development: Experience leveraging modern AI developer tools (Claude Code, Codex, Cursor, Copilot) to accelerate architecture design, prototyping, and production engineering., * Degree in Computer Science, Machine Learning, Electrical Engineering, or a related quantitative field. * Experience building or extending enterprise middleware, integration engines, or API gateways (e.g., Workato, MuleSoft, Apache Camel). * Experience with automated trajectory evaluation tools (LangSmith, Arize, Phoenix, RAGAS) and LLM-as-a-judge methodologies. ## Description ELLKAY is seeking a Staff Machine Learning Engineer, AI Systems Integration who can design, integrate, and operate production-grade, autonomous agentic systems across products, data pipelines, APIs, enterprise applications, and workflow engines. This role sits at the intersection of machine learning engineering, backend systems, integration architecture, and cloud infrastructure. You will be responsible for building the core orchestration runtime, Model Context Protocol (MCP) gateways, enterprise context layers, and automated governance harnesses that allow autonomous and semi-autonomous AI agents to execute multi-step workflows safely and reliably across enterprise ecosystems., 1. Enterprise Context Layer & Data Systems * Design and implement unified Enterprise Context Layers that combine structured relational/NoSQL data and unstructured document stores into high-performance hybrid retrieval engines. * Architect real-time streaming ingestion pipelines (e.g., Kafka, EventBridge) balancing recency (via fast-path memory/cache layers) with long-term semantic relevance (via asynchronous vector indexing). * Enforce granular security policies by embedding Row-Level Security (RLS) and Role-Based Access Control (RBAC) directly into retrieval queries so agents only retrieve context authorized for the active user session. 2. Model Context Protocol (MCP) Gateway Architecture * Design and maintain enterprise Model Context Protocol (MCP) gateways to seamlessly connect heterogeneous tools, REST/gRPC microservices, and enterprise databases to agent ecosystems. * Implement dynamic tool discovery mechanisms, schema validation (using Pydantic v2), and authentication delegation (OAuth2 token exchange and dynamic scope validation). * Build resilient tool execution handles featuring bounded retry loops, fallback routing, circuit breakers, and structured error reporting for agent self-correction during timeouts or malformed tool responses. 3. Agentic Execution Fabric & Orchestration * Structure multi-agent orchestration runtimes supporting both hierarchical manager-worker state graphs and asynchronous, event-driven agent meshes. * Architect state isolation models using thread-level persistence snapshots (Aurora PostgreSQL, DynamoDB) to maintain clear agent context boundaries. * Implement execution circuit breakers (token/depth limits), human-in-the-loop (HITL) escalation boundaries for high-risk actions, and reflection loops for autonomous self-correction. * Establish automated evaluation harnesses utilizing trajectory benchmarking, tool-selection precision evaluation, held-out test sets, and LLM-as-a-judge scoring to continuously govern system performance at scale. 4. Platform, Infrastructure & Developer Tools * Build high-performance backend microservices using Python (FastAPI, Pydantic, AsyncIO) and write production-grade Infrastructure as Code using AWS CDK (TypeScript/Python). * Deploy cloud-native agent runtime environments leveraging AWS Lambda, API Gateway, Step Functions, EventBridge, SQS, ECS Fargate, Aurora Serverless, and DynamoDB. * Leverage modern AI development tools and agentic coding workflows (e.g., Claude Code, OpenAI Codex, GitHub Copilot) to maximize developer velocity, code quality, and implementation speed. * Implement full observability across agent execution trajectories using structured logging, distributed tracing (OpenTelemetry, CloudWatch, X-Ray), and token/cost tracking.