Gen AI Engineer

Alchemy Software Solutions LLC
Mountain View, CA, United States
about 1 month ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
2 years minimum
Working hours
Regular working hours
Job source

Tech stack

Java (Programming Language) Test Suite Application Programming Interfaces (APIs) Artificial Intelligence Amazon Web Services Automation of Tests Continuous Integration Python (Programming Language) Open Source Technology Regression Testing Statistical Process Control (SPC) Test Execution Engine
+9 more
Management of Software Versions WebSocket Large Language Models Caching Generative AI Backend Kubernetes Production Code Api Management

Job description

· Build QE-generation agents on GenRuntime: agents that consume artifacts the GenOS dev flow already produces - requirements, code diffs, API specs, GenUX component usage - and emit functional, API, and regression test assets as validated, structured output.

· Wire QE enablement into the paved road: an app scaffolded through GenOS gets QE agents attached by default - no opt-in ceremony, self-serve onboarding, zero hand-holding.

· Build the agent toolbelt as reusable GenRuntime tools: test execution, self-healing selectors and API contracts, failure triage, defect summarization - including agent-to-agent patterns where test agents interrogate the application’’s own agents.

· Extend AI Workbench eval primitives (LLM Leaderboard, prompt evaluation) into release gates for GenAI applications: golden datasets, LLM-as-judge scoring, statistical quality thresholds enforced in paved-road CI/CD - not just model selection.

· Embed GenSRF coverage into generated tests: safety, privacy, and moderation regressions become executable test cases, not audit findings.

· Integrate with Feature Management so QE-agent rollout is flagged, measured, and adoption-tracked per product team.

· Write well-tested, production-grade code; participate in reviews and design discussions - PR merge velocity and AI-assisted code in PRs are tracked org KPIs.

· Participate in the production support/on-call rotation for the QE capability surface (rotation shape and compensation treatment per Open Items).

· Contribute self-serve onboarding docs and inner-source repos - inner-source contribution is a tracked PDX metric.

Requirements

GenOS is Client’’s Generative AI Operating System - the platform every GenAI experience at Client is built, deployed, and governed on, including the customer-facing Client Assist. The stack:

· GenStudio - LLM sandbox and extensible model catalog; new models onboarded in days.

· AI Workbench - versioned Prompt Management and the LLM Leaderboard for benchmarking.

· GenRuntime - GenOrchestrator (planner, executor, memory, retrieval) plus agents and tools grounding LLMs in Client domain knowledge.

· GenUX - 140+ AI UX components consumed by product teams.

· GenSRF - security, risk, and fraud guardrails as a platform feature, not an afterthought.

· Multi-LLM catalog: Anthropic Claude via AWS Bedrock, Gemini, Llama, and Mistral., · 7+ years backend/platform engineering (Java, Python, or Go) with deep distributed-systems fundamentals: async processing, caching, idempotency, failure handling.

· 2-3+ years building LLM-powered systems in production, not prototypes: agent/orchestration frameworks (LangChain / LlamaIndex / homegrown), structured tool calling, prompt versioning and evaluation.

· Structured-output engineering at production grade - generated tests are code artifacts: schema enforcement, output validation, and repair loops are the daily job.

· Demonstrated QE domain depth: has built or owned test automation architecture - framework design, CI quality gates, flaky-test economics - enough to encode that judgment into agent behavior.

· A platform engineer who has never owned a test suite will build QE agents that generate garbage confidently.

· One major LLM provider at scale - AWS Bedrock strongly preferred - with real operational scars: rate limits, latency variance, provider failover, version drift.

· AWS + Kubernetes deployment depth (services run on Client Kubernetes Service).

· Fluent in AI-assisted development workflows - Copilot-class tooling is the expected daily working mode.

Nice-to-Have

· LLM evaluation engineering: golden datasets, LLM-as-judge calibration, mutation testing or fault injection to validate generated-test quality.

· MCP tool integrations; SSE/WebSocket streaming for agentic responses.

· Vector stores and retrieval in production (OpenSearch / pgvector / Pinecone or equivalent).

· Multi-tenant enterprise platforms under strict security/compliance; fintech background.

· Open-source / inner-source contribution record.

About the company

Since our inception in 2013, Alchemy has been dedicated to reshaping organizational performance through innovative IT services. With a vision to empower businesses seeking a transformative edge, we’ve positioned ourselves at the forefront of digitization and software modernization.

Our name reflects our mission: to transmute technology into gold-standard solutions for our esteemed clients. We proudly serve a diverse range of sectors, including IT and ITES, BFSI, Telecom and Media, Automotive, Manufacturing, Energy, Oil and Gas, Real Estate, Retail, Healthcare, and more.

With a global footprint spanning the USA, India, Europe, Canada, Singapore, Japan, and parts of Central and West Africa, we harness a unique blend of competencies, frameworks, and cutting-edge technologies. Together, we drive growth and innovation across industries, helping organizations turn their visions into reality.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

2:08 min

Essential engineering roles in the generative AI space

Mary Grygleski Mary Grygleski · LIVE

1:52 min

Structuring and scaling the backend engineering team

Stefan Lingler Stefan Lingler +1 · Coffee With Developers

3:15 min

Reversing the caching model for artifact delivery

Thijs Feryn Thijs Feryn · World Congress 2026 Europe

51 sec

Configuring live browser and WebSocket connection endpoints

Suchitra Swain Suchitra Swain · World Congress 2026 Europe

2:36 min

Managing new AI workloads for non-technical employees

Michael Coté Michael Coté · World Congress 2026 Europe

1:12 min

Choosing TypeScript for complex backend applications

Maximilian Otto Maximilian Otto · World Congress 2024

Videos

See all

Related articles

See all