> Markdown version of [/jobs/ext/2203514-sr-llm-python-test-engineer](https://www.wearedevelopers.com/jobs/ext/2203514-sr-llm-python-test-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Sr. LLM / Python Test Engineer - **Company:** BCI-IT, Inc. - **Location:** North Arlington, NJ, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Agile Methodology, Artificial Intelligence, Amazon Web Services, Software Applications, Automation of Tests, Microsoft Azure, Software Quality, Continuous Integration, Database Testing, DevOps, Monitoring of Systems, Apache JMeter, Python (Programming Language), Load Testing, Microsoft UI Automation, NoSQL, Object-Oriented Software Development, Systems Development Life Cycle, Standard Sql, Software Safety, Selenium, SQL Databases, Test Data, Datadog, Cloud Platform System, Large Language Models, Prompt Engineering, Model Validation, Generative AI, Backend, Pandas, Pytest, Playwright, Virtual Agents, Cloudwatch, Restful APIs, Splunk, New Relic (SaaS), Api Management, SDET, Microservices - **Published:** August 23, 2026 - **Apply:** https://www.disabledperson.com/jobs/74440117-sr-llm-python-test-engineer ## About the Role * 8+ years of software quality engineering, test automation, or SDET experience in Agile development environments. * Strong proficiency in Python, Pytest, Pandas, and object-oriented programming principles. * 6+ years of experience with UI automation frameworks such as Playwright or Selenium. * 6+ years of experience testing RESTful APIs and microservices. * Hands-on experience with CI/CD pipelines and DevOps practices. * Experience working with SQL and NoSQL databases. * Experience with AWS or other cloud-based platforms. * Understanding prompt engineering, prompt validation, hallucination detection, bias testing, guardrail validation, and AI output evaluation. * Knowledge of LLMs, Retrieval-Augmented Generation (RAG), AI agents, MCP, and AI-assisted workflows. * Ability to develop QA strategies, test frameworks, and automation solutions for AI-driven applications. * Analytical, problem-solving mindset with continuous learning and quality-first engineering habits., * Experience testing Generative AI and Agentic AI applications. * Ability to validate AI outputs for accuracy, reliability, and business relevance. * Knowledge of AI agent workflows, prompt testing, hallucination detection, and model evaluation. * Understanding AI safety, guardrails, security, compliance, and responsible AI practices. * Experience with LangChain, LangGraph, MCP, OpenAI, Anthropic, Azure OpenAI, Kiro or Amazon Bedrock. * Familiarity with AI governance, risk management, and AI quality metrics. ## Description BCI is seeking a self-driven Test Engineer with experience testing AI-powered applications, Generative AI solutions, and Agentic AI systems. This role is responsible for designing and executing quality engineering activities, developing automation solutions, validating APIs and integrations, supporting CI/CD and release readiness, and promoting a quality-first engineering mindset across traditional and AI-driven platforms. Role is hybrid. Must already be local to the NY/NJ area. No relocation. Must be a US Citizen, GC, Perm Resident, EAD - OPT/STEM. No H1 or sponsorship. No C2C or subcontractors., * Design and execute functional, integration, system, regression, and automation testing activities. * Apply Shift-Left testing practices by participating in requirement reviews and defining test scenarios early in the SDLC. * Build, maintain, and enhance UI automation frameworks using Python, PyTest, and Playwright. * Perform manual and automated API testing using industry-standard tools and frameworks. * Collaborate with developers, product owners, business analysts, and DevOps teams to ensure product quality. * Work with SQL and NoSQL databases for backend validation and data testing. * Support CI/CD pipelines and ensure automated tests are integrated into deployment processes. * Create test plans, test cases, test data strategies, and quality metrics dashboards. * Manage defect lifecycle activities and provide clear reporting of quality risks. * Develop AI test datasets, benchmarks, evaluation frameworks, and automated validation approaches. * Experience with performance and load testing using JMeter or similar tools. * Knowledge of AI observability, model monitoring, and evaluation platforms. * Experience with monitoring tools such as New Relic, Datadog, Splunk, or CloudWatch. ## Related Videos - [Our journey with Spring Boot in a microservice architecture](https://www.wearedevelopers.com/videos/511-our-journey-with-spring-boot-in-a-microservice-architecture) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems) - [NoSQL Data Modeling for Front-end Developers](https://www.wearedevelopers.com/videos/297-nosql-data-modeling-for-front-end-developers) - [How to Avoid LLM Pitfalls - Mete Atamel and Guillaume Laforge](https://www.wearedevelopers.com/videos/1328-how-to-avoid-llm-pitfalls-mete-atamel-and-guillaume-laforge) - [Tomorrow's cloud data platforms - fully managed database-as-a-service (DBaaS)](https://www.wearedevelopers.com/videos/254-tomorrow-s-cloud-data-platforms-fully-managed-database-as-a-service-dbaas) ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this)