Quality Assurance Engineer (Remote)

Who We Are - Motivated by Purpose. Powered by Clinical Expertise.

Founded in 1983, we’re a clinically-driven, tech-enabled utilization management company offering expert clinical reviews, regulatory guidance, and actionable insights to healthcare organizations.

Excellence starts with our people.

WE OFFER

• A competitive compensation package

• Benefits include healthcare, vision, and dental insurance

• A generous 401(k) match

• Paid vacation, PTO, and holidays

• Growth and training opportunities

• An award-winning remote work environment

POSITION OVERVIEW

Quality Assurance Engineer is a corporate function within the Engineering organization responsible for the quality, reliability, and release-readiness of software products built on modern technologies, including cloud-native applications hosted on Microsoft Azure and AI/ML-enabled features.

Roles

• Primary role is testing software applications created or updated by the Application Development team.

• Conduct structured peer and code reviews of test artifacts to ensure adherence to standards and best practices.

• Design, create, and maintain realistic and compliant test data sets, including synthetic data for healthcare (PHI-safe) scenarios.

• Design and execute validation strategies for AI/ML and Retrieval-Augmented Generation (RAG) features, including model quality, grounding, and safety evaluation.

Major Responsibilities or Assigned Duties

• Participate in the test management lifecycle, including test planning, execution tracking, defect management and reporting.

• Provide input on release readiness criteria to Engineering and Product stakeholders.

• Maintain the regression suite after every release.

• Develop and maintain automated test suites in Python (e.g., pytest) and JavaScript/Node.js, integrated into CI/CD pipelines.

• Write and review complex SQL queries, joins, and aggregations to validate data accuracy, completeness, and reconciliation across systems.

• Validate data flows across source systems, transformation layers, and reporting/consumption layers.

• Ensure test data supports both functional automation and AI/ML / RAG evaluation needs.

• Design and execute test strategies for AI/ML and Retrieval-Augmented Generation (RAG) pipelines and AI-assisted features, including retrieval accuracy, grounding, hallucination, output quality, bias/safety, and drift (regression) checks.

• Build and maintain automated evaluation harnesses and datasets for AI/ML features, including prompt/response scoring and metric tracking.

• Validate applications, APIs, and test environments deployed on on-prem data center, Microsoft Azure, including Azure-hosted services, data stores, and CI/CD pipelines.

• Read and interpret React / Node.js / Python / ColdFusion source code, pull requests, and technical designs to identify testable conditions, edge cases, and regression risk.

• Write detailed test cases and test plans that trace directly to code paths, business rules, and acceptance criteria.

• Partner with developers during code review to flag insufficient test coverage, untested branches, and high-risk changes before merge.

• Perform post-deployment validation to confirm application, API, and AI feature stability in production.

• •Investigate production defects and data issues, perform root cause analysis, and drive resolution with Engineering.

Skills and Experience

• 5+ years of experience in automated software testing

• 5+ years of experience in manual software testing

• 5+ years of experience in component-level testing

• 5+ years of experience in end-to-end testing

• Proficiency in  SQL

• Proficiency in  React

• Proficiency in  Node.js

• Proficiency in  Python source code in GitLab

• Proficiency in  testing Artificial Intelligence (AI)

•  Proficiency in  testing Machine Learning (ML)

• Proficiency in  evaluating Artificial Intelligence (AI)

•  Proficiency in  evaluating Machine Learning (ML)

• Proficiency in  testing Retrieval-Augmented Generation (RAG)

• Proficiency in  testing generative AI features

• Proficiency in  testing workloads on Microsoft Azure

Education

• High School Diploma required

• Bachelor’s degree (Advantage)