QA Engineering Manager – AI Systems

QA Engineering Manager – AI Systems

Healthcare AI · Playwright Automation · LLM Evaluation · Clinical Safety Validation

ABOUT THE ROLE

We're building the next generation of AI Quality Engineering for healthcare.

This is a capability-building role, not a maintenance role. Rather than maintaining an existing QA organization, you'll define, build, and lead a Quality Engineering function for AI-powered healthcare products from the ground up — establishing the standards, processes, automation, and AI validation practices that will support our platforms as they scale over the next several years.

This is a player–coach role. You are expected to provide technical leadership while remaining hands-on with automation, AI validation, and quality engineering activities when needed. We are open to candidates currently titled QA Manager, QA Architect, or Senior QA Engineer — what matters is the ability to build and lead this function.

THE MISSION

We are not looking for someone to manage a test backlog. We are looking for someone to build and own an AI Quality Engineering capability that will grow with the organization over the next 2–3 years.

Traditional software QA remains essential. But our long-term differentiator is AI quality — and this role will own it end to end.

CORE OWNERSHIP

• End-to-end software quality across web applications and integrated platform releases

• AI evaluation and validation — relevancy, faithfulness, contextual accuracy, hallucination detection, bias

• Clinical reasoning and safety validation for AI-generated treatment suggestions and clinical decision support

• Hallucination and guardrail testing for AI-driven clinical workflows

• AI regression testing as models, prompts, and workflows evolve

• Automation engineering using Playwright and JavaScript/TypeScript with full CI/CD integration

• Release readiness governance — QA release readiness assessments, risk registers, and sign-off processes

• Design and evolution of scalable AI evaluation frameworks and quality processes

• Quality KPIs, engineering dashboards, defect leakage trends, automation maturity, and executive reporting

• AI quality governance, evaluation standards, and engineering best practices

KEY RESPONSIBILITIES

Organizational leadership

• Build and scale the AI Quality Engineering function — define team structure, career paths, and engineering standards

• Hire, mentor, and develop QA engineers; lead capacity planning and resource allocation

• Establish engineering hiring standards, interview frameworks, and onboarding processes for the QA organization

• Drive consistent quality standards and governance across internal teams, client teams, and external delivery partners

• Represent the organization during client discussions, executive reviews, cross-vendor initiatives, technical escalations, and release governance meetings

• Provide independent quality recommendations and risk assessments to internal leadership and clients, even under demanding delivery timelines

Strategic responsibilities

• Define and execute a multi-year AI Quality Engineering roadmap aligned with product strategy, engineering maturity, and business growth

• Recommend tooling, frameworks, quality standards, and engineering practices to continuously improve AI Quality Engineering

• Partner with Engineering and Product leadership on quality strategy

• Evaluate and introduce emerging AI testing methodologies and tools

• Drive continuous innovation in AI validation practices

• Introduce modern quality engineering practices across the organization

AI and clinical quality engineering

• Design and implement AI evaluation frameworks using DeepEval, RAGAS, LangSmith, Promptfoo, or equivalent

• Validate GenAI/LLM-based clinical features including treatment guardrails, medication recommendations, and clinical task automation

• Develop AI regression test suites that detect quality degradation as models, prompts, and data evolve

• Define clinical safety testing protocols in collaboration with clinical SMEs and product stakeholders

• Build evaluation pipelines that scale across AI model updates and workflow changes

Automation and engineering

• Lead automation initiatives using Playwright and JavaScript/TypeScript for UI, API, and end-to-end testing

• Establish and maintain CI/CD-integrated automation pipelines supporting continuous quality validation

• Develop automation strategies supporting platform-specific and cross-platform testing

• Define and track automation coverage metrics aligned to release risk

Quality strategy and governance

• Define QA strategies, test plans, and quality standards across multiple concurrent product streams

• Manage defect lifecycle, root cause analysis, risk assessment, and QA release readiness assessments

• Validate and assess application performance, reliability, security, usability, and compliance requirements through appropriate quality engineering activities

• Track and report QA metrics, automation coverage, defect leakage trends, AI evaluation maturity, and operational efficiency

• Drive continuous improvement across testing processes, tools, and delivery practices

SUCCESS IN THE FIRST 12 MONTHS

• Establish AI Quality Engineering standards across the organization

• Increase automation coverage and reduce escaped defects

• Implement AI evaluation pipelines for clinical features

• Improve release predictability through governance and readiness processes

• Build and mentor a high-performing QA team

• Establish AI quality scorecards and executive dashboards adopted across engineering leadership

• Improve customer confidence in product quality through predictable releases and transparent quality governance

• Become the trusted quality advisor for clients and engineering leadership

REQUIRED QUALIFICATIONS

• Bachelor's degree in Computer Science, Engineering, or related field

• 8–12+ years of experience in Software Testing / Quality Engineering with demonstrated leadership experience

• Proven track record of building, scaling, or transforming a Quality Engineering function

• Strong hands-on experience building automation frameworks using Playwright (preferred) and JavaScript/TypeScript

• Practical, hands-on understanding of LLMs, GenAI workflows, prompt validation, and AI evaluation concepts

• Experience with LLM evaluation frameworks such as DeepEval, RAGAS, LangSmith, or Promptfoo

• Experience testing AI-powered applications in Healthcare AI, HealthTech, MedTech, Life Sciences, or other regulated industries strongly preferred

• Excellent client-facing communication and stakeholder management skills, including executive-level reporting

• Ability to communicate effectively with engineering teams, product leadership, executive stakeholders, and clients

• Experience coordinating quality across internal teams, client teams, and third-party vendors

• Experience leading geographically distributed engineering teams

• Ability to design quality processes from the ground up in fast-paced, high-growth environments

• Experience in Agile/Scrum methodologies

PREFERRED QUALIFICATIONS

• Experience with clinical safety testing, HIPAA compliance contexts, or healthcare AI validation

• Exposure to cloud platforms — AWS, Azure, or GCP

• Knowledge of performance testing and security testing practices

• Experience building AI evaluation pipelines integrated into CI/CD workflows

• Prior experience in a consulting, technology services, or multi-client delivery environment

WHAT WE'RE LOOKING FOR

• Builder mindset — you want to architect and own a function, not just manage a backlog

• AI quality depth — you understand why hallucination in a dosage recommendation is categorically different from a UI bug

• Technical-commercial balance — you can lead a team, represent the organization in demanding client and executive discussions, and still write a Playwright test when needed

• Healthcare awareness — you appreciate the stakes of clinical AI and bring appropriate rigor to validation

• Delivery focus — you operate in a fast-moving environment without sacrificing quality standards

WHY THIS ROLE MATTERS

This role is more than a QA leadership position.

You will help define how AI-powered healthcare systems are validated, governed, and trusted. The practices you establish will influence product quality, clinical safety, and engineering excellence as our organization scales. If you enjoy building functions, solving complex technical challenges, and shaping the future of AI Quality Engineering in healthcare, we'd like to hear from you.

This role operates in a multi-platform, multi-vendor healthcare AI environment. Candidates with experience coordinating quality across distributed and cross-functional delivery teams are particularly encouraged to apply.