Staff AI QA Engineer, Kaseya Careers, IE

WHAT YOU’LL DO:  

Join the dynamic team at Kaseya as a Staff AI QA Engineer. Ensure the reliability, safety, and ethical integrity of our AI platform by crafting advanced testing strategies and automation frameworks tailored for adaptive, large-scale agentic systems and complex integrations. You’ll play a critical role in validating AI behavior, securing data pipelines, and enforcing quality standards across distributed systems. Your work will help ensure our intelligent agents operate with precision, fairness, and resilience in real-world environments.

WHAT WE ARE LOOKING FOR:  

We’re looking for a quality-focused engineer who can think critically about testing complex, adaptive systems. You’re a strong communicator who can clearly articulate quality risks and testing strategies to both technical and non-technical stakeholders. You thrive in collaborative environments, working closely with developers, data scientists, and product managers, and you bring leadership in QA best practices, mentoring, and ethical testing.

THE SCHEDULE: 
This position is 100% remote.

ESSENTIAL DUTIES AND RESPONSIBILITIES: 

• Design and implement comprehensive test strategies for AI agents, platform services, and integrations
• Develop and maintain automated test frameworks for functional, performance, and security testing
• Validate AI model behavior, including prompt outputs, bias detection, and explainability
• Conduct rigorous testing of distributed systems, microservices, and real-time data pipelines
• Ensure data quality and integrity across training, inference, and integration workflows
• Collaborate with engineering and research teams to define quality gates and ethical testing standards
• Drive continuous testing practices in agile development cycles

WHAT YOU’LL BRING:

• 5–7+ years of experience in QA or Test Engineering
• Mastery of test automation frameworks (e.g., PyTest, Playwright, Selenium, RestAssured)
• Strong API testing skills (manual and automated)
• Experience with performance, load, and high-volume stress testing (e.g., JMeter, k6s, Gatling)
• Deep understanding of AI/ML model evaluation metrics and methodologies
• Experience testing distributed systems and microservices architectures
• Proficiency in data quality validation for AI training and inference workflows
• Experience with contract testing for microservices
• Ability to develop test strategies for non-deterministic and adaptive behaviors in AI systems (e.g., prompt validation, guardrails, bias detection)
• Familiarity with AI model testing tools (e.g., CheckList, Giskard, Deepchecks)
• Experience with test planning, execution, and reporting in agile environments
• Bachelor’s degree in Computer Science, Software Engineering, or a related field
• Preferred: Certifications in QA Automation, AI/ML Testing, or Ethical AI (e.g., ISTQB, Certified Ethical Emerging Technologist)
• Additional training in AI safety, fairness, or model validation is a plus

Nice to Have:

• Experience with AI red teaming and adversarial testing techniques
• Explainability testing and validation for XAI methods (e.g., LIME, SHAP)
• Usability testing for AI-driven UIs and agent interactions
• Familiarity with data privacy testing techniques (e.g., differential privacy validation)
• Security testing for AI models (e.g., model inversion, membership inference, poisoning attacks)
• Experience testing AI ethics, fairness, and bias
• Knowledge of formal verification methods for AI
• Experience building synthetic data generation pipelines for testing