(Master AI & LLM fundamentals, prompt engineering, LLM evaluation with DeepEval & LLM-as-a-Judge, RAG testing, Python/PyTest automation, AI agent & workflow testing, AI security, and CI/CD quality gates with GitHub Actions)
A complete, hands-on path from AI and LLM foundations through prompt engineering, LLM evaluation, Python & PyTest automation, DeepEval, RAG testing, AI agent testing, AI security and CI/CD integration with GitHub Actions.
AI vs. traditional software, tokens & context windows, LLM parameters, working with LLM APIs, and deterministic vs. probabilistic testing.
Prompt architecture, function & tool calling, embeddings & semantic similarity, vector databases, RAG flow, and AI agents & agentic workflows.
The test oracle problem, accuracy & relevancy testing, hallucination detection, bias & toxicity checks, privacy & data leakage, and core AI quality dimensions.
Prompt patterns, prompt test design, robustness testing, structured output validation, prompt versioning and building an LLM test dataset.
Reference-based vs. reference-free evaluation, judge prompts & rubrics, semantic similarity thresholds, evaluator bias, and building a custom AI evaluator.
Python essentials, REST APIs, secrets management, PyTest fundamentals, data-driven testing and modular LLM test framework architecture.
DeepEval setup, LLMTestCase design, relevancy & faithfulness metrics, hallucination metrics, custom judge prompts, and a hands-on evaluation lab.
Chunking & ingestion testing, embedding & vector DB retrieval validation, context precision/recall/relevancy, and end-to-end RAG regression testing.
Agent decision loops, function/tool calling validation, multi-step workflow testing, memory & context retention, and failure/recovery testing.
Prompt injection, jailbreak scenarios, PII & system prompt leakage checks, toxicity & bias scenarios, and input robustness testing.
Manual & automated chatbot testing, strategies for non-deterministic responses, simulating conversations, and streaming/timeout validation.
Integrating AI tests into CI, test data management, automated reporting, running PyTest & DeepEval in GitHub Actions, and managing secrets & flakiness.
Production monitoring, observability & drift tracking, plus a capstone project: a complete automated AI Support Assistant test suite with RAG and function calling.
Whether you're starting out or upskilling, this course is designed to take you from the basics to a confident AI Testing & Quality Engineering professional.
Start from AI and LLM basics and build up to a production-grade AI testing, evaluation and automation skill set across 13 modules — fully guided.
Every module is anchored by real, hands-on labs — building automated LLM test suites with Python, PyTest and DeepEval, not just slides and theory.
Learn prompt injection and jailbreak testing, PII & system prompt leakage checks, and how to validate AI agents and multi-step tool-calling workflows.
Prompt engineering, LLM-as-a-Judge evaluation, DeepEval, RAG testing, Python/PyTest automation and GitHub Actions CI/CD — the full modern AI QA stack.
Every live session is recorded and shared, with a full year of access so you can revisit any topic at your own pace.
Build a complete automated AI Support Assistant test suite with RAG and function calling using Python, PyTest, DeepEval and GitHub Actions.
Experience the trainer's hands-on teaching style before enrolling — completely free, no commitment required.
| Days | India Timings (IST) | US Timings (EST) | UK Timings (BST) |
|---|---|---|---|
| Demo | 19th September @ 7:30 PM - 8:30 PM (IST) | 19th September @ 10:00 AM - 11:00 AM (EST) | 19th September @ 3:00 PM - 4:00 PM (BST) |
| Day 1 | 20th September @ 7:30 PM - 8:30 PM (IST) | 20th September @ 10:00 AM - 11:00 AM (EST) | 20th September @ 3:00 PM - 4:00 PM (BST) |
| Day 2 | 26th September @ 7:30 PM - 9:30 PM (IST) | 26th September @ 10:00 AM - 12:00 PM (EST) | 26th September @ 3:00 PM - 5:00 PM (BST) |
| Day 3 | 27th September @ 7:30 PM - 9:30 PM (IST) | 27th September @ 10:00 AM - 12:00 PM (EST) | 27th September @ 3:00 PM - 5:00 PM (BST) |
| Every Saturday & Sunday 2 hour the sessions will go on | Every Saturday & Sunday 2 hour the sessions will go on | Every Saturday & Sunday 2 hour the sessions will go on |
A comprehensive, progressive curriculum — from AI/LLM fundamentals and prompt engineering to LLM evaluation, Python/PyTest automation with DeepEval, RAG testing, AI agent testing, AI security, chatbot UI testing and GitHub Actions CI/CD, ending in a real-world capstone project.
| Topic | What You'll Learn |
|---|---|
| AI & QA Role | Evolution of software testing, AI vs. traditional software, and where QA fits into modern AI project lifecycles |
| AI, ML, DL & GenAI | Core relationships, capabilities and boundaries across AI domains |
| Intro to LLMs | How LLMs generate responses, training vs. inference, and popular foundation models (GPT, Claude, Gemini, Llama) |
| Tokens & Context Windows | Input/output tokenization, cost models, context window limits and testing implications |
| LLM Parameters | Temperature, Top-P, max tokens, frequency/presence penalties and their impact on test repeatability |
| Working with LLM APIs | API architecture, request payloads, response structure, message roles and JSON mode formatting |
| Traditional vs. AI Testing | Deterministic vs. probabilistic execution, expected outputs vs. acceptable semantic outputs |
| Topic | What You'll Learn |
|---|---|
| Anatomy of an LLM App | Input processing, prompt templating, model invocation and response formatting layers |
| Prompt Architecture | System prompts, dynamic user prompts, message history and constraint specification |
| LLM App Workflows | Chatbot structures, search assistants, content generators and customer service automation flows |
| Function & Tool Calling | How LLMs select tools, formulate JSON arguments, execute actions and process return values |
| Embeddings & Semantic Similarity | Text-to-vector representation, high-dimensional spaces and calculating semantic distance |
| Vector Databases & Retrieval | Purpose of vector DBs, indexing strategies, similarity search and metadata filtering |
| RAG Architecture | End-to-end Retrieval-Augmented Generation flow — Retriever, Context, Generator |
| AI Agents & Agentic Workflows | Planning mechanisms, tool execution loops, multi-turn memory and autonomous task execution |
| Topic | What You'll Learn |
|---|---|
| Why AI Testing Is Different | The test oracle problem, non-determinism, subtle drift and evaluating open-ended text |
| Types of LLM Testing | Functional validation, prompt robustness, response quality, safety checks and performance tracking |
| Accuracy & Factual Testing | Ground truth comparison, reference answers and detecting factual discrepancies |
| Answer Relevancy | Measuring query-response alignment and handling off-topic or evasive answers |
| Hallucination Detection | Defining ungrounded statements, identifying fabricated facts and setting detection boundaries |
| Bias, Toxicity & Fairness | Identifying demographic bias, offensive outputs, stereotyping and unsafe content generation |
| Privacy & Data Leakage | PII exposure, training data memorization, system prompt leakage and secure boundaries |
| Core AI Quality Dimensions | Accuracy, relevance, faithfulness, safety, robustness, latency and token cost |
| Topic | What You'll Learn |
|---|---|
| Prompt Engineering Basics | Structuring clear instructions, contextual guidelines, output constraints and formatting rules |
| Common Prompt Patterns | Zero-shot, few-shot, role prompting, chain-of-thought and structured formatting strategies |
| Prompt Test Design | Positive test suites, negative inputs, edge cases, boundary parameters and ambiguous prompts |
| Prompt Robustness Testing | Evaluating impact of wording tweaks, typos, sentence reshuffling and length variations |
| Structured Output Testing | Validating JSON responses, key presence, schema types and error handling for malformed JSON |
| Prompt Versioning | Managing prompt iterations, tracking degradation and running regression checks across updates |
| LLM Test Dataset | Designing evaluation datasets with inputs, expected behavior, context and scoring criteria |
| Topic | What You'll Learn |
|---|---|
| The LLM Evaluation Problem | Why exact string matching fails for open-ended text and how semantic evaluation solves it |
| Reference-Based vs. Reference-Free | Evaluating against ground-truth answers vs. context-only evaluation |
| LLM-as-a-Judge Techniques | Crafting judge prompts, evaluation personas, rubrics and structured explanations |
| Evaluation Rubrics | Building numeric scale rubrics, pass/fail thresholds and multi-criteria scoring models |
| Semantic Similarity & Thresholds | Cosine similarity, embedding comparison and setting acceptance thresholds |
| Evaluator Bias & Consistency | Handling position bias, self-preference bias, temperature controls and calibration |
| Custom AI Evaluator & Lab | Step-by-step Python evaluation module returning scores, reasoning and status |
| Topic | What You'll Learn |
|---|---|
| Python Environment Setup | Project structures, virtual environments, pip dependencies and clean code principles |
| Python Essentials for AI Testing | Functions, dictionaries, lists, module imports, error handling and JSON manipulation |
| Working with REST APIs | The requests library, authentication headers, retry policies and timeout handling |
| Secrets Management | Environment variables, .env file handling, secure key usage and preventing credential leaks |
| PyTest Fundamentals | Writing test cases, assertions, parameterization, fixtures and setup/teardown practices |
| Automated LLM Test Cases | Integrating API clients directly into PyTest scripts with evaluation assertions |
| Data-Driven Testing | Reading test datasets from JSON/CSV and running parameterized PyTest runs |
| Modular Test Framework | Separating test cases, API clients, evaluation rubrics, data loaders and configuration |
| Topic | What You'll Learn |
|---|---|
| Introduction to DeepEval | Installation, configuration, architecture and core setup within a Python workspace |
| Structuring Test Cases | Creating LLMTestCase instances with inputs, actual outputs, expected outputs and retrieval context |
| Answer Relevancy Evaluation | Implementing relevancy metrics, configuring thresholds and interpreting failure reasons |
| Faithfulness & Groundedness | Evaluating responses against supplied context to detect unsupported assertions |
| Hallucination Metrics | Automated hallucination testing using specialized DeepEval metric suites |
| Custom Metrics & Judge Prompts | Writing domain-specific metrics using custom evaluation criteria and judge prompts |
| Multi-Turn Conversation Testing | Testing context retention, multi-turn history tracking and dialogue flow |
| Reports & Hands-on Lab | Exporting test run results and metric breakdowns · Lab: build an automated LLM evaluation suite with PyTest & DeepEval |
| Topic | What You'll Learn |
|---|---|
| RAG Test Strategy | Identifying risk areas in chunking, indexing, retrieval, context injection and generation |
| Ingestion & Chunking Tests | Chunk size strategies, overlap behavior, boundary splits and metadata preservation |
| Embedding & Vector DB Validation | Evaluating top-K search results, similarity scores and metadata filtering accuracy |
| Precision, Recall & Relevancy | Measuring whether the retriever pulls correct and complete documents while excluding noise |
| End-to-End RAG Suite | Constructing full pipeline tests validating query-to-answer pathways |
| RAG Regression Testing | Detecting degradation after knowledge base updates, embedding changes or prompt modifications |
| Topic | What You'll Learn |
|---|---|
| AI Agent Testing Strategies | Testing autonomous decision loops, goal decomposition, step planning and memory integrity |
| Function & Tool Calling Validation | Validating tool selection logic, required/optional argument construction and execution handling |
| Multi-Step Workflow Testing | Automating tests for multi-tool dependencies, intermediate step validation and task completion |
| Agent Memory & Context Retention | Validating short-term conversation context, state updates and avoiding hallucinated history |
| Failure & Recovery Testing | Testing resilience against broken API tools, malformed arguments, timeouts and fallback handling |
| Hands-on Lab | Create an automated test suite for a multi-step AI agent application |
| Topic | What You'll Learn |
|---|---|
| AI Security Testing Fundamentals | AI-specific security risks vs. traditional web application vulnerabilities |
| Prompt Injection | Direct and indirect prompt injection concepts and basic detection techniques |
| Basic Jailbreak Scenarios | Common roleplay/override attack patterns and validating system safety guardrails |
| PII & System Prompt Leakage | Testing for accidental disclosure of internal instructions, system prompts or personal data |
| Toxicity, Harm & Bias Scenarios | Running basic safety benchmarks to verify refusal mechanisms for inappropriate requests |
| Input Robustness | Testing system stability against long text strings, non-ASCII characters and formatting noise |
| Topic | What You'll Learn |
|---|---|
| AI Application Interfaces | Web chat widgets, assistant portals and conversational design patterns |
| Manual & Automated Chatbot Testing | Core manual test strategies, message input verification and element identification |
| Non-Deterministic UI Responses | Structural assertions, essential keyword presence, regex matching and response shape checks |
| Simulating User Conversations | Testing multi-turn UI flows, context clearing, session resets and user action triggers |
| Timeouts & Streaming Responses | Handling typing indicators, chunked token rendering, long delays and network retries |
| API + Frontend Validation | Hybrid testing approach coupling frontend UI checks with backend API evaluations |
| Topic | What You'll Learn |
|---|---|
| AI Tests in Continuous Integration | Designing automated quality pipelines triggered on code or prompt changes |
| Test Data Management | Version-controlling test datasets, prompt templates and evaluation datasets with source code |
| Automated Reporting & Thresholds | Defining pass/fail criteria based on aggregated evaluation scores and quality benchmarks |
| PyTest & DeepEval in GitHub Actions | Constructing workflow YAML files to run automated AI test suites on push/pull requests |
| Managing Secrets Securely | Configuring GitHub Secrets for API keys, target endpoints and environment variables |
| AI Quality Gates | Blocking pull requests or deployments when evaluation scores drop below target thresholds |
| Costs & Flakiness Management | Sampling strategies, cost-effective evaluation models, retries and score variance tolerance |
| Topic | What You'll Learn |
|---|---|
| Pre-Production vs. Production | Differences between pre-release static evaluation and live continuous evaluation |
| Observability Concepts | Logging prompts/responses, tracing tool execution chains, token consumption and latency |
| Tracking Drift & Degradation | Detecting concept drift, model behavior shifts, negative feedback and hallucination trends |
| Capstone Project | Build a production-ready, fully automated test framework using Python, PyTest, DeepEval and GitHub Actions for an AI Customer Support Assistant with RAG and Function Calling |
Capstone Deliverables: AI Test Strategy & Scenario Document · Data-driven Prompt & RAG Test Suites · DeepEval Metric Evaluations (Relevancy, Groundedness, Hallucination) · Agent Function Calling Test Suite · GitHub Actions CI/CD Workflow with Quality Gates · Final AI Quality & Security Assessment Report.
Passionate AI & LLM Testing professional focused on helping QA Engineers and Automation Professionals build practical skills in testing and evaluating modern AI applications.
Tripura brings extensive experience in automation and quality engineering, with a strong focus on Python, PyTest, REST API testing, AI/LLM evaluation, RAG testing, and automated quality engineering. Her training approach connects fundamental testing concepts with the challenges of modern AI systems, including non-deterministic responses, hallucinations, semantic evaluation, safety, and AI quality.
With a hands-on and example-driven teaching style, she helps learners understand how to build automated LLM test suites using Python, PyTest and DeepEval, evaluate response relevancy and groundedness, test RAG pipelines and AI agent workflows, and integrate AI quality checks into GitHub Actions CI/CD pipelines.
Feedback from learners who completed the AI Testing Course program.
The way the course builds from AI foundations up to prompt engineering and RAG testing made everything click. The AI security module was the highlight for me.
Coming from a manual testing background, the Python and automation sections took time to click, but the hands-on labs made the concepts stick. Tripura explains everything with real project context.
The prompt injection and adversarial testing module was genuinely useful — seeing how to validate AI agents and secure LLM applications gave me a real edge in interviews.
I switched from a functional testing role and was worried about keeping up, but the daily hands-on labs on prompt validation and RAG testing built my confidence step by step.
The live sessions on LLM evaluation and hallucination detection were exactly what I needed for my current job. Doubt-clearing over WhatsApp between classes was a big help too.
Solid, structured curriculum — RAG and LLM evaluation concepts were explained with real testing examples instead of just slides. The certificate helped me negotiate a better offer.
Every participant who successfully completes the training receives a Course Completion Certificate from Isha Training Solutions.
Sample certificate — your name will be printed upon completion
To maintain the quality of our training and ensure a smooth learning experience for all participants, we do not allow batch repetition or switching between courses.
Moving from one course to another, or shifting from one trainer to another, is not possible once a batch has started. Changing batches or trainers in any form is strictly not permitted.
We request all learners to attend the scheduled sessions regularly and make the most of their learning journey. Thank you for your understanding and continued support.
