AI Testing with Generative AI using Python Training

Curriculum:

Module 1: Introduction to AI & GenAI Testing

• What is Generative AI?
• LLM-based applications overview
• Why traditional testing fails for GenAI
• Deterministic vs probabilistic systems
• New testing paradigms for AI systems

Module 2: Fundamentals of LLM Architecture

• How Large Language Models work (high level)
• Tokens, embeddings, context windows
• Prompt → Model → Response lifecycle
• Temperature, top-k, top-p and randomness

Module 3: GenAI Testing Challenges

• Non-deterministic outputs
• Hallucinations
• Bias and toxicity
• Prompt sensitivity
• Data leakage and security risks

Module 4: Python Foundations for AI Testing

• Python testing ecosystem overview
• Using pytest for AI tests
• Structuring AI test suites
• Logging and test result analysis

Hands-On:

• Create Python test cases for AI responses

Module 5: Prompt Testing & Validation

• Prompt engineering basics for testers
• Prompt robustness testing
• Prompt regression testing
• Prompt versioning strategies

Hands-On:

• Prompt mutation testing
• Prompt comparison tests

Module 6: Testing LLM APIs with Python

• Testing GenAI APIs (OpenAI / Azure OpenAI / OSS LLMs – conceptual)
• Request/response validation
• Latency and timeout testing
• Error and fallback testing

Hands-On:

• Python test scripts for LLM API responses

Module 7: AI Output Quality Evaluation

• What makes a “good” AI response?
• Relevance, correctness, completeness
• Semantic similarity testing
• Ground truth vs AI output comparison

Module 8: Hallucination & Safety Testing

• What are hallucinations?
• Hallucination detection techniques
• Fact-based validation
• Toxicity, bias, and safety testing
• Ethical AI testing considerations

Hands-On:

• Hallucination detection tests in Python

Module 9: Introduction to RAG Systems

• What is Retrieval-Augmented Generation (RAG)?
• RAG architecture:
• Ingestion
• Embeddings
• Vector databases
• Retrieval
• Generation
• Why RAG needs specialized testing

Module 10: RAG Testing Strategy

• What to test in RAG pipelines
• Retrieval accuracy vs generation quality
• Chunking and embedding validation
• Context relevance testing

Module 11: RAGAS Framework – Deep Dive

• What is RAGAS?
• RAGAS architecture and metrics
• Core evaluation metrics:
• Faithfulness
• Answer relevance
• Context precision
• Context recall

Hands-On:

• Implement RAGAS evaluation using Python
• Measure RAG quality automatically

Module 12: Automating RAG Evaluation

• Building automated RAG evaluation pipelines
• Integrating RAGAS with Python tests
• CI/CD integration concepts
• Regression testing for RAG systems

Hands-On:

• End-to-end automated RAG testing workflow

Module 13: Performance & Load Testing for GenAI

• Latency testing for LLMs
• Throughput testing
• Cost-aware testing
• Prompt and token optimization testing

Module 14: Monitoring & Observability for AI Testing

• AI test metrics
• Drift detection concepts
• Feedback loop testing
• Continuous evaluation strategies

Module 15: Real-World Case Studies

• Testing a chatbot application
• Testing a GenAI-powered search assistant
• Testing a document-based RAG system