AI Course Assessment Generator

This application generates learning objectives and multiple-choice questions for AI course materials based on uploaded content files. It uses OpenAI's language models to create high-quality educational assessments that adhere to specified quality standards.

Features

• Upload course materials in various formats (.vtt, .srt, .ipynb)
• Generate customizable number of learning objectives
• Create multiple-choice questions based on learning objectives
• Evaluate question quality using an LLM judge
• Save assessments to JSON format
• Track source references for each learning objective and question

Setup

1. Clone this repository
2. Install the required dependencies:

   pip install -r requirements.txt
   

3. Create a .env file in the project root with your OpenAI API key:

   OPENAI_API_KEY=your_api_key_here
   

Usage

1. Run the application:

   python app.py
   

2. Open the Gradio interface in your web browser (typically at http://127.0.0.1:7860)
3. Upload your course materials (.vtt, .srt, .ipynb files)
4. Specify the number of learning objectives to generate
5. Select the OpenAI model to use
6. Generate learning objectives
7. Review and provide feedback on the generated objectives
8. Generate multiple-choice questions based on the approved objectives
9. Review the generated questions and their quality assessments
10. The final assessment will be saved as assessment.json in the project directory

Project Structure

• app.py: Entry point for the application

Modules

• models/: Pydantic data models

  • __init__.py: Exports all models
  • learning_objectives.py: Learning objective data models
  • questions.py: Question and option data models
  • assessment.py: Assessment data models

• ui/: User interface components

  • __init__.py: Package initialization
  • app.py: Gradio UI implementation
  • content_processor.py: Processes uploaded files and extracts content
  • objective_handlers.py: Handlers for learning objective generation
  • question_handlers.py: Handlers for question generation
  • feedback_handlers.py: Handlers for feedback and regeneration
  • formatting.py: Formatting utilities for UI display
  • state.py: State management for the UI

• quiz_generator/: Quiz generation components

  • __init__.py: Package initialization
  • generator.py: Main QuizGenerator class
  • assessment.py: Assessment generation logic
  • question_generation.py: Question generation logic
  • question_improvement.py: Question quality improvement logic
  • question_ranking.py: Question ranking and grouping logic
  • feedback_questions.py: Feedback-based question generation

• learning_objective_generator/: Learning objective generation components

  • __init__.py: Package initialization
  • generator.py: Main generator class
  • base_generation.py: Base generation logic
  • enhancement.py: Enhancement logic
  • grouping_and_ranking.py: Grouping and ranking logic

• prompts/: Prompt templates and components

  • questions.py: Question generation prompts
  • incorrect_answers.py: Incorrect answer generation prompts
  • learning_objectives.py: Learning objective generation prompts

• obsolete/: Deprecated files (not used in current implementation)

• specs.md: Project specifications

• project_flow.md: Detailed description of the project architecture and workflow

Requirements

• Python 3.8+
• Gradio 4.19.2+
• Pydantic 2.8.0+
• OpenAI 1.52.0+
• nbformat 5.9.2+
• instructor 1.7.9+
• python-dotenv 1.0.0+

Install dependencies using uv (recommended):

uv venv -p 3.12
source .venv/bin/activate  # On Windows use: .venv\Scripts\activate
uv pip install -r requirements.txt

Notes

• The application uses XML-style source tags to track which file each piece of content comes from
• Questions are evaluated against quality standards to ensure they meet educational requirements
• Each question includes feedback for both correct and incorrect answers

Prompt Structure

The application's prompt system in prompts.py has been refactored into modular components for better maintainability:

• GENERAL_QUALITY_STANDARDS: Overall quality standards for all generated content
• QUESTION_SPECIFIC_QUALITY_STANDARDS: Standards specific to question generation
• CORRECT_ANSWER_SPECIFIC_QUALITY_STANDARDS: Standards for correct answer options
• INCORRECT_ANSWER_SPECIFIC_QUALITY_STANDARDS: Standards for creating plausible incorrect answers
• EXAMPLE_QUESTIONS: A collection of high-quality example questions for model guidance
• MULTIPLE_CHOICE_STANDARDS: Standards specific to multiple-choice question format
• BLOOMS_TAXONOMY_LEVELS: Educational taxonomy for different levels of learning
• ANSWER_FEEDBACK_QUALITY_STANDARDS: Standards for providing helpful feedback
• LEARNING_OBJECTIVES_PROMPT: Template for generating learning objectives
• LEARNING_OBJECTIVE_EXAMPLES: Examples of well-formulated learning objectives

These components are imported and combined in quiz_generator.py to create comprehensive prompts for different generation tasks. This modular approach makes it easier to:

1. Update individual aspects of the prompt without affecting others
2. Reuse common standards across different generation tasks
3. Maintain consistent quality across all generated content

Detailed Project Flow

Overview

This section provides a more detailed look at how the various components of the system work together to generate educational assessments.

Core Components

1. Content Processing: Handles ingestion of course materials from different file formats
2. Learning Objective Generation: Creates learning objectives from the processed content
3. Question Generation: Produces multiple-choice questions for each learning objective
4. Quality Assessment: Evaluates the generated questions for quality
5. UI Interface: Provides a Gradio-based web interface for user interaction

Application Entry Point (app.py)

• Serves as the entry point for the application
• Loads environment variables (including OpenAI API key)
• Creates and launches the Gradio UI

User Interface (ui/ module)

• Creates the Gradio interface for user interaction

• Organizes functionality into tabs:

  • File upload and learning objective generation
  • Question generation
  • Preview and export

• Key components:

  • app.py: Creates the Gradio interface and defines the UI layout
  • objective_handlers.py: Handles learning objective generation and regeneration
  • question_handlers.py: Handles question generation and regeneration
  • feedback_handlers.py: Handles user feedback and custom question generation
  • formatting.py: Formats quiz data for UI display
  • state.py: Manages state between UI components

Content Processing (ui/content_processor.py)

• ContentProcessor class processes different file types:
  • .vtt and .srt subtitle files
  • .ipynb Jupyter notebook files
• For each file, adds XML source tags to track the origin of content
• Returns structured content for further processing

Quiz Generation (quiz_generator/ module)

• QuizGenerator class is the central component that:
  • Generates learning objectives from processed content
  • Creates multiple-choice questions for each objective
  • Judges question quality
  • Saves assessments to JSON

Learning Objective Generation

1. Takes processed file contents as input
2. Combines content and creates a prompt (utilizing modular components from prompts.py)
3. Uses OpenAI's API with instructor to generate learning objectives
4. Returns structured LearningObjective objects

Question Generation

1. For each learning objective:
   • Retrieves relevant content from source files
   • Creates a prompt by combining modular components from prompts.py
   • Generates a multiple-choice question with feedback for each option
   • Returns a structured MultipleChoiceQuestion object

Data Models (models/ module)

Defines the data structures used throughout the application:

• LearningObjective: Represents a learning objective with ID, text, and source references
• MultipleChoiceOption: Represents an answer option with text, correctness flag, and feedback
• MultipleChoiceQuestion: Represents a complete question with options, linked to learning objectives
• RankedMultipleChoiceQuestion: Extends MultipleChoiceQuestion with ranking information
• GroupedMultipleChoiceQuestion: Extends RankedMultipleChoiceQuestion with grouping information
• Assessment: Collection of learning objectives and questions

Prompt Component Integration

The modular prompt components in the prompts/ directory are imported into the quiz generation modules and assembled into complete prompts as needed:

1. Learning Objective Generation:

   • Components like LEARNING_OBJECTIVES_PROMPT, LEARNING_OBJECTIVE_EXAMPLES, and BLOOMS_TAXONOMY_LEVELS are combined with course content
   • This creates a comprehensive prompt that guides the LLM in generating relevant and well-structured learning objectives

2. Question Generation:

   • Components like GENERAL_QUALITY_STANDARDS, MULTIPLE_CHOICE_STANDARDS, QUESTION_SPECIFIC_QUALITY_STANDARDS, etc. are combined
   • Along with the learning objective and course content, these form a detailed prompt that ensures high-quality question generation

Workflow Summary

1. User uploads content files (notebooks, subtitles) through the UI
2. System processes files and extracts content with source references
3. LLM generates learning objectives based on content
4. User reviews and approves learning objectives
5. System generates multiple-choice questions for each approved objective
6. Questions are presented to the user for review and export

This modular approach makes it easier to maintain, update, and experiment with different prompt components without disrupting the overall system. Any changes to the components in prompts.py will affect how learning objectives and questions are generated, potentially changing the style, format, and quality of the output.