TLDR: A GPU-accelerated Retrieval-Augmented Generation (RAG) system, powered by a quantized Mistral-7B model and deployed as a Telegram bot, provides continuous, personalized academic assistance for “Introduction to Parallel Processing” students. Running on consumer-grade GPUs, this open-source solution ensures privacy, affordability, and responsiveness, demonstrating a practical approach to AI tutoring in HPC education.
In today’s fast-paced academic world, students often need support beyond traditional office hours. This challenge is particularly acute in complex subjects like “Introduction to Parallel Processing.” A new project introduces an innovative solution: a GPU-accelerated, Retrieval-Augmented Generation (RRAG) system deployed as a Telegram bot, designed to offer continuous, on-demand academic assistance to students.
Authored by Guy Tel-Zur from Ben-Gurion University of the Negev, this initiative addresses a critical pedagogical need. The system leverages a quantized Mistral-7B Instruct model, a powerful yet efficient large language model, to deliver real-time, personalized responses grounded in the course materials. A key innovation is the use of GPU acceleration, which significantly reduces the time it takes for the AI to generate responses, making it practical to deploy on readily available consumer hardware. This approach paves the way for affordable, private, and effective AI tutoring in high-performance computing (HPC) education.
Why a Smart Assistant for Students?
With over two decades of teaching Parallel Processing, the author recognized that weekly reception hours are often insufficient, especially during exam periods when students have more questions. Additionally, some students may feel hesitant to ask questions in person. The advent of advanced AI makes it possible to provide a smart agent that is continuously available, 24/7, to bridge this gap.
This project stands out due to several unique features:
- It is built entirely using open-source tools, eliminating licensing issues and costs, and ensuring privacy by running on a standalone computer.
- It uses a Telegram interface, making it accessible from any platform, including desktops, mobile phones, and tablets.
- The bot can run on a commodity computer, ideally with a Graphics Processing Unit (GPU). The project specifically uses an ASUS TUF F17 laptop with an Nvidia GeForce RTX 4060 GPU, demonstrating reasonable response times.
How the System Works
The foundation of the smart agent is its knowledge base, which comprises merged course slides and an electronic textbook. A document preparation pipeline then transforms these materials into a searchable format for the RAG system.
Here’s a breakdown of the core components:
- Embeddings Generator: This component uses the open-source ‘all-MiniLM-L6-v2’ model to convert chunks of course documents into numerical vector embeddings. These vectors capture the semantic meaning of the text, allowing the computer to understand and compare different pieces of text based on their meaning, not just keywords. For example, it understands that “I love dogs” and “I adore canines” have similar meanings.
- Vector Database (FAISS): This specialized database stores the generated embeddings and their metadata, enabling extremely fast similarity searches. When a student asks a question, the system can quickly retrieve the most relevant pieces of information from the knowledge base.
- Retrieval-Augmented Generation (RAG) Pipeline: This is the brain that connects the user’s query to the knowledge base. It takes a user’s question, searches the vector database for the top-k most relevant chunks of information, and then provides both the original text and source references.
- Local LLM Inference (Mistral 7B): The retrieved information, combined with the user’s query, is fed into a quantized version of the Mistral 7B model. This large language model, developed by Mistral AI, is known for its efficiency and power. Running locally with CUDA GPU offloading, it generates coherent, accurate, and contextually grounded answers. Its relatively small size allows it to run efficiently on local hardware while still producing high-quality responses.
- Telegram Bot: This provides the user-friendly conversational interface. It receives messages from students, forwards them to the RAG + LLM pipeline, and sends back the generated answers.
- Orchestration and Containerization: The entire system is encapsulated using Docker and docker-compose, ensuring that it is reproducible and portable. This framework manages all dependencies, from Python to CUDA libraries.
Performance and Responsiveness
A crucial aspect of any interactive AI system is its performance. The most relevant metric for this assistant is Tokens Per Second (TPS), which measures how quickly the model generates output. Benchmarking shows a significant difference across platforms:
- A typical laptop running the model on the CPU achieves approximately 0.5-1.5 TPS.
- This system, running on a laptop with an Nvidia GeForce RTX 4060 GPU, achieves about 16 TPS.
- A powerful cloud server could reach 30-100 TPS, but at a significant hourly cost.
The mean generation speed of around 16 tokens/second on the RTX 4060 Laptop GPU, combined with a low Time To First Byte (TTFB) of about 0.1 seconds (the time until the first output token appears), means the chatbot feels responsive. This performance makes it suitable for smooth chatbots, teaching demonstrations, or personal assistants. For scaling to more users or larger contexts, more VRAM would be beneficial.
Future Optimizations
While the current system performs well, there’s a roadmap for further optimization. This includes fine-tuning parameters such as batch size, the number of GPU layers (how much of the model runs on the GPU), tensor splitting for multi-GPU setups, and the maximum context window (how many tokens the model can “see” at once). Utilizing advanced techniques like Flash Attention, which is supported by the RTX 4060, can also provide significant speedups and lower VRAM usage.
Also Read:
- A New AI System to Support Goat Health and Farm Management
- GeoGPT: Advancing Geoscience Research with Retrieval-Augmented AI
Conclusion
The GPU-accelerated RAG-based Telegram assistant for the “Introduction to Parallel Processing” course is ready for its initial implementation in the forthcoming semester. This project demonstrates a practical and effective way to provide continuous academic support using accessible AI technology. The full details of the research can be found in the paper: A GPU-Accelerated RAG-Based Telegram Assistant for Supporting Parallel Processing Students.
