Local RAG & Gemma3 SLM

If speech/text inputs fall below the command intent threshold, they are routed to the local RAG pipeline to answer Q&A queries based on offline documents.

1. Document Embedding and Chunking

Manuals and technical files (knowledge.pdf) are parsed and split using a sliding chunk window. Each text chunk is encoded into a 384-dimensional vector using MiniLM and compiled into knowledge_db.pkl.

2. Gemma3 Local Inference & KV Cache

The causal language generation runs locally using gemma3.onnx. To support interactive response speeds on the Pi's CPU, the decoding loop employs Key-Value (KV) Caching:

• Static Pre-allocation: Caching attention key-value states in memory avoids re-calculating historical token context during auto-regressive generation.
• Dynamic Specs: KV-cache dimensions are pre-allocated per layer as (1, 1, seq_len, 256) representing a single batch size, 1 KV attention head, current context length, and 256 head dimensions.

RAG Knowledge Pipeline

Sliding Window Embedding & Generator Pipeline