LLM / RAG Software Engineering 2 Semesters

🤖 Campus FAQ Chatbot — Benchmarking Retrieval Strategies

Every faculty office answers the same questions daily: registration, fees, hostel, graduation credits. A chatbot is the obvious fix — but the research is in the retrieval: which search strategy actually finds the right answer in your university's documents?

1The Pitch — Product + Research in One

🛠️ What you build (the demo)

A web chat widget answering student FAQs from real university documents (handbooks, academic circulars, faculty FAQ pages) using Retrieval-Augmented Generation — with source citations shown for every answer.

🔬 What you research (the grade)

RAG quality is decided by retrieval. You benchmark three retrieval strategies on a test set of real student questions and measure which surfaces the correct document — before any LLM ever answers.

2Problem Statement & Objectives

Pernyataan masalah: Student inquiries overload faculty offices and Telegram groups; answers exist in official documents but are scattered and hard to search, and generic chatbots hallucinate university policies.

Build a citation-backed campus FAQ chatbot over a curated document corpus (the MVP).
Benchmark three retrieval strategies — BM25 keyword, dense embeddings, and hybrid — on a labeled question set.
Evaluate retrieval with Hit@k and MRR, and answer quality with a faithfulness rubric, to determine the best pipeline.

3Benchmark Design — the 3 Experiments

Experiment	Setup	What it proves
Eksperimen 1 Keyword (BM25)	Classic BM25 index over chunked documents; retrieve top-5 per question.	The strong classical baseline — surprisingly hard to beat on exact terms (course codes, fee names).
Eksperimen 2 Dense embeddings	Multilingual sentence-embedding model + vector store, same top-5 protocol.	Whether semantic search handles paraphrased and Malay/English mixed questions better.
Eksperimen 3 Hybrid + rerank	BM25 ∪ dense candidates merged, then reranked (reciprocal rank fusion).	Whether combining both closes the gap — the trade-off table (quality vs latency vs cost) examiners want.

📦 Dataset plan

Corpus: 30–60 real documents (student handbook, academic regulations, FAQ pages). Test set: 100–200 real questions collected from students (mini-survey — counts as data collection methodology), each labeled with the document/section holding the answer. Questions split across pure-Malay, pure-English, and mixed to add a language dimension.

4Evaluation Metrics

Metric	Why it matters here
Hit@1 / Hit@5	Was the correct source retrieved at all? The core retrieval score per experiment.
MRR	Rewards putting the right document at the top, not just in the top-5.
Faithfulness rubric (1–5)	Human-scored: does the generated answer stick to the retrieved source? Catches hallucination.
Latency & cost per query	The engineering trade-off dimension — makes Bab 5 recommendations concrete.

5Report Mapping (Bab 1–5)

Chapter	What goes in it (from this package)
Bab 1 — Pengenalan	Problem statement, objectives, scope (one faculty's documents, 3 retrieval strategies), Gantt chart.
Bab 2 — Sorotan Kesusasteraan	Comparison tables: chatbot generations (rule-based → RAG), retrieval methods across recent papers, RAG evaluation frameworks.
Bab 3 — Metodologi	Corpus curation + chunking strategy, question collection survey design, system architecture (indexer → retriever → LLM → citations UI).
Bab 4 — Pembangunan & Pengujian	The 3 experiments, Hit@k/MRR tables, per-language breakdown, faithfulness scoring, chat UI screenshots.
Bab 5 — Dapatan Kajian	Recommended pipeline with evidence, limitations (corpus freshness, rubric subjectivity), future work (multi-faculty scaling).

6Tech Stack

PythonFastAPIrank-bm25 sentence-transformersChroma / FAISSAny LLM API

🔎 Related works — Perplexity API placeholder

Live literature search (RAG evaluation and hybrid retrieval papers, 2022–2026) will be generated here via the Perplexity API once the subscription is active.

⏳ Generate Literature Review (coming soon)

← Back to all FYP ideas · Assignment Helper