NLP Cybersecurity 2 Semesters

🚨 Malay SMS / WhatsApp Scam Detector — ML vs Transformer vs LLM

"Akaun anda telah disekat, sila klik link…" — Malaysians lose millions of ringgit to message scams yearly, and scam text is uniquely local: Malay, English, and rojak mixed with urgency tricks. Detecting it is a genuinely useful, genuinely researchable problem.

1The Pitch — Product + Research in One

🛠️ What you build (the demo)

A web app (or Telegram bot): paste any suspicious SMS/WhatsApp message → scam risk score, highlighted red-flag phrases, and safety advice. Instant, shareable, demo-friendly.

🔬 What you research (the grade)

Three generations of text classification go head-to-head on local scam messages: classical ML, fine-tuned transformer, and zero-shot LLM — accuracy vs cost vs latency.

2Problem Statement & Objectives

Pernyataan masalah: Scam messages targeting Malaysians use code-switched Malay/English and local context (bank names, JPJ, LHDN, parcel delivery); English-centric spam filters miss them, and elderly users can't judge authenticity themselves.

Build a scam-message checker with risk scoring and red-flag explanations (the MVP).
Benchmark classical ML, a fine-tuned multilingual transformer, and a zero-shot LLM on a local scam corpus.
Evaluate accuracy, false-positive cost, latency, and per-1,000-message cost to recommend a deployable pipeline.

3Benchmark Design — the 3 Experiments

Experiment	Setup	What it proves
Eksperimen 1 Classical ML	TF-IDF + character n-grams → SVM vs Random Forest vs Logistic Regression, 80/20 split.	The cheap, fast, offline-capable baseline every comparison needs.
Eksperimen 2 Fine-tuned transformer	mBERT / MalayBERT fine-tuned on the same split.	How much contextual understanding improves detection of subtle scam phrasing.
Eksperimen 3 Zero-shot LLM	An LLM API classifying the same test set with a fixed prompt (no training).	The 2026 question: does prompting beat training? Includes cost + latency trade-off table.

📦 Dataset plan

Collect 1,000–2,000 messages: scam samples from public scam-alert channels (PDRM/MCMC awareness posts, scam-report communities) + legitimate messages (promos, bank alerts, personal texts — anonymized). Label with a guideline; keep a balanced legitimate set including tricky look-alikes (real bank notifications) so false positives are meaningfully tested.

4Evaluation Metrics

Metric	Why it matters here
Recall on scam class	A missed scam is the costly error — this is the headline safety number.
Precision / false-positive rate	Flagging real bank alerts as scams destroys user trust — measured on look-alike messages.
Macro F1	Balanced single number for the 3-way model comparison.
Latency & cost per 1,000 messages	Classical ≈ free, transformer needs a GPU, LLM bills per token — the deployment reality table.

5Report Mapping (Bab 1–5)

Chapter	What goes in it (from this package)
Bab 1 — Pengenalan	Scam statistics in Malaysia, problem statement, objectives, scope (SMS/WhatsApp text only, 3 model families), Gantt chart.
Bab 2 — Sorotan Kesusasteraan	Comparison tables: spam/phishing detection studies, code-switched NLP, LLM-as-classifier papers (2023–2026).
Bab 3 — Metodologi	Data collection + ethics/anonymization note, labeling guideline, preprocessing, system architecture (input → classifier → explanation UI).
Bab 4 — Pembangunan & Pengujian	The 3 experiments, metric tables, false-positive analysis on look-alikes, cost/latency table, app screenshots.
Bab 5 — Dapatan Kajian	Recommended pipeline (e.g., classical filter + LLM confirmation), limitations (dataset recency, voice scams excluded), future work (image scam detection).

6Tech Stack

Pythonscikit-learnHugging Face LLM APIFlask / Telegram Bot API

🔎 Related works — Perplexity API placeholder

Live literature search (scam/phishing detection and LLM classification papers, 2022–2026) will be generated here via the Perplexity API once the subscription is active.

⏳ Generate Literature Review (coming soon)

← Back to all FYP ideas · Assignment Helper