Portfolio · AI engineer (RAG)Yodhimas Geffananda

Telusur.

An internal knowledge assistant (English and Indonesian interface) for mid-size and large companies. Employees ask in plain language and get answers from company documents, with the page cited, drawn only from documents they are allowed to read. Admins get versioning, an append-only audit log, unanswered-question topics, and a measured evaluation of answer quality.

2:54 · no sound · English interface with captions · recorded locally on the fictional PT Nusantara Sejahtera policy set (documents in Indonesian).

Chapters

Measured, not claimed — 22 policy documents, 71 test questions (~30% English)

Retrieval methodRecall@1Recall@5MRRCorrect refusalsFalse refusals
Keyword only0.600.910.71——
Vector only (bge-m3)0.840.950.89——
Hybrid (vector + keyword, RRF)0.720.970.83——
Hybrid + rerank, no query translation0.951.000.9792%10%
Hybrid + rerank, one translation0.971.000.98100%2%
Hybrid + rerank, three paraphrases (production)0.971.000.98100%0%
0.97Faithfulness (LLM judge)
0.94Answer correctness
93%Answers with citations
0 / 3Restricted documents leaked

Under load — concurrent users on one laptop GPU, LLM stubbed to measure this system

Rerank candidatesRecall@1Peak chats / minp95 first token, 20 usersErrors, 20 users
20 (default, quality first)0.974237 s0%
10 (high-traffic setting)0.956521 s0%

What it shows

Retrieval & answers

  • Structure-aware chunking with page numbers
  • Vector + full-text, RRF, cross-encoder rerank
  • Streaming answers, verified citations, refusal threshold

Enterprise controls

  • Multi-tenant with Postgres row-level security
  • Per-collection and per-document access, enforced in SQL
  • Append-only, partitioned audit log

Quality loop

  • Generated test set + manual verification
  • LLM judge on a separate model
  • Thumbs-down feedback becomes a regression test

Screens

Chat answer with a citation; the source panel shows the quoted table and the PDF at page 2
Answer with the source opened at the cited page
The assistant refuses a question about a document the user cannot read
Refusal when the only source is off-limits
Admin overview with document status and seven-day usage
Knowledge admin overview
Evaluation table comparing retrieval methods
Evaluation: methods compared on the same test set