AI Product

How Do You Know a Generated Answer Is Good?

Thirteen years specifying what a system should do and being accountable when it does it wrong. AI product management is the same job with reliability moved from a schedule risk to a runtime one — and the answer is not a better prompt. It is an evaluation harness with a versioned ground-truth set, wired up before the model ships.

Through to May 2026 I led the data-foundation phase (data readiness for agentic AI) on a portal — the unglamorous part where structured and unstructured data get catalogued, described, lineaged and mapped to the business domain, because no agent — and no RAG on top of it — is trustworthy without that groundwork.

Evaluation loop for a transit agent A question goes to the agent, which retrieves and calls tools to produce an answer. The same question is resolved directly against live transit and weather APIs to produce a ground truth. A scorer compares the two and emits metrics. Question "Raining — how do I get there?" Agent retrieval + tools weather · transit Answer route + why Ground Truth the same question, straight from the APIs Scorer compares the two, automatically Retrieval acc. Hallucination Latency · Cost Why transit answers can be scored without a human The same question goes to the agent and to the live transit APIs. The agent produces an answer; the APIs produce the correct answer. Because a correct answer exists, a scorer can compare the two and emit metrics without a human reviewer. ONE QUESTION "Raining. How do I get there?" AGENT SAYS Bus 402, 6 min API SAYS Bus 402, 6 min SCORER no human needed, because right answer SITTING IN AN API RETRIEVAL HALLUCINATION LATENCY COST

Fig. 01 — The capstone in one picture. Most agent outputs need a human to judge them. Transit does not, and that is the whole reason this evaluation set keeps working.

01

What I Actually Measure

Six evaluation metrics — the scoring contract for the system, defined before the build and versioned with the model. Four are automated and reference-based (no human-in-the-loop), which is the only way the eval set survives past its first week without collapsing into a manual review queue.

01

Retrieval Accuracy

Did it fetch the document or API response that actually contains the answer?

Scored automatically

02

Hallucination Rate

Did it invent a bus number, a stop, or a departure time that does not exist?

Scored automatically

03

Tool-Call Success

Did it call the weather and transit APIs, with the right parameters, when it needed to?

Scored automatically

04

Latency P95

How long the slowest realistic request takes. Averages hide the answers that lose users.

Scored automatically

05

Cost per Query

What one answer costs to produce. The number that decides whether a feature survives contact with finance.

Scored automatically

06

Answer Relevance

Correct is not the same as useful. Did it answer the question the person actually asked?

Scored with LLM-as-judge, sampled human calibration
02

What I Can Deliver

AI product roles ask for a fairly consistent set of things, wherever the role sits. This is the set, grouped by what the work actually is. Some of it comes from thirteen years of owning delivery; the rest from graduate coursework in AI and big data — deep learning, natural language processing, generative AI, machine learning and analytics practicums, finished in the 94.9th percentile. I can walk through any of it.

Group 01

Product & Delivery

  • End-to-end product lifecycle: discovery → system design → delivery → operations
  • Business problem → solution hypothesis → validated MVP with an acceptance test
  • Acceptance criteria and evaluation contracts for automated decisions (the scoring bar for shipping)
  • Explainability as a first-class system requirement, not a UI afterthought
  • Guardrails, safe-completion policies and graceful degradation paths as system components
  • Offline eval → shadow deployment → canary → GA, with A/B where the metric warrants it
  • Data-platform foundation for agentic systems — schema, lineage, ownership, DQ and business-domain mapping

Group 02

Build & Prototype

  • RAG system over a real knowledge base — chunking strategy, embeddings, hybrid retrieval, retrieval-quality eval
  • Prompt engineering iterated against a held-out eval set, not by feel
  • Model bake-off per use case, scored on the cost / latency / quality trade-off
  • No-code / low-code AI prototyping for rapid system-shape validation
  • AI-assisted development — Claude Code, Cursor, Codex
  • Python scripting and pipeline automation

Group 03

Evaluate & Govern

  • LLM system evaluation — the six metrics above, on a versioned eval set, tracked across model versions to detect drift
  • Adversarial evaluation — prompt injection, jailbreaks, PII leakage, red-teaming as a release gate
  • Responsible AI — governance framework, model cards, audit trail, incident review
  • Agentic patterns — multi-step tool use, planner-executor topology, guardrail supervision, timeout and cost budgets
  • Fine-tuning vs. retrieval vs. prompt-only — chosen against the reliability bar the system actually has to clear

Working style: embedded in a cross-functional AI systems squad rather than handing specifications over a wall. My value is not writing the model — it is defining the product contract, the definition of correct, and the ship threshold that says whether an output is allowed in front of a real user.

03

Building Now

RAG Chatbot Retrieval + ScoringKnowledge base Q&A

Personal project · In progress

Retrieval Agent With an Evaluation Harness

A question-answering agent over a real document set, built so that retrieval accuracy, faithfulness and hallucination rate are scored automatically rather than judged by eye.

Write-up to follow
LLM App Spec → DraftPlanning automation

Personal project · In progress

Automating the Specification Work I Do by Hand

An LLM tool that drafts and checks the artefacts of product planning — requirements, flows, test scenarios. Built for one user first: me. The interesting part is where it is wrong and why.

Write-up to follow
Capstone Transit AgentMaster's, AI & big data

Korea Cyber University · 2027.02

An Evaluation Framework for Context-Aware Transit Agents

Coursework is finished; this is what remains. The contribution is not the agent — it is the scoring: what counts as a correct answer, and how you catch a fabricated route automatically.

In progress