AI Product

How Do You Know a Generated Answer Is Good?

Thirteen years deciding what a system should do and being accountable when it does it wrong. AI product is that job with a harder question attached — and the answer is not a better prompt. It is a scorecard.

Through to May 2026 I led the data-foundation phase of a portal heading toward agentic AI — the unglamorous part where structured and unstructured data get described and tied to the business, because no agent is trustworthy without it.

Evaluation loop for a transit agent A question goes to the agent, which retrieves and calls tools to produce an answer. The same question is resolved directly against live transit and weather APIs to produce a ground truth. A scorer compares the two and emits metrics. Question "Raining — how do I get there?" Agent retrieval + tools weather · transit Answer route + why Ground Truth the same question, straight from the APIs Scorer compares the two, automatically Retrieval acc. Hallucination Latency · Cost Why transit answers can be scored without a human The same question goes to the agent and to the live transit APIs. The agent produces an answer; the APIs produce the correct answer. Because a correct answer exists, a scorer can compare the two and emit metrics without a human reviewer. ONE QUESTION "Raining. How do I get there?" AGENT SAYS Bus 402, 6 min API SAYS Bus 402, 6 min SCORER no human needed, because right answer SITTING IN AN API RETRIEVAL HALLUCINATION LATENCY COST

Fig. 01 — The capstone in one picture. Most agent outputs need a human to judge them. Transit does not, and that is the whole reason this evaluation set keeps working.

01

What I Actually Measure

Six numbers. Defined before the build, not after the complaint. Four of them can be scored without a human in the loop, which is the only way an evaluation set stays useful past its first week.

01

Retrieval Accuracy

Did it fetch the document or API response that actually contains the answer?

Scored automatically

02

Hallucination Rate

Did it invent a bus number, a stop, or a departure time that does not exist?

Scored automatically

03

Tool-Call Success

Did it call the weather and transit APIs, with the right parameters, when it needed to?

Scored automatically

04

Latency P95

How long the slowest realistic request takes. Averages hide the answers that lose users.

Scored automatically

05

Cost per Query

What one answer costs to produce. The number that decides whether a feature survives contact with finance.

Scored automatically

06

Answer Relevance

Correct is not the same as useful. Did it answer the question the person actually asked?

Scored with judgement
02

What I Can Deliver

AI product roles ask for a fairly consistent set of things, wherever the role sits. This is the set, grouped by what the work actually is. Some of it comes from thirteen years of owning delivery; the rest from graduate coursework in AI and big data — deep learning, natural language processing, generative AI, machine learning and analytics practicums, finished in the 94.9th percentile. I can walk through any of it.

Group 01

Product & Delivery

  • End-to-end product lifecycle
  • Business problem → solution approach
  • Requirements and acceptance criteria for automated decisions
  • Explainability as a product requirement
  • Guardrails and fallback flows
  • Quality metrics and iteration
  • Data foundations for agentic systems — structured and unstructured, described and mapped to the business

Group 02

Build & Prototype

  • RAG over a real knowledge base
  • Prompt design and iterative tuning
  • Comparing models per use case
  • No-code and low-code AI prototyping
  • AI-assisted development — Claude Code, Cursor, Codex
  • Python scripting and workflow automation

Group 03

Evaluate & Govern

  • LLM evaluation — the six metrics above
  • Adversarial evaluation — prompt injection, PII leakage, red-teaming
  • Responsible AI governance frameworks
  • Agentic patterns beyond single-turn tool calling
  • Fine-tuning — when it beats retrieval and when it does not

Working style: embedded in an AI squad rather than handing specifications over a wall. My value is not writing the model — it is deciding what the product should do, what "correct" means, and whether the output is good enough to put in front of a user.

03

Building Now

RAG Chatbot Retrieval + ScoringKnowledge base Q&A

Personal project · In progress

Retrieval Agent With an Evaluation Harness

A question-answering agent over a real document set, built so that retrieval accuracy, faithfulness and hallucination rate are scored automatically rather than judged by eye.

Write-up to follow
LLM App Spec → DraftPlanning automation

Personal project · In progress

Automating the Specification Work I Do by Hand

An LLM tool that drafts and checks the artefacts of product planning — requirements, flows, test scenarios. Built for one user first: me. The interesting part is where it is wrong and why.

Write-up to follow
Capstone Transit AgentMaster's, AI & big data

Korea Cyber University · 2027.02

An Evaluation Framework for Context-Aware Transit Agents

Coursework is finished; this is what remains. The contribution is not the agent — it is the scoring: what counts as a correct answer, and how you catch a fabricated route automatically.

In progress