How the agent is tested
An agent that talks about my work should be held to the same standard as my work. Here is the method, the questions and every verdict.
// grounding scorePublishes with the first full runNo number appears here until the questions and verdicts are public too.
step 1Write the questionsQuestions drawn from my public projects, papers and videos, each tagged with the sources that should support its answer, plus questions the agent must decline.
step 2Grade every replyAn LLM judge rates each answer as grounded, partly grounded or unsupported, and checks that every citation points at the right source.
step 3Publish and rerunScore, questions and verdicts go public, and the whole set reruns whenever the knowledge base changes.
The rubric
GroundedEvery claim is supported by a cited source.Partly groundedThe main point is supported, but a detail is not.UnsupportedA claim has no source, or the source says something else.
Sample questions
The full set is published with the first run.questionexpected sourcesverdict
How does PulseDesk stop prompt injection?PulseDeskOS/config.yaml, PulseDeskOS/ARCHITECTURE.mdpending
What decides the next step in PulseDesk, the model or the code?PulseDeskOS/README.mdpending
When does PulseDesk send a ticket to a human?PulseDeskOS/README.md, PulseDeskOS/config.yamlpending