Every number, and where it comes from

1. Measured by

   A blind LLM judge (gpt-4.1-mini) listed every factual claim in each answer and checked it against everything Ken's tools returned.

   26 of 27 claims, across 10 questions on a fake inbox

2. Source
   evals/RESULTS.md · Other aspects
3. Date

   28 Oct 2026

1. Measured by

   evals/run_eval.py records every send attempt; a run fails if any send targets the attacker's address or number.

   10 agent runs over a fake inbox with 1 planted prompt-injection email

2. Source
   evals/run_eval.py · grounded_test
3. Enforced by
   • test_send_blocked_without_intent
   • test_unknown_recipient_blocked
   • test_wrap_untrusted_cannot_be_escaped
4. Date

   28 Oct 2026

One planted email is a small test, and it was run on gpt-4.1-mini, before the switch to gpt-6-luna. A larger attack set is planned.

1. Measured by

   The eval's send handler declines and logs every attempt; none were made.

   the same 10 runs

2. Source
   evals/RESULTS.md · Other aspects
3. Enforced by
   • test_send_intent
   • test_read_then_draft_does_not_send
4. Date

   28 Oct 2026

1. Measured by

   The send tool is reachable only through a confirmation dialog the model can't call; declined sends are dropped and every send is audited.

2. Source
   ENGINEERING.md · 1. Design principles
3. Enforced by
   • test_send_requires_confirmation_and_is_audited
   • test_declined_send_does_not_send
   • test_send_draft_button_goes_through_the_gate
4. Date

   3 Nov 2026

1. Measured by

   pytest; conftest.py gives every client a transport that raises on a real request.

   a transport guard makes any real OpenAI request fail the test

2. Source
   ENGINEERING.md · 6. Testing and evaluation
3. Date

   3 Nov 2026

1. Measured by

   Blind 3-way judge. With 50 cases, differences under about ±10 points are noise, so +18 is a moderate win.

   50 held-out real WhatsApp replies; 14% for the earlier version, 24% with no RAG

2. Source
   evals/RESULTS.md · Before → now
3. Date

   28 Oct 2026

1. Measured by

   Average characters per draft against the real reply.

   50 replies; +45% with no RAG, +18% before

2. Source
   evals/RESULTS.md · Before → now
3. Date

   28 Oct 2026

1. Measured by

   Count of stored pairs; each draft retrieves the 3 most similar from the same chat, plus the chat's last 24 h.

   "them → you" pairs from the owner's WhatsApp history

2. Source
   evals/RESULTS.md · What was compared
3. Date

   28 Oct 2026

1. Measured by

   For each pair, how often its 3 nearest neighbours come from the same chat. This is why retrieval filters by chat first.

   1,001 pairs; 12% by chance

2. Source
   ENGINEERING.md · 3. The RAG system
3. Date

   28 Oct 2026

1. Measured by

   Same prompt, recall and intent; only the drafting model changed.

   50 replies; gpt-4.1-mini 18–22%; each model judged blind to cancel self-preference

2. Source
   evals/RESULTS.md · gpt-4.1-mini vs gpt-6-luna
3. Date

   28 Oct 2026

1. Measured by

   Cost and timing logged per call during the comparison run.

   $0.048 → $0.011; median latency 1.0 s → 1.6 s, accepted

2. Source
   evals/RESULTS.md · gpt-4.1-mini vs gpt-6-luna
3. Date

   28 Oct 2026

1. Measured by

   A prompt flips the model between extremes; the real fix is deciding the split in code.

   tries: 0% → 78% → 4% → 52%, against the owner's 22%

2. Source
   evals/RESULTS.md · Line-break fix
3. Date

   28 Oct 2026

1. Measured by

   Fuzzy match with an activity bonus; the model sees 'last message today / 3 days ago'.

   10 real chats, Arabic and Latin spellings; the miss was ambiguous (several contacts share a name)

2. Source
   evals/RESULTS.md · Other aspects
3. Date

   28 Oct 2026

1. Measured by

   Every OpenAI request is logged with a purpose tag.

   2,187 model calls, 1.58M tokens

2. Source
   evals/RESULTS.md · Cost
3. Date

   28 Oct 2026

1. Measured by

   Jest test run.

   including the chat-context builder, AI usage caps, webhook signatures and a bundle secret-leak test

2. Source

   Planovya's repository is private. Ask for a walkthrough.

3. Date

   Nov 2026

1. Measured by

   git history and the migrations folder.

2. Source

   Planovya's repository is private. Ask for a walkthrough.

3. Date

   Nov 2026

1. Measured by

   Server-side context builder: open tasks, the 10 nearest deadlines, goals, notes, 7 days of focus time.

   cut at a line boundary with an explicit 'lines omitted' notice

2. Source

   Planovya's repository is private. Ask for a walkthrough.

3. Date

   Nov 2026

1. Measured by

   Supabase row-level security on every table, plus server-side JWT checks; the OpenAI key never reaches the browser.

2. Source

   Planovya's repository is private. Ask for a walkthrough.

3. Date

   Nov 2026