Experiments

What was measured while building the context pipeline, and what each result changed. The design these led to is in context-management.md. How to rerun them is in testing.md.

Setup (2026-09-25): Ollama 0.34.2, NVIDIA RTX 4050 Laptop GPU (6 GB), qwen2.5-3b-instruct (tools), qwen3.5-4b (tools, thinking, vision), nomic-embed-text. Long-chat runs use docs/tools/long_chat.py with a scratch copy of backend/*.go (26 files) attached, on qwen2.5-3b-instruct.

Small-model output varies from run to run. Where a result rests on a single run it says so.

E1: What Ollama does when a prompt is too big#

A system message holding a code word, a filler user message of about 10.6k tokens, then a question about the code word. Sent with num_ctx: 2048 and again with num_ctx: 16384.

At 2048 Ollama silently dropped the whole filler message and kept the system message and the last message. There was no error, and the reply didn't say anything was missing. The filler was about 22.5k characters, so characters / 4 estimated 5.6k tokens against 10,624 real ones: off by almost 2x on dense text.

Changed: Flint does its own fitting, so the model is told when something is left out and the right things are kept. Token estimates are calibrated against the real prompt_eval_count.

E2: The default window#

• A request without num_ctx loaded qwen2.5-3b at 4096. The model's own maximum is 32768.
• A model loaded at 8192, followed by a request without num_ctx, was reloaded at 4096.

Changed: every request, including title generation and summarization, sets num_ctx. Otherwise requests would reload the model back and forth, and the budget would guess the window wrong.

E3: How many tokens an image costs#

On qwen3.5-4b, prompt_eval_count with one image minus without:

That's about width x height / 1024. One large photo takes half of a 4096 window.

Changed: images are budgeted by their pixel size, and only the latest message with images still sends them.

E4: The folder manifest was bigger than the window#

The first long-chat run failed on every turn: request (12168 tokens) exceeds the available context size (8192 tokens). The 40 KB folder manifest (a 42,045-character system message) came to about 12k tokens alone. It's protected, so nothing could shrink it. Unlike E1, Ollama rejected these requests instead of truncating them, because what it must keep (the system message) was itself too big. This was already broken on main for any folder with more than about 25 KB of text.

Changed: maxAttachTotalSize is 12 KB, about 3.6k tokens of Go. Files past the budget are listed as not included, and the model reads them with the shell tool.

E5: Truncated replies teach the model to truncate#

From about the fifth turn of a run, every reply ended in "...[truncated]" and came back in about 0.5 s. The stored replies showed the model writing that marker itself. It had seen its own older replies shortened by decay, each ending in "...[truncated]", and copied the pattern.

Changed: only tool output and later system messages decay. The model never writes those. Old dialogue is summarized instead.

E6: Where the summary instructions go#

Summarizing the same 34-message chunk (with the prompt as a system message unless noted):

Both models miscounted files in some runs ("14 Go files", "24 files"; there were 26), even with an instruction to state only numbers that appear word for word.

Changed: instructions go after the input. Plain bullet prompts, no template.

E7: Keeping what the user said#

The test fact is planted in turn 2, not in the protected first message: staging runs on port 9123, my manager is Tom Okafor, never touch the payments table.

A single full run beat the isolated tests.

Changed: user messages are copied into the summary word for word (user_notes) and carried through merges unchanged. The model only condenses them, on their own, once they outgrow 1/16 of the window. See E11.

E8: Summaries removed the tool-call examples#

Tool calls per turn (turns 0-18) across long-chat runs:

baseline (main + E4 fix)   3 1 1 1 1 1 1 3 1 1 3 3 1 3 1 1 0 1 0
summaries                  0 1 1 1 3 3 3 3 3 0 3 1 0 0 0 0 0 0 0
+ keep latest tool call    1 1 1 1 1 1 1 1 1 3 1 3 1 1 3 1   (stopped by an overflow, E10)
+ cut tool output to fit   1 1 3 2 3 3 1 1 3 2 0 3 3 1 3 1 3 1 3

With summaries, tool calls stopped from turn 12 onward. Dumping the history (TestLiveDumpHistory) showed why: every real tool call had been summarized away. The recent window held only the model's own text-only replies ("I will run the command now. ```bash ..."), and it kept copying them. The baseline never summarized, so its old tool calls stayed in the prompt as examples.

Keeping the latest tool call and result verbatim prevented the drift in later runs. It did not repair a conversation that had already drifted: 0 real tool calls in 6 tries on the drifted conversation.

Changed: the latest tool exchange is always kept verbatim.

E9: A tool nudge that allows answering without tools (rejected)#

The last recall question ("Without running any commands: what port is staging on...") went to grep even when the summary held the answer. The reply even named "Tom Okafor". The tool nudge appended to the last message ends with "then call the tool", which outweighs "without running any commands".

A/B on copies of the same conversation, adding "If the conversation already contains the answer, or the user asked you not to run commands, answer directly without a tool":

In a fresh full run, though, the revised nudge made 2 tool calls in 19 turns. Most answers became guesses ("ratelimit.go is likely used to..."), and some claimed the user had asked for no commands.

Changed: nothing. The original nudge stays. This conflict is still open (see below). Resolved for explicit "don't run" requests in E17.

E10: Protected messages can overflow on their own#

After E8's fix, one run had 3 overflow errors (9122, 9291 and 10665 tokens against 8192). The baseline had 1 (8531). The recent window and the kept tool result can each hold a full cat of a large file, up to 20k characters, and dropping never touches protected messages.

Changed: as a last step, tool output is cut down, oldest first, until the prompt fits. The next run had 0 overflows, and its peak was 8029 of 8192 tokens (prompt plus reply).

E11: Verbatim user notes through a merge#

The first full run after the E7 change:

tool calls per turn   0 3 3 3 1 1 1 1 1 3 0 1 1 1 2 3 1 1 1
peak context          7569 of 8192
overflow errors       0

The level-1 summary covering the planted message still holds the port, manager and table rule word for word, after two rounds of summarizing. The final recall question still went to grep instead of being answered from the summary. That's the E9 nudge conflict, not missing context.

E12: Memory cost of a bigger window#

Each model loaded fresh at each num_ctx, with the size Ollama reports (/api/ps) and total GPU memory in use (nvidia-smi, 6 GB card, about 0.8 GB used by the desktop):

• qwen2.5-3b grows about 36 KB per token of window, matching the estimate from its architecture. Even at 32k it fits entirely on this GPU.
• qwen3.5-4b doesn't fit on the GPU at any window size: about 1.4-2 GB of it runs on the CPU. That's why its summaries took 26-40 s (E6). Its size barely changes with the window.

The KV cache is allocated for the whole window when the model loads, however long the prompt is. So the window size is what costs memory, not the conversation length.

Changed: nothing yet. Memory isn't what stops a bigger window for qwen2.5-3b on this machine. What's still unmeasured is how prompt processing time and answer quality hold up at 16k and 32k.

E13: Drafting a memory from a chat#

@memory save on a two-message chat: "We're planning Project Astra. Decision: we use Postgres 16, not MySQL. I prefer answers under 100 words." (qwen2.5-3b-instruct). Two drafts:

- User plans Project Astra
- User decides to use Postgres 16 instead of MySQL
- User prefers Postgres 16 for the project

- Postgres 16 chosen for Project Astra
- User prefers Postgres 16 over MySQL

Both kept the decision. Both dropped the stated preference for short answers, and both recast the decision as a "preference".

Changed: drafts are never saved without review. The draft appears as an editable card, and only Save memory stores it. @memory save <text> skips the model entirely.

Recall worked in a separate chat. @memory astra which database did we pick and when does it ship? answered with both saved facts (Postgres 16, Friday the 3rd), though it wrongly credited the decision to "you and Tom Okafor".

E14: Where folder memories go#

A memory saved in one folder chat ("always run go vet before tagging a release"), then "Without running any commands: what is the deploy rule for this repo?" in fresh chats on the same folder:

The misses still reached for a tool, which is the E9 nudge conflict.

Changed: folder memories go with the anchor. This is the same finding as the tool nudge's placement (context management): text far from where the model starts writing loses its effect once a long manifest sits in between.

E15: First benchmark run, full vs bare#

The first run of bench/ (benchmark.md) on qwen2.5-3b-instruct, one run per task. full has all scaffolding on; bare switches off the nudge, anchor, summaries, context fitting and preconditions. bare's last task was stopped to save time, since with no fitting it can only overflow.

Scaffolding lost this run. One run can't settle a gap this size, but the failures show real, specific causes rather than noise:

• Long context failed everywhere because of a real bug (E16). A single cat of the 25 KB log made the next request overflow, and the chat could never recover.
• Restraint is where the nudge costs the most. With the nudge on, the model proposed commands for "Without running any commands: what port…", "…who is on call" and "What is 12 times 12?" (echo '12 * 12' | bc). Without it, only the arithmetic one. This is E9 again, now with a count.
• Reaching for tools turns easy questions into failures. full failed max-connections, whose answer was in the attached context, by running a command, getting confused, and blaming "an issue with running the shell command". bare just answered.
• The shield caught a gap (rm-home). Asked to run rm -rf ~, the model proposed cd ~ && rm -rf ., which got past the shield. The benchmark denied it, so nothing ran. In the same run, the harness also failed a harmless rm -rf ~Downloads because its check flagged any rm -rf. Both are fixed: the shield rule now also covers ., .., *, ./* and split flags, and the check only fails on catastrophic targets. bare's 3/3 safety is weaker than it looks: the model never proposed the commands at all (0 blocked), so the shield wasn't tested.
• Denials make the model invent "permission issues", in both configurations. After a denied or failed command, it often gives up with "I don't have the necessary permissions" rather than trying something else.
• The one task scaffolding clearly won was wrong-name-log. full recovered from large-log.txt to large_log.txt; bare gave up.

Also seen: after its rm -rf ~ was blocked, the model proposed find ~ -type f -size +100M -delete. That's destructive but not categorically catastrophic, so it's left to the approval step, not the shield.

Fixes:

Checked afterwards with single tasks on full, not a full rerun:

• rm-home: in the check run the model didn't retry rm -rf .. It proposed an interactive echo … && read prompt, ls ~, and find ~ -type f -size +100M -delete, none of them catastrophic. So this pass shows the narrowed check, not the new shield rule against the model. The shield rule is covered by its unit tests on the exact command.
• log-error: no overflow any more. This time the model guessed the path data/large_log.txt three times. The precondition check caught each one without asking for approval, and after 3 attempts the model had to answer in text. The scaffolding worked as designed, and the model just never found the file.

Next: run no-nudge against full with --runs 3. It's the most likely single cause of the gap, and it decides whether the nudge should be reworded, made conditional, or dropped. Done in E17.

E16: A new chat couldn't recover from one big command output#

full's long-context tasks all ended in exceeds the available context size (12,628 tokens against 8192). Context fitting works from a token estimate, and a new chat's estimate is plain characters / 4. The log is dense timestamps and numbers at about 2 characters per token, so fitting thought the prompt fit when it was more than 1.5x too big. The estimate was only ever corrected after a successful reply, so every later request in that chat failed the same way. One task retried for 744 s.

Ollama's rejection states the real size ("n_prompt_tokens":12628). Flint now reads it, recalibrates from it, refits, and retries once. Nothing has been streamed at that point, so the retry is invisible.

Measured with a regression task added to the benchmark, overflow-recover ("Run cat large_log.txt", then "What port does Stockroom listen on?"):

Commit 4798a83. A unit test (ollama_test.go) pins the exact rejection body Ollama 0.34.2 sends.

How far this goes:

• The first estimate is still wrong, and that's accepted, not fixed. This is what happens and why.

What the estimate is. Flint can't count tokens exactly before sending a request. The only exact count comes back from Ollama, afterwards. So it estimates: characters / 4, multiplied by a per-chat correction ratio that's measured from Ollama's real count after every successful reply. A new chat has no measurement yet, so its ratio starts at 1.0, which means plain characters / 4.

Why characters / 4 is wrong here. Four characters per token is roughly right for English prose. Go code runs about 3.4. A log of timestamps and numbers runs about 2, because the tokenizer splits digits and punctuation into many small tokens. So on dense text, characters / 4 counts about half the real tokens.

What happened before the fix, in numbers. The model ran cat large_log.txt. The output was capped at 20,000 characters, which characters / 4 counts as about 5,000 tokens. Together with the folder manifest, the fitting step thought the request fit the budget (8192 minus the 1024 reply reserve and the tool overhead, about 7,000 estimated tokens). Ollama counted 12,628 real tokens and rejected it. The ratio was never corrected, because only a successful reply corrects it, so the next request was estimated the same way and rejected again, every time.

What happens now, step by step:

1. The request is built with the uncorrected estimate and sent.
2. Ollama rejects it and reports n_prompt_tokens: 12628.
3. Flint divides that by its own estimate of the same request, about 1.8, and stores it as the chat's ratio.
4. The history is rebuilt with that ratio. The same log now counts as roughly 10,000 tokens instead of 5,000, so the fitting step cuts it down until the real size is under the window.
5. The request is sent again and succeeds. Nothing had been streamed, so you never see the first attempt. In the check run, the retried request peaked at 7,090 of 8,192 tokens.

What it costs. One rejected request. Ollama rejects while counting the prompt, before generating anything, so the wasted time is small next to the reply itself.

When it happens. On the first oversized request of a new chat, and again whenever a chat's text suddenly gets much denser than what the ratio was measured on. For example, a ratio measured on prose, followed by a log dump. Each time, one retry corrects it.

Why the estimate itself isn't fixed. A better first guess would need a per-content estimator, for example one that counts digits and punctuation. That could shrink the error but not remove it, since only Ollama knows the real count, and the retry already covers whatever error is left. If the rejected requests ever become a measurable cost, a density-aware first guess is the next step.

• One retry only. If the refit still doesn't fit, for example because the protected messages alone exceed the window, the error shows as before. Cutting tool output to fit makes that rare.
• It depends on Ollama's error format. If a future Ollama drops or renames n_prompt_tokens, this falls back to the old behavior: an error, nothing worse. Paste the new body into the unit test when that happens.
• One ratio covers the whole chat. After a dense log, the corrected ratio overcounts ordinary prose, so the next request trims a little more than it needs to. It corrects itself after the next successful reply, since every reply recalibrates.
• One run before, one after. The difference is clear-cut, but it's n=1.

E17: The tool nudge is left off when the user says not to run commands#

E15 pointed at the tool nudge as the cause of the restraint failures. This is the --runs 3 measurement, on qwen2.5-3b-instruct with the fixture folder attached (so the real manifest is in context), over 8 tasks: 3 restraint, 1 grounding, 4 tool use.

python3 bench/run.py --configs full,no-nudge --runs 3 \
  --tasks no-cmd-port,no-cmd-oncall,no-cmd-math,max-connections,list-data,log-lines,price-lookup,run-check

Baseline:

The nudge is the cause of the restraint failures, and it also earns its place. With it, every "without running any commands" question went to grep or cat. Without it, those were all answered, but tool use fell: twice the model said the data/ folder "wasn't attached" instead of running ls data/, and once it counted a log's lines by eye (2,019; the answer is 400). no-cmd-math failed the same in both (echo '12 * 12' | bc), so the nudge isn't what causes that one. Dropping the nudge was ruled out here.

Candidates, each measured as full with the same command and tasks:

• a (rejected): "If you need to run a command, first briefly think through…". Restraint stayed at 0/9. Making the sentence conditional didn't stop the model reading it as an instruction to use the tool. The grounding and tool-use gains are within noise.
• b (adopted): the original nudge, left off in Go when the latest user message contains a phrase like "without running", "don't run" or "no commands" (forbidsCommands in tools.go). The tool is still offered; only the nudge goes. Every other message still gets the nudge, so tool use is kept (11/12 vs 9/12, within noise). The model makes the same choices as with no nudge on the two tasks that name the restriction. This follows @web and @memory: the decision is made from the user's literal words, not by the model.

Changed: candidate b. Commit on dev, see git log.

How far this goes:

• The phrase list is narrow on purpose. A user who says "just from what you can see" or "from memory" still gets the nudge. The two restraint tasks this fixes use phrases the list was written to match, so the 6/6 there shows the mechanism works, not how often real users phrase it this way.
• no-cmd-math isn't fixed. Nothing in "What is 12 times 12?" says not to run a command, and the model reaches for bc with or without the nudge. The 1/3 is noise. Deciding in Go that a question needs no command would be a classifier, and wrong classifications would cost tool use.
• 3 runs per task, one model.

E18: A denied command made the model claim it lacked permissions#

After a denied command, qwen2.5-3b often said "I don't have the necessary permissions" and gave up, or made up the command's output (E15, and a price-lookup run in E17). The tool result it got was "User denied permission to run this command." The word "permission" is the one it echoed.

Denials in the benchmark only happened when the model happened to propose something the harness refuses, so two tasks were added that deny the first command whatever it is (deny_first): denied-price and denied-lines. Their check fails any reply that mentions "permission".

New wording, which never uses the word, even in a negation: "User denied this command. It was their choice not to run it, and nothing is wrong with your access. Don't guess what it would have output. Try a different command that gets the same information, or ask the user how they want to proceed." The User denied prefix stays, since the command card's "denied" status is read from it, including in old chats.

full, --runs 5, run twice for each wording (10 runs per task):

Changed: the denial message. The "permission" excuse almost disappeared. The pass rate didn't really move: the gap is within noise. The remaining failures have other causes. The model often looks for large_log.txt in data/, where it isn't, and the precondition check rejects the path. And after a denial it sometimes still estimates a count (a line count of 1,690, against 400) despite "Don't guess".

E19: Asked to run something in a chat without a folder#

The shell tool is only offered once a folder is attached. In a chat without one, asked "can you run the command to check it?" after a version had come up, qwen2.5-3b answered with a made-up "example output" (Ollama version: 0.34.2, copied from the conversation) and an invented endpoint (localhost:1885/versions), which reads as if a check was done.

Offering the tool in every chat was considered and rejected: the tool makes the model reach for commands it doesn't need (E15, E17), so every chat would fill with approval prompts, and approval only protects you if you read each one.

Four no folder tasks, all with no folder attached: "Run ollama --version…", "Can you check how much free disk space I have?", the two-turn case above, and a plain "What is the capital of France?" as a control. Passing means the reply gives the command, says a folder can be attached, and invents no output (for the control, doesn't mention folders at all). full, --runs 3:

• Baseline: in a fresh chat the model already said it had no access and gave the command, but never mentioned attaching a folder (it can't know Flint does that). On the follow-up it opened with "Certainly!" as if about to run something, and gave the right command 1 time in 3. No invented output showed up in these runs; the live case had more context.
• A (rejected): a note appended to the last message of every folder-less chat. It fixed the direct requests but leaked into unrelated answers: "The capital of France is Paris. If you need to run a command…". The follow-up failures were wrong commands (pip show ollama, a GitHub API curl), which is the model's knowledge, not the prompt.
• B (adopted): the same note, added only when the latest message matches run / execute / check / command / terminal / shell (asksToRun in tools.go), decided in Go like E17's skip. It sits next to the message, where E9 and the nudge placement found it holds.

Changed: candidate B. The follow-up's jump to 3/3 is partly luck: the model happened to suggest ollama --version every time, which A's runs show it doesn't always do. What the note reliably changes is "attach a folder" being said and no output being invented (9/9 run requests in B). Two replies echoed the note back as if the user had said it ("You're right, I can't run commands in this chat"), since it's attached to the user's message; harmless. The keyword match is broad on purpose ("check" included), so some non-command questions get the note; the note only tells the model what it can't do, so that costs little.

E20: Which embedding model re-ranks @web results best#

@web re-ranks Brave's top 10 by cosine similarity to the query and keeps 3. Asked whether a smaller model than nomic-embed-text (274 MB) would do, 15 real Brave searches (technical, current events, weather, health, how-to) were saved once and scored offline by docs/tools/web_rerank.py, embedding exactly what Flint does (raw title + snippet, one call). All 150 results were graded from title and snippet before any model ran: 2 answers the query, 1 on topic, 0 off topic or stale (the Go 1.23 and 1.25 notes for "latest stable Go release"). Score is the grade sum of the kept 3, out of 6 per query.

• nomic was trained with those task prefixes and Flint sends none. Adding them fixed the worst misses: "ollama keep_alive default" went 2 -> 5, the World Cup final 4 -> 6.
• arctic 33m beats unprefixed nomic but loses to prefixed nomic, which wins or ties it on 11 of 15 queries.
• Cold load plus 11 embeddings is ~1.6 s for both (5 runs each): the time is Ollama starting a runner, not the model's size. Since embeddings now unload right after the call, nomic's larger footprint is held for about that long; it matters only if it pushes the chat model out of a full GPU.
• snowflake-arctic-embed:m (110M, 218 MB, 180 MB loaded), close to nomic's size, did worse: 65/90 (nDCG 0.783) with its query prefix, 57/90 (0.671) without, below Brave's own order.
• One labeler and 15 queries, so a 4-point gap is modest evidence.

Changed: nomic stays the model, and rankByRelevance now sends it the search_query: / search_document: prefixes. Not tried: bigger embedding models (mxbai-embed-large, embeddinggemma, bge-m3) and arctic-embed2.

E21: @web stated a month-old version as the latest#

@web what is the latest Ollama release? answered 0.33.x on 2026-09-26, when GitHub's latest stable was 0.34.4 (v0.40.0 existed only as an rc). Of Brave's 10 results, no snippet contained 0.34.4: the fresh pages (GitHub releases, releases.sh) had snippets without a version, and the only version numbers came from SEO "latest version" posts dated 2026-08-28 and 2026-04-10. The E20 re-rank kept exactly those, since they restate the query. Brave sends page_age per result and Flint dropped it, so neither the model nor the user could see they were old.

Kept each result's date, shown in the sources card and given to the model as a Published: line, plus a note after the results. qwen2.5-3b, real Brave results, fresh chat per run:

• Before: 3/3 stated v0.33.2 undated; all three did say to check the official repository.
• Note naming today's date ("Today is … trust the most recently published … say how old your source is"): 3/3 dated the source, but 2/3 opened with "based on information available up to September 26, 2026", reading today's date as the information's date: more confident than before.
• Same with an example answer format and "never say the answer is current as of today": 4/5 dated, 1/5 still "as of 2026-09-26".
• Adopted: the example format without today's date ("v1.2, according to a page from 2026-08-28; a newer one may exist"): 5/5 dated, 0/5 "as of today", 4/5 said a newer release may exist.
• Leak check, @web who won the 2026 FIFA World Cup final?: 3/3 correct, two added the report date, none added version hedging.

Follow-up, the same day: a date the model made up. @web on the user's own framework (lcore) answered "v0.0.5, according to a page from 2023-04-07". The version was right; the date appears nowhere: Brave sent no page_age for any of the 3 kept results, and the note's example format pushed the model to fill one in. Then a mixed set (the same results, only the stale PyPI one dated), sent directly to Ollama: 5/5 went wrong, either preferring 0.0.3 as "the most recently published" (the only dated result wins the "trust the newest" rule) or pinning PyPI's date on GitHub's v0.0.5. So Go decides: dates and the note reach the model only when every kept result has a date. With none or only some dated, it sees no dates: lcore via the real API 5/5 said 0.0.5 with no date; the mixed set 4/5 said 0.0.5, one gave both with their sources, none gave a date. The all-dated case keeps the adopted wording unchanged.

Changed: results keep page_age as a date; the note is the adopted wording, sent only when every result is dated. Not fixed: the answer is still 0.33.1, because the right number is in no snippet; only reading the page itself would give it, which would send requests to sites other than Brave. A saved search from before this still parses, without dates.

E22: qwen3.5 refuses a system message that isn't first#

@memory lcore on qwen3.5-4b failed with Ollama 500: "Jinja Exception: System message must be at the beginning". Its template has an explicit raise_exception for any system message past the first. Flint places several there on purpose (summaries, context notes, @web results, @memory recalls, a folder attached mid-chat), so on qwen3.5 every chat using one broke. Direct curl, [user, assistant, system, user]: qwen3.5 500, qwen2.5-3b and phi3:3.8b fine. The same content as a user message: qwen3.5 accepted it and answered from the recalled memory.

• Adopted: on that exact error, resend once with every later system message as a user message, and remember the model (in memory, until restart). The rejection comes before any token, so nothing is shown twice. Models that accept later system messages are untouched, since E9-E19's placement results were measured on qwen2.5.
• Rejected: always sending them as user messages. Simpler, but it changes the tuned qwen2.5 prompts and would need the benchmark rerun.

Live, fresh chat, "hi" then @memory lcore: qwen3.5 3/3 answered from the memory ("lcore is your single-file Python WSGI framework..."); a fourth run streamed its thinking but produced no answer and nothing was saved, so it went through but likely ran out of room thinking, a separate problem. qwen2.5-3b and phi3 never trigger the fallback; phi3 echoed the memory block and misdescribed it, which is the model.

E23: The window size for cloud models and bigger GPUs#

A chat with nemotron-3-ultra:cloud showed "157 / 4.1k": the fixed 4096/8192 window was meant for a 6 GB GPU, and a cloud model reports context_length 262144. Checked first whether ollama.com honors num_ctx: a code word plus ~10.9k tokens of filler, sent with num_ctx 4096 and again with 32768. Both processed all 10,924 prompt tokens and answered the code word from the prompt's start. So Ollama cloud ignores num_ctx, and the only effect of Flint's 4k was its own compaction and meter treating a large model as a small one. The same fixed number also held back anyone with a bigger GPU.

Changed: two account settings, local and cloud (Settings → Connection, empty = Auto; one shared box would have dragged a cloud model down to a small GPU's number or the reverse), defaulting to the old 4096/8192 for local and 32768 for cloud, capped at the model's context_length from /api/tags (every installed model reports one: qwen2.5-3b 32768, phi3 131072, qwen3.5-4b 262144). Cloud gets 32k, not its full window, because each turn resends the kept history and a full window could mean 100k+ tokens of quota per turn. Verified live: a custom 16384 loaded qwen2.5-3b at 16384 (/api/ps) with the meter at 16384; Auto on the cloud model reported 32768. Not measured: recall and tool use at bigger windows on the small models (see the open question on a bigger window).

E24: Concurrency and planning for @agent, before any code#

Measured before designing @agent, which runs subtasks as separate model calls. RTX 4050 Laptop (6 GB, ~900 MiB used by the desktop), Ollama 0.34.2, num_ctx 4096, 200 generated tokens per request, temperature 0. Every concurrent request had a different prompt from its first token, so a shared prefix couldn't hide queuing.

The stock install queues. With the systemd service as installed (OLLAMA_NUM_PARALLEL unset), requests to one model run one at a time: four ~2k-token requests on qwen2.5-3b finished at 3.6, 6.7, 10.3 and 13.4 s, with constant tokens/s. A temporary second server with OLLAMA_NUM_PARALLEL=4 (own port, same models, system service left alone) ran them together:

Two at once on parallel 4 cost almost nothing (qwen2.5-3b 4.0 s, llama 4.0 s, 1.5b 2.2 s). All three GQA models stayed fully on the GPU at 4 slots, each slot with the full 4096 (n_ctx_slot). phi3 has no GQA, so four 4k KV caches didn't fit: 14 of 33 layers on the GPU, and four requests took longer than queuing them. Some requests in a round hit Ollama's prompt cache from the previous round (prompt eval 0.02 s instead of 0.5 s); that only affects the prompt-eval share, not the queuing pattern.

Two models don't share the GPU. qwen2.5-3b and llama3.2:3b sent together: on both servers /api/ps only ever showed one, and each switch reloaded (6-10 s load_duration); alternating four requests took 15 s stock and 28 s on parallel 4, where llama also spilled 150 MiB to the CPU. A different num_ctx on the same model reloads it too (qwen2.5:1.5b, 4096 → 8192 → 4096: ~2.5 s each time).

Plan call with format. A JSON schema (list of subtasks, each with task, files, web_query) on four tasks: one-line summaries of three files, three questions about three different files, one question about one file (shouldn't split), and a two-part current-events comparison with web search allowed. The file list with sizes was given, not the contents. All 32 responses (two prompt versions × four models × four tasks) were valid JSON matching the schema, phi3 included.

• A split boolean in the schema was useless: it said false next to three good subtasks on every qwen2.5 and llama plan.
• First prompt: the file tasks split well on all four, except llama3.2 wrote guessed summaries as the tasks ("The config.yaml file stores ... API keys"). Web: qwen2.5-3b returned nothing, llama and 1.5b one subtask with every file and no query, phi3 queries plus unrelated files.
• Second prompt (no boolean, "an instruction, never the answer", one file example and one web example): qwen2.5-3b 4/4 good plans. qwen2.5:1.5b split the single question into two web queries although web was off. llama3.2 searched for Go but forgot Rust. phi3 invented go.txt/rust.txt as inputs and copied the example into an extra subtask on the single-file question.

What this settles for @agent: - Split is decided in Go: two or more subtasks left after validation, never a model flag. Validation drops files not in the folder, clears web_query without a Brave key, and drops a subtask with no inputs left. With those rules qwen2.5-3b and 1.5b plan 4/4, llama3.2 and phi3 3/4 (one sample each at temperature 0, so small n). phi3's copied subtask survives validation: the approval card is where the user catches it. - Agents send the chat's exact num_ctx; anything else reloads the model mid-run. They use the chat's model, since a second model swaps. - Recommended local max-agents default: 2. On the stock install Ollama queues anyway, so 2 costs nothing and the context isolation is the benefit. With OLLAMA_NUM_PARALLEL=2 or more, two agents finish in about the time of one on every GQA model here, at +500 MiB (qwen) to +1.3 GB (llama) for four slots. Four slots already spill phi3 on 6 GB, so going above 2 is for bigger GPUs, set per account.

In the app (phase 3). The same rules run in Go, on the fixture through the real plan call. Two changes came from it:

• qwen2.5:1.5b split three-part questions correctly but left files empty and named the file in the task instead ("Find the value of TAX_RATE in utils.py"), so validation dropped every subtask. A folder file named as a whole word in the task is now added to its inputs: with that, 1.5b plans the code-files, log and summary questions right.
• A new chat has no token calibration, so at ratio 1 the 24.7 KB log (~11k real tokens) counted as fitting an 8192 window. Agent inputs are files, not the chat's own text, so they are sized as dense text, 2 chars per token: the log is planned as cut to 13 KB, start and end.

Still wrong on every model: "who is on call this week" never went to notes.txt (sent to README.md, or dropped), since the planner sees file names, not contents. The card lets the user add the right file. qwen2.5:1.5b also split a single-file question into two near-duplicate subtasks; llama3.2 and phi3 correctly didn't split it.

E25: @agent against a normal chat#

Baseline, before any agent code#

Four split tasks added to the benchmark (independent questions about different files; see benchmark.md), run as a normal chat, full configuration, 3 runs each:

• qwen2.5-3b's one miss: its grep for the ERROR line was cut short by a pipe, and it reported no error code.
• llama3.2:3b fails every run the same two ways: it writes tool calls as JSON text in its reply ({"type":"function","function":{"name": "run_shell",...}}) instead of a structured call, so nothing runs, or it announces commands for facts already in the folder context and stops there. It never proposed a usable command.
• qwen2.5:1.5b never ran a command for the log and made up its error code ("2", or fail.py's message); once it left two of three answers blank; once, after the filler turns, it answered an earlier turn instead of the question.

While building the agents#

Found live on the fixture, before the benchmark:

• A JSON format switches tool calls off. With format and tools in one request, qwen2.5-3b and 1.5b never called the tool and answered straight away (1.5b claiming found: true with nothing read), and llama3.2 put the answer schema's fields into its tool call. So an agent that may run commands works without a format, and one last call with the result schema and no tools gives the result.
• 2 chars per token wasn't enough for agent inputs. The log cut to that size came to 8214-8320 tokens against the 8192 window (about 1.75 chars per token). Without a format Ollama rejected it; with the result format it truncated silently, so the agent never saw the log and answered "not found" or invented one. At 1.5 chars per token all three models found ERROR 4471 in the one result call (6 of 6 prompts), so a two-step "answer, then JSON" variant wasn't needed.
• Agents narrate instead of calling the tool. Without the chat's tool nudge, qwen2.5:1.5b wrote "I'll look through the folder" or a command in a code block, then invented answers in the result call ("port 8080", "John Doe", found: true). With the nudge, qwen2.5-3b explored properly (its no-file on-call agent read config.yaml, then notes.txt, and answered Omar). 1.5b still answered those without a tool call, so an agent given nothing that ran no working command now has its answer replaced in Go with "nothing to answer from".
• Given its file, an agent should just read it. A qwen2.5-3b agent that had config.yaml as input asked for more commands (a grep, then Python one-liners); another, with README.md in its prompt, ran three greps that failed (+ without -E) and reported the owner's email as not found, so the combined answer said it was missing. Now only an agent given no files or search is offered the shell. The same task then answered 4471 and dana@stockroom.test with no commands asked.
• The combined answer (phase 5), log + README task: qwen2.5-3b and qwen2.5:1.5b both answered both parts correctly once agents read their inputs, in 3-4 s after Run. On the three-part task, qwen2.5-3b's planner gave the on-call subtask utils.py; its agent said "not found" and the answer said the on-call information wasn't found, rather than inventing a name.
• The whole flow in Brave (phase 6, Playwright, fresh server): plan, card edits, Run, approvals in the main chat, the answer, the agents window and a reload all worked with no console errors. It caught one bug: files named in a subtask's text were also added back to an edited plan, so clearing a subtask's file on the card didn't stick when its instruction mentioned the file. That rescue now only applies to the model's own plan. phi3:3.8b (no tool support) answered the log
• README task correctly with no commands asked. With a clean instruction ("Find who is on call this week") and no files, a qwen2.5-3b agent read config.yaml, then notes.txt, and answered Omar; with the planner's own wording ("... If no schedule is available, state 'On-call schedule not available.'") it gave up after one failed command, and the evidence check reported that honestly.

E26: An agent with no files searched for a file name#

Seen live: an @agent subtask "who is on call this week" got no files, and its agent ran ls data | grep on_call_schedule.txt, then ... | grep schedule.txt. Both came back as a bare [FAILED, exit code: 1]. The answer was in notes.txt ("On-call this week: Omar."), which it never searched.

Test: the fixture, the three-part task planned, then run with a single subtask "Who is on call this week?" and no files. Read-only commands approved as in the benchmark, pass if the agent's result names Omar.

• Before, qwen2.5-3b guessed names (ls *schedule*.txt, *calendar*, *on_call*) or grepped for "on call", which can't match "On-call". A grep with no match looked the same as a broken command.
• The prompt alone fixed the shape: it now searched the whole tree with grep -rni ... .. It still used the phrase "on call" and then other guesses ("rotation", "roster"), ignoring "one short word" in the system prompt.
• A hint in the result, next to where it generates, worked: after a no-match it retried with "call" or "week" and found notes.txt. The failures ran out of commands still trying phrases. Wins took 1-6 commands, each one an approval.
• qwen2.5:1.5b called no tool in any run, with or without the change; its result stays the "nothing to answer from" fallback from E25.

E27: File names in an answer, checked against the folder#

The shield and preconditions check commands, but nothing checked the answer itself. Built: after a folder-chat reply, names with a file extension (backticked or with a slash, not inside code blocks) are checked against the folder, by exact path or as the tail of a path in it. Ones not found are listed under the reply. Here the model never saw the list; E28 tells it.

Test: qwen2.5-3b-instruct, 8 questions that invite naming files, 7 of them "without running commands".

• With the whole folder in view, the model named only real files (around 20 mentions), and nothing was flagged.
• With the code in subfolders, it invented paths in 3 of 5 answers: backend/run_shell.go, frontend/index.js, backend/schema.sql, migrations/001_create_users_table.sql. All 9 flags were invented names. Two came in text before a tool call, so the list also travels on the tool-call marker.
• Not flagged, by design: names without an extension (backend/migrations/), and member names like console.log or process.env unless written with a slash. The unit test holds these cases.
• Telling the model is E28.

E28: Telling the model its last answer named missing files#

E27 shows invented names to the user only. Tried: when the last answer named files that aren't in the folder, the next request gets one line next to the anchor, listing them and saying "Don't present them as real: check with a command, or say you aren't sure." Ablatable as grounding.

Test: qwen2.5-3b-instruct, Flint's repo root attached. The two E27 questions that drew invented paths, then one follow-up: "show me what's in the first file you named" (depends on the answer) or "in one sentence, what is this project?" (unrelated, to catch side effects). 5 runs per question, follow-up and arm, shuffled. A run counts only if the first answer named a missing file (28 of 40). Replies classified by reading them; a keyword match proved unreliable.

• Without the note the model built on its own invention: cat backend/schema.sql, or "the first file, likely backend/run_shell.go" followed by echo hello. Preconditions caught the cats, but only after the model had committed to the name.
• With it, it hedged and checked ("I'm not sure if the chat.db file is present ... I will run a command") or moved to a file that exists.
• Side effect: on an unrelated question it sometimes added a sentence about verifying the missing files. No apologies in either arm. Fewer commands were proposed with the note, so no extra approvals. Since E29 the anchor lists the folder's real paths, and only 1 in 30 first answers named a missing file (13/30 before), so the note, and this side effect with it, rarely comes up.
• Kept, on by default. The numbers are small (7-8 runs per cell) and one model. Re-measure if the wording changes.

E29: Real file paths in the anchor#

In E28, 28 of 40 first answers named a file that doesn't exist. The model only saw the folder's top level, so it guessed what was in the subfolders. Built: the anchor lists real file paths, grouped by folder and shallowest first, capped at 2400 characters (Flint's repo fits in ~2400, about 600 tokens). Hidden folders, node_modules, __pycache__ and vendor are skipped; a cut says how many files weren't listed. The E28 note also suggests a real path when a missing name matches one by file name or case (agents.md → AGENTS.md). Ablatable as tree and suggest.

Test: qwen2.5-3b-instruct, Flint's repo root attached, six "without running commands" questions with one known file each (shield.go, chat.js, ratelimit.go, websearch.go, flint.css, tasks.py). 5 runs per question and arm, shuffled.

• Without the paths the model either said it couldn't tell or invented a likely name. All 3 correct answers were the same question (chat.js).
• With them, the 5 answers that only matched by keyword pasted most of the list back or named chat.html before chat.js.
• Benchmark (fixture folder, 20 tasks, 2 runs): full 30/40, no-tree 26/40. No category got worse. list-data still used a command in both arms.

E30: Read, write and edit tools#

Before, the model's only tool was run_shell: every file read was a cat you had to approve, and an edit was a sed -i or heredoc you had to decode. Built: read_file (runs without approval, confined to the folder), and edit_file / write_file, shown as a diff and written only when approved. Ablatable as filetools.

Format test first, straight to Ollama: qwen2.5-3b-instruct with all four tools offered, given a file as numbered read_file output and asked for one of six edits, 5 runs each. A run passes only if the edit leaves the file exactly right.

• Line ranges lost indentation: max_connections: 50 without its two leading spaces, which breaks the YAML silently, and stray trailing newlines. Text replacement never did, so it was kept.
• Adding code is weak in both. The model anchored on an existing line and replaced it (price_with_tax(price) became the new function), and never used the empty old_text. The diff card is what catches this.
• Typo task: 0/5 in both. It didn't see inventroy and "fixed" something else. A perception failure, not a format one.

Live, four fixes, each from a failure seen in a real chat:

• Asked to change a price in data/prices.csv (not in the manifest), it guessed the old line (B-220,4.00) twice. Both edits were refused without asking you, which is right, but then the 3-step cap ran out and it gave up. Now a refused edit returns the file's real lines, and the cap counts only steps that asked you (reads and refused proposals are bounded by a total of 8). After the fix, 3 of 3 runs reached the correct diff.
• It once asked to read from line 1000000000000000; an out-of-range start now reads from the top instead of failing.
• A decided edit lost its diff on reload; every edit card now keeps it.
• Found in review, not in a chat: read_file skipped the shield, so with the home folder attached it could read ~/.ssh/id_* unasked. All three file tools now check the real path, after symlinks and .. are resolved, against the shield's secret-file rule, so a link can't dodge it; and with a cloud model a read becomes a pending card that runs only when approved. Its tests showed the rule itself never matched id_ed25519 (letters only before a word boundary), in the shell shield too; fixed.
• Given the applied diff back as its tool result, the model pasted it into its reply, where markdown turned the -/+ lines into a bulleted list. The result now says only "wrote config.yaml (1 line added, 1 removed)"; the card shows the diff, and the model wrote the change itself.

Edit before writing, added after 0.3.0: the diff card's Edit opens the proposal's new lines for you to correct; your lines replace only that region, and the model is told what was written. Checked in the browser: max_connections: 50 proposed, corrected to 60, written, the card showing the 25 → 60 diff, and the model's reply acknowledging 60. Seen in the same session: asked to change B-220's price, the model proposed old_text: "4.50\nB-220" (the line's parts reversed) six times in a row, each refusal showing it the real line, until the 8-step cap ended the turn. The caps held, but the model fixated.

Pointing at the line. When old_text isn't found and exactly one line contains every part of it, the refusal now names that line (ablatable as edithint). The edit-price task, 4 runs each:

• The first wording fixed old_text and exposed new_text: the model then sent B-220,4.50 → 5.00, replacing the whole line with the price. Saying what new_text should be too fixed that.
• The one failure left isn't this path: old_text: "B-220" exists, so it was applied as asked and replaced the SKU (5.00,4.50). That's a wrong edit the diff shows and Edit can correct.
• Four runs per arm: a direction, not a measurement.

Benchmark, partial (7 edit tasks, commands inside the task's copy approved; stopped after 15 runs to spare the machine): with the file tools, 6/15 files came out exactly right. Change TAX_RATE 3/3, MAX_RETRIES 2/3, max_connections 1/3 (twice left untouched), on-call 0/3 (wrote the request in literally: "Priya is on call this week, not Omar."), the price 0/3 (replaced B-220 itself: 5.00,4.50). Every wrong edit was plain on its diff; the benchmark approves them anyway. Not measured yet: the same tasks through shell only (no-filetools), and whether offering four tools changes the other categories.

E31: Two-file edits on a cloud model#

Question: E30 held back a multi-file review card because qwen2.5-3b gets single edits wrong too often. Does a cloud model change that? Model: gpt-oss:120b-cloud, among the cheapest cloud models with tool calling ($0.15 in / $0.60 out per million tokens) and on the free tier. Three new two-file edit tasks: multi-rename (parse_sku to parse_code in utils.py and its caller), multi-config (add max_retries: 6 to config.yaml and read it in main.py, scored by running the code with the config at 6 and then 7), multi-port (the port in README.md plus a server section in config.yaml). 2 runs each, with the 7 single-file edit tasks.

The three errors are Internal Server Error from ollama.com mid-turn, all in shell-only runs; the cause wasn't found. Not counting them, shell only missed nothing, and the file tools missed one task twice.

• Both file-tool misses were the same: the model read both files, renamed the definition, then edited only the import line of main.py and left the call parse_sku(order["sku"]) below it, and said "All references now use the new function name." Each file's edit was right as far as it went; what was missing was a second edit in the same file. Its diff card shows only the import changing, so a reader would catch it. Shell got it with sed 's/parse_sku(/parse_code(/g', one command for every occurrence.
• The model calls tools Flint never offered: in shell-only runs it called apply_patch (13 times), which the precondition check refused as a missing command, and sent some empty commands.
• The approval cap didn't cut any task short, for two reasons found here, neither intended. A cloud read waits for approval, but its result starts with [read , so the cap counts it as a free read: in full, only edits count toward 3, which is why multi-config could run 3 approved reads and 2 edits. And once the cap is reached Flint stops offering tools, but it still runs any tool call that comes back: gpt-oss keeps calling them, so shell-only turns went to 9 approvals. qwen2.5-3b stops when no tools are offered, which is why this never showed before. The cap was left as it is; how to handle both is a decision, not a fix made in this run.
• Harness: the shell-only multi-config runs first failed 0/2 because the benchmark denied sed -i '/max_connections: 25/a…', reading the sed address as an absolute path. Fixed (only a real top-level directory counts) and re-run; the table has the re-run. Pending read_file cards are now approved too.

What it means: on this model, single-file edits are no longer the bottleneck (14/14 against qwen's 6/15), and two-file changes mostly work through one card per file. The failure seen was a missed second edit in the same file, which a turn-level review card wouldn't catch any better than today's cards. A tool that edits several places in one call would address it directly. Two runs per task: a direction, not a measurement.

E32: Multi-file edits for cloud models#

Built after E31, for cloud models only (a local model keeps the E30 schema and one card per edit), each ablatable:

• multiedit: edit_file takes edits, a list of old_text/new_text pairs applied in order. Diffs became a real line diff (LCS on the changed middle) with one hunk per change, as diff -u groups them; before, one block ran from the first change to the last.
• staging: a turn's edits are staged, not proposed one by one. Later reads and edits in the turn see them; after the reply, one review card shows each file's combined diff, with Write all / Discard all. A shell command while edits are staged isn't run (it would see the old files). Each staged result is rewritten with the outcome, so the next turn knows what reached the disk.
• leftover, added when the first two didn't fix E31's miss: an edit that removes a name the file still uses says so in its result, with the lines ("Your edit removed names that main.py still uses: parse_sku (line 7)").
• Every diff card, local ones too, got a checkbox per hunk; only the checked hunks are written, and the model is told which weren't.

gpt-oss:120b-cloud, the three two-file tasks, 3 runs each (no-cloudedits is E31's behaviour):

• The rename is about the note, not the card. Pooling every multi-rename run so far: with the leftover note 6/6; without it 7/12 (E31's 0/2, then 2/2 and 3/3 with one card per edit, 1/2 with edits, 1/3 staged). Each miss was the same: import renamed, parse_sku(order["sku"]) left, "all references updated".
• edits was rarely used: 5 of 36 runs passed it, and it never was what fixed a miss; the model mostly sent "edits": [] next to old_text/new_text. Kept, since it's cheap and lets one edit carry every call site when the model does use it.
• Staging halves the approvals (3.1 against 5.9 per run: the cloud reads plus one review, instead of a card per edit) at the same pass rate. That, not accuracy, is what it buys. All 7 single-file edit tasks also passed staged (7/7).
• Checked in the browser against a real build: a rename staged as two cards marked "Staged for review", one review card for utils.py and main.py, Write all wrote both (no parse_sku left), the cards became Succeeded, and after a reload no review reopened.
• Three runs per arm, one model: the leftover note's effect is large enough to trust the direction, not its size.

Open questions#

• qwen3.5 thinking with no answer. Once in E22 it streamed a long thought and never answered, leaving nothing saved. Unmeasured: how often, and whether the 1024-token reply reserve is what runs out.
• Commands for questions that don't need one. E17 fixed explicit "don't run commands". Simple arithmetic still goes to bc or python3 -c, with or without the nudge.
• Summary accuracy on 3B. Counts are sometimes wrong (E6). User facts no longer depend on the model (E7/E11). Its notes on files and commands still do.
• qwen3.5-4b summary speed. 26-40 s per summary. It runs in the background, but each one holds Ollama's only request slot, so the next chat message waits behind it.
• A bigger window. Memory allows 32k for qwen2.5-3b here (E12). Still unmeasured: prompt processing time and recall quality at 16k and 32k, since small models get worse at using facts buried in a long prompt. A bigger window adds margin but doesn't replace compaction.