A growing chat history can increase the next model request’s input size and eventually exceed its context limit. In LangGraph the fix has two parts. The checkpointer keeps the whole thread, and each model call gets a trimmed view built with trim_messages from langchain-core. In this lesson the code-review assistant aims for an approximate 400-token budget per call. The cost of trimming is that the model forgets. With the same five turns, the thread without a pinned decision lost who reviews #412; the thread that pinned it kept the answer:
When you need this. A reviewer keeps one thread open all week: a decision on Monday, a pasted CI log on Wednesday, a question about that decision on Friday. Trim history to reduce the next model request, and keep selected decisions separately so the application can include them after older turns leave the view.
graph.py · StateGraph(State) · InMemorySaver · gemini-3.8-flash · recorded in runs.json
1. Keep the thread in state and trim only the model’s view
Download the complete lab, extract it and open a terminal in langgraph-context-budget/. The README lists its runtime and recorded examples.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txtFollow the lab setup and run instructions before executing this lesson.
Use the extracted lab and its virtual environment.
"""A code-review chat graph. The checkpointer keeps the whole thread; the model sees a trimmed view."""
import operator
import os
import sys
from typing import Annotated
from langchain_core.messages import AIMessage, AnyMessage, HumanMessage, SystemMessage, trim_messages
from langchain_core.messages.utils import count_tokens_approximately
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_ollama import ChatOllama
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.graph.message import add_messages
from typing_extensions import TypedDict
BUDGET = 400 # approximate tokens the model may see per turn: instructions, pinned notes, history, question
SYSTEM = """You help a developer work through the team's code-review queue. Answer in at most three
sentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."""
MODEL = os.environ.get("REVIEW_MODEL", "gemini-3.8-flash") # REVIEW_MODEL=gemma4:e2b runs it on Ollama
class State(TypedDict):
messages: Annotated[list[AnyMessage], add_messages] # the whole thread; new IDs append, matching IDs replace
pinned: Annotated[list[str], operator.add] # decisions the app pins; never trimmed
view: dict # what the model saw this turn, for the recording
class ContextTooLarge(ValueError):
pass
def count_tokens(messages: list[AnyMessage]) -> int:
"""Heuristic: character estimate plus prior reasoning usage, calibrated against these saved Gemini calls."""
return count_tokens_approximately(messages) + sum(
((m.usage_metadata or {}).get("output_token_details") or {}).get("reasoning", 0)
for m in messages if isinstance(m, AIMessage))
def model_view(state: State, budget: int = BUDGET) -> list[AnyMessage]:
notes = "".join(f"\n- {note}" for note in state.get("pinned", []))
system = SystemMessage(SYSTEM + (f"\n\nPinned decisions:{notes}" if notes else ""))
room = budget - count_tokens([system])
history = trim_messages(state["messages"], max_tokens=room, strategy="last",
token_counter=count_tokens, start_on="human", end_on="human")
if not history: # even the question alone does not fit
raise ContextTooLarge(f"the question needs more than the {room} tokens left after the instructions")
return [system, *history]
def chat_model():
if MODEL.startswith("gemini"):
return ChatGoogleGenerativeAI(model=MODEL) # reads GOOGLE_API_KEY from the environment
return ChatOllama(model=MODEL, temperature=0, reasoning=False)
def build_graph(model=None, budget: int = BUDGET, checkpointer=None):
model = model or chat_model()
def respond(state: State) -> dict:
view = model_view(state, budget)
reply = model.invoke(view)
return {"messages": [reply], "view": {
"sent": len(view) - 1, "stored": len(state["messages"]),
"estimated_tokens": count_tokens(view), "chars_only_estimate": count_tokens_approximately(view),
"server_tokens": (reply.usage_metadata or {}).get("input_tokens")}}
builder = StateGraph(State)
builder.add_node("respond", respond)
builder.add_edge(START, "respond")
builder.add_edge("respond", END)
return builder.compile(checkpointer=checkpointer or InMemorySaver())
if __name__ == "__main__":
graph = build_graph()
config = {"configurable": {"thread_id": "cli"}}
for line in sys.stdin:
state = graph.invoke({"messages": [HumanMessage(line.strip())]}, config)
print(state["view"], "|", state["messages"][-1].text)The graph has one node, but its state has three fields, and each has a job:
messagesis the whole thread.add_messagesappends each turn, and the checkpointer saves it perthread_id, so trimming the model’s view leaves the stored thread intact.pinnedholds decisions the application chose to keep. Its reducer isoperator.add, so each update appends.model_viewputs it in the system message and passes only the history totrim_messages.viewrecords what the model saw on this turn. It exists for the recording and the tests.
The alternative is to trim messages itself, by returning RemoveMessage updates. That removes those messages from the current graph state; older checkpoints can still retain them. Trimming the view costs a little more storage and keeps the choice reversible: a later budget, or a summary, can still use the full thread.
A trimmed decision is forgotten. The decision about #412 was trimmed out of the view, and the model answered: “I do not have the information regarding who reviews #412 or when it must merge.” Pin decisions in their own state field and put them in the system message. Check: test_a_trimmed_decision_is_gone_from_the_view, test_a_pinned_decision_survives_trimming.
2. Trim with trim_messages
With the thread safe in state, we can be strict about the view. model_view builds the system message first, so the history gets only what is left of the budget. Then trim_messages keeps messages from the end of the thread until the next one would not fit:
strategy="last"keeps the most recent messages.start_on="human"drops messages from the front until the view starts with a user turn. A view that starts with an AI reply, or with a tool result whose call was trimmed away, is not a valid conversation for most chat APIs.end_on="human"makes the view end on the current question.- If even the question does not fit, the result is empty.
model_viewraisesContextTooLargeinstead of calling the model with no question.
A question larger than the budget. The 60-line paste raised ContextTooLarge before any model call. Refuse it, or store the paste and send a reference to it. Check: test_a_question_larger_than_the_budget_is_refused_before_the_model_call.
The under-count is the failure to plan for. A budget enforced with an estimate is only as good as the estimate, and the error depends on the text. The reported input count arrives after the call, in usage_metadata, so it can’t stop an oversized request; it can tell you that your margin is too small.
3. Compare estimates with reported usage
trim_messages needs a token_counter. count_tokens_approximately from langchain-core estimates from characters, about four per token by default, and needs no tokenizer or network call. On Gemini it missed a large part of the input. The saved Gemini replies include thought signatures in their content blocks. The reported input usage for resubmitted replies was larger than the character estimate in these runs. On the last turn of the first thread, the request included 3 history messages plus the system message:
| Counter | Tokens |
|---|---|
count_tokens_approximately alone |
109 |
count_tokens in graph.py |
271 |
Reported by Gemini (usage_metadata.input_tokens) |
255 |
In the recorded runs, the extra reported input was close to the reasoning tokens that the replies in the view had used, which LangChain reports in usage_metadata.output_token_details.reasoning. count_tokens adds them to the estimate. That figure is measured from these runs and no documented formula gives it, so the recording keeps checking the estimate against the reported input count.
The estimate misses thought signatures. The character estimate gave 109 tokens for a view reported as 255. Count the reasoning tokens each reply carries back. Check: test_the_counter_adds_the_reasoning_tokens_a_reply_carries_back, test_recorded_run_chars_only_estimate_missed_the_signatures.
Dense text is under-counted. The CI log turn was estimated at 292 tokens and reported at 501, over the 400-token budget. Log lines with paths and numbers have fewer characters per token. Keep the budget well below the hard limit, and alert on reported input above it. Check: test_recorded_log_turn_was_billed_above_the_budget.
4. Run a thread on Gemini
record.py runs the same five turns on two threads. In the second, the application pins the decision about #412 when the user states it. Then it sends a 60-line log on a third thread.
python record.py== thread review pinned=[]
sent 1/2 messages estimate 83 (chars only 83) billed 74 | For #415 we agreed to keep DEFAULT_TIMEOUT a
sent 1/4 messages estimate 78 (chars only 78) billed 72 | mlee reviews #412, and it must merge before
sent 1/6 messages estimate 292 (chars only 292) billed 501 | Here is the CI log from #415. Does anything
sent 1/8 messages estimate 70 (chars only 70) billed 60 | Draft a one-line review comment for #415.
sent 3/10 messages estimate 271 (chars only 109) billed 255 | Who reviews #412, and when must it merge?
last reply: I do not have the information regarding who reviews #412 or when it must merge.
== thread review-pinned pinned=['mlee reviews #412, and it must merge before the 2.5 freeze on November 14.']
sent 1/2 messages estimate 83 (chars only 83) billed 74 | For #415 we agreed to keep DEFAULT_TIMEOUT a
sent 1/4 messages estimate 102 (chars only 102) billed 103 | mlee reviews #412, and it must merge before
sent 1/6 messages estimate 316 (chars only 316) billed 532 | Here is the CI log from #415. Does anything
sent 1/8 messages estimate 94 (chars only 94) billed 91 | Draft a one-line review comment for #415.
sent 3/10 messages estimate 328 (chars only 130) billed 320 | Who reviews #412, and when must it merge?
last reply: mlee reviews #412. It must merge before the 2.5 freeze on November 14.
== a 60-line paste: ContextTooLarge: the question needs more than the 345 tokens left after the instructionssent 3/10 counts different moments. The numerator counts the history messages sent to the model, including the current question but excluding the system message. The denominator counts stored messages after the reply was appended, so it includes a reply that was not yet part of that request. Before the last call, the thread held 9 messages; the request selected 3 of them and added the system message. Every turn fit the estimate, and the thread kept all 10 messages after the final reply.
Compare plain and pinned context
The saved comparison runs two independent five-turn conversations. In the plain thread, trimming removed the reviewer and deadline; the model said it no longer had that information. In the pinned thread, the retained decision let it answer that mlee reviews #412 before the 2.5 freeze on November 14. The checkpointer kept ten messages per thread while the final model view kept three history messages plus the system message. The approximate 400-token budget was exceeded on the log turns, so use provider-specific token counting when enforcing a hard input limit.
Inspect the saved events
[
{
"event": "task",
"question": "Run the same five-turn review conversation in independent plain and pinned threads.",
"turns": [
"For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly.",
"mlee reviews #412, and it must merge before the 2.5 freeze on November 14. Acknowledge briefly.",
"Here is the CI retry log for #415. Summarize what it shows in one sentence.\n09:14:00 tests/test_client.py::test_retry_0 PASSED\n09:14:01 tests/test_client.py::test_retry_1 PASSED\n09:14:02 tests/test_client.py::test_retry_2 PASSED\n09:14:03 tests/test_client.py::test_retry_3 PASSED\n09:14:04 tests/test_client.py::test_retry_4 PASSED\n09:14:05 tests/test_client.py::test_retry_5 PASSED\n09:14:06 tests/test_client.py::test_retry_6 PASSED\n09:14:07 tests/test_client.py::test_retry_7 PASSED\n09:14:08 tests/test_client.py::test_retry_8 PASSED\n09:14:09 tests/test_client.py::test_retry_9 PASSED\n09:14:10 tests/test_client.py::test_retry_10 PASSED\n09:14:11 tests/test_client.py::test_retry_11 PASSED\n09:14:12 tests/test_client.py::test_retry_12 PASSED\n09:14:13 tests/test_client.py::test_retry_13 PASSED\n09:14:14 tests/test_client.py::test_retry_14 PASSED",
"Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines.",
"Who reviews #412, and when must it merge?"
],
"pinned_decision": "mlee reviews #412, and it must merge before the 2.5 freeze on November 14.",
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
}
},
{
"event": "turn_input",
"thread": "plain",
"turn": 1,
"question": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly.",
"pin": false
},
{
"event": "model_request",
"ordinal": 1,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly."
}
]
},
{
"event": "model_response",
"ordinal": 1,
"message": {
"type": "ai",
"text": "Understood. I have noted that for #415, `DEFAULT_TIMEOUT` should remain at 10s and a per-client timeout override should be added instead.",
"tool_calls": [],
"usage": {
"input_tokens": 76,
"output_tokens": 218,
"total_tokens": 294,
"input_token_details": {
"cache_read": 0
},
"output_token_details": {
"reasoning": 181
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "plain",
"turn": 1,
"sent": 1,
"stored": 1,
"estimated_tokens": 85,
"chars_only_estimate": 85,
"server_tokens": 76,
"reply": "Understood. I have noted that for #415, `DEFAULT_TIMEOUT` should remain at 10s and a per-client timeout override should be added instead."
},
{
"event": "turn_input",
"thread": "plain",
"turn": 2,
"question": "mlee reviews #412, and it must merge before the 2.5 freeze on November 14. Acknowledge briefly.",
"pin": false
},
{
"event": "model_request",
"ordinal": 2,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly."
},
{
"role": "ai",
"text": "Understood. I have noted that for #415, `DEFAULT_TIMEOUT` should remain at 10s and a per-client timeout override should be added instead."
},
{
"role": "human",
"text": "mlee reviews #412, and it must merge before the 2.5 freeze on November 14. Acknowledge briefly."
}
]
},
{
"event": "model_response",
"ordinal": 2,
"message": {
"type": "ai",
"text": "Understood. I have recorded that mlee is reviewing #412 and that it must be merged before the 2.5 freeze on November 14.",
"tool_calls": [],
"usage": {
"input_tokens": 326,
"output_tokens": 166,
"total_tokens": 492,
"input_token_details": {
"cache_read": 0
},
"output_token_details": {
"reasoning": 132
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "plain",
"turn": 2,
"sent": 3,
"stored": 3,
"estimated_tokens": 334,
"chars_only_estimate": 153,
"server_tokens": 326,
"reply": "Understood. I have recorded that mlee is reviewing #412 and that it must be merged before the 2.5 freeze on November 14."
},
{
"event": "turn_input",
"thread": "plain",
"turn": 3,
"question": "Here is the CI retry log for #415. Summarize what it shows in one sentence.\n09:14:00 tests/test_client.py::test_retry_0 PASSED\n09:14:01 tests/test_client.py::test_retry_1 PASSED\n09:14:02 tests/test_client.py::test_retry_2 PASSED\n09:14:03 tests/test_client.py::test_retry_3 PASSED\n09:14:04 tests/test_client.py::test_retry_4 PASSED\n09:14:05 tests/test_client.py::test_retry_5 PASSED\n09:14:06 tests/test_client.py::test_retry_6 PASSED\n09:14:07 tests/test_client.py::test_retry_7 PASSED\n09:14:08 tests/test_client.py::test_retry_8 PASSED\n09:14:09 tests/test_client.py::test_retry_9 PASSED\n09:14:10 tests/test_client.py::test_retry_10 PASSED\n09:14:11 tests/test_client.py::test_retry_11 PASSED\n09:14:12 tests/test_client.py::test_retry_12 PASSED\n09:14:13 tests/test_client.py::test_retry_13 PASSED\n09:14:14 tests/test_client.py::test_retry_14 PASSED",
"pin": false
},
{
"event": "model_request",
"ordinal": 3,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "Here is the CI retry log for #415. Summarize what it shows in one sentence.\n09:14:00 tests/test_client.py::test_retry_0 PASSED\n09:14:01 tests/test_client.py::test_retry_1 PASSED\n09:14:02 tests/test_client.py::test_retry_2 PASSED\n09:14:03 tests/test_client.py::test_retry_3 PASSED\n09:14:04 tests/test_client.py::test_retry_4 PASSED\n09:14:05 tests/test_client.py::test_retry_5 PASSED\n09:14:06 tests/test_client.py::test_retry_6 PASSED\n09:14:07 tests/test_client.py::test_retry_7 PASSED\n09:14:08 tests/test_client.py::test_retry_8 PASSED\n09:14:09 tests/test_client.py::test_retry_9 PASSED\n09:14:10 tests/test_client.py::test_retry_10 PASSED\n09:14:11 tests/test_client.py::test_retry_11 PASSED\n09:14:12 tests/test_client.py::test_retry_12 PASSED\n09:14:13 tests/test_client.py::test_retry_13 PASSED\n09:14:14 tests/test_client.py::test_retry_14 PASSED"
}
]
},
{
"event": "model_response",
"ordinal": 3,
"message": {
"type": "ai",
"text": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively.",
"tool_calls": [],
"usage": {
"input_tokens": 433,
"output_tokens": 39,
"total_tokens": 472,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "plain",
"turn": 3,
"sent": 1,
"stored": 5,
"estimated_tokens": 271,
"chars_only_estimate": 271,
"server_tokens": 433,
"reply": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively."
},
{
"event": "turn_input",
"thread": "plain",
"turn": 4,
"question": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines.",
"pin": false
},
{
"event": "model_request",
"ordinal": 4,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines."
}
]
},
{
"event": "model_response",
"ordinal": 4,
"message": {
"type": "ai",
"text": "Please update the client constructor to accept an optional `timeout_seconds` argument that falls back to the existing default when omitted to maintain backward compatibility, and add a regression test covering both the explicit value and default fallback cases.",
"tool_calls": [],
"usage": {
"input_tokens": 151,
"output_tokens": 44,
"total_tokens": 195,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "plain",
"turn": 4,
"sent": 1,
"stored": 7,
"estimated_tokens": 195,
"chars_only_estimate": 195,
"server_tokens": 151,
"reply": "Please update the client constructor to accept an optional `timeout_seconds` argument that falls back to the existing default when omitted to maintain backward compatibility, and add a regression test covering both the explicit value and default fallback cases."
},
{
"event": "turn_input",
"thread": "plain",
"turn": 5,
"question": "Who reviews #412, and when must it merge?",
"pin": false
},
{
"event": "model_request",
"ordinal": 5,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines."
},
{
"role": "ai",
"text": "Please update the client constructor to accept an optional `timeout_seconds` argument that falls back to the existing default when omitted to maintain backward compatibility, and add a regression test covering both the explicit value and default fallback cases."
},
{
"role": "human",
"text": "Who reviews #412, and when must it merge?"
}
]
},
{
"event": "model_response",
"ordinal": 5,
"message": {
"type": "ai",
"text": "I do not have the information regarding who reviews #412 or when it must merge.",
"tool_calls": [],
"usage": {
"input_tokens": 210,
"output_tokens": 19,
"total_tokens": 229,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "plain",
"turn": 5,
"sent": 3,
"stored": 9,
"estimated_tokens": 281,
"chars_only_estimate": 281,
"server_tokens": 210,
"reply": "I do not have the information regarding who reviews #412 or when it must merge."
},
{
"event": "turn_input",
"thread": "pinned",
"turn": 1,
"question": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly.",
"pin": false
},
{
"event": "model_request",
"ordinal": 6,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it."
},
{
"role": "human",
"text": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly."
}
]
},
{
"event": "model_response",
"ordinal": 6,
"message": {
"type": "ai",
"text": "Understood. I have noted to keep `DEFAULT_TIMEOUT` at 10s and add a per-client timeout override for #415.",
"tool_calls": [],
"usage": {
"input_tokens": 76,
"output_tokens": 191,
"total_tokens": 267,
"input_token_details": {
"cache_read": 0
},
"output_token_details": {
"reasoning": 159
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "pinned",
"turn": 1,
"sent": 1,
"stored": 1,
"estimated_tokens": 85,
"chars_only_estimate": 85,
"server_tokens": 76,
"reply": "Understood. I have noted to keep `DEFAULT_TIMEOUT` at 10s and add a per-client timeout override for #415."
},
{
"event": "turn_input",
"thread": "pinned",
"turn": 2,
"question": "mlee reviews #412, and it must merge before the 2.5 freeze on November 14. Acknowledge briefly.",
"pin": true
},
{
"event": "model_request",
"ordinal": 7,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it.\n\nPinned decisions:\n- mlee reviews #412, and it must merge before the 2.5 freeze on November 14."
},
{
"role": "human",
"text": "For #415 keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead. Acknowledge briefly."
},
{
"role": "ai",
"text": "Understood. I have noted to keep `DEFAULT_TIMEOUT` at 10s and add a per-client timeout override for #415."
},
{
"role": "human",
"text": "mlee reviews #412, and it must merge before the 2.5 freeze on November 14. Acknowledge briefly."
}
]
},
{
"event": "model_response",
"ordinal": 7,
"message": {
"type": "ai",
"text": "Understood. I have noted that mlee reviews #412 and that it must merge before the 2.5 freeze on November 14.",
"tool_calls": [],
"usage": {
"input_tokens": 330,
"output_tokens": 201,
"total_tokens": 531,
"input_token_details": {
"cache_read": 0
},
"output_token_details": {
"reasoning": 169
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "pinned",
"turn": 2,
"sent": 3,
"stored": 3,
"estimated_tokens": 328,
"chars_only_estimate": 169,
"server_tokens": 330,
"reply": "Understood. I have noted that mlee reviews #412 and that it must merge before the 2.5 freeze on November 14."
},
{
"event": "turn_input",
"thread": "pinned",
"turn": 3,
"question": "Here is the CI retry log for #415. Summarize what it shows in one sentence.\n09:14:00 tests/test_client.py::test_retry_0 PASSED\n09:14:01 tests/test_client.py::test_retry_1 PASSED\n09:14:02 tests/test_client.py::test_retry_2 PASSED\n09:14:03 tests/test_client.py::test_retry_3 PASSED\n09:14:04 tests/test_client.py::test_retry_4 PASSED\n09:14:05 tests/test_client.py::test_retry_5 PASSED\n09:14:06 tests/test_client.py::test_retry_6 PASSED\n09:14:07 tests/test_client.py::test_retry_7 PASSED\n09:14:08 tests/test_client.py::test_retry_8 PASSED\n09:14:09 tests/test_client.py::test_retry_9 PASSED\n09:14:10 tests/test_client.py::test_retry_10 PASSED\n09:14:11 tests/test_client.py::test_retry_11 PASSED\n09:14:12 tests/test_client.py::test_retry_12 PASSED\n09:14:13 tests/test_client.py::test_retry_13 PASSED\n09:14:14 tests/test_client.py::test_retry_14 PASSED",
"pin": false
},
{
"event": "model_request",
"ordinal": 8,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it.\n\nPinned decisions:\n- mlee reviews #412, and it must merge before the 2.5 freeze on November 14."
},
{
"role": "human",
"text": "Here is the CI retry log for #415. Summarize what it shows in one sentence.\n09:14:00 tests/test_client.py::test_retry_0 PASSED\n09:14:01 tests/test_client.py::test_retry_1 PASSED\n09:14:02 tests/test_client.py::test_retry_2 PASSED\n09:14:03 tests/test_client.py::test_retry_3 PASSED\n09:14:04 tests/test_client.py::test_retry_4 PASSED\n09:14:05 tests/test_client.py::test_retry_5 PASSED\n09:14:06 tests/test_client.py::test_retry_6 PASSED\n09:14:07 tests/test_client.py::test_retry_7 PASSED\n09:14:08 tests/test_client.py::test_retry_8 PASSED\n09:14:09 tests/test_client.py::test_retry_9 PASSED\n09:14:10 tests/test_client.py::test_retry_10 PASSED\n09:14:11 tests/test_client.py::test_retry_11 PASSED\n09:14:12 tests/test_client.py::test_retry_12 PASSED\n09:14:13 tests/test_client.py::test_retry_13 PASSED\n09:14:14 tests/test_client.py::test_retry_14 PASSED"
}
]
},
{
"event": "model_response",
"ordinal": 8,
"message": {
"type": "ai",
"text": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively within 14 seconds.",
"tool_calls": [],
"usage": {
"input_tokens": 464,
"output_tokens": 44,
"total_tokens": 508,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "pinned",
"turn": 3,
"sent": 1,
"stored": 5,
"estimated_tokens": 295,
"chars_only_estimate": 295,
"server_tokens": 464,
"reply": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively within 14 seconds."
},
{
"event": "turn_input",
"thread": "pinned",
"turn": 4,
"question": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines.",
"pin": false
},
{
"event": "model_request",
"ordinal": 9,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it.\n\nPinned decisions:\n- mlee reviews #412, and it must merge before the 2.5 freeze on November 14."
},
{
"role": "human",
"text": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines."
}
]
},
{
"event": "model_response",
"ordinal": 9,
"message": {
"type": "ai",
"text": "Please update the client constructor to accept an optional `timeout_seconds` argument that preserves the existing default when omitted to ensure backward compatibility, and include regression tests covering both the explicit and omitted timeout cases.",
"tool_calls": [],
"usage": {
"input_tokens": 182,
"output_tokens": 40,
"total_tokens": 222,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "pinned",
"turn": 4,
"sent": 1,
"stored": 7,
"estimated_tokens": 219,
"chars_only_estimate": 219,
"server_tokens": 182,
"reply": "Please update the client constructor to accept an optional `timeout_seconds` argument that preserves the existing default when omitted to ensure backward compatibility, and include regression tests covering both the explicit and omitted timeout cases."
},
{
"event": "turn_input",
"thread": "pinned",
"turn": 5,
"question": "Who reviews #412, and when must it merge?",
"pin": false
},
{
"event": "model_request",
"ordinal": 10,
"policy": {
"lesson_id": "langgraph-context-budget",
"model": "gemini-3.8-flash",
"max_model_calls": 10,
"max_output_tokens": 4096,
"thinking_level": "low",
"timeout_seconds": 45,
"max_retries": 0,
"vertexai": false
},
"sent_view": [
{
"role": "system",
"text": "You help a developer work through the team's code-review queue. Answer in at most three\nsentences, from the conversation and the pinned decisions only. If a fact is in neither, say you do not have it.\n\nPinned decisions:\n- mlee reviews #412, and it must merge before the 2.5 freeze on November 14."
},
{
"role": "human",
"text": "Draft a one-line review comment asking for a per-client timeout override. The proposed API accepts an optional timeout_seconds argument on the client constructor; when absent, it keeps the existing default. Ask for a regression test that covers both omitted and explicit values. Keep backward compatibility for existing callers. Do not claim the code was changed or tests were executed. The supplied test log is a fixture for discussion, not a fresh CI result. Keep the comment about the timeout API only; do not discuss reviewers or deadlines."
},
{
"role": "ai",
"text": "Please update the client constructor to accept an optional `timeout_seconds` argument that preserves the existing default when omitted to ensure backward compatibility, and include regression tests covering both the explicit and omitted timeout cases."
},
{
"role": "human",
"text": "Who reviews #412, and when must it merge?"
}
]
},
{
"event": "model_response",
"ordinal": 10,
"message": {
"type": "ai",
"text": "mlee reviews #412. It must merge before the 2.5 freeze on November 14.",
"tool_calls": [],
"usage": {
"input_tokens": 237,
"output_tokens": 24,
"total_tokens": 261,
"input_token_details": {
"cache_read": 0
}
},
"finish_reason": "STOP"
}
},
{
"event": "turn_result",
"thread": "pinned",
"turn": 5,
"sent": 3,
"stored": 9,
"estimated_tokens": 302,
"chars_only_estimate": 302,
"server_tokens": 237,
"reply": "mlee reviews #412. It must merge before the 2.5 freeze on November 14."
},
{
"event": "completion",
"status": "completed_requires_semantic_review",
"model_calls": 10,
"approximate_input_budget": 400,
"checks": {
"stored_both_threads": true,
"plain_decision_trimmed": true,
"pinned_system_retains_decision": true,
"plain_answer_does_not_name_mlee": true,
"pinned_answer_names_mlee": true,
"pinned_answer_names_date": true
},
"threads": [
{
"thread": "plain",
"stored_messages": 10,
"last_view_has_decision": false,
"system_has_pinned_decision": false,
"turns": [
{
"turn": 1,
"sent": 1,
"stored": 1,
"estimated_tokens": 85,
"chars_only_estimate": 85,
"server_tokens": 76,
"reply": "Understood. I have noted that for #415, `DEFAULT_TIMEOUT` should remain at 10s and a per-client timeout override should be added instead."
},
{
"turn": 2,
"sent": 3,
"stored": 3,
"estimated_tokens": 334,
"chars_only_estimate": 153,
"server_tokens": 326,
"reply": "Understood. I have recorded that mlee is reviewing #412 and that it must be merged before the 2.5 freeze on November 14."
},
{
"turn": 3,
"sent": 1,
"stored": 5,
"estimated_tokens": 271,
"chars_only_estimate": 271,
"server_tokens": 433,
"reply": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively."
},
{
"turn": 4,
"sent": 1,
"stored": 7,
"estimated_tokens": 195,
"chars_only_estimate": 195,
"server_tokens": 151,
"reply": "Please update the client constructor to accept an optional `timeout_seconds` argument that falls back to the existing default when omitted to maintain backward compatibility, and add a regression test covering both the explicit value and default fallback cases."
},
{
"turn": 5,
"sent": 3,
"stored": 9,
"estimated_tokens": 281,
"chars_only_estimate": 281,
"server_tokens": 210,
"reply": "I do not have the information regarding who reviews #412 or when it must merge."
}
]
},
{
"thread": "pinned",
"stored_messages": 10,
"last_view_has_decision": true,
"system_has_pinned_decision": true,
"turns": [
{
"turn": 1,
"sent": 1,
"stored": 1,
"estimated_tokens": 85,
"chars_only_estimate": 85,
"server_tokens": 76,
"reply": "Understood. I have noted to keep `DEFAULT_TIMEOUT` at 10s and add a per-client timeout override for #415."
},
{
"turn": 2,
"sent": 3,
"stored": 3,
"estimated_tokens": 328,
"chars_only_estimate": 169,
"server_tokens": 330,
"reply": "Understood. I have noted that mlee reviews #412 and that it must merge before the 2.5 freeze on November 14."
},
{
"turn": 3,
"sent": 1,
"stored": 5,
"estimated_tokens": 295,
"chars_only_estimate": 295,
"server_tokens": 464,
"reply": "The log shows that all 15 retry tests (`test_retry_0` through `test_retry_14`) in `tests/test_client.py` passed consecutively within 14 seconds."
},
{
"turn": 4,
"sent": 1,
"stored": 7,
"estimated_tokens": 219,
"chars_only_estimate": 219,
"server_tokens": 182,
"reply": "Please update the client constructor to accept an optional `timeout_seconds` argument that preserves the existing default when omitted to ensure backward compatibility, and include regression tests covering both the explicit and omitted timeout cases."
},
{
"turn": 5,
"sent": 3,
"stored": 9,
"estimated_tokens": 302,
"chars_only_estimate": 302,
"server_tokens": 237,
"reply": "mlee reviews #412. It must merge before the 2.5 freeze on November 14."
}
]
}
],
"boundary": "Approximate trimming is not a provider token cap. Generated replies require semantic review; reading is not mastery."
}
]How it works
Each invoke with the same thread_id loads the thread’s last checkpoint, appends the new message through the reducer, and runs respond. Here respond passes [system, *kept], rebuilt from the full list on every turn, so the view changes as the thread grows: on turn 2 of the first thread the long first reply already pushed turn 1 out.
Trimming does not require an extra model call. LangChain’s create_agent offers a summarizing alternative, SummarizationMiddleware, which replaces old messages with a model-written summary. A summary can retain information from removed turns, but it costs a model call and can lose or distort a decision. Pinning is the explicit version: the application, not a model, decides what must survive.
Test the trimming without a model
test_graph.py runs the real graph with a scripted chat model that records the view it receives, and checks the recorded Gemini run as a fixture.
python -m pytest -v test_graph.pytest_graph.py::test_the_thread_keeps_every_message_while_the_model_sees_a_trimmed_view PASSED
test_graph.py::test_the_view_starts_on_a_user_turn PASSED
test_graph.py::test_a_trimmed_decision_is_gone_from_the_view PASSED
test_graph.py::test_a_pinned_decision_survives_trimming PASSED
test_graph.py::test_the_counter_adds_the_reasoning_tokens_a_reply_carries_back PASSED
test_graph.py::test_a_question_larger_than_the_budget_is_refused_before_the_model_call PASSED
test_graph.py::test_recorded_run_pinning_kept_the_answer PASSED
test_graph.py::test_recorded_run_chars_only_estimate_missed_the_signatures PASSED
test_graph.py::test_recorded_log_turn_was_billed_above_the_budget PASSED
============================== 9 passed ===============================All 9 tests pass.
Read the test and recording scripts
"""Offline tests: a scripted chat model replaces Gemini; the recorded Gemini run is checked as a fixture."""
import itertools
import json
from pathlib import Path
import pytest
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.messages.utils import count_tokens_approximately
from graph import BUDGET, ContextTooLarge, build_graph, count_tokens
RUN = json.loads(Path(__file__).with_name("runs.json").read_text())
FILLER = "We went through the sync design again. " * 12
class Scripted(GenericFakeChatModel):
seen: list = []
def invoke(self, messages, *args, **kwargs):
self.seen.append(messages)
return super().invoke(messages, *args, **kwargs)
def chat(turns, pin=None):
model = Scripted(messages=(AIMessage(f"Noted {i}.") for i in itertools.count()), seen=[])
graph = build_graph(model)
config = {"configurable": {"thread_id": "t"}}
for i, text in enumerate(turns):
update = {"messages": [HumanMessage(text)]}
if pin and i == 0:
update["pinned"] = [pin]
state = graph.invoke(update, config)
return state, model.seen[-1]
def test_the_thread_keeps_every_message_while_the_model_sees_a_trimmed_view():
state, view = chat([f"Decision {i}. {FILLER}" for i in range(6)])
assert len(state["messages"]) == 12
assert state["view"]["sent"] < 12
assert count_tokens(view) <= BUDGET
def test_the_view_starts_on_a_user_turn():
_, view = chat([f"Decision {i}. {FILLER}" for i in range(6)])
assert isinstance(view[1], HumanMessage)
def test_a_trimmed_decision_is_gone_from_the_view():
_, view = chat(["mlee reviews #412.", *[f"Note {i}. {FILLER}" for i in range(4)], "Who reviews #412?"])
assert not any("mlee" in m.text for m in view)
def test_a_pinned_decision_survives_trimming():
_, view = chat(["mlee reviews #412.", *[f"Note {i}. {FILLER}" for i in range(4)], "Who reviews #412?"],
pin="mlee reviews #412.")
assert "mlee reviews #412." in view[0].text
def test_the_counter_adds_the_reasoning_tokens_a_reply_carries_back():
reply = AIMessage("Noted.", usage_metadata={"input_tokens": 9, "output_tokens": 230, "total_tokens": 239,
"output_token_details": {"reasoning": 220}})
assert count_tokens([reply]) == count_tokens_approximately([reply]) + 220
def test_a_question_larger_than_the_budget_is_refused_before_the_model_call():
model = Scripted(messages=iter([AIMessage("never")]), seen=[])
with pytest.raises(ContextTooLarge):
build_graph(model).invoke({"messages": [HumanMessage("log line 09:15:01 ok " * 200)]},
{"configurable": {"thread_id": "big"}})
assert model.seen == []
def test_recorded_run_pinning_kept_the_answer():
plain, pinned = RUN["runs"]
assert "do not have" in plain["turns"][-1]["reply"]
assert "mlee" in pinned["turns"][-1]["reply"]
def test_recorded_run_chars_only_estimate_missed_the_signatures():
last = RUN["runs"][0]["turns"][-1] # three messages sent, one of them a Gemini reply
assert last["chars_only_estimate"] < last["server_tokens"] / 2 < last["estimated_tokens"]
def test_recorded_log_turn_was_billed_above_the_budget():
log_turn = RUN["runs"][0]["turns"][2]
assert log_turn["estimated_tokens"] <= RUN["budget"] < log_turn["server_tokens"]"""Run one code-review thread on Gemini twice, without and with a pinned decision, and save runs.json."""
import json
from langchain_core.messages import HumanMessage
from graph import BUDGET, MODEL, ContextTooLarge, build_graph
LOG = "\n".join(f"09:14:{s:02d} tests/test_client.py::test_retry_{s} PASSED [{7 * s:3d}%]" for s in range(15))
DECISION = "mlee reviews #412, and it must merge before the 2.5 freeze on November 14."
TURNS = [
"For #415 we agreed to keep DEFAULT_TIMEOUT at 10s and add a per-client timeout override instead.",
DECISION,
f"Here is the CI log from #415. Does anything look wrong?\n{LOG}",
"Draft a one-line review comment for #415.",
"Who reviews #412, and when must it merge?",
]
def thread(graph, name: str, pin: bool) -> dict:
config = {"configurable": {"thread_id": name}}
turns = []
for text in TURNS:
update = {"messages": [HumanMessage(text)]}
if pin and text == DECISION:
update["pinned"] = [DECISION]
state = graph.invoke(update, config)
turns.append({"question": text.splitlines()[0], **state["view"], "reply": state["messages"][-1].text})
return {"thread": name, "pinned": state.get("pinned", []), "stored_messages": len(state["messages"]), "turns": turns}
def main() -> None:
graph = build_graph()
runs = [thread(graph, "review", pin=False), thread(graph, "review-pinned", pin=True)]
paste = "Check this log:\n" + "\n".join(f"09:15:{s:02d} tests/test_client.py::test_retry_{s} PASSED" for s in range(60))
try:
graph.invoke({"messages": [HumanMessage(paste)]}, {"configurable": {"thread_id": "paste"}})
overflow = "no error"
except ContextTooLarge as error:
overflow = f"{type(error).__name__}: {error}"
with open("runs.json", "w") as f:
json.dump({"model": MODEL, "budget": BUDGET, "runs": runs, "overflow": overflow}, f, indent=1)
for run in runs:
print(f"== thread {run['thread']} pinned={run['pinned']}")
for t in run["turns"]:
print(f" sent {t['sent']:>2}/{t['stored'] + 1:<2} messages estimate {t['estimated_tokens']:>3} "
f"(chars only {t['chars_only_estimate']:>3}) billed {t['server_tokens']:>3} | {t['question'][:44]}")
print(" last reply:", run["turns"][-1]["reply"])
print("== a 60-line paste:", overflow)
if __name__ == "__main__":
main(){
"question": "Who reviews #412, and when must it merge?",
"sent": 3,
"stored": 9,
"estimated_tokens": 271,
"chars_only_estimate": 109,
"server_tokens": 255,
"reply": "I do not have the information regarding who reviews #412 or when it must merge."
}Read the explanation
Trimming keeps recent messages, not important ones. Decide explicitly what must survive.
What’s next
- Add timeouts and token budgets to a LangGraph agent: limit how long each call may take and how many tokens a whole run may use.
References
Official sources checked October 22, 2026, against the installed LangGraph and langchain-core packages.
- LangGraph: Add memory: short-term memory with a checkpointer, and trimming with
trim_messagesandcount_tokens_approximately. - LangGraph: Persistence: threads,
thread_idand checkpoints. - LangChain: Short-term memory: trimming and summarizing messages, including
SummarizationMiddleware. - LangChain: ChatGoogleGenerativeAI: the Gemini chat model and its usage metadata.
- Gemini API: Thinking and signatures: reasoning metadata and the distinction between generateContent and Interactions.
- The
trim_messagesandcount_tokens_approximatelyparameters match the installed langchain-core interfaces; the lockfile pins the versions.