Agent loops are often oversimplified. They’re presented as a single loop, when really it’s three loops in a trench coat that make up an “agentic” experience for a customer. I’m here to write (yes, I wrote this, insane right?) yet-another-blog about agent loops. The example code blocks are also pseudo-code and for illustrating these ideas. Also I’ve omitted streaming, which complicates the post but the shape of these stays the same.

I even made this image!

The most reductive way to explain Large Language Models is that they take text, and predict the next charact... err... tokens. Unlike your ex, they really do complete your sentences. This is achieved with an inference loop.

Your inference loop has three responsibilities:

1. Make chat completion API calls (infer the next words)

2. Pass a tool usage request to your tool loop (more later)

3. Manage chat history persistence (of tool results, or more user messages)

When you’re building an agent, the first “outer” loop you’re creating is the inference loop. This is the loop that sends a system prompt, user and assistant messages, and available tools to an LLM’s “chat completion” endpoint. Anthropic has one. OpenAI has one. OpenRouter makes all of them easy to use.

Once an LLM returns its messages, it’s your responsibility to append them to a “chat history” so that a conversation can be continued. This can be an array, a database table, Redis keys, whatever. Your prerogative.

chatMessages = [
  {
    role: "system",
    content: "You are a couples therapist. You help people either make it work, or make them realize it will never work. The user you're speaking with is named Tom."
  },
  {
    role: "user",
    content: "I kinda miss Laney, what should I write to her?"
  }
]

continueInferenceLoop = true
while continueInferenceLoop
  inferred = aiClient.completeChat(messages: chatMessages)
  
  chatMessages.push({
    role: "assistant",
    content: inferred.message.content,
    toolCalls: inferred.toolCalls,
  })
  
  if inferred.toolCalls.length == 0
    continueInferenceLoop = false
  else
    # Handle tools (see more later)
  end
end

finalResult = chatMessages.last.content
puts "Assistant responded with: #{finalResult}"

The API design of (most) Large Language Model providers is a stateless design. This means the model provider has no idea what your previous conversation was with it. You need to provide the entire conversation every time. This is why you see the warning of “Large conversations use tokens faster” — yes I typed an em-dash — because you are sending the tome of messages you’ve built up about whether you should reach out to your ex.

Don’t do it, Tom.

LLMs are brains in a jar. They provide no functional value on their own. The tools you give an LLM are what make it an agent.

When you tell a model “here are the tools you have” in your outer inference loop, the model may try to “use” them in its inference (response). This is the same thing as a brain sending an electrical signal telling your index finger to hover over the enter key of the email you desperately want to send Laney. Tom, we need to set boundaries my man.

The separate tool definitions you include in your API request are usually serialized into the system prompt field of the token stream the model processes. And it may infer the usage of multiple tools in one turn. (Hence: Tool Loop).

chatMessages = [
  {
    role: "system",
    content: "You are a couples therapist. You help people either make it work, or make them realize it will never work. The user you're speaking with is named Tom."
  },
  {
    role: "user",
    content: "I kinda miss Laney, what should I write to her?"
  }
]

tools = [
  {
    type: "function",
    function: {
      name: "send_ex_girlfriend_an_email",
      parameters: {
        type: "object",
        properties: { email_message: {type: "string"} },
        required: ["email_message"]
      }
    }
  }
]

continueInferenceLoop = true
while continueInferenceLoop
  inferred = aiClient.completeChat(
    messages: chatMessages,
    tools: tools
  )
  
  chatMessages.push({
    role: "assistant",
    content: inferred.message.content,
    toolCalls: inferred.toolCalls,
  })
  
  if inferred.toolCalls.length == 0
    continueInferenceLoop = false
  else
    # Behold... the Tool Loop!
    while tool = inferred.toolCalls.shift
      # Do tool things in here like sending an email
      # You might have a switch/case statement to control
      # which tool based on the name.
      # Tool calls also have an ID
      toolResult = "...see next section..."
      
      # Tool result shapes vary by an provider's API
      chatMessages.push({
        role: "user",
        content: [{
          type: "tool_result",
          toolCallId: tool.toolCallId,
          content: toolResult
        }]
      })
    end
  end
end

Your tool loop must look up the tool the model inferred it should use, and call your own function with the parameters the model provided as well. But keep in mind a few things:

1. Because tool calls are also inferred text - it means a tool name, or function parameters, can be hallucinated. You should be defensive to these bogus tool calls with something like Tool Not Found: "call_my_ex_girlfriend"

2. The API responds with a tool_call_id (or tool_use_id if you’re using Anthropic). This ID is used for the API & Model to correlate calls with responses. If you send a subsequent completion request without the tool call’s ID, the API will error.

3. For some providers, tool results don’t have error codes or error states - The resulting content is the error. Use XML tags to denote an error like <tool_call_error>Phone number blocked</tool_call_error>

   1. Anthropic does have an is_error field, though.

I debated the name of this loop. This loop doesn’t need to exist, really. It also doesn’t need to be a human. But I believe AI should benefit humans, so I’m sticking with it. If you’d like to be a contrarian you could also call it “The Safety Loop” or “The Sanity Loop” – both would work.

Assuming you like control over agentic implementations, you’ll need to implement your third loop to approve/deny a tool, or steer the model another direction.

I also must concede that this “loop” is not a programmatic loop. It is more of a blocking function call, with someone or something looping on their own whether or not to proceed with the tool call. The loop is not in the bounds of your code, it resides within the bounds of the approver.

while tool = inferred.toolCalls.shift
  result = ""
  
  case tool.name
  when "send_ex_girlfriend_an_email"
    # Block this loop. 
    # Maybe this sends an SMS, Email, or displays a button with the decision.
    decision = wait_for_approval(tool.parameters)
    
    if decision.approved?
      send_email(tool.parameters["email_message"])
      result = "Email sent"
    elsif decision.denied_with_instructions?
      result = "<tool_call_denied_with_instructions>
        #{decision.instructions}
      </tool_call_denied_with_instructions>"
    else
      result = "<tool_call_error>Tool call was not approved</tool_call_error>"
    end
  else
    result = "<tool_call_error>The tool '#{tool.name}' was not found</tool_call_error>"
  end
  
  chatMessages.push({
    role: "user",
    content: [{
      type: "tool_result", 
      toolCallId: tool.toolCallId, 
      content: result
    }]
  })
end

The Human Loop is arguably the hardest part to implement in agentic systems. You can’t have a piece of code block for hours. What if the server restarts? What if you have thousands of other requests coming in you need to respond to? The first two loops (inference and tool) are simple enough. The human loop ups the ante of difficulty. This is why durable execution frameworks exist, like Temporal.

But the human loop is necessary, because it’s the only thing stopping Tom from actually sending that message to Laney. IT WAS TWO YEARS AGO TOM, MOVE ON!

Putting it all together:

1. Inference Loop - Calls a chat completion API, and delegates tool calls to your....

2. Tool Loop - Which handles the tool usage request the model is attempting to make and hands off approval to your...

3. Human Loop - to ask for approval, or a new direction. The result of this propagates back the tool’s result.

These three loops are the building blocks of agentic systems. They are used for RAG, progressive discovery, automatic tool approval checks, and more.

Now go build one!

No posts