Prompting agents: What works and why

Nolan Sullivan

September 22, 2025 - 18 min read

As chatbots, large language models are surprisingly human-like and effective conversation partners. They mirror whoever is chatting with them and have a knack for small-talk like no other - sometimes even appearing too compassionate.

In contrast, working with an LLM agent (rather than a chatbot) often feels like you’re pushing a string, only to realize too late that the string has folded over itself in kinks and loops, and you need to start over again. Agents that lack clear success and failure criteria or explicit direction can be expensive, slow, and in the worst case, destructive.

In this guide, we unpack the different layers of prompting agents, and explore proven methods for improving how we prompt agents.

But first, why focus on agents if there is already so much written about prompts and context engineering in general?

Chatbots are for loops; agents are while loops

Chatbots take discrete turns, with a human available to steer the bot at each turn. They work on a single task per turn. Claude, ChatGPT, and Grok are examples of chatbots. There are many prompting guides available online for these popular chatbots.

Agents, on the other hand, work continuously in a loop to achieve complex goals. Agents are usually employed in situations where they have access to tools that influence the real world. Examples include Claude Code, ChatGPT Operator (soon to be replaced by ChatGPT agent mode), and Gemini CLI. Agents are so complex, layered, and multifaceted that most agent prompting guides only scratch the very surface - how an end user should ask an agent to do things.

A note on terminology

The AI world hasn’t settled on standard terms yet (even calling LLMs “AI” is sometimes frowned upon), so let’s be clear about what we mean:

Agents are AI systems that can take actions through tools - they can run commands, manipulate files, call APIs, and change things in the real world. Claude Code executing terminal commands is an agent. When Cursor edits your files or Gemini CLI runs Python scripts, they are also acting as agents.

Agent interfaces like Claude Code, ChatGPT Operator, and Gemini CLI are the products you interact with. They combine an underlying model (Claude 3.5 Sonnet, GPT-4, Gemini) with tools and a user interface.

Chatbots just generate text responses. They can’t execute code, access your filesystem, or take actions beyond returning text. Regular Claude.ai and ChatGPT (without plugins) are chatbots.

When we talk about “prompting agents” we mean getting AI to actually do things, not just talk about doing them.

When prompting an agent, start at the right layer

An agent’s prompt doesn’t start at the point when a user asks a question. An agent prompt is a larger entity, which we can break up into the following distinct layers, all of which influence how well an agent performs, and each of which is as important as the others to get right:

1. The model’s platform-level instructions: At the highest layer are the platform-level instructions. These are set by the platform, like OpenAI or Anthropic. For example, even if we use the OpenAI API, the model’s API responses won’t include copyrighted media, illegal material, or information that could cause harm.

2. Developer instructions: These are often called the system prompt, and for most developers, this is the highest level of authority their prompts could have. Examples include proprietary system prompts, like Claude Code or Cursor’s system prompt, or prompts for open-source agents.

3. User rules: Some agents support rules files that the user can set for all instances of their agent. For example, Claude Code reads the file at ~/.claude/CLAUDE.md, or any parent directory of the project you’re working on, and consistently applies your rules to its actions.

4. Project rules: These are instructions an agent applies within a specific directory. For example, a CLAUDE.md file in the project root directory, or a child directory.

5. User request: This is the actual prompt entered by the user, for example, “Fix the race condition in agents/tasks.py”.

6. Tool specifications: At the lowest level are the descriptions and guidelines from tool developers, which include input/output formats, constraints, and best practices.

These different prompt levels are strung together in the agent’s context, and changing any one may have an effect on the agent’s performance.

Understanding how system prompts shape agent behavior

You can’t change the system prompt of Claude Code or ChatGPT Operator, but understanding what’s happening behind the scenes helps explain why agents sometimes behave in unexpected ways.

System prompts are the hidden instructions that make agents work. They’re written by the companies building these tools and run thousands of words long. When your agent refuses to do something reasonable or insists on doing something you didn’t ask for, the system prompt is often at work.

Here’s what these gigantic prompts typically contain:

1. Identity and role boundaries

Most agents start with a defined identity that constrains what they will and won’t do. This is why Claude Code won’t help you write malware, even if you have a legitimate security testing reason.

2. Tool usage patterns and guardrails

Agents have extensive instructions about how to use their tools correctly. This is why Claude Code often checks file contents before editing, or why it might refuse certain filesystem operations.

3. Domain-specific behaviors

Agents come preloaded with opinions about best practices. When v0 generates a React component, it follows specific instructions about which libraries to use and how to structure code.

The massive size and impact of system prompts

Modern AI coding agents use system prompts of around tens of thousands of characters. These prompts encode years of engineering wisdom, safety rules, and behavioral patterns.

• Your instructions compete with these prompts: If you ask for something that conflicts with the system prompt, the system prompt usually wins.
• Weird behaviors often trace back here:
• You can work around them once you know they exist:

Learning from open-source agents

If you’re building your own agent or want to understand how they think, these open-source system prompts are goldmines:

• The Goose system prompt
• The Cline system prompt

Now that you understand the hidden forces shaping agent behavior, let’s look at what you can actually control: your own prompts.

Prompting as a user: Techniques that improve agent performance

Let’s look at some examples of prompting techniques.

1. Give agents clear success criteria

2. Use the “think” tool pattern

3. Be specific

4. Create custom user rules in a convention file

5. Learn real-world safety from production disasters

6. Practice constraint-based prompting

7. Use AI to improve your prompts

8. Use XML tags in prompts

As agents get smarter, prompts become more important, not less

1. Create a CLAUDE.md (or equivalent file) for your projects with explicit rules.
2. Include units, formats, and examples in your prompts.
3. Test prompts with actual outputs before trusting them in production.
4. When something goes wrong, make your prompt more specific, not longer.

Last updated on April 24, 2026.