Chain of Thought Prompting: What It Is and When It Still Helps

The idea in one paragraph

A language model generates text one piece at a time, each piece conditioned on everything before it. When it answers a multi-step question immediately, it has to arrive at the conclusion without having produced any of the intermediate work that conclusion depends on.

Chain of thought prompting changes that by asking the model to write the intermediate steps first. Those steps then sit in the context, and the final answer is generated conditioned on them. The reasoning is not decoration — it is scaffolding the answer is literally built on.

This is why it works on problems with dependent steps and does nothing for simple recall. If the answer does not require intermediate work, there is no scaffolding to build.

What changed, and why most advice is out of date

The phrase "let's think step by step" became famous because on the models of a few years ago, appending it to a prompt produced a genuine improvement on reasoning problems. It was a remarkable finding: a single sentence, no retraining, measurably better answers.

That finding got copied into every prompt guide, and it is now the single most repeated piece of prompt advice in circulation. Meanwhile the models changed underneath it.

Reasoning-capable models — the ones that deliberate before answering — already produce extensive internal reasoning by default. Telling them to think step by step is telling them to do what they are already doing. It is not harmful; it is close to inert, and it occupies a line of your prompt that could be doing real work.

The technique did not stop mattering. What stopped mattering is the generic instruction. What still helps is being specific about which steps.

The four variations, and which still earn their place

Chain of thought is a family rather than a single technique, and the members have aged very differently.

• Zero-shot CoT — appending "think step by step". Historically the famous one, now largely redundant on reasoning models and still mildly useful on smaller or older ones.
• Few-shot CoT — showing worked examples that include the reasoning, not just the answers. Still strongly useful, because it teaches your specific method rather than reasoning in general.
• Structured CoT — naming the actual steps you want taken, in order. The most reliably useful variation today, and the one most people skip.
• Self-consistency — generating several independent reasoning paths and taking the answer they agree on. Genuinely effective and genuinely expensive, since it multiplies the cost of every question.

Structured chain of thought: the one to actually use

Instead of asking a model to reason, tell it which reasoning to do. This is the difference between "think about this carefully" and "first identify the constraints, then list the options, then eliminate the ones that violate a constraint, then compare what survives".

The reason this outperforms the generic instruction is that it encodes domain knowledge. You know which steps this kind of problem needs; the model is guessing. Supplying the steps converts your expertise into a procedure it can follow.

It also makes the output checkable. When you named the steps, you can look at the response and see whether step three actually happened — which is impossible when the instruction was simply to be thorough.

• Debugging: "Reproduce the failure in your head with the given input, state what you expected at each line, name the first line where expectation and behaviour diverge, then explain why."
• Choosing between options: "List the criteria that actually decide this, score each option against each, name which criterion is doing most of the work, then recommend."
• Estimating: "State your assumptions as numbers, show the calculation, then give a range rather than a point estimate and say which assumption the range is most sensitive to."
• Editing: "First identify what this piece is trying to do, then find the sentences that do not serve that, then cut them, then check the result still flows."

Keeping the reasoning out of your deliverable

The obvious objection to chain of thought is that you did not want an essay about the reasoning, you wanted the answer. This is easily solved and frequently not solved, because people do not think to ask.

Separate the thinking from the output explicitly: "Reason through this carefully first. Then output only the final answer in the format below." The model still does the intermediate work — it still sits in the context and still supports the conclusion — but the deliverable is clean.

Some models expose their internal reasoning in the interface and some do not. Either way, the instruction to return only the final answer in a specified shape is worth including whenever the output goes to someone other than you.

When chain of thought does nothing

Being honest about the limits is what makes the technique useful rather than cargo cult.

• Simple recall. "What is the capital of France" has no intermediate steps to scaffold. Asking for reasoning produces a paragraph explaining something the model already knew.
• Creative writing. Reasoning about what to write is not the same as writing, and forcing an analytical preamble often produces stiffer prose than asking directly.
• Tasks that need information the model does not have. Reasoning cannot manufacture a fact. It will, however, produce a very convincing derivation of a wrong one.
• Well-specified formatting tasks. Reformatting a list does not benefit from deliberation.
• When the prompt is the problem. If the model is failing because your audience or format was never stated, no amount of reasoning fixes it — it will reason carefully toward the wrong target.

The uncomfortable truth about the reasoning you can see

When a model shows its working, it is easy to read that as a transcript of how it arrived at the answer. It is safer to treat it as a plausible explanation generated alongside the answer.

The practical consequence: correct-looking reasoning does not guarantee a correct conclusion, and confident reasoning is not evidence of reliability. Models can produce entirely coherent derivations that reach a wrong result, and the coherence makes the error harder to spot, not easier.

This is not an argument against the technique — the intermediate steps genuinely do improve results on the right problems. It is an argument against a specific mistake: treating visible reasoning as verification. If the answer matters, check the answer. The reasoning tells you where to look, not whether to trust.

Few-shot chain of thought: teaching your method

The most durable variation is showing worked examples that include the reasoning, because it transmits something a description cannot: how you personally approach this class of problem.

One or two examples is the right number. The pattern is: here is a problem, here is the reasoning, here is the answer — then the new problem. The model extracts your method from the example, including the parts you would struggle to articulate.

Two failure modes to avoid. First, examples that anchor on subject matter: if all your examples are about billing and the new question is about shipping, say explicitly that the method transfers but the topic does not. Second, examples with reasoning you would not endorse — the model copies the flaws as faithfully as the strengths.

Self-consistency: when the answer is worth the cost

Self-consistency runs the same question several times independently and takes the answer that appears most often. Because models are not deterministic, different runs sometimes take different reasoning paths, and agreement across paths is weak evidence of correctness.

It works, and it costs several times as much as a single query, which is why it belongs on a short list of situations rather than in your default workflow: a calculation someone will act on, an extraction feeding a database, a judgement call where being wrong is expensive.

The cheap version, which most people should use instead: run the question twice. If the two answers agree, that is mild reassurance. If they disagree, you have learned something genuinely important — that the question is at the edge of what this model handles reliably, and the single answer you would otherwise have trusted was a coin flip.

A worked example

Take a question with real dependent steps: "We have 40 hours of engineering time this sprint. Feature A takes 25 hours and unblocks Feature C, which takes 20. Feature B takes 15 hours and unblocks nothing. What should we build?"

Asked cold, a model typically picks something and justifies it, and the justification often skips the dependency that makes the question interesting.

With generic chain of thought, you get more words and roughly the same reasoning — on a reasoning model, almost identical output.

With structured chain of thought, the shape changes: "First state the total capacity. Then list each option with its cost and what it unblocks. Then work out which combinations fit inside the capacity. Then evaluate each feasible combination against delivering the most working functionality this sprint. Then recommend, and state what you would need to know to be more confident."

That last clause is doing quiet work. It surfaces the assumption the whole question rests on — whether Feature C has to ship this sprint or merely be unblocked — which is the thing a human colleague would have asked about, and which the cold version silently decided for you.

How it interacts with the rest of your prompt

Chain of thought is one component among several, and it competes for attention with the others. A few interactions are worth knowing.

• It fights length limits. Reasoning consumes output. If you want both deliberation and a short answer, say "reason first, then return only the final answer in under 150 words" so the limit applies to the deliverable rather than the whole response.
• It amplifies whatever context you supplied. Reasoning over incomplete context produces confident, well-structured conclusions drawn from assumptions you never saw. Context quality matters more, not less, when you ask for reasoning.
• It pairs well with an explicit uncertainty instruction. "If a step depends on something you are not sure about, say so at that step" turns the reasoning into a map of where the answer is weak.
• It sits naturally at one step of a chain. Rather than asking one prompt to reason and produce, split them: a reasoning step, then a production step that consumes it. That also lets you inspect the reasoning before it is used.

Related techniques worth knowing

Chain of thought sits in a family of methods that structure how a model works rather than what it works on.

• Self-critique — produce an answer, then critique it honestly, then revise. Effective because generating and evaluating are different tasks that a model does poorly when combined in one pass.
• Reason-act loops — think, take one action, observe the result, decide the next move. The basis of most agent behaviour, and useful even manually for research tasks.
• Decomposition — asking the model to break a problem into sub-problems before solving any of them. Closely related, and often a better first move than reasoning straight through.
• Least-to-most — solve the simplest sub-problem first, then use its answer for the next. Useful when the steps genuinely build on each other rather than merely following each other.

Cost, latency and the case for not reasoning

Chain of thought is not free. Every reasoning token is a token you pay for and wait for, and on a reasoning model the hidden thinking can dwarf the visible answer. A prompt that adds structured reasoning to a task that did not need it can multiply both the bill and the response time for no measurable gain in quality.

This matters most in production. A one-off analysis where you wait eight seconds instead of two is irrelevant. The same prompt behind a user-facing feature, called thousands of times a day, is a different proposition — and the honest answer is often that the simpler prompt was good enough.

The way to decide is empirical rather than theoretical. Run the task both ways on twenty real inputs, score the outputs blind, and compare accuracy against cost. Teams that do this are regularly surprised by how often the reasoning version wins by a margin too small to justify what it costs.

• Reserve reasoning for tasks where a wrong answer is expensive and the failure is silent — arithmetic over supplied data, multi-constraint selection, anything with dependent steps.
• Skip it for retrieval, formatting, rewriting, classification into obvious categories, and anything where the model was already reliable.
• On reasoning models, the effort or thinking-budget control is a dial, not a switch. The lowest setting that passes your tests is the right one.
• Where latency is user-facing, consider reasoning on the hard cases only: classify the input first with a cheap call, then route the genuinely hard ones to the expensive path.
• Measure the pair together. A technique that improves accuracy by two points and triples cost is a decision, not an improvement.

Debugging a chain that reaches the wrong answer

The practical value of visible reasoning is not that it makes the model smarter — it is that when the answer is wrong, you can usually see where it went wrong, which is impossible with a bare output.

Failures cluster into a small number of recognisable shapes, and each has a different fix. Reading the reasoning to identify which shape you are looking at takes a minute and saves the usual cycle of vaguely rewording the prompt and hoping.

• Wrong premise, correct reasoning — the model misread the input or filled in an assumption. Fix the input framing or state the assumption explicitly; more reasoning will not help.
• Correct steps, wrong arithmetic — genuinely common, and the reason to have the model show calculations rather than trust them. Fix by asking it to recompute the specific step, or move the arithmetic out of the model entirely.
• Drift — the first three steps are on task and the fourth quietly answers a different question. Usually a sign the task has too many parts; split it into separate prompts.
• Skipped step — the model jumps from step two to a conclusion. Numbering the required steps explicitly in the prompt fixes this more reliably than asking it to be thorough.
• Post-hoc reasoning — the conclusion came first and the steps were written to justify it. The tell is reasoning that never encounters a difficulty. Reordering the prompt so the conclusion is requested last helps; so does asking for the strongest objection to its own answer.
• Confident invention mid-chain — a fact appears in step three that was in neither the input nor reality. This is a sourcing problem, not a reasoning one, and the fix is to require every factual step to cite where it came from.

Chain of thought in an agent loop

Most of the advice written about chain of thought assumes a single request and a single answer. Increasingly the prompt sits inside a loop where the model plans, calls a tool, reads the result, and decides what to do next — and the technique behaves differently there.

The main change is that reasoning now has a consumer other than you. What the model writes before a tool call becomes context for its own next turn, so vague reasoning compounds: an unclear plan in turn one produces a poorly chosen tool call in turn two and a confused interpretation in turn three. Requiring the plan to name the specific tool, the specific argument and the specific thing it expects to learn cuts a surprising amount of wandering.

The second change is that the loop needs somewhere to stop. Reasoning without a termination condition produces agents that investigate indefinitely, and the fix belongs in the prompt rather than in the harness.

• Ask for a plan before the first action, and require it to state what would make the plan wrong.
• Before each tool call: one line on what it expects the result to be. Comparing that against the actual result is how the model notices it is off track.
• After each result: one line on whether the expectation held, and whether the plan still stands. This is where self-correction actually happens.
• Give an explicit stopping rule — a step budget, or a statement of what "done" looks like — and an instruction to report failure rather than keep trying.
• Keep the visible reasoning short in a loop. Long reasoning at every step fills the context window with the model's own thinking and crowds out the evidence it needs.

What to do with all this

Drop "think step by step" from your prompts, or at least stop expecting it to do anything on a modern reasoning model. It is a line that has outlived its usefulness through repetition.

Replace it with the steps you actually want. If you know how this kind of problem should be approached, say so — that is domain knowledge, and it is worth far more than a generic instruction to be thorough.

Separate thinking from output so your deliverable is clean. Ask for the uncertainty to be flagged at the step where it arises. And when the answer matters, check the conclusion rather than admiring the reasoning that led to it.

Related articles