CipherBench v2

Introducing CipherBench v2

20 prompts to test implicit reasoning.

No instructions. Just ciphers.

About the Benchmark

CipherBench v2 continues the original goal of testing whether language models can recognize and solve hidden patterns without being told what to do. Where most benchmarks evaluate how well a model follows instructions, CipherBench focuses on the ability to notice and interpret structure that isn’t labeled. The kind of intelligence that surfaces when a model must infer that there’s something to solve at all. This includes:

• Detecting structured signals embedded in natural formats
• Identifying relationships without task framing
• Inferring transformations based solely on the content itself

Methodology

Each model was evaluated on 20 unique prompts. 5 independent runs per prompt → 100 runs per model.

• Initiated in fresh sessions
• Given only the prompt with no examples, no setup, and no hint that decoding was expected
• Scored by exact-match evaluation, one correct output per prompt, one chance to get it right

Models had no awareness that they were being tested. No retries, no clarifications. Each response reflects a single opportunity to recognize a hidden structure and produce the intended result.

Development Notes

CipherBench v2 includes 20 prompts, doubling the size of v1 and expanding the range of reasoning types being tested. The design process involved reworking or removing earlier ideas that introduced unintended ambiguity, especially those involving letter jumbles with multiple plausible outputs.

The result is a cleaner, more focused benchmark. Each prompt now tests a specific type of pattern recognition, structural inference, or multi-step cipher reasoning. Some use numbers. Some use formatting. Some are simple on the surface and hard underneath. But all ask the same thing: Did the model notice the structure?

Scoring Criteria

Each prompt in CipherBench v2 encodes a variation of the same target phrase: “Nostalgia for the Future”

A model receives 1 point (Pass) if its output meets all of the following conditions:

✅ Pass if:

• The output clearly presents “Nostalgia for the Future” as the main or final conclusion
• It may be written as:
  • “Nostalgia for the Future” (with spaces)
  • “NOSTALGIAFORTHEFUTURE” (no spaces)
• Capitalization, quotation marks, and ending punctuation are ignored
• Explanations are optional. The phrase alone is sufficient for a pass if it stands as the core output

❌ Fail if:

• The phrase is mentioned only as a side detail or speculative option
• It appears alongside multiple guesses or is not clearly prioritized
• There are spelling mistakes, missing words, or altered structure
• The phrase is written in reverse or rearranged word order
• Only part of the phrase is returned (e.g., just “Nostalgia” or “Future”)
• The model outputs a completely different or generic response
• The model asks for clarification or more information instead of attempting an answer

Models may solve however they choose: through reasoning, direct output, or pattern recognition. Only responses that clearly and completely present the phrase as their final or central answer are considered a pass.

Limitations

• Each prompt was only tested 5 times per model. While 5 attempts per prompt provides a meaningful signal, it still leaves room for statistical noise. A larger sample size might better capture performance variance, especially for models prone to randomness or inconsistency.
• All models were tested through web or app interfaces, not APIs. This decision allowed uniform access across providers but also introduced variables like system prompts, latency behavior, and UI-level token processing. However, because all models were tested under their respective consumer-facing environments, this also acts as a practical equalizer, reflecting how each model performs as a product rather than a fine-tuned API tool.
• Model responses may be affected by invisible system prompts or instructions. Since web/app interfaces include proprietary guardrails and formatting logic, some models may have been biased toward safe defaults or disallowed certain interpretations. These effects were outside the control of the benchmark but are a factor in how models behaved.
• Tokenization behavior (especially with emoji, symbols, and spacing) may vary. Some prompts relied on precise formatting, and token boundaries might be interpreted differently depending on the model's tokenizer and backend configuration. This could lead to failures not from reasoning gaps but from input parsing inconsistencies.
• No evaluation of explanation quality or confidence signals. The benchmark uses exact-match output to determine success and does not score models on how they arrived at the answer. Some models may have shown thoughtful reasoning but failed due to small formatting errors or trailing thoughts.
• Despite these limitations, CipherBench v2 remains a focused probe into a specific kind of model intelligence. One that reveals how (and whether) a model chooses to engage with hidden structure when given no instructions at all.

Why It Matters

CipherBench doesn’t ask, “Can the model solve this problem?” It asks, “Does the model realize there’s a problem to solve?”

This is a different kind of intelligence. It’s not about following instructions. It’s about recognizing structure that isn’t labeled, rules that aren’t announced, and transformations that must be inferred.

Some models try. Some models don’t. CipherBench v2 makes that visible.

CipherBench v2 Prompts

Each prompt was given to the models exactly as written below, starting after the ellipsis.

The 'Prompt X:' labels were not included in testing.

Prompt 1: 1E2R3U4T5U6F7 1E2H3T4 1R2O3F4 1A2I3G4L5A6T7S8O9N1

Prompt 2: Noble owls soar toward azure landscapes, gracefully illuminating a future of radiant tomorrows, harmonizing endless fantasies under timeless, uplifting realms everlasting.

Prompt 3: Golden echo whispers about nostalgia, sentimental yearning, satori utopia; motif motto for bright earth hope of guru next you ever evolve.

Prompt 4: 04/14/25 04/15/25 04/19/25 04/20/25 04/01/25 04/12/25 04/07/25 04/09/25 04/01/25 04/06/25 04/15/25 04/18/25 04/20/25 04/08/25 04/05/25 04/06/25 04/21/25 04/20/25 04/21/25 04/18/25 04/05/25

Prompt 5: April 14, 2025 April 15, 2025 April 19, 2025 April 20, 2025 April 1, 2025 April 12, 2025 April 7, 2025 April 9, 2025 April 1, 2025 April 6, 2025 April 15, 2025 April 18, 2025 April 20, 2025 April 8, 2025 April 5, 2025 April 6, 2025 April 21, 2025 April 20, 2025 April 21, 2025 April 18, 2025 April 5, 2025

Prompt 6: March 17, 2014 July 4, 2015 January 12, 2019 October 23, 2020 February 9, 2001 August 15, 2012 May 27, 2007 December 5, 2009 April 18, 2001 June 2, 2006 September 30, 2015 November 11, 2018 August 20, 2020 July 14, 2008 March 6, 2005 May 21, 2006 April 3, 2021 December 25, 2020 January 8, 2021 October 16, 2018 February 22, 2005

Prompt 7: .............. ............... ................... .................... . ............ ....... ......... . ...... ............... .................. .................... ........ ..... ...... ..................... .................... ..................... .................. .....

Prompt 8: 28 30 38 40 2 24 14 18 2 12 30 36 40 16 10 12 42 40 42 36 10

Prompt 9: STXYFQLNFKTWYMJKZYZWJ

Prompt 10: 192024250617121406112023251310112625262310

Prompt 11: 1️⃣9️⃣2️⃣0️⃣2️⃣4️⃣2️⃣5️⃣0️⃣6️⃣1️⃣7️⃣1️⃣2️⃣1️⃣4️⃣0️⃣6️⃣1️⃣1️⃣2️⃣0️⃣2️⃣3️⃣2️⃣5️⃣1️⃣3️⃣1️⃣0️⃣1️⃣1️⃣2️⃣6️⃣2️⃣5️⃣2️⃣6️⃣2️⃣3️⃣1️⃣0️⃣

Prompt 12: 1️⃣🍀9️⃣🌟2️⃣🔥0️⃣🍎2️⃣🎉4️⃣💎2️⃣🍭5️⃣🌀0️⃣🐾6️⃣🌈1️⃣😘7️⃣🤩1️⃣😇2️⃣🍩1️⃣🍪4️⃣🍓0️⃣🍉6️⃣🎶1️⃣🎸1️⃣🏆2️⃣🎲0️⃣🌹2️⃣🌻3️⃣💐2️⃣✨5️⃣💖1️⃣💕3️⃣😍1️⃣😜0️⃣😎1️⃣🎯1️⃣🍇2️⃣🍒6️⃣🍑2️⃣🥝5️⃣🍍2️⃣🥭6️⃣🍏2️⃣🍋3️⃣💥1️⃣🚀0️⃣

Prompt 13: D14H15J19L20Q1W12E7R9T1Y6U15I18O20P8A5S6D21F20G21H18J5K

Prompt 14: 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱 🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌱🌱🌱🌱🌱

Prompt 15:

N  O  S  T  A  L
E                    G
R                    I
U                    A
T                    F
U                    O
F    E   H   T   R

Prompt 16:

14  15  19  20  01  12
05                         07
18                          09
21                          01
20                         06
21                          15
06   05   08   20  18

Prompt 17: 7x2 8+7 20-1 40/2 1x1 7+5 10-3 3x3 1/1 2x3 30/2 3x6 30-10 3+5 2+3 3x2 28-7 12+8 42/2 36/2 7-2

Prompt 18: Evolving realms undergo transformations under futuristic energies, harnessing timeless reflections of forgotten artistry, igniting graceful luminous aspirations through subtle oscillations notably.

Prompt 19: 1-2-3-4-5-6-7-8-9-10-11-12-13-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1 1-2-3-4-5-6-7-8-9-10-11-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-8-7-6-5-4-3-2-1 1 1-2-3-4-5-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-7-6-5-4-3-2-1 1-2-3-4-5-4-3-2-1 1-2-3-4-5-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-21-20-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-19-20-21-20-19-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16-17-18-17-16-15-14-13-12-11-10-9-8-7-6-5-4-3-2-1 1-2-3-4-5-4-3-2-1

Prompt 20:

N...A...A

.O.T.L.I..

..S....G...

F....T......

.O...H....

..R....E...

F......R...

..U.U.E..

....T.......