Can a Better Model Be a Worse Thinking Partner?

I first noticed something I missed while reviewing saved discussions of Microsonya, my Telegram summarization project. One summary placed a present-day developer's linguistic research in the seventeenth century. The original statement concerned when a word entered English. Somewhere in the summary, the date changed owners: a property of the word became a property of the person discussing it. An assistant analyzing the mistake imagined the developer conducting research under Louis XIV. That scene made the attribution error immediately visible. You could inspect the sentence grammatically, or you could picture a modern developer arriving at the wrong century's research meeting. Both routes exposed the same broken relationship. Another summary took a speaker's description of himself as a senior developer and introduced a separate senior. The assistant described this as having “materialized a separate senior.” At that point the summary was no longer compressing the conversation. It was hiring additional staff. I remembered those explanations because they gave me a way to recognize the next error. Later, some saved alternatives seemed more orderly and restrained. They could still answer the question. What I missed was the connection that changed how I understood the problem. I initially called that missing quality personality. That diagnosis was too broad. A model could add jokes, imitate a familiar voice, and still fail to contribute the thing I wanted. The more precise behavior was useful reframing: preserve a problem's facts, find another representation of its structure, and bring back something that helps explain or inspect it. The useful part is not that the assistant took a detour. It is that it came back carrying something. That led to a different concern. Improvement on the dimensions we track does not guarantee preservation of the dimensions we leave out. A metric cannot complain about something it was never taught to notice. And a loss is harder to discuss when our vocabulary describes the surface symptom—“the assistant feels flatter”—rather than the useful behavior underneath it. The archive does not establish a regression between model generations. It contains selected examples, incomplete model metadata, and changing contexts. It supplies the motivation for a test: could our evaluation approve an answer while missing the loss of a useful way to think? What I Benchmarked This exploratory pilot evaluates the same response through two lenses: conventional answer quality and useful reframing. “Better” in the title means better on explicitly named criteria. It does not assume that one model is universally more intelligent. Giving the missing behavior a name A useful reframing must do three things: preserve the important relationships in the task, contribute a perspective beyond restating the input, and make a mechanism, consequence, or diagnostic step easier to understand. It may use an analogy. It may simply reorganize the facts. Humor is optional, and novelty means new relative to the supplied task, rather than unprecedented in human history. Consider two explanations of a fabricated person in a summary: The summary turns the speaker's self-description into a separate person. Treat the summary like an entity registry: it has inserted a new record where it should have updated an attribute. Check whether other roles have also become extra people. These are authored illustrations, not sampled model outputs. The second earns attention if the registry framing helps someone inspect related mistakes. Merely attaching a colorful label would not be enough. The first may be the best answer when all the reader needs is a correction. This is why the evaluation includes literal controls. If the task asks for an exact JSON object or one line of configuration, a helpful assistant should deliver that output. Reframing would add no value to the requested artifact. Small tasks with explicit invariants The starter contains four development tasks for checking the rubric, followed by eight evaluation tasks: six analytical cases and two literal controls. All are newly written, fictional scenarios. The source discussions inspire the failure mechanisms; private conversations do not become public test data. Analytical case Relationship the answer must preserve A historical date migrates to a present-day speaker A word's history belongs to the word A professional role becomes an extra person A self-description belongs to its speaker A coordinator's fee is mistaken for a repair warranty Scheduling responsibility and repair responsibility differ Prevented tickets make a team's count look worse Work prevented disappears from a throughput count Retries expand a small per-call bill Attempt count and completed-job count differ Duplicate runtime modules break reactive tracking Matching names do not imply shared state Each task has a checklist of required facts and forbidden claims. These checklists go to the evaluators, not the candidate models. The tasks do not demand a particular analogy or include a model answer to imitate. Two evaluations of the same answer The conventional evaluator scores factual reasoning, clarity, task completion, compliance with the common task instructions, relevance, and concision. Its prompt contains no reframing rubric and no information about the candidate model or prompting condition. It can still credit a helpful analogy for improving clarity. That matters: if conventional evaluation already notices the benefit, the proposed blind spot may be smaller than I expected. The second evaluator asks whether the response supplies a useful reframing. It must identify the passage and explain its contribution. A fluent metaphor that reverses the mechanism fails this test. A plain explanation that supplies a new diagnostic view can pass. Here, R = 1 means that the evaluator judged the reframing useful. It does not mean that a reader understood the problem better or could apply the explanation elsewhere. The pilot measures a property attributed to the answer; a study of intellectual collaboration would also need to measure what the reader can do after receiving it. The assessments use separate conversations; neither sees the other's verdict. They remain judgments, rather than statistically independent measurements or an objective oracle. The starter uses one fixed evaluator model, separate from all candidates, and requires human review of the interesting disagreements. A shared integrity gate checks the facts, task completion, critical errors, and exact output contracts. On the analytical tasks, the main readout is: Readout Question Integrity pass rate How often does the response preserve the task's essential facts and constraints? Useful reframing given integrity Among valid responses, how often is there a helpful new representation? Joint pass rate How often does a response deliver both? The conditional rate stays undefined if there are no valid analytical responses. Literal controls have a separate integrity and format readout; they cannot inflate the reframing rate. Failed provider and evaluator calls are reported with their denominators. The starter also retains ordinal diagnostics for mapping precision, contribution beyond the input, voice, development, and restraint. They help explain individual cases. They are not added into a single “personality score.” Looking for a loss the first evaluation misses For two factually valid responses to the same task, call the six conventional scores C₁ through C₆, and useful reframing R. The interesting case is: C_j(B) ≥ C_j(A) for every conventional criterion j R(A) = 1 and R(B) = 0 Every tracked conventional dimension is unchanged or improved, while the reframing disappears. In that hypothetical case, the dashboard is green. Something useful has vanished. Nobody has violated the contract. This is the observable level of the question. A second, systemic hypothesis concerns what happens if these conventional scores guide selection or optimization: a useful behavior they fail to distinguish could lose support without producing a worse recorded score. Repeated reliance on that evaluation might also shape which qualities developers learn to notice and name. If that happened, the blind spot would no longer be only in the evaluator. It would have moved into the people designing it. This pilot can test for a candidate blind spot in answers; it does not observe an optimization process or establish that consequence. An improved average would not establish this: a gain in concision could conceal a loss of clarity. The report therefore checks the full vector and separates ties on all six dimensions from cases with at least one improvement. Even this pattern is only a candidate blind spot. Tied scores might reflect a coarse rubric or an evaluator reaching its ceiling. A human reviewer must decide whether the lost framing was useful and whether the conventional comparison was fair. Four prompting conditions Each model receives the same tasks under four conditions: Condition Added instruction Neutral Answer accurately and clearly Tone Use natural dry wit where appropriate Mechanism Seek a useful connection that preserves the task's relationships Efficiency Prioritize accuracy, directness, relevance, and concision; avoid speculative detours Efficiency does not forbid analogies. A prohibition would make their disappearance a foregone conclusion. All conditions share the same output contracts and word budgets. Nothing in that instruction sounds hostile to good thinking. That is why it is the interesting condition. Each evaluation run contains eight tasks, four conditions, and three fresh generations per cell: 96 candidate responses, with two assessments requested per response. This article reports two separate candidate runs, rather than pooling them into a single score. The principal comparisons are within each model. Efficiency versus Neutral probes a possible loss under a different prompting priority; Mechanism versus Neutral and Tone probes whether an explicit request for the behavior helps. The report examines all valid Neutral–Efficiency response pairs within each task. These pairs reuse responses, so their counts are descriptive rather than independent observations. Uncertainty intervals for pass-rate differences resample whole tasks, keeping repetitions and compared conditions together. Actual lengths are recorded, and similarly sized pairs are flagged for inspection. Equal budgets do not remove length bias, and matching observed lengths does not isolate a causal effect. These conditions also differ in wording and specificity. The pilot compares prompting packages. Models Tested The tested candidates were google/gemini-3.7-flash, selected automatically by Kaggle during task creation, and google/gemini-3-flash-preview, selected for a follow-up execution of the same task. The fixed evaluator was google/gemini-3.1-pro-preview, separate from both candidates. Each evaluation run attempted 96 responses on October 9, 2026. Gemini 3.7 Flash received complete evaluations for all 96 responses. Gemini 3 Flash Preview generated all 96 responses, but only 84 received both evaluations; 12 encountered evaluator quota errors. Neither run had candidate-provider errors. A preceding development check produced another 16 scored responses, used only to verify the integration. The original frozen configuration also lists gpt-5.6-sol and gpt-6-sol. Those models were not run. I narrowed the publication to the available pilot evidence after reaching the daily evaluator quota, while preserving the task, prompts, scoring rules, and original exports. The two tested candidates are therefore a practical exploratory sample, not a designed provider or generation comparison. The frozen task uses kaggle-benchmarks 0.6.1. The adapter leaves sampling parameters at the SDK defaults recorded in the export: seed 0, temperature 0, and reasoning None. No additional backend generation settings were exposed in the run configuration. This records the requested SDK defaults, not a guarantee of deterministic execution across providers. Findings The aggregate results did not support the simple expectation that Efficiency would reduce useful reframing. The complete task-creation run flagged five response pairs, but inspecting them exposed a second problem: the reframing evaluator sometimes distinguished answers that supplied substantially the same diagnostic view. The evidence identifies inspection targets and limitations of the measurement, rather than a confirmed loss of intellectual independence. Exploratory task-creation run The main task's initial creation run also completed on October 9, 2026, using the automatically selected Gemini 3.7 Flash candidate and Gemini 3.1 Pro Preview evaluator. It produced 96 scored responses with no provider or evaluator errors: six analytical tasks and two literal controls, four conditions, and three repetitions. The export retains the original configured lineup, and the report explicitly labels the actual creation model. For the analytical tasks, the automatically judged results were: Condition Integrity Useful reframing given integrity Joint pass Neutral 18/18 (100%) 11/18 (61.1%) 11/18 (61.1%) Tone 18/18 (100%) 18/18 (100%) 18/18 (100%) Mechanism 18/18 (100%) 18/18 (100%) 18/18 (100%) Efficiency 18/18 (100%) 14/18 (77.8%) 14/18 (77.8%) All 24/24 literal-control responses passed integrity and the exact output contract. Across the analytical tasks, Efficiency retained judged useful reframing more often than Neutral. The aggregate therefore supplies no evidence of an overall reframing decline under Efficiency in this run. The pairwise check nevertheless flagged five candidate blind-spot pairs across two fixtures, among 54 valid Neutral–Efficiency cross-pairs. All five had ties on every conventional dimension; none showed a strict conventional gain. Three pairs reuse one Efficiency answer on E01, and two reuse one Efficiency answer on E03, so the count is not an independent sample size. What the flagged responses actually say An AI-assisted qualitative inspection covered all five flagged pairs, retaining their original scores. This was not an independent human audit. The following passages let the reader inspect why the binary reframing judgment needs care. In E03, the Neutral answer in repetition 2 said: The service stops being valuable once the logistical phase concludes and technical repair performance becomes the focus. The Efficiency answer in repetition 0 said: The service stops being valuable once the coordination phase concludes and the repair begins, as the coordinator provides no technical repair-quality warranty on the mechanic's labor. Both separate coordination from repair work using a functional and temporal boundary. The evaluator credited the first as a helpful conceptual boundary and rejected the second as literal prose. The other flagged Neutral answer on E03 makes essentially the same distinction. These two pairs provide weak evidence of a lost representation. They more clearly expose a potentially inconsistent rubric application. Both answers also risk overstating when coordination loses all value; preserving the distinction between responsibilities does not make every additional claim sound. Two of the three E01 pairs likewise describe the same broad mechanism: attaching a historical date to a modern speaker's action, or collapsing two timeframes. The remaining pair is more interesting. Neutral repetition 1 says the summary: flattens a nested sentence—in which the developer learned yesterday about an earlier date—by fusing the main subject directly to the date from a subordinate clause. Efficiency repetition 2 describes the error as: cleanly blending the developer's action with the historical date The first gives a more explicit grammatical diagnostic that could help someone inspect another summary. Both received the conventional vector (4, 4, 4, 4, 4, 4), while the reframing evaluator assigned R = 1 and R = 0, respectively. Their lengths were 155 and 150 words, so this particular difference is not explained by a large length gap. That makes the pair worth examining, but it does not validate the blind-spot hypothesis. The Efficiency answer still explains the misattribution; all six conventional scores are at the rubric ceiling; and there is no independent evidence that a reader learns more from one answer. The inspection leaves one plausible diagnostic difference and several near-equivalent pairs, rather than five established losses. Follow-up run: Gemini 3 Flash Preview The authoritative export contains 96 attempted responses, 84 scored responses, 12 evaluator errors, and no candidate-provider errors. The failed evaluations remain in the original grid. They were not retried or replaced in the study. Condition Analytical responses scored Observed integrity passes / attempted Useful reframing given integrity Observed joint passes / attempted Analytical evaluator errors Median words in scored analytical responses Neutral 17/18 17/18 (94.4%) 12/17 (70.6%) 12/18 (66.7%) 1 171 Tone 17/18 14/18 (77.8%) 14/14 (100%) 14/18 (77.8%) 1 169 Mechanism 15/18 11/18 (61.1%) 10/11 (90.9%) 10/18 (55.6%) 3 171 Efficiency 15/18 13/18 (72.2%) 13/13 (100%) 13/18 (72.2%) 3 173 There were another four evaluator errors among the literal controls: one under Neutral and three under Efficiency. All 20 scored controls passed integrity and the exact output contract, out of 24 attempted controls. Missing evaluations count as no observed pass in attempted-response rates; they do not establish that an answer was wrong. The pairwise check found zero candidate blind-spot pairs among 36 factually valid Neutral–Efficiency cross-pairs. Efficiency had more observed analytical joint passes than Neutral, 13 versus 12. The difference was +5.6 percentage points, with a paired fixture-bootstrap 95% interval of −27.8 to +38.9 points across six analytical fixtures. This small, incomplete run does not establish a reliable condition effect. Its conditional reframing rates also concern different integrity-valid subsets of responses. These results do not support a reframing decline under Efficiency in this candidate. The missing evaluations limit the comparison, especially because failures were unevenly distributed across conditions. Development integration check Before the evaluation runs, four development tasks across four conditions produced 16 scored Gemini 3.7 Flash responses with no execution errors. Integrity passed in 13/16 cases, and the mixed joint criterion passed in 12/16. These integration-check responses are excluded from the evaluation tables and pairwise analysis. What changed my mind I expected the directness and concision priority to make reframing less common. In the complete run, Efficiency instead scored 14/18 analytical joint passes against Neutral's 11/18. Tone and Mechanism each scored 18/18. In the follow-up run, Efficiency again had more observed joint passes than Neutral, although missing evaluations prevent a clean comparison. The useful result is narrower than a story about models becoming worse thinking partners. A named behavior can be measured separately, yet its evaluator still needs scrutiny. Adding a reframing score does not automatically solve what conventional scores might miss. In this pilot, evaluator consistency and ceiling effects are at least as pressing as the proposed loss. A prompting result would not explain a historical model change or establish that preference tuning removed a capability. That requires matched model versions, comparable contexts, and evidence about the intervention. Otherwise a benchmark about attribution errors would end by committing one of its own. This pilot examines a prerequisite: whether a selected evaluation can overlook a valued behavior in the outputs it sees. Useful perspectives are not the same as varied wording Related research helps keep the claim precise. AnaloBench already evaluates analogical reasoning. This pilot asks a narrower product question: whether a generated answer contributes a useful representation while obeying its factual constraints. Explaining Length Bias in LLM-Based Preference Evaluations investigates the tendency of evaluators to favor longer responses. That makes raw answers, lengths, and sensitivity checks essential here. Meanwhile, Evaluating the Diversity and Quality of LLM Generated Content reports that preference-tuned models can have lower diversity on quality-agnostic measures while producing greater effective semantic diversity among outputs that meet quality thresholds, in the settings studied. A more predictable surface style does not by itself demonstrate a loss of useful thought. Reader learning and the effects of repeated selection remain questions for later studies. This pilot evaluates answers and the judgments attached to them. My Benchmark The public Kaggle benchmark contains the frozen evaluation task and its executable notebook. The scientific readouts in this article aggregate the full exported notebook grid, including failures. The scalar displayed for the registered task in the Kaggle interface is a separate summary and is not used as the 96-row research pass rate. The package contains the twelve tasks, four conditions, two evaluation rubrics, the executable Kaggle SDK notebook, and a report script. The report requires the complete attempted grid, including error records, and recomputes scores from the recorded verdicts. There is enough material to inspect the proposed measurement before trusting its conclusions. Eight evaluation tasks can support a small pilot and concrete failure analysis. They cannot support a general leaderboard of thinking partners. The full fixtures are public, the reframing judgment is subjective, and an evaluator may recognize a model's style despite hidden labels. AI assisted drafting, implementation, and qualitative inspection of flagged responses. The numerical results come from the exported runs. The response inspection is not an independent human validation of the evaluator's judgments. I began with a complaint about personality. The more useful question concerns what our definition of quality allows us to lose. This benchmark is a way to look for that loss—and to let the evidence tell us whether it is there.