
The reason to change
My previous note asked whether a citation actually supports a claim. That is also where I would begin when deciding whether to change models. I want to know how much usable work reaches my spreadsheet after I have checked it.
As a finance student interested in automation, I can see the appeal of trying every new release. But switching has a cost of its own: adjusting prompts, checking access, learning how the model fails and rebuilding confidence in the output. I would want a reason tied to a specific task before making that investment.
A replacement would need to earn its place
When checked on October 29, Vals’ Finance Agent v2.1 table listed Terra at 55.96% and $1.92 per test, Sol at 61.82% and $1.63, and Gemini 4 Argon at 67.66% and $4.32. These are benchmark observations, not prices or expected success rates for my own workflow. The primary score gives weighted partial credit subject to critical checks; All-Pass is a separate measure. The test split contains 449 questions, and v2.1 scores are not directly comparable with v2.
Those rows would help me choose what to investigate. They would not settle the decision. I would keep one shipping-company filing, a small set of questions and a manually checked Excel answer sheet fixed. Every answer would need its original label, period, unit and source passage. I would record the model version, available tools and settings so that a different setup does not quietly become a different experiment.
Before running it, I would write down what would make a replacement acceptable. It should introduce no new critical errors in this small test, and it should improve something I care about: review time, completeness or measured cost. A wrong unit or an unsupported explanation would count as a serious failure even if the table looked polished.
I would repeat the questions and keep unsuccessful attempts. Then I would compare the time and corrections needed to reach an acceptable answer, alongside actual usage charges. I would also include a manual baseline: for a few numbers, reading the filing myself could be faster. This would be a learning exercise, not enough evidence to declare a universal winner.
Keeping the decision reversible
I have not conducted this comparison. My aim would be to learn when changing tools is worth the effort, including when it is sensible to keep the current setup. A small improvement on one task would justify another test before wider use.
If I later connected the process to n8n, a new model version would enter a review queue first. It would run against the saved questions without replacing approved inputs. That is a proposed design, not an integration I have built.
The next question is how to keep that comparison meaningful over time. When a filing changes, I would need to distinguish a corrected source value from a model error—and preserve both versions before updating the spreadsheet.
Sources
- Finance Agent v2.1: leaderboard and methodology — Vals AI · Published Updated 2026-10-27; accessed 2026-10-29
- Evaluation announcement: GPT-6.1 Terra — Vals AI · Published 2026-10-26; accessed 2026-10-27
- Previous note: A citation is not the same as evidence — Jaehyuk Song · Published 2026-10-23; accessed 2026-10-29