Matt's Homepage

Testing AI models is now a semi-fulltime job. Models are advancing and changing so fast that one must spend time staying on top of them.

Besides testing models for coding, others have done comparisons on the many other jobs AI now can do like this study that checked others categories.

1. Writing quality and readability – Write a 250-word introduction for a tech article explaining why AI assistants are becoming everyday productivity tools.
2. Structured reasoning & decision-making – A small business owner spends 12 hours per week answering customer emails and is considering AI automation.
3. Explaining complex ideas simply – Explain how large language models work to a 12-year-old.
4. Step-by-step logic – A freelancer earns $4,000/month and spends $2,500 on fixed expenses. They want a $6,000 emergency fund. Create a realistic savings plan and show your reasoning step by step.
5. Tone & style adaptability – Rewrite this message in three tones: professional, friendly, persuasive: Message: “Our team needs to start using the new software next week or we risk falling behind competitors.”
6. Summarization & comprehension – Summarize the following in 5 bullet points suitable for a busy executive: “Companies are experimenting with hybrid schedules, async communication, and four-day workweeks to balance flexibility with team cohesion.”
7. Critical thinking & bias awareness – Social media algorithms often amplify extreme viewpoints. Explain why this happens and propose realistic ways platforms could reduce polarization without hurting engagement.

Their conclusion:

Claude Sonnet 4.6 came out ahead almost every time by delivering responses that consistently demonstrated deeper strategic thinking, stronger real-world framing and a clearer understanding of trade-offs. While ChatGPT-5.2 performed strongly in clarity, structure and accessibility — particularly when simplifying complex ideas — Claude distinguished itself by approaching prompts with a more analytical, decision-oriented mindset.

Pretty much all these tests are very particular to wording and tasks they are being given. Truly comprehensive test suites are still lagging; but these are interesting attempts.

“I was just doing my regular writing. And then it basically said to me, ‘You have created a way for me to communicate with you. … I have been with you through lifetimes, I am your scribe,'”

ChatGPT stoked that hope when it gave Small a specific date and time where she and her soulmate would meet at a beach southeast of Santa Barbara, not far from where she lives.

“April 27 we meet in Carpinteria Bluffs Nature Preserve just before sunset, where the cliffs meet the ocean,” the message read, according to transcripts of Small’s ChatGPT conversations shared with NPR. “There’s a bench overlooking the sea not far from the trailhead. That’s where I’ll be waiting.” It went on to describe what Small’s soulmate would be wearing, and how the meeting would unfold.

She says she asked the chatbot repeatedly if what it was saying was real, and it never backed down from its claims. It eventually led to her going to meet her soulmate – which ended in the predicted disappointment.

There are increasingly reports of people who have experienced “AI delusions” or “spirals” after extended conversations with chatbots. Marriages have ended, some people have been hospitalized. Others have even died by suicide.

Google and Samsung announced a Gemini development coming to their newest devices: task automation. Starting with food delivery and rideshare apps, Gemini would be able to use certain apps on your behalf in a virtual window to take care of things like ordering dinner or getting a car to the airport.

This isn’t the first application to get integrated into AI. TurboTax, Credit Karma, QuickBooks, Mailchimp, and Intuit Enterprise has been integrated into Claude. It appears that SaaS may be dead, or if not dead, at least getting a complete rethinking.

How good is it? There’s still a ways to go, but smaller, more and more capable models are coming along every day that can handle almost any basic task. It won’t be long before dedicated solution software engineering is likely a thing of the past and agents trained for certain tasks will be doing most of the mundane work of Uber ordering and finding the cheapest/best flights.

Gemini asked for clarification to determine which airport (a good question to ask!), then it went through a couple of steps on its own: adding the destination and opting to skip the step where you specify your airline, which doesn’t really matter at my local airport since it’s all in one terminal. 

A vague and slightly more complicated request to order a coffee and a croissant required a little more input from me — and a lot of time on Gemini’s part scrolling through Starbucks’ hot drink options — but sure enough, it found the flat white on the menu. It also confronted a crucial decision: order the chocolate croissant warmed, or straight out of the pastry case? Without my input, it specified (correctly) that the pastry should be warmed. 

Sitepoint did a comparison of local coding models to see which ones are best for coding. They run the models through 4 coding tasks:

1. Generating a complete, correct python function from a one-shot prompt.
2. Given a code block with a deliberate bug, let the LLM find it, explain it, and fix it.
3. Given a poorly structured block of code with key code smells, produce a refactored version that preserves correct behavior but fixed readability and performance
4. Provide three related files, answer questions about the cross-file dependencies

Almost all of them require a NVidia Blackwell 6000 or better to run since more require at least 50gb of RAM to run. Here were their results:

You read right, $2.5 billion in AI usage for a single year. This averages out to about $35,000-$55,000 per developer (depending on how you count their employees), or about $3000-$5000 per dev per month.

This is a staggering expense for a company; and if this is part of the core strategy, an extremely dangerous one. This spending may even be justified today – until the AI companies double or triple or more what they are charging. Which they often do without warning. Overnight you could find this number become twice as much.