In today’s column, I examine the hot new trend in the AI community decreeing that generative AI and large language models (LLMs) can be rated or assessed via their respective vibes. Yes, that’s right, rather than using conventional ironclad quantitative metrics to compare the best AI LLMs, the latest groovy thing to do is claim that your AI has really good vibes. This is a growing movement that ought to be raising eyebrows and forthrightly made visible to its superficiality and smacks of a Jedi-trick look-the-other-way hand-waving.
Let’s talk about it.
This analysis of an innovative AI breakthrough is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining various impactful AI complexities (see the link here).
Vibes Enters Into The AI World
The usual way to assess generative AI and LLMs consists of measuring various key metrics and then comparing how each such AI performs accordingly (such as calculating the TTFT, time to first token, and the TPS, tokens per second, etc.). There are ongoing debates about which metrics are the most suitable or satisfactory, but at least the matter entails quantitative measurements. It is somewhat possible for AI makers to try and rig their results though once third parties get access to the AI wares, the gig is nearly up. Those third parties can potentially obtain different results and report that something seems to be amiss. In the end, the numbers tell the story.
In recent months, a more nebulous and vacuous kind of assessment has been entering into the AI vocabulary. It has to do with vibes. An AI maker might say that their LLM exhibits great vibes and suggest that even if the hardcore metrics don’t seem to be up to snuff, that’s okay since the vibes are hopping instead.
To be clear, vibes as a measurement tool are entirely in the eye of the beholder.
That’s a pretty convenient way to measure things for the AI maker. No one can fully refute such a claim. If someone contests that the AI isn’t showcasing vibes, an AI maker can just shrug their shoulders and say that, well, they feel it, and not everyone necessarily is going to have that same gestalt sense of the AI. Meanwhile, the AI maker keeps touting the beautiful vibes that their AI is emitting.
If this trend continues to gain steam, the ability to accurately gauge the progress of advancing AI is going to be undercut, sadly so.
What Got The Vibes Ball Rolling
A notable impetus toward adopting the vibes criteria can be generally traced to OpenAI’s Sam Altman’s tweets in March and February of this year. He’s not the only AI luminary to go this route (as an aside, in an upcoming posting, I’ll be discussing the rise of “vibe coding” that was initiated by another AI luminary and has become quite a phenomenon in the prompting of generative AI).
Here is an example of a vibes reference by Sam Altman in a posting on X (March 11, 2025):
- “We trained a new model that is good at creative writing (not sure yet how/when it will get released). this is the first time i have been really struck by something written by AI; it got the vibe of metafiction so right.”
His postings a few weeks earlier on X (taking place on February 27, 2025) set the stage for considering the “magic” of AI, along with a semblance that AI is somehow transforming or transmuting into exhibiting a “different kind of intelligence”:
- “GPT-4.5 is ready! good news: it is the first model that feels like talking to a thoughtful person to me. i have had several moments where i’ve sat back in my chair and been astonished at getting actually good advice from an AI.”
- “a heads up: this isn’t a reasoning model and won’t crush benchmarks. it’s a different kind of intelligence and there’s a magic to it i haven’t felt before. really excited for people to try it!”
Please know that some in the AI community are welcoming the vibes moniker with open arms. The ardent belief is that quantitative measures alone do not tell the full story. You wouldn’t measure a person solely by their height, weight, and other quantifiable metrics. The overall chemistry or vibes of the person would be an important additional dimension. Ergo, we should recognize the same kind of need for assessing AI.
Hogwash, exhort the vibes skeptics. Trying to draw a parallel to how we gauge people is plainly wrong. It is disgraceful and another prime example of the ongoing disconcerting effort to anthropomorphize AI. This vibe conjecture has got to be stopped, right away, before it becomes infused into the AI community. Getting it excised will be almost impossible once it has cemented into AI lore.
A heated debate is underway amidst the for-vibes versus the against-vibes camps.
Reasons To Support Vibes
In discussing the expanding vibes conception with my fellow AI researchers, I’ve come up with four major reasons that there is a strident belief that vibes are handy and well worth continued usage:
- (1) Vibes provide a visceral essence: When using generative AI, some of the LLMs do an amazing job of seemingly exhibiting an emotional bonding with users and this functionality ought to be acknowledged, thus employing vibes for this purpose makes perfectly good sense.
- (2) Vibes denote upbeat connotations: It is customary to associate positivity with the aura of human vibes and this same upbeat connotation helps encourage people to leverage the tremendous strengths of modern-era generative AI.
- (3) Vibes allow for variability. When various LLMs are rated about the same on the customary quantitative measures, the vibes considerations can instructively aid in differentiating one LLM from another.
- (4) Vibes is an ideal catch-all. The word “vibes” is succinct, and everyone already knows what it connotes. If the AI community came up with some new word or technical phrasing to describe the same thing, people wouldn’t know what we were talking about. In short, it’s an ideal catch-all.
Go ahead and take a quiet contemplative moment to mull over those keystone reasons to support the vibes movement. Perhaps do so accompanied by a glass of fine wine.
Reasons To Reject Vibes
In the same discussion with my fellow AI researchers, I came up with four major reasons to reject the vibes inclination:
- (1) Vibes are dodgy. Whether those favoring vibes are genuine or trying to be underhanded, it doesn’t matter either way per se, since the bottom line is that vibes as terminology in assessing AI are entirely amorphous and outright dodgy.
- (2) Vibes promulgate the AI mystique. AI is just bits and bytes. By associating vibes with AI, doing so is an outrageous and shifty means of subliminally implying that AI is mysterious, perhaps bordering on possessing sentience. Shame, shame, shame.
- (3) Vibes mask AI slowdowns. Rather than fessing up to releasing an LLM that doesn’t showcase much in the way of new AI advances, it is all too easy to make a big hullabaloo about the vibes. This is a showmanship distractor and does a disservice to honestly understand the progress of AI.
- (4) Vibes skirt comparability. The loosey-goosey nature of vibes allows a newly released LLM to skirt around the tangible metrics that are used to compare the latest in AI. Soon, we will be comparing generative AI almost entirely on vibes, which will have little or no substantive meaning and become nothing more than a hollow opinion-based shouting match.
Grab yourself another glass of wine and give those notable points some devout attention for a few moments.
Making Vibes Real
To be frank, the odds of stopping the use of AI vibes as a claim are extremely low. It is just too inviting. It is warm and fuzzy. Those carping about the vibes are not going to convince the vibes proponents to back down.
Vibes are here to stay.
This is especially worrisome because AI developers might be tempted to start shaping LLMs toward giving off more and more vibes. Perhaps attention to other important technological advances in AI will be waylaid. The vibes pursuit gobbles up all the air in the room when it comes to making AI better than it is now.
The old saying that it sometimes is useful to fight fire with fire comes to mind here. Suppose that we tried stridently to turn vibes into something quantifiable. In the context of AI, vibes could be a set of quantified measurable elements.
Allow me to briefly outline this faithful proposition.
First, we would need to break down the essence of AI vibes into distinctive measurement components.
I’ll spitball some ideas. Conversational flow could be calculated based on the responsiveness of the user, along with the number of dialogue turns, and so on. Another factor could be a sentiment alignment score, measuring the tone of the AI and the tone of the user. There might also be engagement and likeability scoring based on ratings by users post-conversation with an LLM. Those are just a few suggestions and are meant to get the creative juices going.
Second, the distinctive measurement components would need to be deliberated and ultimately turned into a usable standard that the AI community could readily implement. The idea is that a standard would keep each AI maker honest in their assessments and not allow them to craft their own proprietary measures (they still could do so, but their feet would still be held to the fire by having to presumably report on the standard measures too).
Third, once the standards measures are available, AI leaderboards will hopefully start to report on the vibes metrics. The beauty too is that third parties would be able to perform the same assessments. AI makers that perhaps stretched their vibes claims could be found out. Other competing AI makers would certainly welcome that kind of “keep the playing field level” approach.
A final thought for now.
Some believe that vibes speak louder than words. Thus, seeking to remake vibes into a quantified and calculable instrument is the antithesis of what vibes are all about. You either have vibes or give off vibes, or you don’t.
For me, I have good vibes about making AI vibes into something real, especially since AI vibes appear to be here for good, whether we like it or not.
Read the full article here
