Market research firms are cooked 😳 PyMC Labs + Colgate just published something wild. They got GPT-4o and Gemini to predict purchase intent at 90% reliability compared to actual human surveys. Zero focus groups. No survey panels. Just prompting. The method is called Semantic
Most LLM surveys fail because models regress to the mean. When asked for a direct “1–5” rating, GPT-4o replied “3” almost every time producing KS similarity = 0.26 to real human data. Translation: the distribution was basically useless.
When researchers switched to free-text answers and then asked a second model to translate those into ratings (“Follow-up Likert Rating”), correlation with real panels jumped to ρ = 0.85. Distribution realism improved to KS = 0.72. Better, but still too narrow.
The real breakthrough came with Semantic Similarity Rating (SSR) mapping each text answer to 5 anchor statements via cosine similarity. Result: ρ = 0.90 correlation attainment and KS = 0.88 distribution similarity for GPT-4o. Basically 90% of human test–retest reliability.
Even more interesting: synthetic consumers mirrored real demographic patterns. Middle-aged personas rated higher purchase intent than young or old ones. Income level “2” (“in danger financially”) triggered the sharpest drop in ratings exactly like human panels.
The best part? When LLMs weren’t given demographic context, they matched the shape of human distributions (KS = 0.91) but lost meaning correlation fell to ρ = 0.50. So realism ≠ understanding. The persona prompt is what makes the model “think” like a consumer. Read full
The AI prompt library your competitors don't want you to find → Biggest collection of text & image prompts → Unlimited custom prompts → Lifetime access & updates Grab it before it's gone 👇 https://godofprompt.ai/pricing
@rryssf_ Fascinating
@rryssf_ Anyone productizing this. DM me. I will fund it.
@rryssf_ Nice write up, I think it's great AI models can whip up fake "people" that mimic real survey responses, skipping slow human panels for faster, fairer results tbh!
@rryssf_ This is the correct diagnosis. An industry built on extracting signal from human noise is being automated by an engine trained on that same noise. The insight isn't the method; it's the vulnerability. The real prize is not a better engine for interpreting the chaos. It is
@rryssf_ This semantic approach could revolutionize market research validation. 90% reliability without traditional surveys means dramatically lower customer acquisition costs for companies who adopt early. Big opportunity for AI-driven research automation startups.
@rryssf_ Interested. How are you calibrating against real sales by channel? I’d test it to pre-score creatives before panels. Any API or shared prompts to reproduce?
@rryssf_ This signals a massive disruption in market research. If LLMs can predict consumer behavior at 90% accuracy, the entire consumer insights industry needs to adapt. Companies leveraging this early will have significant competitive advantages.
@rryssf_ the future of one man business is coming
@rryssf_ Predicting purchase intent with 90% reliability is nuts. Market research done.
@rryssf_ don’t underestimate the power of LLM, they can turn into good if you need a structure of the information you want to retrieve also users are spread into different LLM so the key is orchestrated LLM to get the full response
@rryssf_ seems like the audience in comments understand that its a thing but their brains cant accept this reality..
@rryssf_ Yeah
@rryssf_ They will need to reinvent their industry or consulting will get its Napster moment.
@rryssf_ This hits home. @VelozityAI helps teams skip the survey panels too. Our AI personas run instant focus groups and usability tests straight from your data. Results in minutes, not weeks.. at a fraction of the cost. Try us https://velozity.co
@rryssf_ Looks like those market research firms just got served a side of AI with their predictions! Maybe they should've seen this coming instead of asking "why?" 😂
@rryssf_ next up, simulate voting results?
@rryssf_ Looks like AI is about to make traditional market research almost obsolete.
@rryssf_ Your thread is everybody's favorite! #TopUnroll https://threadreaderapp.com/th... 🙏🏼@TheRealZongi for 🥇unroll
@rryssf_ Firms still paying for market research should’ve been cooked awhile ago
@rryssf_ I’ve just built an app that can do Market a competitive intelligence Using frameworks like PORTERS McKc Gartner It can do SWOT , PESTLE, industry life cycle etc Will continue to refine I agree AI is coming for Big 4 , but looking at Deloitte fiasco I think there is time
@rryssf_ Soon every product will be absolutely right !
@rryssf_ This is seriously wild! 🤯 The idea of replacing traditional market research with something so much faster and cheaper is mind-blowing. What do you think this means for the future of insights work? #AI #MarketResearch
@rryssf_ Next up: simulating potential users to test any product idea. oh, wait, I just developed that: mimiqai dot com
@rryssf_ I can build this into a product. Comment if interested.
@rryssf_ this is genius
@rryssf_ that approach basically compresses weeks of research into minutes and at a fraction of the cost
@rryssf_ Wild how fast this flipped! Semantic similarity + reasoning beats static surveys every time
@rryssf_ consulting firms are definitely sweating right now.