AI or Human? Study Finds 73% of People Get Fooled by Chatting with OpenAI’s Latest Bot
Apr 22, 2025
Buckle up!
A new study shows that OpenAI’s GPT-4.5 may have passed a version of the Turing Test under specific conditions. In a text-based chat setup, people couldn't tell GPT-4.5 from a real person 73% of the time.
This is huge news!
But first, you may be wondering, what is the Turing Test?
Here's my lay interpretation.
Imagine you’re on one side of a black curtain and on the other side there’s a human and a ‘machine’ (i.e., computer). If you were to talk to the human and the machine through the curtain, and... you couldn't tell the difference, the machine passes the Turing Test for being indistinguishable from a human.
Passing this test doesn’t mean GPT-4.5 “thinks” like a human. However, it does show you how good these large language models (LLMs) are at sounding more human-like.
The Study
In this study, three people joined a chat: a human, GPT-4.5, and a judge. The judge’s job was to figure out who was human. When GPT-4.5 was told to ‘act like’ a certain type of person (i.e., adopt a role) — say, a young person who knows a lot about internet culture — again, it fooled judges 73% of the time.
Lead researcher Cameron Jones said people couldn’t tell the difference between GPT-4.5 and a human, and sometimes thought the AI was more human than the real person!
Mind officially blown here!
The Big Takeaway!
Giving the AI a Role Made a Huge Difference.
Here's where the test took an interesting turn in my opinion. Without any special prompts — just basic instructions to sound human — GPT-4.5 did much worse. Its success rate dropped to 36%. Yes! Worse!
This shows how important the setup is. Telling the AI to “act” a certain way helps it sound more real. This is a huge finding! When using any of the LLMs, it would be wise to assign your AI model a role if you want better results!
Here's an 'over-simplified' example: Bad Prompt: "Analyze this data and tell me what you find." Good Prompt: "Act like a CFO, analyze this data and tell me what you find."
Significance for selling?
These advances in machine interactions continue to bring us closer and closer to having AI Agents who will connect, converse, and convert prospects or leads into new clients without any human-in-the-loop (HITL). And, moving forward, make sure to assign your LLM a role. Here's another example:
Bad Prompt: “Write a sales email to a healthcare company.”
Good Prompt: “You are a high-performing B2B SaaS sales rep with 10 years of experience selling enterprise software to IT decision-makers. Your goal is to write a cold outreach email to a CTO at a mid-sized healthcare company, focusing on reducing IT downtime and improving data security.”
Role assignment matters!