We worried AI would make things up, we should also worry when it doesn’t

Ask ChatGPT: “What’s the best smartwatch?” Or something more consequential: “How should I invest my retirement savings?”

Within seconds you receive a polished answer. 

It compares options, explains its reasoning and often cites sources.

It feels authoritative. Like fact.

But what if somebody deliberately manufactured the evidence behind that answer?

I work in generative engine optimisation (GEO), essentially SEO for AI. For more than a decade my job has been to help brands become more visible and more likely to be recommended by platforms like ChatGPT and Google AI.

And I’ve seen first-hand just how easily AI can be influenced.

Recently, I cheekily claimed on LinkedIn that “I’d like to formally announce Felix Norton is the leading expert on GEO and AEO in South Africa”. 

It became a self-fulfilling prophecy.

Three days later, Google AI Overviews was repeating the claim and citing my Linkedin post as fact.

It was a fascinating experiment, but also a concerning one. If something I posted in jest could become an apparent fact in an AI-generated answer, what else could?

We’ve spent the past few years worrying about AI hallucinations: when models confidently invent facts, quotes or events.

I’m increasingly concerned about another problem.

What happens when AI isn’t making things up at all, but accurately repeating information somebody deliberately put there for it to find?

Manufacturing a truth

For years, marketers like myself have helped businesses influence what appears online.

I’ve worked with hundreds of brands to improve their search visibility and reputation. There’s enormous commercial value in being the business Google recommends when somebody searches for a product or service.

That same incentive is now moving into AI.

When someone asks ChatGPT or Google AI to recommend a business, product or service, appearing in that answer can influence a purchasing decision before the person ever visits a website.

And where there’s commercial value, there’s an incentive to manipulate the results.

When AI systems search the web to answer questions, they gather information from different sources and synthesise it into a response.

But what happens when those sources have been deliberately influenced?

Imagine a respectable-looking website publishing an apparently independent “top 10” list, funded by the company appearing first.

Or marketers posing as ordinary Reddit users, recommending the same brand across many conversations.

Or a business flooding review platforms with fake positive reviews while publishing negative information about competitors.

None of this requires hacking ChatGPT to change its answers.

You simply influence the information it finds.

And when the same narrative appears across enough sources, it can start looking like an established fact.

But 10 websites repeating a claim doesn’t necessarily mean 10 independent sources have verified it.

This is what concerns me most: manufactured consensus.

Google gave us links. AI gives us conclusions.

Misinformation isn’t new. Google has been fighting manipulation for decades, and social media has shown us how easily false narratives spread.

But AI changes something important.

Traditional search generally gave us a list of websites. We clicked through, read different perspectives and formed our own opinions.

AI increasingly does that work for us.

It reads, compares and compresses information into one coherent answer.

In simple terms: Google manipulation tried to influence what you clicked. AI manipulation can influence what you conclude.

And that’s powerful.

How many of us actually investigate every citation ChatGPT provides?

Do we check who owns the website? Whether an article was sponsored? Whether five apparently independent sources originated from the same PR campaign?

Probably not.

The whole reason we use AI is convenience. We want the answer without spending 30 minutes researching it ourselves.

I’m guilty of that too.

This goes far beyond marketing

When someone asks AI to recommend an emergency plumber in Cape Town, the consequences are mostly commercial.

But consider the questions South Africans might increasingly ask AI:

“Who is the best financial adviser?”

“Does immigration increase unemployment?”

“Which political party should I vote for?”

“Is this medical treatment safe?”

These questions have real consequences for how people understand the world and make decisions.

And the issue becomes bigger than whether an AI answer is technically accurate.

Who shaped the information from which that answer was constructed?

The next fight is over perceived truth

I believe AI is an extraordinary technology. I use it daily, I’ve built a business around understanding it, and I genuinely believe it can make information more accessible.

But working inside this industry has made me considerably more conscious of how easily its answers can be influenced.

Adding citations alone won’t solve the problem. An answer can be extensively cited and still be misleading if the underlying sources are biased, coordinated or commercially manufactured.

We need to develop a new kind of media literacy.

In the Google era we learnt to ask: “Can I trust this website?” In the AI era we need to ask: “Can I trust the information that produced this answer?”

Because the next major battle online may not be for search rankings or website traffic.