Hallucinations: When Your Model Confidently Invents a Citation

· 2 min read · Syed Omar Faruk Towaha
Hallucinations: When Your Model Confidently Invents a Citation

A student once showed me a literature review with twelve beautifully formatted references. Authors, journal names, volume numbers, DOIs. Three of them didn't exist. The authors were real researchers. The journals were real journals. The papers were pure imagination, assembled by a language model that knew exactly what a citation looks like.

That's a hallucination: fluent, plausible, confident and false.

Why it happens

A language model is trained to predict the next piece of text. It learns patterns: what follows "According to a 2019 study in", what a DOI looks like, which researchers write about which topics. When you ask for a specific fact it doesn't reliably know, it doesn't have a built-in "I don't know" reflex unless it has been trained and prompted carefully to use one. It produces the most plausible continuation, and plausible is not the same as true.

The model isn't lying. Lying requires knowing the truth. It's more like a student in an oral exam who has read a lot and is very good at sounding right.

Where the risk is highest

Risk by task
Not measured data — a rule of thumb for where to be careful.

Hallucinations are rare when you give the model the text and ask it to transform it. They're common when you ask it to recall precise, niche details: exact numbers, quotes, citations, page numbers, recent events, small companies, obscure people.

How to protect yourself

1. Give it the source. Instead of "what does the latest guideline say?", paste the guideline and ask questions about it. This is the whole idea behind retrieval-augmented generation: ground the answer in documents.

2. Ask for evidence and check it. "Quote the sentence that supports this" makes errors easier to spot. Then actually look the quote up.

3. Verify every citation yourself. Search the DOI. Open the paper. If it takes you more than a minute to find, assume it doesn't exist.

4. Permit uncertainty. Add "If you're not sure, say you're not sure" to your prompt. It helps more than you'd expect.

5. Use tools with search when facts matter, and still check the links. A link can be real while the summary of it is wrong.

6. Be most suspicious of numbers. Percentages, dates and statistics are where fluent nonsense hides best.

The academic angle

In research, a fake citation isn't a small error. It can be treated as misconduct, even if a tool produced it. Your name is on the paper, not the model's. Use AI to brainstorm, outline, rephrase and explain, but every factual claim and reference in your final work should be something you've seen with your own eyes.

The healthy mindset

Treat a language model like a brilliant friend who has read everything and remembers it imperfectly. Fantastic for ideas, explanations and first drafts. Terrible as the only source for anything you'll be held accountable for. Trust, but verify. Actually, mostly verify.

// related

// prefer the terminal?

Open the terminal blog and type read ai-hallucinations-confident-citations.