LumaBlog

Can you trust the sources in an AI medical answer?

Not automatically. The sources in an AI-generated medical answer can point to papers that do not exist, or to real papers that do not say what the answer claims, so each one has to be checked before you rely on it.

AI answers can still be useful. Treat each citation as a lead to follow, and look for answers where every claim is tied to a real paper that supports it, or clearly flagged when no such paper turns up.

Why do AI medical answers cite papers that do not exist?

A large language model writes by predicting which words are likely to come next, based on patterns in the text it was trained on. It does not consult a library while it writes unless it has been connected to one, and even a connection does not guarantee that the final text matches what was retrieved.

A reference is a very regular kind of text: author surnames, a title, a journal, a year, a volume, page numbers. A model that has seen many references can produce a new one in the same shape, with a title that fits the question and a journal that fits the field. The result looks like a citation because it was built from the pattern of citations. Nothing in that process checks that the paper exists.

We cover the mechanics in more detail in what a fabricated citation is and why AI tools produce them.

How often are AI-generated citations fabricated?

One measurement comes from a 2023 study by Walters and Wilder in Scientific Reports. The authors examined citations in literature reviews written by an earlier version and a newer version of a widely used chatbot: 636 citations across 84 generated literature reviews on 42 topics. The topics came from many fields, so the study was not specific to medicine.

  • 55 percent of the citations from the earlier version were fabricated.
  • 18 percent of the citations from the newer version were fabricated.
  • Among the citations that were real, 43 percent of the earlier version's and 24 percent of the newer version's contained substantive errors.

The newer version did better, but it still fabricated part of its reference list, and its real citations carried errors too.

For biomedicine specifically, a 2026 audit by Topaz and colleagues in The Lancet reports that prior studies estimate 30 to 69 percent of references generated by large language models in biomedical contexts are fabricated.

Are fake references already in published research?

Yes. The Topaz audit checked 97.1 million structured references across about 2.5 million biomedical papers and found 4,046 fabricated references across 2,810 papers. Roughly one in 277 papers published in the first seven weeks of 2026 cited a paper that does not exist, up from one in 458 in 2025 and one in 2,828 in 2023.

At the time of the audit, 98.4 percent of the affected papers had received no publisher action. A reference copied from a published paper can therefore still lead nowhere.

What is a misattributed citation?

Fabrication is the easier failure to catch, because a search for the paper comes back empty. The harder one is the misattributed citation: the paper is real, the details are correct, and the paper does not support the sentence that cites it.

Misattribution takes a few familiar forms:

  • The study looked at a different population, such as adults when the claim is about children.
  • It measured a different outcome than the one the claim describes.
  • It reported a tentative finding, and the answer states it as settled.
  • The finding appears in the paper's background section, where the authors were summarizing someone else's work.

The Walters and Wilder error rates among real citations show why a reference that exists still needs reading. For more on this failure, see why a real citation can still be the wrong citation.

What does a trustworthy AI medical answer look like?

A trustworthy answer makes checking easy. Look for these features:

  1. Each factual claim stands on its own, so you can check it separately instead of judging the answer as a whole.
  2. Each supported claim links to a specific paper with a working identifier, such as a PubMed ID (PMID) or a digital object identifier (DOI).
  3. The linked paper actually supports the claim: same population, same outcome, and a conclusion as firm as the sentence.
  4. Claims that could not be supported are flagged plainly instead of being given a nearby citation.
  5. You can open each source and read it yourself.

The first point is the idea behind claim-level verification. A mostly right answer can hide one wrong claim, and a verdict on the whole answer will miss it.

How can you check the sources yourself?

You do not need special access. PubMed, the free search engine for biomedical literature maintained by the National Library of Medicine at the National Institutes of Health, is open to anyone.

For each citation:

  1. Search the PMID or the exact title on PubMed.
  2. Confirm that the authors, journal, year and volume match the record.
  3. Read the abstract and ask whether it supports the specific sentence that cites it.
  4. Look for a retraction notice on the record.

Our guide on how to check if a PubMed citation is real walks through each step and the warning signs of a fake.

To see claim-by-claim checking on a real question, Luma's demo separates an answer into claims, searches published research for each one, links the supported claims to their sources and flags what it cannot verify.

Frequently asked questions

Can AI tools make up medical references?

Yes. A large language model can produce a reference with realistic authors, title and journal for a paper that does not exist, because it generates text from patterns instead of looking sources up. Every AI-supplied reference should be checked against a database such as PubMed.

Are newer AI models better at citing sources?

In a 2023 study by Walters and Wilder, a newer version of a widely used chatbot fabricated 18 percent of its citations, compared with 55 percent for an earlier version. Fewer fabrications still leaves some, so references from newer tools need the same checks.

Does a PMID prove a citation is real?

No. An identifier can be invented, or it can belong to a different paper than the one described. Look up the PMID and confirm that the title, authors and journal on the record match the citation.

Should patients trust the sources in an AI health answer?

Patients can use AI answers to learn terms and prepare questions, but the sources should not be taken on trust. Bring the answer and any questions it raises to your clinician, who can tell you what applies to you.

References

Check an AI medical answer

Paste an answer or ask a question. Luma splits it into claims, checks each one against published research, links what the evidence supports and flags what it cannot verify.

Try the demo

More from the blog

# Can you trust the sources in an AI medical answer?

Published October 1, 2026

Not automatically: AI medical answers can cite papers that do not exist or real papers that say something else, so every source needs a check before you use it.

Not automatically. The sources in an AI-generated medical answer can point to papers that do not exist, or to real papers that do not say what the answer claims, so each one has to be checked before you rely on it.

AI answers can still be useful. Treat each citation as a lead to follow, and look for answers where every claim is tied to a real paper that supports it, or clearly flagged when no such paper turns up.

## Why do AI medical answers cite papers that do not exist?

A large language model writes by predicting which words are likely to come next, based on patterns in the text it was trained on. It does not consult a library while it writes unless it has been connected to one, and even a connection does not guarantee that the final text matches what was retrieved.

A reference is a very regular kind of text: author surnames, a title, a journal, a year, a volume, page numbers. A model that has seen many references can produce a new one in the same shape, with a title that fits the question and a journal that fits the field. The result looks like a citation because it was built from the pattern of citations. Nothing in that process checks that the paper exists.

We cover the mechanics in more detail in [what a fabricated citation is and why AI tools produce them](/blog/what-is-a-fabricated-citation).

## How often are AI-generated citations fabricated?

One measurement comes from a 2023 study by Walters and Wilder in Scientific Reports. The authors examined citations in literature reviews written by an earlier version and a newer version of a widely used chatbot: 636 citations across 84 generated literature reviews on 42 topics. The topics came from many fields, so the study was not specific to medicine.

- 55 percent of the citations from the earlier version were fabricated.
- 18 percent of the citations from the newer version were fabricated.
- Among the citations that were real, 43 percent of the earlier version's and 24 percent of the newer version's contained substantive errors.

The newer version did better, but it still fabricated part of its reference list, and its real citations carried errors too.

For biomedicine specifically, a 2026 audit by Topaz and colleagues in The Lancet reports that prior studies estimate 30 to 69 percent of references generated by large language models in biomedical contexts are fabricated.

## Are fake references already in published research?

Yes. The Topaz audit checked 97.1 million structured references across about 2.5 million biomedical papers and found 4,046 fabricated references across 2,810 papers. Roughly one in 277 papers published in the first seven weeks of 2026 cited a paper that does not exist, up from one in 458 in 2025 and one in 2,828 in 2023.

At the time of the audit, 98.4 percent of the affected papers had received no publisher action. A reference copied from a published paper can therefore still lead nowhere.

## What is a misattributed citation?

Fabrication is the easier failure to catch, because a search for the paper comes back empty. The harder one is the misattributed citation: the paper is real, the details are correct, and the paper does not support the sentence that cites it.

Misattribution takes a few familiar forms:

- The study looked at a different population, such as adults when the claim is about children.
- It measured a different outcome than the one the claim describes.
- It reported a tentative finding, and the answer states it as settled.
- The finding appears in the paper's background section, where the authors were summarizing someone else's work.

The Walters and Wilder error rates among real citations show why a reference that exists still needs reading. For more on this failure, see why a real citation can still be the wrong citation.

## What does a trustworthy AI medical answer look like?

A trustworthy answer makes checking easy. Look for these features:

1. **Each factual claim stands on its own**, so you can check it separately instead of judging the answer as a whole.
2. Each supported claim links to a specific paper with a working identifier, such as a PubMed ID (PMID) or a digital object identifier (DOI).
3. The linked paper actually supports the claim: same population, same outcome, and a conclusion as firm as the sentence.
4. Claims that could not be supported are flagged plainly instead of being given a nearby citation.
5. You can open each source and read it yourself.

The first point is the idea behind [claim-level verification](/blog/what-is-claim-level-verification). A mostly right answer can hide one wrong claim, and a verdict on the whole answer will miss it.

## How can you check the sources yourself?

You do not need special access. PubMed, the free search engine for biomedical literature maintained by the National Library of Medicine at the National Institutes of Health, is open to anyone.

For each citation:

1. Search the PMID or the exact title on PubMed.
2. Confirm that the authors, journal, year and volume match the record.
3. Read the abstract and ask whether it supports the specific sentence that cites it.
4. Look for a retraction notice on the record.

Our guide on [how to check if a PubMed citation is real](/blog/how-to-check-if-a-pubmed-citation-is-real) walks through each step and the warning signs of a fake.

To see claim-by-claim checking on a real question, Luma's [demo](/demo) separates an answer into claims, searches published research for each one, links the supported claims to their sources and flags what it cannot verify.

## Frequently asked questions

### Can AI tools make up medical references?

Yes. A large language model can produce a reference with realistic authors, title and journal for a paper that does not exist, because it generates text from patterns instead of looking sources up. Every AI-supplied reference should be checked against a database such as PubMed.

### Are newer AI models better at citing sources?

In a 2023 study by Walters and Wilder, a newer version of a widely used chatbot fabricated 18 percent of its citations, compared with 55 percent for an earlier version. Fewer fabrications still leaves some, so references from newer tools need the same checks.

### Does a PMID prove a citation is real?

No. An identifier can be invented, or it can belong to a different paper than the one described. Look up the PMID and confirm that the title, authors and journal on the record match the citation.

### Should patients trust the sources in an AI health answer?

Patients can use AI answers to learn terms and prepare questions, but the sources should not be taken on trust. Bring the answer and any questions it raises to your clinician, who can tell you what applies to you.

## References

- Topaz M, Roguin N, Gupta P, Zhang Z, Peltonen LM. The Lancet. 2026;407(10541):1779-1781. doi:10.1016/S0140-6736(26)00603-3. PMID 42107362. https://pubmed.ncbi.nlm.nih.gov/42107362/
- Walters WH, Wilder EI. Scientific Reports. 2023;13(1):14045. doi:10.1038/s41598-023-41032-5. PMID 37679503. https://pubmed.ncbi.nlm.nih.gov/37679503/