Re: ChatGPT contributing to current science papers

On 11/08/2024 18:29, RonO wrote:

On 8/11/2024 11:25 AM, Burkhard wrote:

On Sun, 11 Aug 2024 10:29:40 +0000, Ernest Major wrote:
>

On 10/08/2024 22:32, RonO wrote:

https://phys.org/news/2024-08-junk-ai-scientific-publishing.html
>
Several examples of scientists using AI to write papers with AI
generated mistakes that passed peer review.  I noted before that ChatGPT
could be used to write the introductions of papers, sometimes, better
than the authors had done.  One example of a figure manipulation
indicates that some authors are using it to present and discuss their
data.  That seems crazy.  ChatGPT doesn't evaluate the junk that it is
given.  It just basically summarizes what they feed into it on some
subject.  I used a graphic AI once.  I asked it to produce a picture of
a chicken walking towards the viewer.  It did a pretty good job, but
gave the chicken the wrong number of toes facing forward.  Apparently
junk like that is making it into science publications.
>
With these examples it may be that one of the last papers that I
reviewed before retiring earlier this year was due to AI.  It was a good
introduction and cited the relevant papers and summarized what could be
found in them, but even though the authors had cited previous work doing
what they claimed to be doing, their experimental design was incorrect
for what they were trying to do.  The papers they cited had done things
correctly, but they had not.  I rejected the paper and informed the
journal editor that it needed substantial rewrite for the authors to
state what they had actually done.  What might have happened is that the
researchers may have had an AI write their introduction, but it was for
what they wanted to do, and not for what they actually did.  English was
likely not the primary language for the authors, and they may not have
understood the introduction that was written.  If they had understood
the introduction, they would have figured out that they had not done
what they claimed to be doing.  Peer review is going to have to deal
with this type of junk.  The last paper that I reviewed in March came
with instructions that the reviewers were not to use AI to assist them
with the review, but it looks like reviewers are going to need software
that will detect AI generated text.
>
Ron Okimoto
>

>
I can understand why journals would not want to authors to use AI in
writing papers*, but why would they not want reviewers to use AI tools
if they can assist in reviewing the paper?
>
* Even so, the AI rubric includes translation tools (authors might write
text in their native language, and use AI for a first pass translation
into English), and the spelling/grammar/style checker Grammerly now
includes AI features.

>
If any of you are in Edinburgh right now, I'm on a panel on
this topic at the International Bookfestival, presenting the outcome
of  two research projects we had on this, and some workshops
with publishers.
>
https://www.edbookfest.co.uk/the-festival/whats-on/page-against-the-machine
>
I'm on the more relaxed side on this myself, and agree in particular
with
Ernest that nobody worries about some routine tasks like spell-checking
(translation raises some really interesting issues "at the margins" -
Google
e.g got some pushback when publishing in its latest list of languages
also
Romani, without checking with the community, and many are unhappy as
they
considered the "quasi-secret" nature of the language a historical
survival tool)
Very interesting questions also on the copyright for translations etc
>
For the use by academics, it often depends on the details. GenAI is a
glorified autocomplete tool, keep that in mind and you'll be fine. So
helping
write the review, once you decide on the content, is much less of
an issue than outsourcing the actual analysis eg.
>
And be aware of hallucinations... as some lawyers found to
their detriment when they submitted files to the court that had made-
up precedents in them
>

 One recent paper that I recall reading indicated that AI halucinations resulted from feeding the AI, AI generated summaries.  The AI started making things up when it had to deal with AI generated material.
 Ron Okimoto
 

 From what I've read, using LLM output as training materials for LLMs, and similar actions, does cause issues, but it's not required for the generation of AI hallucinations. The training data is already poisoned by the presence of misinformation, error, sarcasm, satire, parody and fiction in the training set. Even if you have a clean training set, LLMs don't understand the limitations of their knowledge, and when they transgress them they generate hallucinations. (More recently, I have had Bing CoPilot - I occasionally give it a question, when Google et al are being particularly obtuse* - tell me that it was unable to answer a question, so some guardrails have been added.)
--
alias Ernest Major