Re: ChatGPT contributing to current science papers

On 8/10/2024 11:38 PM, JTEM wrote:

  RonO wrote:
 

Several examples of scientists using AI to write papers with AI generated mistakes that passed peer review.

 If you're only finding out now that "Peer Review" is heavily
flawed, you haven't been paying attention.
 THE BIGGEST PROBLEM is that when one ever hear of one type of
mistake:  When "Peer Review" validates garbage. What also
happens and you NEVER hear about is when "Peer Review"
silences good science, keeps it out of print.
  

Peer review has it's flaws, but there is absolutely no doubt that it is the best means we have for giving research it's first pass evaluation. Peer review can be manipulated (Sternberg and Meyer), and groups of researchers have been exposed for recommending each others papers for peer review (some journals ask the authors to recommend possible peer reviewers in their field).
Peer review doesn't just catch flawed research, but reviewer suggestions nearly always make a paper better than it initially was.  It is rare for any paper to be accepted without reviewer comments that need to be acted on.  I have only had 2 papers accepted as submitted without modification.  They are my 3rd and 4th most cited papers (576 and 360 citations according to google scholar).  The first paper was the first such example for all involved even my major professor.  The second was the first such example for all but 3 of 16 authors.  My guess is that the majority of researchers never experience such an event.  I wrote the initial draft of the first paper, and my major professor suggested a bunch of revisions, and then he suggested revisions of the revisions for multiple subsequent drafts.  He often did this.  One of his students screamed in his face after an extended run of revisions for what she was writing.  The result was a paper where we had gone over the results and conclusions so many times that there was nothing left to add, and nothing that was out of place.  The second example occurred because some of the authors did not want to accept some of the conclusions from the data for reasons that should not have affected the scientific conclusions.  As a result we had an intensive internal peer review where the manuscript was revised, and the methods were thoroughly evaluated with the limitations of the methodology clearly described in the paper. Wording of the conclusions and discussion were gone over multiple times to try to satisfy the dissenters.  In the end the dissenters asked to be removed as authors for a paper that was submitted to and accepted without revisions by PNAS, and became a widely cited paper in the field.
Usually peer review catches things that need to be revised, but for those two papers we did it ourselves.
Ron Okimoto