The tech companies advise putting it in everything. It can do anything — transcribe, write your stories, scour the internet for sources, compose your emails and analyze large databases — all while you set your fantasy football lineup.
In reality, there is no intelligence in AI. It’s simply a gigantic pattern machine trying to predict the next word or pixel based on the countless amounts of data it has consumed (either legally or not). It’s less Hal-9000 and more parrot, spouting out words it’s not capable of understanding on its own. Freelance writer Jen Miller has taken to calling it a “lying plagiarism machine” due to AI’s tendency to hallucinate false facts.
While I’m an AI skeptic on Wall Street’s view that the technology will change everything, AI tools can have a place in your journalist’s toolkit. There’s one problem: without a set of standards or guidelines, journalists are in the Wild West when it comes to choosing which software to use.
For instance, one of my favorite tools is the AI-powered transcription service Otter. Not only does Otter provide a searchable transcription, allowing me to go straight to the audio segment I’m looking for, but it also does real-time transcription, a handy function during breaking news situations.
Yet Otter is facing a federal lawsuit claiming the company “deceptively and surreptitiously” recorded private conversations without permission from its users. It also uses recordings to train its AI transcription service, which users agree to in the terms of service. That’s problematic for journalists who may be conducting off-the-record conversations or attempting to protect the anonymity of a source. It also may run afoul of a newsroom’s specific policies when it comes to AI ethics guidelines.
It’s all so overwhelming and confusing. It would be nice if there were a Wirecutter or Consumer Reports for AI tools — a trusted source with real human beings testing and researching these tools to provide a guide for journalists. Jeremy Caplan, the director of teaching and learning at the Craig Newmark Graduate School of Journalism, does an excellent job breaking down AI tools in his Wonder Tools newsletter, but more rigorous testing would be welcome.
Hilke Schellmann could be on that path. Schellmann, an assistant professor of journalism at New York University, recently set out to test how well AI tools work for journalism. A frequent attendee of journalism conferences, Schellmann conceived the idea after speaking with reporters and hearing about the limited time they had to experiment with AI-powered software.
“I don’t think it’s super efficient if every journalist does their own little vibe testing at home,” Schellmann said. ”Wouldn’t it be more helpful if one or two people test the tools with larger experiments so we can authoritatively say, ‘This tool works really well for this use case.’”
So Schellmann and a team of academics, journalists and research assistants went to work to see which AI tools were best for summarizing the transcripts of local government meetings, then compared the results to similar summaries produced by humans.
And the results surprised her.
Schellmann provided the full methodology of her test in a piece published in the Columbia Journalism Review, but trust me, it was rigorous. Out of the four chatbots tested, ChatGPT-4 delivered the most accurate short summaries, around 200 words, using the simple prompt, “Give me a short summary of this document.” Its summaries also retained more facts than the human-written ones, which took three or four hours to complete, compared to the minute it took for the chatbot.
But that all fell apart when Schellmann increased the size of the summary to around 500 words.
“The accuracy went down about 50%, meaning the tools only found about 50% of the facts,” Schellmann said. “That makes it really hard for a journalist to use, because a lot of important information inside the transcripts of the meetings was not retained.”
“For long summaries, I wouldn’t recommend chatbots at all, either for news gathering or research, and absolutely not for publication,” Schellmann added. “Because we can’t trust the tools to be accurate.”
When it came to a separate study of AI-powered software that generates literature reviews of scientific papers — tools that could be really helpful to science journalists — the results were so bad they shocked Shellmann.
“The results were really alarming,” Schellmann said. “The accuracy rate was so low [in the reviews of scientific papers], I would not recommend them at all.”
In a different study conducted last year, Schellmann discovered OpenAI’s popular transcription tool, Whisper, had a major flaw — it created lines of transcribed text not present in the original audio. Such a significant hallucination issue is problematic on many fronts, including the fact that medical centers use Whisper-based tools to transcribe patients’ meetings with their doctor.
Schellmann, who currently holds a research professor fellowship at Princeton University, hopes to enlist the data scientists at the school’s Center for Information Technology Policy to automate this type of testing. Possibly on her to-do list would be to find repetitive work journalists might want to get rid of and test AI tools to make their work more efficient. Or even identifying future threats journalists may face and finding solutions that aren’t currently available to individual reporters on a deadline, such as the best tool to detect whether an image is authentic or created by AI.
“How can we make sure the person we’re interviewing on Zoom is actually a real person and not someone’s avatar?” Schellmann said. “We can’t ask individual journalists to find solutions to that. It’s not doable.”
What are some effective uses of AI-powered technology that journalists can benefit from at this point?
Schellmann said chatbots can perform several functions helpful to journalists, from improving their text to doing data analysis to even scouring the internet for background information. She’s had success using chatbots to uncover a tricky source’s email address by simply asking, “Find me the email address of this person working at this company.” Oftentimes, it’s an address Schellmann could have found on her own, but the chatbot saved her the time and trouble of going down Google rabbit holes.
One of her favorite uses is to drop a story into a chatbot and ask what she’s missing. Depending on her prompt, the chatbot sometimes shoots back slightly better headlines or improvements to her story that you might expect from an editor.
“Sometimes it’s not really all that helpful, but other times it’s beneficial, and it cost me about two seconds,” Schellmann said.
“It can be a helpful technology in the way an intern is helpful,” Schellmann said. “They can help you do stuff, compile things and summarize things, but you always have to make sure it’s accurate.”
Rob Tornoe is a cartoonist and columnist for Editor and Publisher, where he writes about trends in digital media. He is also a digital editor and writer for The Philadelphia Inquirer. You can reach him at robtornoe@gmail.com.
Comments
No comments on this item Please log in to comment by clicking here