Opinion: Politico’s troubles show some of the issues ahead for AI deployments

By Alisa Cromer

At  NichePublisher.Biz,  we are always looking for ways to add value to a membership.  

What if paid members could type a question into a search bar or bot and get a summarized answer from our content, with links to the reports that answered the question? 

Our model was the database  CrunchBase, which reinvented its home page as a single AI-powered search bar, without a list of categories at all.

This kind of tool in chatbox form has also already been built at the Washington Post and the Financial Times.

So we went on a hunt for a tool and found Direqt. With a single line of code deployed, our new chatbot would answer questions from our content with links to the articles, for around $50 a month. We deployed it on the development server to see how it would do. 

Direqt did surface the right articles, and a summary. But when asked a question from a real publisher, “Do B2B publishers still use editorial calendars?” the BOT answered “yes,” even though there are no articles mentioning B2B editorial calendars on our website. Was it using external information? 

Another issue is the style of the answers. We have a very particular voice – if not a style guide  – to make information specific and actionable. The BOT answers were wordy and abstract. There was no easy way to edit these results by, say, using prompts. 

So for now, the tool – and the project –  remains on hold, stalled on the dev server.

Trouble at Politico

Politico had a similar experience using AI to power search and summaries of its  owned content.

An article for Neiman Journalism Lab by Andrew Deck provides an inside view of the accuracy issues that still occur. When  The PEN Guild, which represents over 250 workers at Politico and its sister publication, E&E News, filed a claim of contract violation for AI issues, Deck got access to transcripts including all documents  – 300 pages worth – from the July 11 hearing. 

Yes, they had an AI  policy

Politico execs had an AI policy. It was even codified into the editorial union contract with The Pen Guild. 

Specifically, editors are supposed to review and greenlight AI-assisted content. However,  their innovative AI usage did not neatly fit the policies created. 

Consider Politico’s  AI-powered Report Builder, available to Politico’s top-tier Pro Members. The tool builds paginated reports – complete with charts and maps from a query, using information in Politico’s own archive. 

According to the article, Jeremy Bowers, Politico’s global chief technology officer, said this feature was highly requested by Politico Pro subscribers, which include lobbyists, aides, and NGOs. 

So it would seem like a simple decision to provide this valuable, additional service to paid members. A quick search on Microsoft’s Report Builder shows the premium per user starts at $20 per month, so perhaps economical, too. 

But according to documents in the transcripts, Report Builder made mistakes.

Aspects of the Biden administration’s oil and gas policies were attributed to the Trump administration, and  information on the  Farm Bill became co-mingled with issues related to Social Security, Medicare, and Medicaid, the article said. 

 Ouch. 

Speech to Text Event Coverage, too

Politico also used a tool called LETO to  record, transcribe and summarize all speeches during last fall’s Democratic National Convention (DNC) in something like real time – posted summaries of each speech appeard  home page of Politico every 10 to 15 minutes. 

Apparently, LETO works a lot like Otter.ai, which most of us use, and posts on the home page were, in fact, labeled as AI-generated. 

With a prepared speech, not an interview, the almost real time updating seemed to be a valuable, more neutral upgrade to what users wanted to know, separate from the thousands of opinion blogs. 

But…Politico’s official editorial policy said that a team of editors is supposed to review and copy-edit AI-generated content, the article noted.

The automated workflow designed by the engineers never sent the content through the CMS, so editors did not even see it, much less edit it, before publication.

As a publisher, I would have been be enticed to add this live stream of updates to the home page anyway, in a high competitive political news market during one of the biggest stories of the year. The transcripts were of live speeches – those are real – and everyone knows that AI is not perfect, right? 

Plus, can  AI summaries can even be considered “articles” and, if labeled,  doesn’t everyone know the difference? 

And the speech-to-text tool only resulted in minor errors like mis-spelled names. It’s  only too easy to get on this slippery slope. 

The issue of trust

What also came up during Politico’s arbitration hearing, was how their policy on corrections and retractions – a key feature of building trust in news media –  suddenly became inapplicable. How do you retract the answer to a question – or report generated – by a bot? Should you issue a formal correction to a published, labeled AI  summary?

So these are issues we all will probably need to consider in the future.

Interestingly, according to Decks, Politico’s deputy editor-in-chief Joe Schatz  testified on behalf of management, about the AI-produced summaries.

“I would never want to publish this as an article, because it’s not an article,” he said.