Robojournalists vs. Humans

Automated Insights and Narrative Science are two companies that specialize in producing software that can write news articles. But how do these news articles stack up to the human-written competition? A German researcher attempts to find out.

In the peer-reviewed article “Enter the robot journalist: users’ perceptions of automated content”, researchers randomly gave German undergraduate students one of two articles. One article was written by a human, and another was written by a program called “Statsheet” (which was created by Automated Insights). The German students were to read the article, rate it, and then say whether the article was written by a human being or a bot.

The researchers asked two questions:

1) Can the research students consistently identify whether an article is written by a human or a bot? - The answer is a resounding “NO”. While a majority of students correctly identified the bot-written article as being written by a bot, a majority of students also identified the human-written article as being written by a bot. There is no statistically significant difference between how the students ‘evaluated’ whether the article was written by a human or a bot.

2) Are bots able to write prose of equal quality and crediblity as that of humans? - The answer is “Yes, with one exception”. There was no statistically significant difference between the quality of the journalism of the human and the bot…except for the fact that the human’s article was seen as more pleasant to read.

(As a tangentical side note, the students rated the bot’s article was being more informative, accurate, trustworthy, and objective…though the bot’s article was also rated as more boring to read too. The students also rated the human’s article as being well-written and coherent.

The reason this note is tangentical is that these results are not “statistically significant”; if you were to run this same experiment with a different population, you would likely get different results.)

Conclusion: The “robojournalists” are able to produce journalism that is able to equal that of the human competition. There are limits to the robojournalists though. They are unable to write prose that is ‘pleasent to read’. This may be a sign of limits to robojournalists’ creativity, which would give humans the edge. For now.