Observational study – how do journalists really verify UGC?
The importance of user generated content (UGC) in journalism is growing relentlessly [1] [2] [3]. Smartphones have high quality cameras able to take excellent eyewitness images, record videos of events as they happen and even stream videos in real-time. Eyewitness media will often appear on Social Networks before any mention occurs on mainstream news channels.
What is an observatory study?
To better understand the real-world strategies and workflows employed by journalists the University of Southampton IT Innovation Centre and Deutsche Welle have worked together to conduct an observational study of journalists doing UGC verification. We worked with 3 experienced journalists for a day, filming them and asking them to ‘think aloud’ as they performed several different UGC verification tasks. We watched what they did and asked them why they did it.
The results are summarized in this article and provide a window into the reality of UGC verification today. Our long term aim is that these results will allow future verification tools to be developed that better support what journalists really do day to day. Here is a short video introducing the study and discussing its findings:
Participants
The three participants were all experienced DW journalists with many years of experience. Some work with UGC on a daily basis whilst others have more limited exposure in their daily role. They all came from different backgrounds and had different education and experience levels. The participants were:
- Julia Bayer – DW Social Media Journalist
- Zulfikar Abbany – DW Science and Technology Journalist
- Ruben Bouwmeester – DW Innovation UGC Researcher
Tasks
We prepared three tasks, each of which contained 5-10 content items and/or profiles. For all tasks participants followed their standard verification processes taking up to 20 minutes per item. Each task ended with a short verbal summary per item stating if they would consider using it in a news story and what evidence is behind this verification decision.
- Task 1: Assess credibility and bias of a UGC source profile. Decide whether a source can be trusted to report sufficiently unbiased facts about a given topic.
- Task 2: Verify a UGC claim. Decide whether a UGC claim is “good enough” (i.e. credible) to be used in a news story.
- Task 3: Verify content. Decide whether a UGC image or video is “good enough” (i.e. authentic) to be used in a news story.
It was made clear that this was not a test and we were interested only in the process and the evidence that was accepted/discarded, not the final conclusion or the time needed. Observations were made in a training room at Deutsche Welle’s premises not at a live UGC news desk.
Results – Participant approaches
Julia Bayer – ‘get the big picture’
This journalist was driven by the bigger picture more than by small details. Whilst she did have first impressions, she collected all possible information about the author who generated the content and this heavily influenced her decision. Independent of which task she did, an extensive profile check was done to decide whether the author in question was credible or not.
Zulfikar Abbany – ‘trust your gut’
This journalist was able to decide quickly whether an item was worth pursuing further or not. Whilst still carrying out the required checks he was able to process the suggested items much quicker than the two other journalists relying more on instinct.
Ruben Bouwmeester – ‘the devil is in the detail’
This journalist had a more structured approach, following the same workflow with each item. The biggest selection of tools was used of all the participants and the level of detailed examined was highest. This meant the verification accuracy was higher than the other two journalists, with little things spotted that the others missed. This journalist used all of the time allotted unlike the others before making a decision.
A video containing three perspectives on video verification from these participants is available as part of work from the InVID project.
Results – Verification strategies and tools used by the participants
After analysis of the verification activities performed by the three participants, Stefanie Wiegand from The IT Innovation Centre created a schematic overview of the verification activities that she identified, which you can see below. The complexity of the activities show that verification of UGC is a nonlinear process that can be very different from person to person.
Overview of the UGC verification strategies and tools used by participants
Challenges
One of the biggest challenges for journalism is the quickly changing environment in which they are expected to carry out their work. There are many sources of UGC (e.g. Twitter, Facebook, Instagram or whatever Social Network emerges tomorrow) and each has a set of tools needed to analyse them.
We found that all of the tools used in this study were web-based, because journalists are often limited installing desktop tools on their work PC’s for security reasons and because of company policies. The majority of these tools were used in their free (i.e. functionally reduced) versions because of the multitude of existing tools, each covering just a niche verification task. For some tools companies may acquire licenses but they are often expensive and only pay off for larger institutions.
All journalists are under constant time pressure and need to make quick decisions about whether UGC can be used or not. This is particularly difficult given that any mistakes reduce the journalists own reputation.
Work such as this article, understanding the scattered tooling and strategies required to facilitate high level UGC verification, is essential when looking at the future direction of tools to effectively support the verification process.
Acknowledgement
The work presented in this article is part of the research and development in the REVEAL project (grant agreement 610928), supported by the 7th Framework Program of the European Commission.
References
[1] Wiegand, S. Middleton, S.E. 2016. Veracity and velocity of social media content during breaking news: Analysis of november 2015 paris shootings. Third Workshop on Social News On the Web (SNOW-2016), WWW 2016, Companion of the 25th International World Wide Web Conference
[2] Silverman, C. (Ed.), 2013. Verification Handbook. European Journalism Centre
[3] Silverman, C. 2015. Lies, Damn Lies, and Viral Content. How News Websites Spread (and Debunk) Online Rumors, Unverified Claims, And Misinformation. Tow Center for Digital Journalism, Columbia Journalism School
Related Posts
Image Verification Corpus Released September 11, 2014 | Jochen Spangenberg
Verify This Week (13/2014) April 9, 2014 | Linda Rath-Wiggins
Pants on fire: content verification tools and other ways to deal with the fake n... December 13, 2016 | Aleksandra Kuczerawy
Verify this week (29/2014) October 8, 2014 | Linda Rath-Wiggins
