Overview of the Classifying Concerning Tweets study

This study aimed to determine whether suicide-related Twitter posts could be classified as ‘strongly concerning’ based solely on the content of the post, as judged by human coders.

We wished to determine whether such content could be legitimately considered as an indicator of suicide risk. From the 18th February 2014 to the 23rd April 2014, Twitter was monitored for a series of suicide-related phrases and terms using the public Application Program Interface (API). Matching tweets were stored in a data annotation tool developed by the Commonwealth Scientific and Industrial Research Organisation (CSIRO).

From 18th Feb 2014 – 23 April 2014 Twitter was monitored for a series of suicide related phrases  and terms using an API interface. During this time 14,701 suicide related tweets were collected. 14% were randomly selected and divided into 2 equal sets for coding by human researchers. Machine learning processes were then applied to assess whether we could identify a “concerning tweet”, automatically and in real time.

Strongly Concerning

14%

Possibly Concerning

56%

Safe to Ignore

28%

Human Coder Agreement

76%

Social media monitored

Twitter

Meet the Researcher

Dr. Bridianne O’Dea

The machine learned classifier correctly identified 80% of ‘strongly concerning’ tweets and showed increasing gains in accuracy; however, future improvements are necessary as a plateau was not reached.

The current study demonstrated that it is possible to distinguish the level of concern among suicide-related tweets, using both human coders and an automatic machine classifier. Importantly, the machine classifier replicated the accuracy of the human coders.

These findings confirm that Twitter is used by individuals to express suicidality and that the proposed method has advanced our ability to automatically and reliably detect suicidality among Twitter users.

Contributors

Bridianne O’Dea
Cecile Paris
Helen Christensen
David Milne
Mark E. Larsen
Tjeerd Boonstra
Philip Batterham
Stephen Wan

Funding

  • NHMRC John Cade Fellowship  APP1056964
  • NSW Mental Health Commission