Overview of the Classifying Concerning Tweets study

This study aimed to determine whether suicide-related Twitter posts could be classified as ‘strongly concerning’ based solely on the content of the post, as judged by human coders.

We wished to determine whether such content could be legitimately considered as an indicator of suicide risk. From the 18th February 2014 to the 23rd April 2014, Twitter was monitored for a series of suicide-related phrases and terms using the public Application Program Interface (API). Matching tweets were stored in a data annotation tool developed by the Commonwealth Scientific and Industrial Research Organisation (CSIRO).

From 18th Feb 2014 – 23 April 2014 Twitter was monitored for a series of suicide related phrases  and terms using an API interface. During this time 14,701 suicide related tweets were collected. 14% were randomly selected and divided into 2 equal sets for coding by human researchers. Machine learning processes were then applied to assess whether we could identify a “concerning tweet”, automatically and in real time.

Strongly Concerning


Possibly Concerning


Safe to Ignore


Human Coder Agreement


Social media monitored


Meet the Researcher

Dr. Bridianne O’Dea

The machine learned classifier correctly identified 80% of ‘strongly concerning’ tweets and showed increasing gains in accuracy; however, future improvements are necessary as a plateau was not reached.

The current study demonstrated that it is possible to distinguish the level of concern among suicide-related tweets, using both human coders and an automatic machine classifier. Importantly, the machine classifier replicated the accuracy of the human coders.

These findings confirm that Twitter is used by individuals to express suicidality and that the proposed method has advanced our ability to automatically and reliably detect suicidality among Twitter users.


Bridianne O’Dea
Cecile Paris
Helen Christensen
David Milne
Mark E. Larsen
Tjeerd Boonstra
Philip Batterham
Stephen Wan


  • NHMRC John Cade Fellowship  APP1056964
  • NSW Mental Health Commission