A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries

There is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task depend on standardized and reproducible evaluation methodologies for comparing systems. However, timeline summary evaluation is still in its infancy, with competing methodologies currently being explored in international evaluation forums such as TREC. One area of active exploration is how to explicitly represent the units of information that should appear in a 'good' summary. Currently, there are two main approaches, one based on identifying nuggets in an external 'ground truth', and the other based on clustering system outputs. In this paper, by building test collections that have both nugget and cluster annotations, we are able to compare these two approaches. Specifically, we address questions related to evaluation effort, differences in the final evaluation products, and correlations between scores and rankings generated by both approaches. We summarize advantages and disadvantages of nuggets and clusters to offer recommendations for future system evaluations.

Gaurav Brauah, Richard McCreadie and Jimmy Lin.
A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries
In Proceedings of CIKM, 2017.

PDF
Dataset

Analyzing Disproportionate Reaction via Comparative Multilingual Targeted Sentiment in Twitter

Global events such as terrorist attacks are commented upon in social media, such as Twitter, in different languages and from different parts of the world. Most prior studies have focused on monolingual sentiment analysis, and therefore excluded an extensive proportion of the Twitter userbase. In this paper, we perform a multilingual comparative sentiment analysis study on the terrorist attack in Paris, during November 2015. In particular, we look at targeted sentiment, investigating opinions on specific entities, not simply the general sentiment of each tweet. Given the potentially inflammatory and polarizing effect that these types of tweets may have on attitudes, we examine the sentiments expressed about different targets and explore whether disproportionate reaction was expressed about such targets across different languages. Specifically, we assess whether the sentiment for French speaking Twitter users during the Paris attack differs from English-speaking ones. We identify disproportionately negative attitudes in the English dataset over the French one towards some entities and, via a crowdsourcing experiment, illustrate that this also extends to forming an annotator bias.

Karin Sim Smith, Richard McCreadie, Craig Macdonald, and Iadh Ounis
Analyzing Disproportionate Reaction via Comparative Multilingual Targeted Sentiment in Twitter
In Proceedings of ASONAM, 2017.

PDF
BIBTEX

Transfer Learning for Multi-language Twitter Election Classification

Both politicians and citizens are increasingly embracing social media as a means to disseminate information and comment on various topics, particularly during significant political events, such as elections. Such commentary during elections is also of interest to social scientists and pollsters. To facilitate the study of social media during elections, there is a need to automatically identify  posts that are topically related to those elections. However, current studies have focused on elections within English-speaking regions, and hence the resultant election content classifiers are only applicable for elections in countries where the predominant language is English. On the other hand, as social media is becoming more prevalent worldwide, there is an increasing need for election classifiers that can be generalised across different languages, without building a training dataset for each election. In this paper, based upon transfer learning, we study the development of effective and reusable election classifiers for use on social media across multiple languages. We combine transfer learning with different classifiers such as Support Vector Machines (SVM) and state-of-the-art Convolutional Neural Networks (CNN), which make use of word embedding representations for each social media post. We generalise the learned classifier models for cross-language classification by using a linear translation approach to map the word embedding vectors from one language into another. Experiments conducted over two election datasets in different languages show that without using any training data from the target language, linear translations outperform a classical transfer learning approach, namely Transfer Component Analysis (TCA), by 80% in recall and 25% in F1 measure.

Xiao Yang, Richard McCreadie, Craig Macdonald, and Iadh Ounis
Transfer Learning for Multi-language Twitter Election Classification
In Proceedings of ASONAM, 2017.

PDF
BIBTEX

Searching the Internet of Things

Richard McCreadie, Dyaa Albakour, Jaranna Manotumruka, Craig Macdonald, and Iadh Ounis
Searching the Internet of Things
Building Blocks for IoT Analytics, 2016.

Book
BIBTEX

EAIMS: Emergency Analysis Identification and Management System

Social media has great potential as a means to enable civil protection and law enforcement agencies to more effectively tackle disasters and emergencies. However, there is currently a lack of tools that enable civil protection agencies to easily make use of social media. The Emergency Analysis Identification and Management System (EAIMS) is a prototype service that provides real-time detection of emergency events, related information finding and credibility analysis tools for use over social media during emergencies. This system exploits machine learning over data gathered from past emergencies and disasters to build effective models for identifying new events as they occur, tracking developments within those events and analyzing those developments for the purposes of enhancing the decision making processes of emergency response agencies.

Richard McCreadie, Craig Macdonald, and Iadh Ounis
EAIMS: Emergency Analysis Identification and Management System
In Proceedings of SIGIR, 2016.

PDF
BIBTEX

Comparing Overall and Targeted Sentiments in Social Media during Crises

The tracking of citizens' reactions in social media during crises has attracted an increasing level of interest in the research community. In particular, sentiment analysis over social media posts can be regarded as a particularly useful tool, enabling civil protection and law enforcement agencies to more effectively respond during this type of situation. Prior work on sentiment analysis in social media during crises has applied well-known techniques for overall sentiment detection in posts. However, we argue that sentiment analysis of the overall post might not always be suitable, as it may miss the presence of more targeted sentiments, e.g. about the people and organizations involved (which we refer to as sentiment targets).  Through a crowdsourcing study, we show that there are marked differences between the overall tweet sentiment and the sentiment expressed towards the subjects mentioned in tweets related to three crises events.

Saul Vargas, Richard McCreadie, Craig Macdonald, and Iadh Ounis
Comparing Overall and Targeted Sentiments in Social Media during Crises
In Proceedings of ICWSM, 2016.

PDF
BIBTEX

SUPER: Towards the use of Social Sensors for Security Assessments and Proactive Management of Emergencies

Social media statistics during recent disasters (e.g. the 20 million tweets relating to `Sandy' storm and the sharing of related photos in Instagram at a rate of 10/sec) suggest that the understanding and management of real-world events by civil protection and law enforcement agencies could benefit from the effective blending of social media information into their resilience processes. In this paper, we argue that despite the widespread use of social media in various domains (e.g. marketing/branding/finance), there is still no easy, standardized and effective way to leverage different social media streams -- also referred to as social sensors -- in security/emergency management applications. We also describe the EU FP7 project SUPER (Social sensors for secUrity assessments and Proactive EmeRgencies management), started in 2014, which aims to tackle this technology gap.

Richard McCreadie, Karolin Kappler, Magdalini Kardara, Andreas Kaltenbrunner, Craig Macdonald, John Soldatos and Iadh Ounis
SUPER: Towards the use of Social Sensors for Security Assessments and Proactive Management of Emergencies
In Proceedings of SWDM 2015 (at WWW 2015).

PDF
BIBTEX