Detection of Disaster-Affected Cultural Heritage Sites from Social Media Images Using Deep Learning Techniques

Authors: Pakhee Kumar, Ferda Ofli, Muhammad Imran, Carlos Castillo
ACM Journal on Computing and Cultural Heritage (JOCCH), 13(3):23.1-23.31, 2020.
https://doi.org/10.1145/3383314

Abstract: This paper describes a method for early detection of disaster-related damage to cultural heritage. It is based on data from social media, a timely and large-scale data source that is nevertheless quite noisy. First, we collect images posted on social media that may refer to a cultural heritage site. Then, we automatically categorize these images according to two dimensions: whether they are indeed a photo in which a cultural heritage resource is the main subject, and whether they represent damage. Both categorizations are challenging image classification tasks, given the ambiguity of these visual categories; we tackle both tasks using a convolutional neural network. We test our methodology on a large collection of thousands of images from the web and social media, which exhibit the diversity and noise that is typical of these sources, and contain buildings and other architectural elements, heritage and non-heritage, damaged by disasters as well as intact. Our results show that while the automatic classification is not perfect, it can greatly reduce the manual effort required to find photos of damaged cultural heritage by accurately detecting relevant candidates to be examined by a cultural heritage professional.

Demo: https://aidr-heritage.qcri.org/

sites covered by our dataset
Heritage and not-heritage sites covered in our dataset

Dataset This resource contains images and expert annotations for the detection of damaged heritage sites. The images were downloaded from Google and labeled as “heritage” vs. “not-heritage”. We tested our trained models on a Twitter dataset that we also annotated to identify whether a heritage site shows damage or not.

Dataset (~4GB)


Please cite the following paper, if you use this resource in your research.
Pakhee Kumar, Ferda Ofli, Muhammad Imran, Carlos Castillo. Detection of Disaster-Affected Cultural Heritage Sites from Social Media Images Using Deep Learning Techniques. Accepted in the ACM Journal on Computing and Cultural Heritage (JOCCH), 2020.