Actionable Requests and Offers on Social Media
The project aims to enhance crisis response by leveraging Large Language Models (LLMs) to classify social media posts into actionable "requests" and "offers" with fine-grained categories. A hierarchical taxonomy and Query-Specific Few-Shot Learning (QSF Learning) approach are employed to systematically identify and prioritize relevant posts for humanitarian organizations. On this page, we give a brief overview of the project and provide access to related material, including datasets, taxonomy, and code.
Table of Contents:
Taxonomy
Our taxonomy is a structured and compact knowledge tree, synthesized from classification schemas developed by humanitarian organizations and academic institutions. It provides a detailed yet bounded framework to guide LLMs toward relevant and context-specific understanding. This helps constrain the LLM’s attention to actionable and mission-critical content during classification tasks.
Actionability
We define actionability as crisis-related information that demands a response or constitutes an offer. Actionability is determined by critical details such as time, location, urgency, and specificity (e.g., quantities or concrete actions). Not all urgent posts are actionable — a post may lack the details required to trigger an effective response.
Query-Specific Few-Shot (QSF) Learning
We introduce a novel prompting strategy called Query-Specific Few-Shot (QSF) Learning to enhance LLM classification accuracy. We observed that the relevance of few-shot examples to the input query significantly impacts model performance. When examples closely resemble the query post, the LLM performs better — similar to receiving a custom cheat sheet for the task at hand.
Prompts
To ensure high-quality model outputs, we employ a variety of prompt engineering techniques. These include few-shot prompting, chain-of-thought prompting, and iterative refinement based on experimental findings. Our goal is to maximize performance on classification tasks by optimizing the prompt design.
Downloads:
You can download all the project material as a single archive below. The archive contains:
- taxonomy.json: The complete taxonomy used for classification, derived from various humanitarian documents.
- resources.txt: A file containing the source documents and references used to construct the taxonomy.
- synthetic_data.json: A collection of synthetic, labeled data generated for testing and training.
- real_data.json: Real-world social media posts annotated with fine-grained labels.
- prompts/: A folder containing the prompt templates used in our experiments (BL1 through QSF Learning).
Download here: requests_offers_data.zip
Citation
Coming soon.
Terms of Use
The requests offers dataset is licensed under CC BY-NC-SA 4.0, allowing usage for non-commercial research purposes. Please review the full Terms of Use.