jakobrogstadius / crisistracker

CrisisTracker is an open-source web platform that extracts situation awareness reports from public tweets during humanitarian disasters. It combines automated processing with crowdsourcing to quickly detect new events and bring structure to millions of updates in real-time.

CrisisTracker

CrisisTracker is a software platform that extracts situational awareness reports from torrents of public tweets during humanitarian disasters. It combines different kinds of automated processing with crowdsourcing to quickly detect new events and bring together related evidence into stories.

As of 2016, the project is no longer maintained, as there is no funding and I need to pay the bills. Contact me if you are interested in sponsoring further development. -Jakob Rogstadius

Distributed sensing via social media

During humanitarian crises in recent years, online social media (mainly Twitter, Facebook and YouTube) have emerged as a means for affected local populations to communicate their experiences to the world. With increasing technology adoption and public access to most posted messages, it is now possible to tap into real-time reports from thousands or millions of people on the ground.

The Twitter microblogging service saw 500 million tweets being posted daily in October 2012, by over 200 million active users. Unlike for instance Facebook and SMS, the vast majority of these tweets is shared publicly and can be accessed in real-time though an application programming interface (API). The challenge however is sense-making. With so much content being generated, maintaining overview and history, and detecting patterns and actionable information, requires specialized information management tools. This is what CrisisTracker aims to provide.

Technology

The CrisisTracker platform uses an automated real-time clustering algorithm based on Locality Sensitive Hashing (LSH) to group together tweets that are textually very similar. A cluster of messages (a "story") typically refers to a single well-defined event, such as artillery shelling of a location, a disease outbreak in a refugee camp, a bombing, etc.

Although individual tweets are both extremely brief (maximum 140 characters) and difficult to verify independently, stories in CrisisTracker capture the event from multiple angles and provide a real-time index of published evidence in the form of images, video and news articles.

Events during a time period can be summarized by looking at those stories that were mentioned by the highest number of Twitter users.

Technologies by module

• The clustering back-end is implemented in C# and MySQL
• The machine learning module uses Java, Weka, MySQL and Redis
• The web front-end uses d3 (for content) and jQuery (for page layout and tabs), with a PHP/JSON API

AIDR Integration

AIDR - Artificial Intelligence for Emergency Response - is an open-source platform developed at QCRI that provides automated real-time topic classification of document streams. It is a supervised classifier, meaning that it learns how to classify documents based on examples provided by human curators.

The AIDR platform has been integrated into CrisisTracker to provide the topic filters seen on the explore page. You can help teach the system how to better classify content by visiting the training page.

Free and open source

CrisisTracker is free and open source, so that you can deploy your own instance or integrate it with your own analysis software. The developers will be happy to hear from you if you set up a deployment!

Resources

• Example deployment
• Installation instructions
• Publications
• More documentation is available on the wiki

Contact

Please contact Jakob Rogstadius for inquiries.

Acknowledgements

Development of CrisisTracker has been made possible through funding and computational resources by M-ITI, IBM Research, University of Oulu and QCRI.

Active contributors:

• None

Past contributors:

• Jakob Rogstadius - Lead developer
• Claudio Teixeira - Developer
• Vassilis Kostakos - Advisor
• Maja Vukovic - Advisor
• Taha Kass-Hout - Advisor
• Evangelos Karapanos - Advisor
• Ko-Hsun Huang - Advisor
• Carlos Castillo - Advisor

crisistracker's Issues

Your project seems to be using an outdated Sarissa version.

Hi,

I noticed your project using an outdated version of the Sarissa JavaScript/AJAX library. The version in your code base has version 0.9.9.5. The latest official version is 0.9.9.6 and it includes some fixes for IE9, among other things. Even that version is old and outdated. I exported the latest CVS trunk version from the official SourceForge CVS repository and exported it to Git and pushed it to GitHub. I made some fixes to the project's Maven POM and I'm trying to collect good quality third party patches for inclusion into the 0.9.9.7-SNAPSHOT branch that I started today. If you are still maintaining any projects that rely on Sarissa in any way, you might want to take a look: https://github.com/volkertb/sarissa/tree/0.9.9.7-SNAPSHOT

Have fun and carry on! :-)

Build failing on CentOS 6.3 with Mono 2.4

Hello,

this is Mathieu, we discussed already via the SBTF.

I'm trying to build CrisisTracker on CentOS 6.3 64 bits (Red Hat Enterprise Linux community rebuild).

I have cloned the GIT repository, run the build (commands and full output below) and get the following errors:

TrackFilter.cs(64,98): error CS0241:  Default parameter specifiers are not permitted
TwitterStreamConsumer.cs(254,25): error CS1526:  A new expression requires () or [] after type
TwitterStreamConsumer.cs(253,17): error CS0210:  You must provide an initializer in a fixed or using statement declaration
TwitterStreamConsumer.cs(254,106): error CS0201:  Only assignment, call, increment, decrement, and new object expressions can be used as a statement
TwitterStreamConsumer.cs(254,107): error CS1525:  Unexpected symbol `,', expecting `;'
TwitterStreamConsumer.cs(251,13): error CS1524:  Expected catch or finally
TwitterStreamConsumer.cs(263,17): error CS1518:  Expected `class', `delegate', `enum', `interface', or `struct'
TwitterStreamConsumer.cs(267,9): error CS8025:  Parsing error
WordCount.cs(71,67): error CS0241:  Default parameter specifiers are not permitted

Any idea of what could cause this?
Is it something in the code or should I change some build arguments.

I am an experienced Java developer, but I have very little experience with C# :(

Cheers,

Mathieu

[mbaudier@mostar Server]$ xbuild CrisisTracker.sln
XBuild Engine Version 2.4.3.1
Mono, Version 2.4.3.1
Copyright (C) Marek Sieradzki 2005-2008, Novell 2008-2009.

Build started 26/08/2012 11:03:47.

---

Project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/CrisisTracker.sln" (default target(s)):

Target ValidateSolutionConfiguration:
    Building solution configuration "Debug|Mixed Platforms".

Target Build:
    Project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/Common/Common.csproj" (default target(s)):

        Target _ValidateEssentialProperties:

        Target PrepareForBuild:
            Configuration: Debug Platform: AnyCPU

        Target SplitProjectReferencesByExistent:

        Target ResolveProjectReferences:

        Target ResolveAssemblyReferences:

        Target CopyFilesMarkedCopyLocal:

        Target AssignTargetPaths:

        Target SplitResourcesByCulture:

        Target CreateManifestResourceNames:

        Target GenerateResources:

        Target GenerateSatelliteAssemblies:
        Skipping target "GenerateSatelliteAssemblies" because its outputs are up-to-date.

        Target CoreCompile:
            Tool /usr/bin/gmcs execution started with arguments: /noconfig /debug:full /debug+ /optimize- /out:obj/Debug/CrisisTracker.Common.dll BinaryHeap.cs English.cs Helpers.cs Output.cs Properties/AssemblyInfo.cs Region.cs Settings.cs TrackFilter.cs TwitterStreamConsumer.cs WordCount.cs WordStemCache.cs /target:library /define:DEBUG;TRACE /reference:../MySql.Data.dll /reference:/usr/lib64/mono/2.0/System.dll /reference:/usr/lib64/mono/2.0/System.Core.dll /reference:/usr/lib64/mono/2.0/System.Xml.Linq.dll /reference:/usr/lib64/mono/2.0/System.Data.DataSetExtensions.dll /reference:/usr/lib64/mono/2.0/System.Data.dll /reference:/usr/lib64/mono/2.0/System.Xml.dll /reference:/usr/lib64/mono/2.0/mscorlib.dll /reference:/usr/lib64/mono/2.0/System.Configuration.Install.dll /reference:/usr/lib64/mono/2.0/System.Transactions.dll /reference:/usr/lib64/mono/2.0/System.Drawing.dll /warn:4

TrackFilter.cs(64,98): error CS0241: Default parameter specifiers are not permitted
TwitterStreamConsumer.cs(254,25): error CS1526: A new expression requires () or [] after type
TwitterStreamConsumer.cs(253,17): error CS0210: You must provide an initializer in a fixed or using statement declaration
TwitterStreamConsumer.cs(254,106): error CS0201: Only assignment, call, increment, decrement, and new object expressions can be used as a statement
TwitterStreamConsumer.cs(254,107): error CS1525: Unexpected symbol ,', expecting;'
TwitterStreamConsumer.cs(251,13): error CS1524: Expected catch or finally
TwitterStreamConsumer.cs(263,17): error CS1518: Expected class',delegate', enum',interface', or `struct'
TwitterStreamConsumer.cs(267,9): error CS8025: Parsing error
WordCount.cs(71,67): error CS0241: Default parameter specifiers are not permitted
Task "Csc" execution -- FAILED
Done building target "CoreCompile" in project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/Common/Common.csproj".-- FAILED

        Target _GetCompileOutputsForClean:

        Target _RecordCleanFile:

    Done building project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/Common/Common.csproj".-- FAILED

Task "MSBuild" execution -- FAILED
Done building target "Build" in project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/CrisisTracker.sln".-- FAILED

Done building project "/home/mbaudier/dev/floss/git/CrisisTracker/Server/CrisisTracker.sln".-- FAILED

Build FAILED.
Errors:

/home/mbaudier/dev/floss/git/CrisisTracker/Server/CrisisTracker.sln (default targets) ->
(Build target) ->
/home/mbaudier/dev/floss/git/CrisisTracker/Server/Common/Common.csproj (default targets) ->
(CoreCompile target) ->

TrackFilter.cs(64,98): error CS0241:  Default parameter specifiers are not permitted
TwitterStreamConsumer.cs(254,25): error CS1526:  A new expression requires () or [] after type
TwitterStreamConsumer.cs(253,17): error CS0210:  You must provide an initializer in a fixed or using statement declaration
TwitterStreamConsumer.cs(254,106): error CS0201:  Only assignment, call, increment, decrement, and new object expressions can be used as a statement
TwitterStreamConsumer.cs(254,107): error CS1525:  Unexpected symbol `,', expecting `;'
TwitterStreamConsumer.cs(251,13): error CS1524:  Expected catch or finally
TwitterStreamConsumer.cs(263,17): error CS1518:  Expected `class', `delegate', `enum', `interface', or `struct'
TwitterStreamConsumer.cs(267,9): error CS8025:  Parsing error
WordCount.cs(71,67): error CS0241:  Default parameter specifiers are not permitted

 0 Warning(s)
 9 Error(s)

Time Elapsed 00:00:00.2399310