mixedemotions / twitter_crawlers

MixedEmotions module that connects to the Twitter Stream API in order to retrieve Tweets regarding certain keywords or phrases

MixedEmotions

The MixedEmotions platform is a Big Data Toolbox for multilingual and multimodal emotion analysis. It is built around stand alone docker modules with an orchestrator that links the modules into analysis workflows utilising MESOS for scalable cloud deployment. Core capabilities include emotion extraction from text, audio and video with many other capabilities, such as sentiment analysis, social network analysis, entity detection and linking and sophisticated data visualisation.

Contact

[email protected]

Citation

if you use any of the modules, please cite the following paper: http://ieeexplore.ieee.org/abstract/document/8269329/

Demonstration Site

http://mixedemotions.insight-centre.org/

Modules

The MixedEmotions platform has been developed with several modules in it. Those modules provide diverse functionalities. Mainly, the toolbox is focused on extracting emoitions from different modalities: text, audio and video. However it also provides other kind of functionalities such as Social Network analysis, Knowledge graphs, entity linking and many others.

Some functionalities are provided with open source modules. Others are provided with proprietary modules.

The list of modules can be found in these tables below, with links pointing to resources where you can have more information about them.

Open Source Modules

Id	Functionality	Modality	Language	Source	Download
m1	Sentiment Extraction	Text	EN	github	Dockerhub
m2	Sentiment Extraction	Text	EN, CS	github	[Dockerhub]
m4	Sentiment Extraction	Text	EN, ES	github	Dockerhub
m5	Emotion recognition	Text	EN	[github]	Dockerhub
m6	Emotion recognition	Audio	EN	github	Dockerhub
m7	Emotion recognition	Text	EN, ES, Multiple	github	Dockerhub
m8	Entity Extraction	Text	ES	github	Dockerhub
m10	Entity Extraction and Linking	Text	EN	github	Dockerhub
m13	Topic Extraction	Text	ES	github	Dockerhub
m16	Suggestion mining	Text	EN	github	Dockerhub
m20	Twitter media crawler	Text	n/a	github	n/a
m21	Fusion	Text/Audio/Video	n/a	[github]	[Dockerhub]
m22	Social Network Analysis	graph	n/a	github	Dockerhub
m25	Social semantic Knowledge graph	graph	n/a	github	Dockerhub
m27	Emotion recognition from Video	Video	n/a	github	+info
m28	Analytics module “Kibi”	-	-	github	kibi
m32	Youtube crawler	Text/Video	n/a	github	n/a

Proprietary Modules

Id	Functionality	Modality	Language	Proprietary	More info
m3	Sentiment Extraction	Text	EN, IT	ExpertSystem	+info
m9	Entity Extraction	Text	IT, EN	ExpertSystem	+info
m11	Topic Extraction	Text	IT, EN	ExpertSystem	+info
m12	Topic Extraction	Text	EN	NUIG	+info
m15	Entity Linking	Text	IT, EN	ExpertSystem	+info
m17	Speech to text	Audio	EN	Phonexia	+info
m18	Machine translation	Text	CS, ES, DE, IT	NUIG	+info
m23	Emotion recognition from Audio	Audio	DE, EN, CS	Phonexia	+info
m24	Recommendation engine	Text	EN	ExpertSystem	+info
m28	Age estimation from audio	Audio	n/a	Phonexia	+info
m29	Gender identification from audio	Audio	n/a	Phonexia	+info

Custom Module

You can use your REST service or your own Docker module within the platform. For how to use a Docker module within the platform check the wiki

Orchestrator

Additionally an open source orchestrator has been developed as an starting point on using the MixedEmotions Toolbox. You can find it here.

More information

More information about the platform can be found on this project's wiki page.

Partners

Partner	Country
NATIONAL UNIVERSITY OF IRELAND, GALWAY	Ireland
UNIVERSIDAD POLITECNICA DE MADRID	Spain
UNIVERSITAT PASSAU	Germany
EXPERT SYSTEM S.P.A.	Italy
PARADIGMA DIGITAL SL	Spain
VYSOKE UCENI TECHNICKE V BRNE	Czech Republic
SINDICE LIMITED	Ireland
DEUTSCHE WELLE	Germany
PHONEXIA SRO	Czech Republic

Acknowledgement

This development has been funded by the European Union through the MixedEmotions Project (project number H2020 655632), as part of the RIA ICT 15 Big data and Open Data Innovation and take-up programme.

http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/index.html

twitter_crawlers's Issues

use recent version of Twarc

It looks like the crawler is using Twarc version 0.3.3

The current version of Twarc on pypi (1.0.9) has a different API, where a Twarc object has a filter method (with different/more parameters) instead of the stream method used here.

We can keep a branch with the older Twarc while the paradigma demo site is still using it.

problems with finding keywords

The re searches in these lines of code:

https://github.com/MixedEmotions/twitter_crawlers/blob/master/twitter_crawler_project.py#L133-L141

have a few problems:

• they find sub-words (ie: synonym "trump" will also find "trumpet", "@Trump" will find "@trumpIsBad"). Twitter search queries don't do this.
• why is a user mention more important than a word occurrence? As far as I know, you can include user mentions in twitter search queries.

crontab entry to ensure crawler restarts if something goes wrong

To ensure continuous crawling, here's a linux crontab entry that would check every 5 minutes to see if it's running:

*/5 * * * * campaigns_twitter_crawler/start_project_twitter_crawler.sh start >> campaigns_twitter_crawler/cronlog.txt

This entry expects campaigns_twitter_crawler to be a subfolder of the users home folder. If this works well, we should add it somewhere, either in the readme or as a comment in the respective launch scripts.

add a requirements file

should we be flagging hash tags also?

the current twitter_crawler_project.py script does not flag when hash tags have been identified as synonyms. From my perspective, knowing if a keyword has been detected as a hash tag vs normal word is important, as they can have quite different meanings.

I'd like to change the code that keeps user mentions (eg: @mrfoo) found in the tweet to also keep hash tags (currently only the word part of the tag is reported).

It'd mean two changes:

• r"@%s\b" changed to r"[#@]%s\b"
• "@%s" % keyword changed to "%s%s" % (word_match.group()[0], keyword)

Thoughts?