Comments (4)
Descriptions
The purpose of tweet parsing is to split a tweet JSON data into several associated data structures, e.g., tweet, url, twitter_user, hashtags and etc. If we would like to save these data structures into a database, the relationships of tables should also be taken considered. For relationships like one-to-one (e.g., table tweet
vs. ass_tweet
) and one-to-many (e.g., table twitter_user
vs. table tweet
), we should finish the insertion of tables with no foreign key dependencies (e.g., table twitter_user
) because of the unknown of the foreign keys. After the insertion, the foreign keys become known, then we can use them to insert the depending tables (e.g., table tweet
). For many-to-many relationship (e.g., table tweet
vs. url
), things become a little more complicated. An intermediated table is needed (e.g., table ass_tweet_url
), which is many-to-one to the other two table that associate with it. To accomplish the insertion of URLs of a tweet, we need to first finish the insertion of the tweet
table and the url
table, then fetch these inserted primary id
s and insert them into ass_tweet_url
table correspondingly.
One Per Time Implementation
In twitter streaming, the tweets are received one by one. Thus, the straightforward idea is to parse and save a tweet one by one. In this implementation, the parsing and saving operations are working interactively. For example, when we parse the necessary data of the twitter_user
, we insert it and get its primary id
in the database. In the following parsing and saving operations, we will use this primary id
when necessary.
Bulk Implementation
From above, we can see that saving the parsed objects from a tweet JSON data into the database require many database queries operations. The performance of the one-per-time implementation is very limited, which cannot consume large number tweets in a short time. In the twitter streaming, the consuming of tweets must be fast enough to keep the connection of the stream. To overcome it, in the current implementation, we use a queue to cache the coming tweets and then parse them. Even though, the one-per-time implementation would generate so many queries that may overload the shared database server. Moreover, sometimes we want to reparse the tweets (e.g., when we have requirements of new tables or bug fix), the performance of the parser would be the bottleneck.
Therefore, we propose this bulk implementation. In this implementation, the parsing and saving operations are separated. The parsing operation will split a tweet into different objects and have no interaction with the database. In this way, we can parse a large block of tweets and merge the parsed objects of the same kind together. For each kind of parsed objects, the saving operations would take the whole block and use one query to save them into the database. Please note that the saving operation should also take care of tables with foreign keys.
Working progress
- The parsing operation
- The saving operation
- Tests
from hoaxy-backend.
@shaochengcheng will change requirements; pandas, networkx and newspaper3k will stay '==' and others will change to '>='.
from hoaxy-backend.
@chathuriw will update server after this update is pushed to master.
from hoaxy-backend.
It seems everything is working fine; no errors. Closing. @shaochengcheng let us know if there are any remaining tasks related to this issue.
from hoaxy-backend.
Related Issues (20)
- GET Articles queries only return 100 results (Hoaxy API) HOT 5
- Make API server configurable HOT 1
- Docker build fail on jcc install HOT 1
- Add AltNews.in And BoomLive.in as fact checkers HOT 1
- One-time update of source list HOT 1
- Make it easier and document how to update source list HOT 4
- Docker compose not working HOT 1
- Facebook interface? (and CI) HOT 4
- Some of Hoaxy Web APIs need switch to HTTP POST HOT 1
- Comply with new Twitter API HOT 2
- Fix for Botometer-v4 HOT 3
- A bug in network construction HOT 2
- Hoaxy Community AMI HOT 5
- Error when running Hoaxy config HOT 2
- Error when running hoaxy init HOT 2
- New Error when running hoaxy init HOT 21
- Error building the lucene index HOT 5
- Dockerfile link update HOT 1
- error when building using the dockerfile HOT 2
- docker-compose up Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hoaxy-backend.