Comments (18)
Thanks for reporting this. Is it a large JSON file? Can you share it with me at [email protected]?
from hydrator.
I got it. It's interesting, of the 621,567 lines there appears to be one line (307,400) that has invalid JSON on it. It looks truncated in some way. Do you remember if you happened to have a storage problem or shutdown/stop the Hydrator about midway through hydrating this tweet id dataset? It shouldn't be a problem to start/restart but I should test it out to make sure.
Have you been able to hydrate successfully before?
from hydrator.
@opendatasurgeon could you share your ID file with me too if it's not too much trouble? This should hopefully help me figure out what might have happened.
from hydrator.
Ok, I will take a look. I rehydrated the tweet ids and also have tested generating CSV from hydrated data that I tried start/stopping, quitting and stopping the network during and wasn't able to get a corrupted jsonl file.
from hydrator.
Will do sir. I unfortunately did';t capture the screenshot if the % lost. I will however re-run this. And report it to you. Thank you :)
from hydrator.
Well that's a relief. If you notice it again and can figure out a way to reproduce please let us know. I will leave this ticket open until the app reports the JSON parse error better.
from hydrator.
Sure sir, thank you for looking into this. I will share it with you. I put a 20MB text file in the Hydrator, and in return I got 2.8GB JSONl file.
from hydrator.
Hello @edsu , just sent that json file which lead me to this error. Thank you & your team for making this tool!
from hydrator.
I got it. It's interesting, of the 621,567 lines there appears to be one line (307,400) that has invalid JSON on it. It looks truncated in some way. Do you remember if you happened to have a storage problem or shutdown/stop the Hydrator about midway through hydrating this tweet id dataset? It shouldn't be a problem to start/restart but I should test it out to make sure.
Have you been able to hydrate successfully before?
No sir, I got no warning or note while hydrating. It was after when I hit the CSV button, json output gave no error. Also my computer didn't sleep, nor I have a storage issue. I have ~280 GB left on the drive where I ran this program. I didn't stop the hydration, I just waited until I get an option to export. It was a continuous process on my end. Only thing missing hardware wise is a dedicated audio device. Whenever I need to hear something, I just put a headset.
from hydrator.
@opendatasurgeon could you share your ID file with me too if it's not too much trouble? This should hopefully help me figure out what might have happened.
Sure Mr. Summers. Will email the ID file which caused this error and also give you the id file and the json file where I was able to get an output without an error.
from hydrator.
Mr. Summers, sent you an email containing id files. Please let me know if there is anything else you need. Appreciate your help again!
from hydrator.
Thanks @opendatasurgeon. I'm confused about the relationship between the output-2020-03-22.jsonl
file you sent me (which has 621,567 lines) and the output-2020-03-22.txt
file you sent (which has 1,069.472 lines). I would expect the numbers to be much closer together unless a large number had been deleted? Can you see the % deleted value on the dataset detail view? It looks like it didn't finish hydrating, or a significant chunk of the file was lost? Can you please try hydrating it again and see if you have the same problem? I can also try hydrating here and see what happens.
from hydrator.
Oh yes, before I forget to tell you. The source of the data I am using to hydrate is here: https://github.com/echen102/COVID-19-TweetIDs
This text file (output-2020-03-22.txt) containing twitter IDs is from March (2020-03) directory. You will see name difference because I combined all the chunks pertaining to same days together in one text file. I believe they are separate in the original data source, because GitHub only allows 25mb uploads, that's why they split each giant texts files in parts. But I am re-running it now.
from hydrator.
Hello @edsu Just finished hydrating the text file. No error this time. Same computer, same process (meaning continuous hydration, no-computer sleeping, no disconnects or starting or stopping), I don't get it. The CSV and JSON files are surely smaller in size. Please see the screenshots below.
from hydrator.
@opendatasurgeon hydration worked for me too, 53% of the tweets have been deleted! That is a shockingly large percentage for such a recent set of tweets (just over a month old). I guess it's not surprising given that these appear to be COVID-19 related tweets, and there have been widespread disinformation campaigns about it.
Do you think it's possible you may have overwritten an output file accidentally when hydrating two files at the same time?
I'm going to leave this ticket open because Hydrator should report an error in the JSON rather than throwing an exception.
from hydrator.
I closed this by accident and want to keep it open until Hydrator reports errors better.
from hydrator.
Do you think it's possible you may have overwritten an output file accidentally when hydrating two files at the same time?
I don't think I did overwrite any of my file sir. I always name my files differently, and I believe I just merged the files, that were chunk of same days, together into one. I in fact left duplicates IDs in these files because I was going to clean CSV files anyways. So don't know what could have happened.
Do you think maybe there is limit a I should test with a May file? I know for sure first time around the deletion % wasn't above 30-40%. Let me report it you. Thanks again for helping out Mr Summers!!
from hydrator.
I am not getting this error these days even when hydrating >1500000 twitter ids text file. I will keep you in loop Mr. Summers. Thank you!
from hydrator.
Related Issues (20)
- Cannot link to my account HOT 3
- Not recognizing new consumer key and secret key HOT 1
- Contents of tweets are not complete HOT 7
- Reconstructing Threads HOT 7
- Losing lots of tweets when hydrating HOT 1
- javascript error HOT 1
- Number of hydrated tweets vastly smaller than rows in resulting CSV HOT 4
- When the Tweet ID file has windows line endings, the app throws an error
- Perpetually "Verifying Microsoft Excel.app" HOT 5
- Respect HTTP Proxy environment variables
- [Security] Workflow main.yml is using vulnerable action actions/checkout
- Cannot link to twitter account in 3.0 version. HOT 3
- Only 5 % tweets hydrated HOT 10
- problem with Hydrator HOT 1
- Hydrator slowing down HOT 1
- ids in jsonl file don't match original tweet ids HOT 1
- Can't find .env file HOT 1
- The link Twitter account is invalid HOT 5
- Hydrator Error HOT 6
- Invalid Tweet ID on Line 1 HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hydrator.