drawrowfly / tiktok-scraper Goto Github PK
View Code? Open in Web Editor NEWTikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
hi, i have a idea/solution for downloading video without watermark
i sniffing in app tiktok in android and i found new api.
but i dont know how to get video id
example :
https://www.tiktok.com/@vikaberk/video/6721248223035411718 >> video id right
but how to convert to
v09044310000bl3ammlqa7lrlgrvj9tg
this is old api to get no wm https://api2.musical.ly/aweme/v1/play/?video_id=v09044310000bl3ammlqa7lrlgrvj9tg&line=0&ratio=720p&watermark=0&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0&logo_name=tiktok
but right now, not work again with videoid 6721248223035411718
I saw that online tiktok downloaders allows to download videos without a watermark. Is it possible with this lib?
Is it possible to use authorName for the downloaded file?
The scraper doesn't seem to work when the keyword (e.g. the hashtag I search for) is in Hebrew script. The word does show up in Hebrew alphabet in Terminal and I have Unicode (UTF-8) text encoding enabled, so I don't think the issue is with Terminal. Can you advise? Thanks!
Code app.js:
const TikTokScraper = require('tiktok-scraper');
let posts = TikTokScraper.hashtag("singing", { event: true, number: 200});
posts.on('data', (json) => {
console.log(json)
})
posts.on('done', () => {
console.log("done")
})
posts.on('error', (error) => {
console.log(error)
})
posts.scrape();
Version:
"dependencies": {
"tiktok-scraper": "^1.0.9"
}
Error:
TypeError: Cannot read property 'itemListData' of undefined
at TikTokScraper.mainLoop (/node_modules/tiktok-scraper/lib/index.js:394:53)
at async TikTokScraper.scrape (/node_modules/tiktok-scraper/lib/index.js:297:9)
As of 9:50am CET, I can't download video's anymore. What worked the day before, doesn't work now, I'm affraid.
command use: tiktop-scraper user thisisanexampleuser -n 9999 -d -store
commands used to download by music association does not work, either.
Firstly, thanks for all your awesome work on this!
I'm using the latest version (0.0.16), so I do not think it's a duplicate of this issue #44
CLI works just fine. e.g.
tiktok-scraper user USERNAME -n 20 -d -store
Using the node method returns 0 results (was previously working). Any ideas?
const posts = await TikTokScraper.user(USER_ID, {
by_user_id: true,
user_data: true,
number: 10,
});
Command does not do anything. Am I correctly inserting the music id?
URL: https://www.tiktok.com/music/The-Box-6768646921029470981?lang=en
Tried
tiktok-scraper music The-Box-6768646921029470981 -n 50 -d
tiktok-scraper music 6768646921029470981 -n 50 -d
tiktok-scraper music The-Box -n 50 -d
none is downloading
I see the following message when I try to run the scraper and it exits without downloading any data.
⠋ TikTok Scraper Started(node:19861) ExperimentalWarning: The fs.promises API is experimental
Have you bumped into this?
_Even when setting -n 0, i can only scrape 20-30 videos depending on each users. Redownloading the same user would get the same number of videos.
_I did scraped different users during 24h window, here's just an example:
Username: chiyuu
Total videos: 446.
Downloaded: 28.
In older version i've got 415.
Current version 1.0.12
Is it possible to get a user’s following list?
I’ve seen this on github:
https://github.com/botsphp/tiktok_crawler
And didn’t understand how to use it correctly
Hi i tested today,
i cant scraping profile, result collector length is 0
i run this project in RDP AWS
but i getting result
collector[]
and i test _getUserId() but getting result {"statusCode":200,"contentType":"text/html","content":""}
how to fix that?
Hello,
my setup :
Ubuntu 18.04
NodeJS : I tried version 12 and 13
tiktok scraper : 1.0.12
I keep getting an error message when I'm trying to use a proxy-server . Tried different proxy-servers.
Only appears when I'm trying to use a proxy.
tiktok-scraper trend -d -n 5 -p https://115.23.28.168:8080
Error: Error: [object Object]
at TikTokScraper.extractTac (/usr/local/lib/node_modules/tiktok-scraper/lib/index.js:311:19)
When using the module example, the collector returns an empty array.
Module not working:
(async () => {
try {
const posts = await TikTokScraper.user({ USER_ID: SOME_USER_ID_STRING }, { number: 20, by_user_id: true });
console.log(posts); // { collector: [] }
} catch (error) {
console.log(error);
}
})();
CLI works:
tiktok-scraper user lebeats_211
Any idea what could be causing the difference?
I`m just curious should we be worried from this warning we see every time we use the scrapper:
(node:61341) ExperimentalWarning: The fs.promises API is experimental
this is the error:
the command:
tiktok-scraper hashtag tiktokchefs -d -n 30 -store
C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\multipleBar.js:42
this.stream.clearLine();
^
TypeError: this.stream.clearLine is not a function
at Multibar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\help ers\multipleBar.js:42:21)
at ProgressBar.bar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l ib\helpers\multipleBar.js:33:22)
at ProgressBar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\node_modul es\progress\lib\node-progress.js:103:10)
at Multibar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\m ultipleBar.js:56:17)
at ProgressBar.bar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\he lpers\multipleBar.js:28:18)
at Request.r.get.on.on.chunk (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l ib\index.js:117:34)
at Request.emit (events.js:198:13)
at IncomingMessage.<anonymous> (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper \node_modules\request\request.js:1073:12)
at IncomingMessage.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at IncomingMessage.Readable.push (_stream_readable.js:224:10)
at HTTPParser.parserOnBody (_http_common.js:122:22)
at TLSSocket.socketOnData (_http_client.js:442:20)
at TLSSocket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at TLSSocket.Readable.push (_stream_readable.js:224:10)
at TLSWrap.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
It downloads a file(a corrupt zip file) with one video in it, but just only that.
Thank you for providing such a useful tool, but when I used it, I found that the videos are watermarked. There are two suggestions to see if it can be achieved:
Scraping started returning an empty collector. I'm guessing the signature is out of data again.
(Btw, how are you reverse engineering the signature? Happy to help maintain if you point me in the right direction.)
I'm getting following error:
Error: Data should not be empty or the "fields" option should be included
at JSON2CSVParser.preprocessData (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1484:15)
at JSON2CSVParser.parse (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1450:32)
at Bluebird.fromCallback.cb (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:102)
at tryCatcher (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/util.js:16:23)
at Function.Promise.fromNode.Promise.fromCallback (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/promise.js:209:30)
at Promise (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:44)
at processTicksAndRejections (internal/process/next_tick.js:81:5)
is it possible to download videos without a watermark?
Tried a few http proxy and it will always fail. (I suspect it might be related to https over http).
just tried today and it doesn't work - doesn't even create cvs file for trending.
It works perfectly for other API - get hashtag info etc.
Hello drawrowfly,
I want to integrate your scraper into php website, can you give me your email? Are you available for a custom paid project? Thank you!
The result does not include the likes count of a post.
In terminal:
https://www.tiktok.com/@maevaa.ghennam/video/6805591629471173894
Bad url format. Correct format: https://www.tiktok.com/@USERNAME/video/ID
In code regex:
if (!/^https://www.tiktok.com/@(\w+)/video/(\d+)$/.test(this.input)) {
throw Bad url format. Correct format: https://www.tiktok.com/@USERNAME/video/ID
;
}
Part w+ equals [a-zA-Z0-9]+
It seems that the scraper doesn't work anymore, I tried to debug it and I received status code 200 in the body response of video requests (not 0), maybe the signature or something else has changed in tiktok system? hope it help
Hi there,
Can you please add number of followers in your api reponse.
When runnning with nodejs v10.18.1 and installed with Yarn or npm, I receive this error when trying $tiktok-scraper --help
internal/modules/cjs/loader.js:638
throw err;
^
Error: Cannot find module 'Bluebird'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:13:18)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/instance.js:3:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/bin/cli.js:4:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)
Please add support to scrape multiple users from a text file
Thank you for such amazing library.
is there any way to get user profile info ? Like following/followers count ?
Script
const TikTokScraper = require('tiktok-scraper');
(async () => {
try {
const posts = await TikTokScraper.hashtag('singing', { number: 100 });
console.log(posts);
} catch (error) {
console.log(error);
}
})();
Version:
"tiktok-scraper": "^1.0.13"
Error:
...\node_modules\jsdom\lib\jsdom\living\helpers\dates-and-times.js:235
} catch {
^
SyntaxError: Unexpected token {
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:607:28)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\form-controls.js:18:5)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\focusing.js:5:24)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
Hi @drawrowfly
Currently, your node module supports for searching by hashtag, user, trend, music. Do you know a way to search by keyword? Are you planing to add "search by keyword" to this module?
Best,
-Young
When viewing TikTok videos via the mobile app, sometimes there is a
This sound isn't available.
message above the video. I noticed that these videos do not get scraped. Is it possible to include them?
IDK if it helps in figuring out the issue, but I've been able to share them to Instagram via the mobile app and download/hear the sound that way.
Love the scraper, thanks!
Is it possible to use authorName for the downloaded file?
I keep getting this error when running the package and it stopped working for some reason. Reading from here that "node-uuid" dependency should be swapped out for "uuid"
jotform/jotform-api-nodejs#20
Also, is the trending function call supposed to be random? I'm just downloading 3 videos and download the same videos 2 weeks ago and today.
This isn't really an issue, I'm just not sure where to put questions.
Is it possible to easily delete stored download history per user?
For instance, I want to re-download a user's videos and store the download history.
How do I delete their old download history?
I'm using ubuntu. I can navigate to the tmp folder but I'm not seeing how to identify which history file belongs to which user.
Thanks
can you update the signature? seems like not working again.
Hey there, when downloading posts, how does the program handle new posts for an already downloaded user? Ideally, I'd be able to just run the script again for a user and it would download said user's new posts and their metadata without having to redownload everything. Is this a possibility?
Apologies for opening an issue for this, I couldn't see a way to contact you, drawrowfly.
Any chance to get user posts by user id ?
Trending hashtags are a good way to seed the scraper. These hashtags appear to be different than the hashtags present in the trending posts. The app itself seems to have an infinite scroll of these trending hashtags, but the web interface only loads a fixed number.
https://www.tiktok.com/discover
A feature request, something to investigate when you have time. :)
Code
const TikTokScraper = require('tiktok-scraper');
let posts = TikTokScraper.hashtag("singing", { event: true, number: 10});
posts.on('data', (json) => {
console.log(json)
})
posts.on('done', () => {
console.log("done")
})
posts.on('error', (error) => {
console.log(error)
})
posts.scrape();
Version:
"tiktok-scraper": "^1.0.14
Error:
Error: RequestError: Error: ESOCKETTIMEDOUT
at TikTokScraper.scrapeData (*/node_modules/tiktok-scraper/lib/index.js:616:19)
done
This morning I tried to scrape a profile and the program immediately exited with no other message after displaying the "TikTok Scraper Started" message.
I updated the program and repeatedly receive the same message. Could it be that my IP is banned/limited?
Thanks
Hello, Drawrowfly,
Is there a way to get post info by the post id?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.