🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
Home Page: https://discord.gg/uFNUXvFFmg
License: GNU General Public License v3.0
This is fairly minor as far as features go but the Stop button seems to only stop the current job. I can understand that being a useful feature in some situations but generally if I'm trying to Stop a job I want all of them stopped, and it usually takes a few moments before the active job can properly stop (which adds up quickly if you've hit download on many profiles).
I imagine this basically being a Stop All button, acting like the current Stop button for the active job but then just removing all the following jobs from the queue before they even start, so it should only take about the same time as the current Stop routine (though its possible that code-wise this is more complicated than that)
getting "Twitter has invalid credentials" any ideas? have token and auth,.. need put in cookie ? thx
Sorry I never replied to your fix of issue #25, it did fix the original issue, I just hadn't had time to properly try it this past week.
There seems to be a couple smaller issues still with them though, from what I've seen:
Tagged Photos: doesn't add date/time to name if file is part of a "set" of photos (one instagram post containing multiple photos).
Stories: seems to sometimes leave duplicates behind after adding the date/time (i.e. a file named "filename123.png" exists, as well as a file named "220410_173000_filename123.png", if it was taken at 17:30:00 on april 10th 2022). Doesn't always happens but seems to happen at least a third of the time on the accounts I checked it on.
left the program running overnight to download several new instagram users, came back to find it had stopped at the first profile and hadn't yet downloaded anything due to a warning notification that the user has a large number of tagged photos "taking a long time" (I expected it to take a long time. that's was why I left it overnight). When I confirmed and let it continue, it stopped again at the next profile with the same warning, and the next, and the next...
Would very much like the option to disable this and any other notifications that prevent downloads until being acknowledged. alternatively perhaps have it automatically resume if not acknowledged for 2 minutes
Firstly, great little tool you’ve built here! Simple to use and it just… works!
One question though: it appears files (at least from reddit) have a random name assigned, with the option to add a date in settings.
Similar tools have used the name of the post as the filename. Is it possible to add this as an option?
Describe the bug
SCrawler does not download media at full-resolution (4K).
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I'd expect SCrawler in the original raw resolution (maximum attainable).
Screenshots
Release information (please complete the following information):
Additional context
I've only seen this issue with images, I have not tested if videos are downloaded in full.
It seems that if a profile has any pinned posts, all of the posts present at initial download will get downloaded as expected. However, posts added afterwards aren't downloaded when updating those users.
I'm new to the program and in trying it out and downloading some posts, I find that an Instagram post that I download with SCrawler will have dimensions of 1080 x 1350 whereas if I manually find the source of the image and download in browser that it has dimensions of 1440 x 1800. I'd like to know if there's anything I can do to make sure SCrawler gets the highest quality image possible
Is your feature request related to a problem? Please describe.
-
Describe the solution you'd like
Describe alternatives you've considered
-
Additional context
-
I admittedly haven't used this feature for a while and only noticed recently, so I'm not sure what version it stopped working, but I tried using it in version 2022.9.13.0_x64 and the following issues occurred:
I have a couple other suggestions relating to this feature that aren't bugs so much as possible improvements:
Hello, first I want to thank you for the nice program you developed. I'm trying to download my instagram saved posts, but when I click to start the download it says "Instagram Images: 0; Videos: 0" as shown below:
Steps to reproduce the behavior:
There were no logs in the log window.
Release information (please complete the following information):
About the .NET Framework version:
I checked in my Windows/Microsoft.NET folder and the highest version listed is v4.0.30319. However, when I downloaded the .NET runtime v4.6.1 (in this link: https://dotnet.microsoft.com/en-us/download/dotnet-framework/net461) and tried to install it, this message shows:
PS:
I triple checked all my credentials.
I also copied the ffmpeg.exe to the folder.
Downloading my own pictures from my profile worked fine.
Expected result:
Download all images and videos of user
Issue:
Grabs only 24 files
Tested on this user link:
https://www.reddit.com/user/Fantastic-Skill5156
Wondering if it may be easier and more helpful to add a file's original upload date/time as file metadata (I'm specifically thinking applying it as the "Date Taken" metadata, but could maybe be a selectable option in settings). Personally it would be more useful to me than adding the date/time to the filename, as I can easily sort by "Date Taken" instead.
Is your feature request related to a problem? Please describe.
When a twitter profile uploads a gif the program detects it as just a jpeg image and downloads it like that rather than a gif.
Describe the solution you'd like
would be nice if it can download it as a gif.
Describe alternatives you've considered
maybe convert it to a mp4 at least.
allow imgur / other third party link support to grab images/videos/gifs hosted from imgur also as it skips those, for twitter and reddit and most people use imgur to upload!
"Not accurate" is a loose term that comes to mind. Let me elaborate.
Before I updated just now, the Settings and Saved! folders were pointing to a folder on my network. Not over a network drive, and for reasons will not use a network drive.
(Typed file names are examples, not actual, refer to screenshot.)
Network listing would be \Truenas\sharename\downloads\RedditStuff\SCrawler
Saving a navigated to spot, and reopening Settings or similar: C:\file\path\to\Scrawler\Truenas\sharename\downloads\RedditStuff\SCrawler
Both the Settings and Reddit.
The only reason I split the two, I have SCrawler saved in my dropbox location, while where it saves content to is a network drive.
Moving SCrawler to my network share, and running it from, results in basically same file pathing. \Truenas\sharename\downloads\RedditStuff\SCrawler\Truenas\sharename\downloads\RedditStuff\SCrawler.
As far as I can, without actually testing it, SCrawler must run and save on the local machine. If I had to guess, the main harddrive at worst?
This is after copying SCrawler to the same network share as the saves. Then changing settings to point to the new location, closing, then opening settings. Restarting SCrawler does not resolve it (Closing by the icon by the clock).
Before copying to network drive, running on second local harddrive.
Back on 2.0.1:
Here's logs
-----InnerException 1-----
-----InnerException 2-----
Message: The remote server returned an error: (400) Bad Request.
HelpLink:
StackTrace: at System.Net.HttpWebRequest.GetResponse()
at PersonalUtilities.Tools.WEB.Response.GetResponse(String URL, String PayLoad, ErrorsDescriber e)
2021-12-22 15:37:06: Twitter - *: data downloading error [https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=*&count=200&exclude_replies=false&include_rts=1&tweet_mode=extended]
[Response.GetResponse]:
400 (BadRequest)
URL: [https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=*&count=200&exclude_replies=false&include_rts=1&tweet_mode=extended]
The remote server returned an error: (400) Bad Request.
2021-12-22 15:37:06: Message: [Response.GetResponse]:
400 (BadRequest)
URL: [https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=*&count=200&exclude_replies=false&include_rts=1&tweet_mode=extended]
The remote server returned an error: (400) Bad Request.
HelpLink:
StackTrace: at PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage _Message, Object OptionalValueReturn)
at PersonalUtilities.Tools.WEB.Response.ReturnError(Exception& ex, ErrorsDescriber e, String URL)
at PersonalUtilities.Tools.WEB.Response.GetResponse(String URL, String PayLoad, ErrorsDescriber e)
at SCrawler.API.Twitter.UserData.DownloadData(String POST, CancellationToken Token)
-----InnerException 1-----
-----InnerException 2-----
Message: The remote server returned an error: (400) Bad Request.
HelpLink:
StackTrace: at System.Net.HttpWebRequest.GetResponse()
at PersonalUtilities.Tools.WEB.Response.GetResponse(String URL, String PayLoad, ErrorsDescriber e)
Cloning the project in visual studio shows that its missing a file called InfoMod.vb and its also making many references to a different project namespace called PersonalUtilities.
Due to the missing files you cannot build this, run it locally or create changes.
Removing the user name/account from the crawler deletes the folder with all the pics and videos
It'd be better if the folder remains in the SCrawler_2.0.0.4_x64\Data\ subfolder
As the tile says, does the program track and download any new content published on the profile added to it?
Can find any settings on how to do that...
The program only seems to grab the 5 most recent story "bubbles" (not sure what to call them, but the icons visible above posts) for a given user. All stories within each "bubble" are grabbed without issue and stored in a named folder without issue, but if a user has more than 5 "bubbles" anything after the 5th one is ignored.
Added a quick screengrab of what I mean in case my phrasing is too confusing. green-checkmarked "bubbles" are saved properly by the program, but every bubble after that is ignored.
On August 31, RedGifs changed the API requirements. These changes cause an error on downloading. Unfortunately, at this time, I still cannot resolve this issue.
Starting from version 2022.9.8.0, downloading data from the RedGifs site is temporary disabled. You can still add RedGifs profiles, but they won't download. You will be able to download them when this issue is fixed.
If you know how to solve this issue, you can post a suggestion in a comment on this post.
Having the same issue in latest version 2022.7.7.0 date not adding to file name
Settings.zip
Hey All,
I read the guide multiple times but do not get how this actually works. None of the features is settling in my congested mind. Please help.
I would like to know how to use this app. This thread might be long and sorry for that. I will be posting the question as we go, I am a system admin and should not be much more difficult to explain to me.
Twitter: (I have added both tokens and cookies and see no entry in the log)
General:
4. What is the actual meaning of "User"? Does that mean actual individual usernames we would like to get content from? Example Reddit-reddit.com/u/xyz?
5. What is Channel? Why there is a Reddit icon for that? Does this mean subreddit?
Once I am able to get point 1 and then I will explore further.
Version:SCrawler_3.0.0.10_x64
Edit: I have removed some questions as reverting back to 3.0.0.9 solved it for now.
I wish there is a video explaining how it works.
To Reproduce
Steps to reproduce the behavior:
Release information (please complete the following information):
Checked in both 3.0.0.8 and 3.0.0.9. Last version I know it definitely still worked properly in was 3.0.0.6, I skipped 3.0.0.7 so unsure if it was an issue then.
Steps taken:
Fresh install, changed settings as below:
added an instagram user (haven't checked Twitter etc. yet) and downloaded.
confirmed that downloaded images don't have date/time added in filename
A few days ago, Instagram started requiring a new authorization token. The name of this token is x-csrftoken. I'll update the release soon, but for now you can add the token to the Instagram settings file - Responser_Instagram.xml. You can find this file in the SCrawler settings path.
You can find this token in the same way as other authorization headers.
This file looks like this:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Response>
<UserAgent>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36</UserAgent>
<Method>GET</Method>
<Encoding Use="1">utf-8</Encoding>
<EncodingPayload Use="1">utf-8</EncodingPayload>
<UseWebClient>0</UseWebClient>
<UseWebClientAdditionalHeaders>0</UseWebClientAdditionalHeaders>
<UseWebClientCookies>0</UseWebClientCookies>
<Host Use="0"></Host>
<KeepAlive Use="0">0</KeepAlive>
<Referer Use="0"></Referer>
<Timeout Use="0">0</Timeout>
<UseDefaultCredentials Use="0">0</UseDefaultCredentials>
<Pipelined Use="0">0</Pipelined>
<PreAuthenticate Use="0">0</PreAuthenticate>
<ContentType Use="0"></ContentType>
<ContentLength Use="0">0</ContentLength>
<Accept Use="0"></Accept>
<LatestURL></LatestURL>
<LatestPayLoad></LatestPayLoad>
<CookieDomain>instagram.com</CookieDomain>
<CookieHeader>set-cookie</CookieHeader>
<Decoders></Decoders>
<Header Name="x-ig-app-id"></Header>
<Header Name="x-ig-www-claim"></Header>
</Response>Add a new line at the end:
<Header Name="x-csrftoken">PASTE YOUR TOKEN HERE</Header>Your file should now look like this:
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Response>
<UserAgent>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36</UserAgent>
<Method>GET</Method>
<Encoding Use="1">utf-8</Encoding>
<EncodingPayload Use="1">utf-8</EncodingPayload>
<UseWebClient>0</UseWebClient>
<UseWebClientAdditionalHeaders>0</UseWebClientAdditionalHeaders>
<UseWebClientCookies>0</UseWebClientCookies>
<Host Use="0"></Host>
<KeepAlive Use="0">0</KeepAlive>
<Referer Use="0"></Referer>
<Timeout Use="0">0</Timeout>
<UseDefaultCredentials Use="0">0</UseDefaultCredentials>
<Pipelined Use="0">0</Pipelined>
<PreAuthenticate Use="0">0</PreAuthenticate>
<ContentType Use="0"></ContentType>
<ContentLength Use="0">0</ContentLength>
<Accept Use="0"></Accept>
<LatestURL></LatestURL>
<LatestPayLoad></LatestPayLoad>
<CookieDomain>instagram.com</CookieDomain>
<CookieHeader>set-cookie</CookieHeader>
<Decoders></Decoders>
<Header Name="x-ig-app-id"></Header>
<Header Name="x-ig-www-claim"></Header>
<Header Name="x-csrftoken">PASTE YOUR TOKEN HERE</Header>
</Response>Copy & Pasting, a Link that automatically gets detected and added instead of having to Click on "Add"
I would love if there was an option in the program to fully reset a user's data to allow it to be re-downloaded
Possibly should include the option to compare what files the data XML file and posts TXT file thinks the program has against the list of files already present in the data folders, and then attempt to redownload any files that don't exist in the data folders.
I've had to reset users several times now, either due to files being downloaded without date/time, or on a couple occasions I've had issues where during a longer download (a user with several thousand posts, for example) it's been running for enough hours that either I forget and shut down my computer mid-download, or on one occasion one of my authorizations expired mid-download and it seemed to think it had downloaded everything when I reset my authorizations but many files hadn't yet been downloaded.
While you do have instructions in the FAQ on how to manually reset a user so that their content can be re-downloaded, it seems to not always work cleanly, and given the changes with each new version I've been finding it easier to start fresh each time rather than try and get the program to recognise that there are photos I want it to grab
Per your FAQ on how to "re-download all data"
Double-click on the user you want to redownload. In the opened window open folder setting. Delete the files ending with _Data.xml and _Posts.txt. Download this user again.
I have found that something in the user XML file (not the data one) also seems to need to be deleted or reset before the program will redownload everything. I'm not entirely sure what it is, possibly the "last downloaded" date prevents it for looking for earlier posts, but I've taken to completely deleting the entire user folder and then re-building it using the program, which isn't ideal.
It would be preferable to just have a button that reset the data without resetting user info/preferences.
Doesn't download all reddit user files
Example URL (NSFW):
https://old.reddit.com/user/exhale_jay
Version:
Latest version 3.0.0.9
how can i access via token on twitter?
Is your feature request related to a problem? Please describe.
Add a feature to reddit to be able to downland from imgur.com posts
Describe the solution you'd like
downland from imgur.com posts
Additional context
add a checkbox to create user panel for reddit for downloading imgur.com posts
Not downloading anything even though I pasted in the Token and Authorization.
LOG
2022-04-01 01:26:20: [PluginHost.New]
Could not load file or assembly 'SCrawler.Plugin.LPSG, Version=1.0.0.1, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
2022-04-01 01:26:20: Message: Could not load file or assembly 'SCrawler.Plugin.LPSG, Version=1.0.0.1, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
HelpLink:
StackTrace: at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at SCrawler.Plugin.Hosts.PluginHost..ctor(SFile AssemblyFile, XmlFile& _XML, SFile GlobalPath, XMLValue`1& _Temp, XMLValue`1& _Imgs, XMLValue`1& _Vids)
-----InnerException 1-----
-----InnerException 2-----
Message: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
HelpLink:
StackTrace:
2022-04-01 01:26:20: [PluginHost.New]
Could not load file or assembly 'SCrawler.Plugin.XVIDEOS, Version=1.0.0.2, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
2022-04-01 01:26:20: Message: Could not load file or assembly 'SCrawler.Plugin.XVIDEOS, Version=1.0.0.2, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
HelpLink:
StackTrace: at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at SCrawler.Plugin.Hosts.PluginHost..ctor(SFile AssemblyFile, XmlFile& _XML, SFile GlobalPath, XMLValue`1& _Temp, XMLValue`1& _Imgs, XMLValue`1& _Vids)
-----InnerException 1-----
-----InnerException 2-----
Message: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
HelpLink:
StackTrace:
2022-04-01 01:32:18: AConvert: Converting Error [19.03.2022 20:27:37] - [String] --> [DateTime]
[AConvertPartial] Error converting from [String] to [DateTime]
2022-04-01 01:32:18: Message: [AConvertPartial] Error converting from [String] to [DateTime]
HelpLink: AConvertPartial
StackTrace: at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
at PersonalUtilities.Functions.ArgConverter.AConvert(Object Value, AModes AMode, Type DestinationType, Boolean IsChecking, Boolean ReturnNullArg, Object NullArg, IFormatProvider Provider, ErrorsDescriber e)
-----InnerException 1-----
-----InnerException 2-----
Message: Format Provider [ANumbers] does not supported
HelpLink: 0
StackTrace: at PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage _Message, Object OptionalValueReturn)
at PersonalUtilities.Functions.ArgConverter.ADateTime.Parse(String Value, Type DestinationType, String Pattern, IFormatProvider Provider, Nullable`1 Style, TimeModes TimeMode, Object NothingArg, ErrorsDescriber e)
at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
2022-04-01 01:35:32: AConvert: Converting Error [19.03.2022 20:27:37] - [String] --> [DateTime]
[AConvertPartial] Error converting from [String] to [DateTime]
2022-04-01 01:35:32: Message: [AConvertPartial] Error converting from [String] to [DateTime]
HelpLink: AConvertPartial
StackTrace: at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
at PersonalUtilities.Functions.ArgConverter.AConvert(Object Value, AModes AMode, Type DestinationType, Boolean IsChecking, Boolean ReturnNullArg, Object NullArg, IFormatProvider Provider, ErrorsDescriber e)
-----InnerException 1-----
-----InnerException 2-----
Message: Format Provider [ANumbers] does not supported
HelpLink: 0
StackTrace: at PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage _Message, Object OptionalValueReturn)
at PersonalUtilities.Functions.ArgConverter.ADateTime.Parse(String Value, Type DestinationType, String Pattern, IFormatProvider Provider, Nullable`1 Style, TimeModes TimeMode, Object NothingArg, ErrorsDescriber e)
at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
Since this program runs in the tray, can we get a setting to automatically grab new content every X minutes?
Went to change the default settings for instagram. The value defaults to 6000 but starts out immediately highlighted red and no matter what I enter that doesn't seem to change. When I press OK I get this error.
Default settings seem to work, but this error prevents me from changing default settings.
This is for version 3.0.0.2, unsure if this issue existed in the prior 3.0.0.x versions as I only just now went to update, but I don't remember this being an issue in 2.0.0.4 which was the last version I used.
There is some lossy frames when downloading videos from twitter as it strictly converts videos to 30 fps.
Requesting compatibility with ifunny.co.
I don't know to much about web APIs. but here's what I found:
https://www.npmjs.com/package/ifunny-web-api
I'd like to see mine and my friends old account content from back in 2013, but their site is so janky, you can't hardly scroll down a week back without issue.
I totally understand if you don't want to touch this site! I just thought I'd try to reach out.
I have a suspicion that this is due to instagram changing something on their end again.
I got an error with my instagram credentials, but when I went to update them, the Hash field wouldn't fill itself out when the button (circled below) was clicked (in the past I've done this many times across many versions without any issue)
From latest version (fresh install) to clarify exactly what I mean.
I tried using cookies from a different account, and I tried on a fresh install and some past versions, and all had the same issue.
I'm not familiar enough with the ins-and-outs of instagram credentials to know how to manually locate my Hash unfortunately, so I haven't been able to try that
Describe the bug
Instagram credentials keeps expired. When I go back to check the cookies, there is a rur cookie that produce error so I think this might be the case
To Reproduce
Steps to reproduce the behavior:
rur, shbid, shbts, sessionid, csrftoken, ig_nrcb, ds_user_id, ig_did, mid.Hash2, x-ig-app-id, ix-ig-www-claim and Saved posts user. I only reload to get the Hash.https://www.instagram.com/SomeUserNameDownload selected and I see nothing being downloaded. Checking log show me that Instagram credentials have expiredrur cookie (which has the value of "EAG\0543019457599\0541683004771:01f7a6bc68dbceae15c578c56a7373aae35b5d8976bc686833b96e4cf94606cdb3647372") I see this error:Message: Value of '1/1/0001 12:00:00 AM' is not valid for 'Value'. 'Value' should be between 'MinDate' and 'MaxDate'.
Parameter name: Value
HelpLink:
StackTrace: at System.Windows.Forms.DateTimePicker.set_Value(DateTime value)
at PersonalUtilities.Forms.Controls.TextBoxExtended.set_Value(Object NewValue)
at PersonalUtilities.Tools.WEB.CookieEditorForm.CookieEditorForm_Load(Object sender, EventArgs e)
Checking the log produce this:
2022-05-02 12:48:11: AConvert: Converting Error [19.03.2022 20:27:37] - [String] --> [DateTime]
[AConvertPartial] Error converting from [String] to [DateTime]
2022-05-02 12:48:11: Message: [AConvertPartial] Error converting from [String] to [DateTime]
HelpLink: AConvertPartial
StackTrace: at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
at PersonalUtilities.Functions.ArgConverter.AConvert(Object Value, AModes AMode, Type DestinationType, Boolean IsChecking, Boolean ReturnNullArg, Object NullArg, IFormatProvider Provider, ErrorsDescriber e)
-----InnerException 1-----
-----InnerException 2-----
Message: Format Provider [ANumbers] does not supported
HelpLink: 0
StackTrace: at PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage _Message, Object OptionalValueReturn)
at PersonalUtilities.Functions.ArgConverter.ADateTime.Parse(String Value, Type DestinationType, String Pattern, IFormatProvider Provider, Nullable`1 Style, TimeModes TimeMode, Object NothingArg, ErrorsDescriber e)
at PersonalUtilities.Functions.ArgConverter.AConvertPartial(Object Value, Type DestinationType, IFormatProvider Provider)
2022-05-02 12:48:11: Form loading error
Value of '1/1/0001 12:00:00 AM' is not valid for 'Value'. 'Value' should be between 'MinDate' and 'MaxDate'.
Parameter name: Value
2022-05-02 12:48:11: Message: Value of '1/1/0001 12:00:00 AM' is not valid for 'Value'. 'Value' should be between 'MinDate' and 'MaxDate'.
Parameter name: Value
HelpLink:
StackTrace: at System.Windows.Forms.DateTimePicker.set_Value(DateTime value)
at PersonalUtilities.Forms.Controls.TextBoxExtended.set_Value(Object NewValue)
at PersonalUtilities.Tools.WEB.CookieEditorForm.CookieEditorForm_Load(Object sender, EventArgs e)
Expected behavior
Getting cookie to work so I run the download.
Release information (please complete the following information):
If I did anything wrong I apologize for wasting anyone time. I have checked the issue I find nothing. Thank you for any help
I used version 2.0.0.4 now I downloaded and started version 3.0.0.6 and nothing works.
How to fix that. I read that I have to archive the user.xml, but the bat file isn't working.
Do I have to delete all data and start from the beginning?
If a user is added to a collection by mistake, there seems to be no option to remove the user from the collection.
Additionally, when selecting a collection and using the "Delete user / collection" option, selecting the user from the list of users within the collection does not delete the user account (in fact, nothing appears to happen). The similar menu for editing a user within a collection seems to work fine.
I am using 2.0.0.4, 64x architecture.
Hi! I love this program so far, but I've noticed that on instagram it only pulls from a users posts , when there's a lot more content that could be grabbed from there. I don't know how big of an addition this might be but I'd love if this worked with more than just the posts.
Specifically I would love to see the following added, perhaps in separate sub-folders within the users folder (like how videos are, by default):
I've attached an image of what areas on a profile I mean in case my terminology is wrong or unclear. Cheers!
Is your feature request related to a problem? Please describe.
I feel like a lot of users of SCrawler use it to make account backups. However, SCrawler only saves the media and date acquired; this is fine as I can go back to their Twitter account and look for more information. However, when accounts are deleted/removed, I often want to know the context behind the images (i.e. title, uploader, date acquired etc.).
Describe the solution you'd like
It would be cool if a feature was added in the SCrawler "create user" window that allowed an option to embed metadata into each individual image/video file. This could include Title (caption), Authors (username), Program Name (SCrawler), Date acquired (post date & time).
Additional context
I know this is a continuation of issue #35 and issue #36. But I thought I'd make a feature request if you're interested in implementing it in the future.
I have it set to add date and time to the beginning of filenames. This is working for other things as normal (at least for twitter and instagram posts, both photos and videos), but this info does not get added for the newly-added instagram tagged photos or stories.
I see you have no LICENSE file for this project. The default is copyright.
I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.
I decided to start fresh with 3.0.0.5 and set everything up again from scratch cause I did something (not sure what but definitely just user error) that broke my previous version. While setting it up and going through the process for setting up cookies and authorization tokens I ran into a handful of minor issues that I was able to work around but figured I would mention here anyhow to make it easier for new users in future.
Issue 1: Twitter cookie "invalid"
I went to add my Twitter Cookie from Chrome as described in the wiki and received an error stating the following (I replaced the original string of numbers with 0 for privacy):
The 'Value'='{"i_p":0000000000000,"i_l":1}' part of the cookie is invalid.
This value was the "g_state" value for the cookie. I simply removed that line and it worked fine, but perhaps that is something the program could do automatically in future.
Issue 2: on the 'Settings' wiki page for "How to find Twitter tokens" in step 6 it mentions "a query that starts with 'TwitterDetail?'...". There is no such query, and I believe it is supposed to be 'TweetDetail?' instead (which seems to work for the remaining steps).
Issue 3: It may be helpful to clarify in various places on the wiki and the program that x-ig-app-id and x-ig-www-claim are needed for instagram tagged photos and stories (and only needed for them). It took me a little while to realise I was missing them and that was why I couldn't get tagged photos or stories to work.
Issue 4: I wasn't able to find Insta hash 2 by following the guide on the wiki. No text with "query_hash=" appeared when I followed the steps. I personally don't use the saved posts functionality (I was only looking for it because I thought perhaps it was needed along with x-ig-app-id and x-ig-www-claim in order to get tagged/stories) so I didn't put much additional effort in once the instructions failed but wanted to mention it here in case it's one of those things that instagram quietly changed and you need to adjust for.
The setting to add the date/time to the beginning of the saved file doesn't seem to work on versions past 3.0.0.6.
twiter doesn't download any images no matter of who is added as the user.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.