reluce / szurubooru-toolkit Goto Github PK
View Code? Open in Web Editor NEWPython package and script collection to manage your szurubooru image board.
License: GNU General Public License v3.0
Python package and script collection to manage your szurubooru image board.
License: GNU General Public License v3.0
Hi!
Big fan of the toolkit since there's not a lot out there for szurubooru importing, management etc.
One thing I would like to see, however, is URL support. Although it is nice to be able to scrape results from a query, I would like to be able to insert one or multiple (or hundreds via a url list .txt or something) links to a booru post and scrape that way.
Essentially, being able to do import-from-booru gelbooru gelbooru-urls.txt
would be a dream come true, but even import-from-booru gelbooru https://gelbooru.com/index.php?page=post&s=view&id=1234567
would be great, and the end user could make a shellscript or similar to handle mass imports from there.
Would it be possible to also add implications when using create-tags?
convert_image in upload_media.py will attempt to convert (shrink) files that it has no business modifying.
EDIT:
Here is the error:
$ poetry run upload-media
[INFO] [23.05.2022, 17:44:58 EDT]: Found 43 file(s). Starting upload...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 292, in main
upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 230, in upload_post
post.image = convert_image(file, file_ext, file_to_upload)
File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 223, in convert_image
return image
UnboundLocalError: local variable 'image' referenced before assignment
Flake8 reports a complexity of 32.
While the upload-media
script currently does its designed job, it would be nice to have the option to directly tag the post after the upload.
For that, the auto_tagger
script's main function could accept the uploaded/returned post id and use it to tag the post.
The script should take a search query for various Boorus as an input and downloads + tags all matched posts for the query.
Something like: import-from --booru [danbooru,gelbooru,konachan,yandere] query
The function get_posts_from_booru
in utils.py skips a page of posts. Starting the range at 0 in the for loop seems to fix this.
Edit: Although this fixed this issue I was having, it tried to import posts twice in some cases.
Hello,
I'm receiving the following error when attempting to run any variation of the auto_tagger python script:
Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py 'id:126'
Could not process your query: 'total'.
Traceback (most recent call last):
File "auto_tagger.py", line 129, in <module>
main()
File "auto_tagger.py", line 28, in main
post_ids, total = api.get_post_ids(user_input.query)
TypeError: cannot unpack non-iterable NoneType object
{
"szurubooru": {
"url": "https://booru.mydomain.com/",
"api_token": "my api token",
"public": "False"
},
"auto_tagger": {
"saucenao_api_key": "my api token",
"hide_progress": "False",
"local_tmp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
"use_saucenao": "True",
"deepbooru_enabled": "False",
"deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
"deepbooru_threshold": "0.7",
"boorus": {
"danbooru": {
"user": "None",
"api_key": "None"
},
"gelbooru": {
"user": "None",
"api_key": "None"
},
"konachan": {
"user": "None",
"password": "None"
},
"yandere": {
"user": "None",
"password": "None"
},
"pixiv": {
"user": "None",
"password": "None",
"token": "None"
}
}
},
"upload_images": {
"upload_dir": "I:/Images/Booru/posts",
"hide_progress": "False",
"cleanup": "False",
"tags": ["tagme"]
}
}
Implement new script delete-posts
with argument --mode=query|range
.
The query mode deletes all posts returned from the input query while the range mode accepts a range like "100..200", which deletes the posts starting from id 100, ending with id 200.
Hello,
It looks like create-tags feature does not work anymore:
root@szurubooru-toolkit-7b59658595-g9s96:/szurubooru-toolkit# /usr/local/bin/create-tags --overwrite --query genshin*
[INFO] [25.04.2023, 12:53:54 UTC]: Fetching tags from URL https://danbooru.donmai.us/tags.json?search[post_count]=>10&search[name_matches]=genshin*&limit=100&page=1...
[CRITICAL] [25.04.2023, 12:53:54 UTC] [danbooru.download_tags]: Could not fetch tags: Expecting value: line 1 column 1 (char 0)
[SUCCESS] [25.04.2023, 12:53:54 UTC]: Script finished creating tags!
Thanks!
I'm getting an invalid token error when trying to run upload_images.py
This is my full terminal output:
Found 1 images. Starting upload...
0%| | 0/1 [00:00<?, ?it/s]
An error occured while getting the image token: Invalid token.
An error occured during the similarity check: Invalid token.
Traceback (most recent call last):
File "B:\Downloads\szurubooru-scripts-main_fork\upload_images.py", line 204, in <module>
main()
File "B:\Downloads\szurubooru-scripts-main_fork\upload_images.py", line 184, in main
post.exact_post, similar_posts = check_similarity(api, post.image_token)
TypeError: cannot unpack non-iterable NoneType object
I'm running szurubooru on unraid as a docker container. I generated the token by running echo -n admin:token | base64
in the dockers console.
I'm running the python scripts on a Windows machine
My full config file with personal info redacted:
{
"szurubooru": {
"url": "https://szurubooru.domain.com",
"api_token": "my_token",
"public": "False"
},
"auto_tagger": {
"saucenao_api_key": "my_api_key",
"show_progress": "True",
"local_tmp_path": "/tmp/",
"use_saucenao": "False",
"deepbooru_enabled": "False",
"deepbooru_model": "./misc/deepbooru/model-resnet_custom_v3.h5",
"deepbooru_threshold": "0.7",
"boorus": {
"danbooru": {
"user": "my_username",
"api_key": "my_api_key"
},
"gelbooru": {
"user": "None",
"api_key": "None"
},
"konachan": {
"user": "None",
"password": "None"
},
"yandere": {
"user": "None",
"password": "None"
},
"pixiv": {
"user": "None",
"password": "None",
"token": "None"
}
}
},
"upload_images": {
"upload_dir": "P:/phone/szurubooru",
"show_progress": "True",
"tags": ["auto_tagme"]
}
}
Implement new script reset-posts
with argument --mode=query|range
.
The query mode resets all posts returned from the input query while the range mode accepts a range like "100..200", which resets the posts starting from id 100, ending with id 200.
Reset means that all tags and sources will be removed from those posts.
Hello,
I have a (big) library of pictures that I'd like to upload (around 11k items).
It uploaded quite a lot, but now when it starts, it fails with the following error (at 483/11566):
[CRITICAL] [27.04.2023, 13:03:41 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/usr/local/bin/upload-media", line 6, in <module>
sys.exit(main())
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 300, in main
upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 237, in upload_post
post.exact_post, similar_posts = check_similarity(szuru, post.token)
TypeError: cannot unpack non-iterable NoneType object
Thanks!
Basically, add a script that:
-implication-count:0
)Right now, szurubooru doesn't retroactively update implications and it's not a feature that's planned to be added (see https://github.com/rr-/szurubooru/issues/189 ), but it's a rather heavy maintenance burden for creating proper tag implication structures, especially for tags that encompass large amounts of posts.
Hi,
Just wrote a script to convert all exported tags from my old hydrus instance into a format szurubooru-toolkit could import, and I followed the documentation here, where it says to format <tag_name>,<tag_number>. Upon doing this in my text file, the script immediately throws up a large error ended with Exception: Tag category '0' not found.. As soon as I find and replace ", 0" to ", default", it works fine, now throwing up category '3' not found. It's not a huge issue, and it's a relatively easy fix on my end, but it does seem a bit confusing, following the documentation.
Hello, I've been getting this issue from the docker container when I try to run the auto-tagger from the docker container :(
szuru-toolkit | Traceback (most recent call last):
szuru-toolkit | File "/usr/local/bin/auto-tagger", line 3, in <module>
szuru-toolkit | from szurubooru_toolkit.scripts.auto_tagger import main
szuru-toolkit | File "/szurubooru-toolkit/src/szurubooru_toolkit/__init__.py", line 12, in <module>
szuru-toolkit | from .twitter import Twitter # noqa F401
szuru-toolkit | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
szuru-toolkit | File "/szurubooru-toolkit/src/szurubooru_toolkit/twitter.py", line 4, in <module>
szuru-toolkit | import tweepy
szuru-toolkit | ModuleNotFoundError: No module named 'tweepy'
Through some trial and error, we have gotten most of everything work but I'm still having some difficulty getting a specific command to work.
Running this command
python3 auto_tagger.py 'date:today'
outputs the following error
Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py 'date:today'
Could not process your query: 'total'.
Traceback (most recent call last):
File "auto_tagger.py", line 129, in <module>
main()
File "auto_tagger.py", line 28, in main
post_ids, total = api.get_post_ids(user_input.query)
TypeError: cannot unpack non-iterable NoneType object
Specifying a post ID does work, allowing it to find the exact image and updating the tags. It just seems that specifying a date does not.
Current config:
{
"szurubooru": {
"url": "https://booru.mydomain.com",
"api_token": "[redacted]",
"public": "False"
},
"auto_tagger": {
"saucenao_api_key": "[redacted]",
"hide_progress": "False",
"local_temp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
"use_saucenao": "True",
"deepbooru_enabled": "False",
"deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
"deepbooru_threshold": "0.7",
"boorus": {
"danbooru": {
"user": "None",
"api_key": "None"
},
"gelbooru": {
"user": "None",
"api_key": "None"
},
"konachan": {
"user": "None",
"password": "None"
},
"yandere": {
"user": "None",
"password": "None"
},
"pixiv": {
"user": "None",
"password": "None",
"token": "None"
}
}
},
"upload_images": {
"upload_dir": "I:/Images/Importing",
"hide_progress": "False",
"cleanup": "False",
"tags": ["tagme"]
}
}
Hello again! I'm receiving this error when auto tagging and I can't seem to track down what image/video is causing it. I had thought videos were the problem since I don't think saucenao/deepbooru accept videos but I had it scan just a video of mine and it just skipped it, which i assume is the intended behavior. And didn't error out at all.
The error:
Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py tag-count:0
Found 51 posts. Start tagging...
0%| | 0/51 [00:00<?, ?it/s]Could not get result from SauceNAO with uploaded image: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "auto_tagger.py", line 129, in <module>
main()
File "auto_tagger.py", line 65, in main
tags, source, rating, limit_short, limit_long = saucenao.get_metadata(post)
File "C:\Users\shrev\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\syncer.py", line 44, in run
return asyncio.get_event_loop().run_until_complete(f(*args, **kwargs))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "Z:\Misc\szurubooru\szurubooru-scripts-main\classes\saucenao.py", line 64, in get_metadata
if results.results:
AttributeError: 'NoneType' object has no attribute 'results'
I know I'm not being rate limited by Saucenao as I'm tracking my API usage and I still have about 1500 hits left.
Any ideas to narrow down what exactly is causing the issue?
Config:
{
"szurubooru": {
"url": "https://booru.mydomain.com",
"api_token": "token",
"public": "False"
},
"auto_tagger": {
"saucenao_api_key": "token",
"hide_progress": "False",
"local_temp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
"use_saucenao": "True",
"deepbooru_enabled": "True",
"deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
"deepbooru_threshold": "0.7",
"boorus": {
"danbooru": {
"user": "None",
"api_key": "None"
},
"gelbooru": {
"user": "None",
"api_key": "None"
},
"konachan": {
"user": "None",
"password": "None"
},
"yandere": {
"user": "None",
"password": "None"
},
"pixiv": {
"user": "None",
"password": "None",
"token": "None"
}
}
},
"upload_images": {
"upload_dir": "I:/Images/Importing",
"hide_progress": "False",
"cleanup": "False",
"tags": ["tagme"]
}
}
For some reason, when importing from url from sankaku.app, only 11 tags are imported, but an identical search on chan.sankakucomplex.com will retrieve all tags. Oddly, the first one actually imports the source url, but the later doesn't.
The auto-tagger generates an error and does not actually update the post. Deepdanbooru itself is working, I can see via the debug logs that it actually generated tags. Here is what I see in the console:
$ auto-tagger.exe "tagme"
[INFO] [30.10.2022, 00:34:30 Eastern Daylight Time]: Initializing script...
[INFO] [30.10.2022, 00:34:34 Eastern Daylight Time]: Retrieving posts from XXXX with query "tagme"...
[INFO] [30.10.2022, 00:34:37 Eastern Daylight Time]: Found 2 posts. Start tagging...
0%| | 0/2 [00:00<?, ?it/s]
[INFO] [30.10.2022, 00:34:38 Eastern Daylight Time]: Your daily SauceNAO limit has been reached. Consider upgrading your account.
[INFO] [30.10.2022, 00:34:38 Eastern Daylight Time]: Continuing tagging with Deepbooru only...
[WARNING] [30.10.2022, 00:34:48 Eastern Daylight Time] [szurubooru.update_post]: Could not edit your post: Name must satisfy regex '^\\S+$'.
50%|██████████████████████▌ | 1/2 [00:10<00:10, 10.65s/it]
[WARNING] [30.10.2022, 00:34:54 Eastern Daylight Time] [szurubooru.update_post]: Could not edit your post: Name must satisfy regex '^\\S+$'.
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Script has finished tagging.
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Total: 2
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Tagged: 0
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Deepbooru: 2
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Untagged: 0
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Skipped:
Here is a scrubbed log file:
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_user = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_api_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_user = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_api_url = XXXX
[95m[INFO][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Initializing script...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.parse_args][0m: sankaku_url = None
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.parse_args][0m: query = tagme
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[saucenao.__init__][0m: Using SauceNAO API token
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[danbooru.__init__][0m: Using Danbooru user XXXX with API key
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[gelbooru.__init__][0m: Using Gelbooru without user and API key
[95m[INFO][0m [92m[30.10.2022, 00:39:23 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Retrieving posts from XXXX with query "tagme"...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:23 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Getting post from query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Got a total of 2 results
[95m[DEBUG][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Searching across 1 pages
[95m[INFO][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Found 2 posts. Start tagging...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning tags: []
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning sources:
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning rating: safe
[95m[INFO][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[auto_tagger.parse_saucenao_results][0m: Your daily SauceNAO limit has been reached. Consider upgrading your account.
[95m[INFO][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[auto_tagger.parse_saucenao_results][0m: Continuing tagging with Deepbooru only...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed following tags: ['XXXX']
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[utils.convert_rating][0m: Converted rating rating:questionable to sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed rating sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Updating following post: Post(id: XXXX, source: Deepbooru, content_url: XXXX, version: 3, relations: [{'id': XXXX, 'thumbnailUrl': 'XXXX'}], tags: ['XXXX'], safety: sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using payload: {"version": 3, "tags": ["XXXX"], "source": "Deepbooru", "safety": "sketchy"}
[95m[WARNING][0m [92m[30.10.2022, 00:39:36 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Could not edit your post: Name must satisfy regex '^\\S+$'.
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed following tags: ['XXXX']
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[utils.convert_rating][0m: Converted rating rating:safe to safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed rating safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Updating following post: Post(id: XXXX, source: Deepbooru\nXXXX\n, content_url: XXXX, version: 4, relations: [], tags: ['XXXX'], safety: safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using payload: {"version": 4, "tags": ["XXXX"], "source": "Deepbooru\nXXXX\n", "safety": "safe"}
[95m[WARNING][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Could not edit your post: Name must satisfy regex '^\\S+$'.
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Script has finished tagging.
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Total: 2
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Tagged: 0
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Deepbooru: 2
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Untagged: 0
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Skipped: 0
For what it's worth I am tagging a post with pre-existing tags.
If it is significant, although redacted in the the log above I noticed that on the lines containing "Updating following post" and "Using payload" the tags
array appears to contain some tags with spaces and some with underscores between words.
Hello, I seem to be getting these error logs while trying to run the autotagger automatically in the docker container:
[INFO] Retrieving posts from https://szuru.bubbletea.dev with query " -deepbooru"...
[CRITICAL] [szurubooru.get_posts] Could not process your query: 'total'
Thank you!
After the first run of this command I get a rate limit error instantly:
Command: auto-tagger.exe --remove-tags "tagme" "id:208"
Scrubbed log shows 5 requests to Saucenao in a short time frame:
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_user = exampleuser
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_url = https://example.com
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_api_url = https://example.com/api
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_user = exampleuser
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_url = https://example.com
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_api_url = https://example.com/api
[INFO] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.main]: Initializing script...
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: sankaku_url = None
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: query = id:208
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: remove_tags = ['tagme']
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [saucenao.__init__]: Using SauceNAO API token
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [danbooru.__init__]: Using Danbooru user exampleuser with API key
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [gelbooru.__init__]: Using Gelbooru without user and API key
[INFO] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.main]: Retrieving posts from https://example.com with query "id:208"...
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.get_posts]: Getting post from query_url: https://example.com/api/posts/?query=type%3Aimage%2Canimation+id%3A208
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [szurubooru.get_posts]: Got a total of 1 results
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [szurubooru.get_posts]: Searching across 1 pages
[INFO] [24.11.2022, 22:34:52 Eastern Standard Time] [auto_tagger.main]: Found 1 posts. Start tagging...
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:53 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:53 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:54 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[WARNING] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_result]: Could not get result from SauceNAO with image URL "https://example.com/data/posts/208_xxx.png": <strong>Search Rate Too High.</strong><br /><br />exampleuser, basic accounts share an IP based usage pool. Your IP (x.x.x.x) has exceeded the basic account type's rate limit of 4 searches every 30 seconds.<br />Account upgrades provide a per-user usage pool, and can be used to increase this limit.<br /><br />Please check the <a href='user.php?page=account-upgrades'>upgrades page</a> to find available upgrades.
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning tags: []
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning sources:
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning rating: safe
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Updating following post: Post(id: 208, source: https://example.org, content_url: https://example.com/data/posts/208_xxx.png, version: 3, relations: [], tags: ['highres', 'tagme'], safety: safe
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Using query_url: https://example.com/api/post/208
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Using payload: {"version": 3, "tags": ["highres", "tagme"], "source": "https://example.org", "safety": "safe"}
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Script has finished tagging.
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Total: 1
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Tagged: 0
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Deepbooru: 0
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Untagged: 1
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Skipped: 0
Will be useful for future doc generation based on Docstrings (mkdocs).
There are extra sets of square brackets in the example config for the booru section (line 21-33). They were messing with the toml parser on my end.
The upload-media script should have an additional option like upscale <factor>
. I've looked into a few available waifu2x Python projects, but none seem to easily work out of the box (CUDA or WSL problems).
i have the autotagger set up and it seems to throw an error when trying to add a tag that doesn't exist in the szurubooru instance yet. here's an example stack trace for a tag, and please let me know if this needs to be a bug reported in the szurubooru application or pyszuru instead!
thank you!
Traceback (most recent call last):
File "/usr/local/bin/auto-tagger", line 6, in <module>
sys.exit(main())
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/auto_tagger.py", line 288, in main
szuru_tag = szuru.api.getTag(tag)
File "/usr/local/lib/python3.11/site-packages/pyszuru/__init__.py", line 29, in getTag
t.pull()
File "/usr/local/lib/python3.11/site-packages/pyszuru/resource.py", line 105, in pull
data = self._api._call("GET", self._get_instance_urlparts())
File "/usr/local/lib/python3.11/site-packages/pyszuru/api.py", line 131, in _call
self._check_api_response(response)
File "/usr/local/lib/python3.11/site-packages/pyszuru/api.py", line 51, in _check_api_response
raise SzurubooruHTTPError(msg)
pyszuru.api.SzurubooruHTTPError: TagNotFoundError: Tag 'easter_egg' not found.
I use Conda on Windows 10 for a virtual Python environment in which I installed szurubooru-toolkit via pip. tag-posts
fails due to using the working directory of the terminal as the "query" argument.
After I open CMD I run conda activate booru
and then these two commands which fail:
tag-posts.exe --update-implications "id:1"
tag-posts.exe --add-tags "example" "id:1"
Log file:
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: add_tags = None
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: remove_tags = None
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: update_implications = True
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: query = C:\Users\USER\.conda\envs\booru\Scripts\tag-posts
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.get_posts]: Getting post from query_url: https://example.booru/api/posts/?query=type%3Aimage%2Canimation+C%3A%5CUsers%5CUSER%5C.conda%5Cenvs%5Cbooru%5CScripts%5Ctag-posts
[CRITICAL] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.get_posts]: SearchError: Unknown named token: 'c'. Available named tokens: ['ar', 'area', 'aspect-ratio', 'comment', 'comment-count', 'comment-date', 'comment-time', 'content-checksum', 'creation-date', 'creation-time', 'date', 'disliked', 'edit-date', 'edit-time', 'fav', 'fav-count', 'fav-date', 'fav-time', 'feature-count', 'feature-date', 'feature-time', 'file-size', 'flag', 'height', 'id', 'image-ar', 'image-area', 'image-aspect-ratio', 'image-height', 'image-width', 'last-edit-date', 'last-edit-time', 'liked', 'md5', 'note-count', 'note-text', 'pool', 'rating', 'relation-count', 'safety', 'score', 'sha1', 'source', 'submit', 'tag', 'tag-count', 'time', 'type', 'upload', 'uploader', 'width'].
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: add_tags = example
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: remove_tags = None
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: update_implications = False
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: query = C:\Users\USER\.conda\envs\booru\Scripts\tag-posts
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.get_posts]: Getting post from query_url: https://example.booru/api/posts/?query=type%3Aimage%2Canimation+C%3A%5CUsers%5CUSER%5C.conda%5Cenvs%5Cbooru%5CScripts%5Ctag-posts
[CRITICAL] [31.10.2022, 15:48:26 Eastern Daylight Time] [szurubooru.get_posts]: SearchError: Unknown named token: 'c'. Available named tokens: ['ar', 'area', 'aspect-ratio', 'comment', 'comment-count', 'comment-date', 'comment-time', 'content-checksum', 'creation-date', 'creation-time', 'date', 'disliked', 'edit-date', 'edit-time', 'fav', 'fav-count', 'fav-date', 'fav-time', 'feature-count', 'feature-date', 'feature-time', 'file-size', 'flag', 'height', 'id', 'image-ar', 'image-area', 'image-aspect-ratio', 'image-height', 'image-width', 'last-edit-date', 'last-edit-time', 'liked', 'md5', 'note-count', 'note-text', 'pool', 'rating', 'relation-count', 'safety', 'score', 'sha1', 'source', 'submit', 'tag', 'tag-count', 'time', 'type', 'upload', 'uploader', 'width'].
Currently, even if deepbooru_enabled
in config.toml
is set to false
, __init__.py
tries to import the deepbooru module and its Tensorflow dependencies.
And even if the Tensorflow package is installed and deepbooru_enabled
is set to false
, the scripts might fail in case the hosts processor doesn't have the AVX instruction set (common for NAS systems).
Is the currently a way to import images from a folder with sidecars that programs like gallery-dl make with --write-tags instead of having to set the tag to "tagme" and relying on the auto tagger? Additionally, would it be possible to have a boolean in the config so that if the similarity is too high, the tags get copied over anyway like if the same image was downloaded twice, but from two different boorus with two different sets of tags?
I'm unable to use the delete-posts
tool to delete video posts. In szurubooru.py
, in the get_posts
method, which delete-posts
uses, videos are excluded from the results:
szurubooru-toolkit/src/szurubooru_toolkit/szurubooru.py
Lines 61 to 65 in 5d94b77
Is there a reason videos are explicitly excluded? Could we have an option to include them?
Thanks
When I use auto-tagger to tag a post with Saucenao results, it appears to remove any existing tags and replaces them fully with the new ones.
There are instances where I have custom tags, and it would be good to keep those, but still be able to use auto-tagger. Could this be an option in the config file?
The type:image,animation
restriction doesn't seem to be working, which results in the list of returned posts to include mp4s and other media, which can cause errors when trying to resize what should only be static images, etc.
I've already fixed this issue on my docker
branch here, namely this commit.
However, my docker branch also includes a lot of other minor fixes/changes including:
Would you be interested in any of these changes? I could cherrypick any of the changes out and open a pr if you'd like! Thank you!
Unfortunately, it seems like the example command in crontab_sample
for the docker image cannot run, and needs to be modified to the following:
*/30 * * * * cd /szurubooru-toolkit && /usr/local/bin/auto-tagger --add-tags deepbooru "-deepbooru -deepbooru" >/proc/1/fd/1 2>&1
Also, it seems like if you only want to search for a inverted tag (like -deepbooru
), you have to append it twice, otherwise it errors with auto-tagger: error: the following arguments are required: query
.
Thank you for all your work on this project!
I would absolutely love for better logging in these scripts. Particularly the auto-tagger but really all of them could use it. As an example, I keep getting this error and it does nothing but tell me 'image dimensions too small'. While that is nice to know, it doesn't really help me narrow down which image in particular is the one that is too small.
Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py tag-count:0
Found 117 posts. Start tagging...
0%| | 0/117 [00:00<?, ?it/s]image dimensions too small...
Could not get result from SauceNAO with uploaded image: image dimensions too small...
Traceback (most recent call last):
File "auto_tagger.py", line 129, in <module>
main()
File "auto_tagger.py", line 65, in main
tags, source, rating, limit_short, limit_long = saucenao.get_metadata(post)
File "C:\Users\shrev\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\syncer.py", line 44, in run
return asyncio.get_event_loop().run_until_complete(f(*args, **kwargs))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "Z:\Misc\szurubooru\szurubooru-scripts-main\classes\saucenao.py", line 64, in get_metadata
if results.results:
AttributeError: 'NoneType' object has no attribute 'results'
So some improved logging could really go a long way with these scripts. You may already be working on something but wanted to throw it out there anyways.
Hi, I've been trying for a while now to get import-from-twitter working.
I enabled the developer account on twitter, generated the consumer key and secret, and generated the account token and secret, and put them into the config.toml
Whenever I run the command for the same user-ID, I keep getting either "you don't have permission" or a error 403. I've tried regenerating everything several times and still can't get it working. Any help would be greatly appreciated.
as part of this commit, the [twitter]
section in config.toml has been removed, but it seems like the code checking for that option is still there
sorry i am not making a pr because i am also lazy -3-
Hello,
Is there a way to change the default setting to safe instead of unsafe when uploading using the upload-media script?
Regards,
Hello,
The skipping that occus when similarity checks occurs is now broken:
[WARNING] [11.12.2023, 03:09:42 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0). Skipping post...
Traceback (most recent call last):
File "/usr/local/bin/upload-media", line 6, in <module>
sys.exit(main())
^^^^^^
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 327, in main
success, saucenao_limit_reached = upload_post(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable bool object
Thanks
When running the script auto-tagger (or any using it) and the SauceNAO limit has been reached, the script handles it (probably) correctly if the limit is being hit during the run. If the script starts another run, it won't detect that the limit has been reached and runs into a few exceptions.
If limit is reached, pysaucenao return the String Limit reached
which we can be used to handle the correct continuation of the script.
Downloading from any tag with more than 100 posts won't go above 100, even when using --limit setting.
--limit works for setting it to a lower number, but not above 100.
Implement new script tag-posts
with arguments --add-tags=TAGS
, --remove-tags=TAGS
, --mode=append|overwrite
and query
.
Basically a manual implementation of the auto-tagger
script. While that script already has those functionalities as well, it won't execute unless one automatic option like saucenao_enabled
or deepbooru_enabled
is set.
With --mode
we can specify if existing tags of the post should be kept or overwritten. In that case the --remove-tags
argument makes only sense with --mode append
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.