reluce / szurubooru-toolkit Goto Github PK

View Code? Open in Web Editor NEW

39.0 2.0 12.0 723 KB

Python package and script collection to manage your szurubooru image board.

License: GNU General Public License v3.0

Python 97.35% Dockerfile 0.37% Shell 0.07% JavaScript 1.36% HTML 0.33% CSS 0.52%

szurubooru image-board automation python bash scripts szuru tagging uploader auto-tagging

szurubooru-toolkit's People

Contributors

Stargazers

Watchers

Forkers

manicarus ipchar maxgames17 rodolfofigueroa prof-m joshuaaferguson parallax4 myalow orendcross nitori-dev gljames24

szurubooru-toolkit's Issues

Import from URL

Hi!

Big fan of the toolkit since there's not a lot out there for szurubooru importing, management etc.
One thing I would like to see, however, is URL support. Although it is nice to be able to scrape results from a query, I would like to be able to insert one or multiple (or hundreds via a url list .txt or something) links to a booru post and scrape that way.

Essentially, being able to do import-from-booru gelbooru gelbooru-urls.txt would be a dream come true, but even import-from-booru gelbooru https://gelbooru.com/index.php?page=post&s=view&id=1234567 would be great, and the end user could make a shellscript or similar to handle mass imports from there.

Create tags: implications

Would it be possible to also add implications when using create-tags?

convert_image tries to convert MP4s

convert_image in upload_media.py will attempt to convert (shrink) files that it has no business modifying.

EDIT:
Here is the error:

$ poetry run upload-media
[INFO] [23.05.2022, 17:44:58 EDT]: Found 43 file(s). Starting upload...
Traceback (most recent call last):                                              
  File "<string>", line 1, in <module>
  File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 292, in main
    upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
  File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 230, in upload_post
    post.image = convert_image(file, file_ext, file_to_upload)
  File "/home/user/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 223, in convert_image
    return image
UnboundLocalError: local variable 'image' referenced before assignment

Reduce complexity of auto-tagger's main function

Flake8 reports a complexity of 32.

Add option to tag post directly after upload

While the upload-media script currently does its designed job, it would be nice to have the option to directly tag the post after the upload.

For that, the auto_tagger script's main function could accept the uploaded/returned post id and use it to tag the post.

Add script to import posts from Boorus

The script should take a search query for various Boorus as an input and downloads + tags all matched posts for the query.

Something like: import-from --booru [danbooru,gelbooru,konachan,yandere] query

Importing from booru skips 100 posts when limit is above 100

The function get_posts_from_booru in utils.py skips a page of posts. Starting the range at 0 in the for loop seems to fix this.
Edit: Although this fixed this issue I was having, it tried to import posts twice in some cases.

szurubooru-toolkit/src/szurubooru_toolkit/utils.py

Line 400 in 612cd3a

for page in range(1, pages_needed + 1):

TypeError: cannot unpack non-iterable NoneType object

Hello,

I'm receiving the following error when attempting to run any variation of the auto_tagger python script:

Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py 'id:126'
Could not process your query: 'total'.
Traceback (most recent call last):
  File "auto_tagger.py", line 129, in <module>
    main()
  File "auto_tagger.py", line 28, in main
    post_ids, total = api.get_post_ids(user_input.query)
TypeError: cannot unpack non-iterable NoneType object

{
    "szurubooru": {
        "url": "https://booru.mydomain.com/",
        "api_token": "my api token",
        "public": "False"
    },
    "auto_tagger": {
        "saucenao_api_key": "my api token",
        "hide_progress": "False",
        "local_tmp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
        "use_saucenao": "True",
        "deepbooru_enabled": "False",
        "deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
        "deepbooru_threshold": "0.7",
        "boorus": {
            "danbooru": {
                "user": "None",
                "api_key": "None"
            },
            "gelbooru": {
                "user": "None",
                "api_key": "None"
            },
            "konachan": {
                "user": "None",
                "password": "None"
            },
            "yandere": {
                "user": "None",
                "password": "None"
            },
            "pixiv": {
                "user": "None",
                "password": "None",
                "token": "None"
            }
        }
    },
    "upload_images": {
        "upload_dir": "I:/Images/Booru/posts",
        "hide_progress": "False",
        "cleanup": "False",
        "tags": ["tagme"]
    }
}

New script: delete-posts

Implement new script delete-posts with argument --mode=query|range.

The query mode deletes all posts returned from the input query while the range mode accepts a range like "100..200", which deletes the posts starting from id 100, ending with id 200.

Create tags is broken

Hello,

It looks like create-tags feature does not work anymore:

root@szurubooru-toolkit-7b59658595-g9s96:/szurubooru-toolkit# /usr/local/bin/create-tags --overwrite --query genshin*
[INFO] [25.04.2023, 12:53:54 UTC]: Fetching tags from URL https://danbooru.donmai.us/tags.json?search[post_count]=>10&search[name_matches]=genshin*&limit=100&page=1...
[CRITICAL] [25.04.2023, 12:53:54 UTC] [danbooru.download_tags]: Could not fetch tags: Expecting value: line 1 column 1 (char 0)
[SUCCESS] [25.04.2023, 12:53:54 UTC]: Script finished creating tags!

Thanks!

An error occured while getting the image token: Invalid token.

I'm getting an invalid token error when trying to run upload_images.py

This is my full terminal output:

Found 1 images. Starting upload...
  0%|                                                     | 0/1 [00:00<?, ?it/s]
An error occured while getting the image token: Invalid token.

An error occured during the similarity check: Invalid token.
Traceback (most recent call last):
  File "B:\Downloads\szurubooru-scripts-main_fork\upload_images.py", line 204, in <module>
    main()
  File "B:\Downloads\szurubooru-scripts-main_fork\upload_images.py", line 184, in main
    post.exact_post, similar_posts = check_similarity(api, post.image_token)
TypeError: cannot unpack non-iterable NoneType object

I'm running szurubooru on unraid as a docker container. I generated the token by running echo -n admin:token | base64 in the dockers console.
I'm running the python scripts on a Windows machine

My full config file with personal info redacted:

{
    "szurubooru": {
        "url": "https://szurubooru.domain.com",
        "api_token": "my_token",
        "public": "False"
    },
    "auto_tagger": {
        "saucenao_api_key": "my_api_key",
        "show_progress": "True",
        "local_tmp_path": "/tmp/",
        "use_saucenao": "False",
        "deepbooru_enabled": "False",
        "deepbooru_model": "./misc/deepbooru/model-resnet_custom_v3.h5",
        "deepbooru_threshold": "0.7",
        "boorus": {
            "danbooru": {
                "user": "my_username",
                "api_key": "my_api_key"
            },
            "gelbooru": {
                "user": "None",
                "api_key": "None"
            },
            "konachan": {
                "user": "None",
                "password": "None"
            },
            "yandere": {
                "user": "None",
                "password": "None"
            },
            "pixiv": {
                "user": "None",
                "password": "None",
                "token": "None"
            }
        }
    },
    "upload_images": {
        "upload_dir": "P:/phone/szurubooru",
        "show_progress": "True",
        "tags": ["auto_tagme"]
    }
}

New script: reset-posts

Implement new script reset-posts with argument --mode=query|range.

The query mode resets all posts returned from the input query while the range mode accepts a range like "100..200", which resets the posts starting from id 100, ending with id 200.

Reset means that all tags and sources will be removed from those posts.

Issue with upload_media feature

Hello,

I have a (big) library of pictures that I'd like to upload (around 11k items).
It uploaded quite a lot, but now when it starts, it fails with the following error (at 483/11566):

[CRITICAL] [27.04.2023, 13:03:41 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/bin/upload-media", line 6, in <module>
    sys.exit(main())
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 300, in main
    upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 237, in upload_post
    post.exact_post, similar_posts = check_similarity(szuru, post.token)
TypeError: cannot unpack non-iterable NoneType object

Thanks!

Suggestion: add implications autotagger

Basically, add a script that:

Goes through all tags with at least one implication (-implication-count:0)
Loops over the posts that belong to these tags.
Check if the post has the implications properly added.
If not, edit the post to add those tag implications.

Right now, szurubooru doesn't retroactively update implications and it's not a feature that's planned to be added (see https://github.com/rr-/szurubooru/issues/189 ), but it's a rather heavy maintenance burden for creating proper tag implication structures, especially for tags that encompass large amounts of posts.

Number-based Category tagging in a tags.txt

Hi,

Just wrote a script to convert all exported tags from my old hydrus instance into a format szurubooru-toolkit could import, and I followed the documentation here, where it says to format <tag_name>,<tag_number>. Upon doing this in my text file, the script immediately throws up a large error ended with Exception: Tag category '0' not found.. As soon as I find and replace ", 0" to ", default", it works fine, now throwing up category '3' not found. It's not a huge issue, and it's a relatively easy fix on my end, but it does seem a bit confusing, following the documentation.

ModuleNotFoundError: No module named 'tweepy'

Hello, I've been getting this issue from the docker container when I try to run the auto-tagger from the docker container :(

szuru-toolkit  | Traceback (most recent call last):
szuru-toolkit  |   File "/usr/local/bin/auto-tagger", line 3, in <module>
szuru-toolkit  |     from szurubooru_toolkit.scripts.auto_tagger import main
szuru-toolkit  |   File "/szurubooru-toolkit/src/szurubooru_toolkit/__init__.py", line 12, in <module>
szuru-toolkit  |     from .twitter import Twitter  # noqa F401
szuru-toolkit  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
szuru-toolkit  |   File "/szurubooru-toolkit/src/szurubooru_toolkit/twitter.py", line 4, in <module>
szuru-toolkit  |     import tweepy
szuru-toolkit  | ModuleNotFoundError: No module named 'tweepy'

TypeError: cannot unpack non-iterable NoneType object

Through some trial and error, we have gotten most of everything work but I'm still having some difficulty getting a specific command to work.

Running this command

python3 auto_tagger.py 'date:today'

outputs the following error

Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py 'date:today'
Could not process your query: 'total'.
Traceback (most recent call last):
  File "auto_tagger.py", line 129, in <module>
    main()
  File "auto_tagger.py", line 28, in main
    post_ids, total = api.get_post_ids(user_input.query)
TypeError: cannot unpack non-iterable NoneType object

Specifying a post ID does work, allowing it to find the exact image and updating the tags. It just seems that specifying a date does not.

Current config:

{
    "szurubooru": {
        "url": "https://booru.mydomain.com",
        "api_token": "[redacted]",
        "public": "False"
    },
    "auto_tagger": {
        "saucenao_api_key": "[redacted]",
        "hide_progress": "False",
        "local_temp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
        "use_saucenao": "True",
        "deepbooru_enabled": "False",
        "deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
        "deepbooru_threshold": "0.7",
        "boorus": {
            "danbooru": {
                "user": "None",
                "api_key": "None"
            },
            "gelbooru": {
                "user": "None",
                "api_key": "None"
            },
            "konachan": {
                "user": "None",
                "password": "None"
            },
            "yandere": {
                "user": "None",
                "password": "None"
            },
            "pixiv": {
                "user": "None",
                "password": "None",
                "token": "None"
            }
        }
    },
    "upload_images": {
        "upload_dir": "I:/Images/Importing",
        "hide_progress": "False",
        "cleanup": "False",
        "tags": ["tagme"]
    }
}

Random errors when auto-tagging

Hello again! I'm receiving this error when auto tagging and I can't seem to track down what image/video is causing it. I had thought videos were the problem since I don't think saucenao/deepbooru accept videos but I had it scan just a video of mine and it just skipped it, which i assume is the intended behavior. And didn't error out at all.

The error:

Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py tag-count:0
Found 51 posts. Start tagging...
  0%|                                                    | 0/51 [00:00<?, ?it/s]Could not get result from SauceNAO with uploaded image: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "auto_tagger.py", line 129, in <module>
    main()
  File "auto_tagger.py", line 65, in main
    tags, source, rating, limit_short, limit_long = saucenao.get_metadata(post)
  File "C:\Users\shrev\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\syncer.py", line 44, in run
    return asyncio.get_event_loop().run_until_complete(f(*args, **kwargs))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "Z:\Misc\szurubooru\szurubooru-scripts-main\classes\saucenao.py", line 64, in get_metadata
    if results.results:
AttributeError: 'NoneType' object has no attribute 'results'

I know I'm not being rate limited by Saucenao as I'm tracking my API usage and I still have about 1500 hits left.

Any ideas to narrow down what exactly is causing the issue?

Config:

{
    "szurubooru": {
        "url": "https://booru.mydomain.com",
        "api_token": "token",
        "public": "False"
    },
    "auto_tagger": {
        "saucenao_api_key": "token",
        "hide_progress": "False",
        "local_temp_path": "Z:/Misc/szurubooru/szurubooru-scripts-main/tmp",
        "use_saucenao": "True",
        "deepbooru_enabled": "True",
        "deepbooru_model": "Z:/Misc/szurubooru/szurubooru-scripts-main/misc/deepbooru/model-resnet_custom_v3.h5",
        "deepbooru_threshold": "0.7",
        "boorus": {
            "danbooru": {
                "user": "None",
                "api_key": "None"
            },
            "gelbooru": {
                "user": "None",
                "api_key": "None"
            },
            "konachan": {
                "user": "None",
                "password": "None"
            },
            "yandere": {
                "user": "None",
                "password": "None"
            },
            "pixiv": {
                "user": "None",
                "password": "None",
                "token": "None"
            }
        }
    },
    "upload_images": {
        "upload_dir": "I:/Images/Importing",
        "hide_progress": "False",
        "cleanup": "False",
        "tags": ["tagme"]
    }
}

Tags not all importing depending on website

For some reason, when importing from url from sankaku.app, only 11 tags are imported, but an identical search on chan.sankakucomplex.com will retrieve all tags. Oddly, the first one actually imports the source url, but the later doesn't.

Auto-tagger error: Name must satisfy regex '^\\S+$'

The auto-tagger generates an error and does not actually update the post. Deepdanbooru itself is working, I can see via the debug logs that it actually generated tags. Here is what I see in the console:

$ auto-tagger.exe "tagme"
[INFO] [30.10.2022, 00:34:30 Eastern Daylight Time]: Initializing script...
[INFO] [30.10.2022, 00:34:34 Eastern Daylight Time]: Retrieving posts from XXXX with query "tagme"...
[INFO] [30.10.2022, 00:34:37 Eastern Daylight Time]: Found 2 posts. Start tagging...
  0%|                                                     | 0/2 [00:00<?, ?it/s]
[INFO] [30.10.2022, 00:34:38 Eastern Daylight Time]: Your daily SauceNAO limit has been reached. Consider upgrading your account.
[INFO] [30.10.2022, 00:34:38 Eastern Daylight Time]: Continuing tagging with Deepbooru only...

[WARNING] [30.10.2022, 00:34:48 Eastern Daylight Time] [szurubooru.update_post]: Could not edit your post: Name must satisfy regex '^\\S+$'.
 50%|██████████████████████▌                      | 1/2 [00:10<00:10, 10.65s/it]
[WARNING] [30.10.2022, 00:34:54 Eastern Daylight Time] [szurubooru.update_post]: Could not edit your post: Name must satisfy regex '^\\S+$'.
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Script has finished tagging.
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Total:     2
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Tagged:    0
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Deepbooru: 2
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Untagged:  0
[SUCCESS] [30.10.2022, 00:34:54 Eastern Daylight Time]: Skipped:

Here is a scrubbed log file:

[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_user = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_api_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_user = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_url = XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[szurubooru.__init__][0m: szuru_api_url = XXXX
[95m[INFO][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Initializing script...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.parse_args][0m: sankaku_url = None
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[auto_tagger.parse_args][0m: query = tagme
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[saucenao.__init__][0m: Using SauceNAO API token
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[danbooru.__init__][0m: Using Danbooru user XXXX with API key
[95m[DEBUG][0m [92m[30.10.2022, 00:39:19 Eastern Daylight Time][0m [93m[gelbooru.__init__][0m: Using Gelbooru without user and API key
[95m[INFO][0m [92m[30.10.2022, 00:39:23 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Retrieving posts from XXXX with query "tagme"...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:23 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Getting post from query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Got a total of 2 results
[95m[DEBUG][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[szurubooru.get_posts][0m: Searching across 1 pages
[95m[INFO][0m [92m[30.10.2022, 00:39:26 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Found 2 posts. Start tagging...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning tags: []
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning sources: 
[95m[DEBUG][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[saucenao.get_metadata][0m: Returning rating: safe
[95m[INFO][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[auto_tagger.parse_saucenao_results][0m: Your daily SauceNAO limit has been reached. Consider upgrading your account.
[95m[INFO][0m [92m[30.10.2022, 00:39:27 Eastern Daylight Time][0m [93m[auto_tagger.parse_saucenao_results][0m: Continuing tagging with Deepbooru only...
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed following tags: ['XXXX']
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[utils.convert_rating][0m: Converted rating rating:questionable to sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:34 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed rating sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Updating following post: Post(id: XXXX, source: Deepbooru, content_url: XXXX, version: 3, relations: [{'id': XXXX, 'thumbnailUrl': 'XXXX'}], tags: ['XXXX'], safety: sketchy
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:35 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using payload: {"version": 3, "tags": ["XXXX"], "source": "Deepbooru", "safety": "sketchy"}
[95m[WARNING][0m [92m[30.10.2022, 00:39:36 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Could not edit your post: Name must satisfy regex '^\\S+$'.
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed following tags: ['XXXX']
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[utils.convert_rating][0m: Converted rating rating:safe to safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[deepbooru.tag_image][0m: Guessed rating safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Updating following post: Post(id: XXXX, source: Deepbooru\nXXXX\n, content_url: XXXX, version: 4, relations: [], tags: ['XXXX'], safety: safe
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using query_url: XXXX
[95m[DEBUG][0m [92m[30.10.2022, 00:39:41 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Using payload: {"version": 4, "tags": ["XXXX"], "source": "Deepbooru\nXXXX\n", "safety": "safe"}
[95m[WARNING][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[szurubooru.update_post][0m: Could not edit your post: Name must satisfy regex '^\\S+$'.
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Script has finished tagging.
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Total:     2
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Tagged:    0
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Deepbooru: 2
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Untagged:  0
[95m[SUCCESS][0m [92m[30.10.2022, 00:39:42 Eastern Daylight Time][0m [93m[auto_tagger.main][0m: Skipped:   0

For what it's worth I am tagging a post with pre-existing tags.
If it is significant, although redacted in the the log above I noticed that on the lines containing "Updating following post" and "Using payload" the tags array appears to contain some tags with spaces and some with underscores between words.

Cryptic error message when using an invalid API token

Hello, I seem to be getting these error logs while trying to run the autotagger automatically in the docker container:

[INFO] Retrieving posts from https://szuru.bubbletea.dev with query " -deepbooru"...
[CRITICAL] [szurubooru.get_posts] Could not process your query: 'total'

Thank you!

SauceNao rate limit of 4 searches every 30 seconds

After the first run of this command I get a rate limit error instantly:
Command: auto-tagger.exe --remove-tags "tagme" "id:208"
Scrubbed log shows 5 requests to Saucenao in a short time frame:

[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_user = exampleuser
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_url = https://example.com
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_api_url = https://example.com/api
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_user = exampleuser
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_url = https://example.com
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.__init__]: szuru_api_url = https://example.com/api
[INFO] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.main]: Initializing script...
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: sankaku_url = None
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: query = id:208
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.parse_args]: remove_tags = ['tagme']
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [saucenao.__init__]: Using SauceNAO API token
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [danbooru.__init__]: Using Danbooru user exampleuser with API key
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [gelbooru.__init__]: Using Gelbooru without user and API key
[INFO] [24.11.2022, 22:34:50 Eastern Standard Time] [auto_tagger.main]: Retrieving posts from https://example.com with query "id:208"...
[DEBUG] [24.11.2022, 22:34:50 Eastern Standard Time] [szurubooru.get_posts]: Getting post from query_url: https://example.com/api/posts/?query=type%3Aimage%2Canimation+id%3A208
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [szurubooru.get_posts]: Got a total of 1 results
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [szurubooru.get_posts]: Searching across 1 pages
[INFO] [24.11.2022, 22:34:52 Eastern Standard Time] [auto_tagger.main]: Found 1 posts. Start tagging...
[DEBUG] [24.11.2022, 22:34:52 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:53 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:53 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:54 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_result]: Trying to get result from content_url: https://example.com/data/posts/208_xxx.png
[WARNING] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_result]: Could not get result from SauceNAO with image URL "https://example.com/data/posts/208_xxx.png": <strong>Search Rate Too High.</strong><br /><br />exampleuser, basic accounts share an IP based usage pool. Your IP (x.x.x.x) has exceeded the basic account type's rate limit of 4 searches every 30 seconds.<br />Account upgrades provide a per-user usage pool, and can be used to increase this limit.<br /><br />Please check the <a href='user.php?page=account-upgrades'>upgrades page</a> to find available upgrades.
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning tags: []
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning sources: 
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [saucenao.get_metadata]: Returning rating: safe
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Updating following post: Post(id: 208, source: https://example.org, content_url: https://example.com/data/posts/208_xxx.png, version: 3, relations: [], tags: ['highres', 'tagme'], safety: safe
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Using query_url: https://example.com/api/post/208
[DEBUG] [24.11.2022, 22:34:55 Eastern Standard Time] [szurubooru.update_post]: Using payload: {"version": 3, "tags": ["highres", "tagme"], "source": "https://example.org", "safety": "safe"}
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Script has finished tagging.
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Total:     1
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Tagged:    0
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Deepbooru: 0
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Untagged:  1
[SUCCESS] [24.11.2022, 22:34:59 Eastern Standard Time] [auto_tagger.main]: Skipped:   0

Improve Docstrings & Type Annotations

Will be useful for future doc generation based on Docstrings (mkdocs).

Extra square brackets in sample config

There are extra sets of square brackets in the example config for the booru section (line 21-33). They were messing with the toml parser on my end.

Add image upscaling option

The upload-media script should have an additional option like upscale <factor>. I've looked into a few available waifu2x Python projects, but none seem to easily work out of the box (CUDA or WSL problems).

pyszuru.api.SzurubooruHTTPError: TagNotFoundError

i have the autotagger set up and it seems to throw an error when trying to add a tag that doesn't exist in the szurubooru instance yet. here's an example stack trace for a tag, and please let me know if this needs to be a bug reported in the szurubooru application or pyszuru instead!
thank you!

Traceback (most recent call last):

File "/usr/local/bin/auto-tagger", line 6, in <module>
  sys.exit(main())
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/auto_tagger.py", line 288, in main
  szuru_tag = szuru.api.getTag(tag)
File "/usr/local/lib/python3.11/site-packages/pyszuru/__init__.py", line 29, in getTag
  t.pull()
File "/usr/local/lib/python3.11/site-packages/pyszuru/resource.py", line 105, in pull
  data = self._api._call("GET", self._get_instance_urlparts())
File "/usr/local/lib/python3.11/site-packages/pyszuru/api.py", line 131, in _call
  self._check_api_response(response)
File "/usr/local/lib/python3.11/site-packages/pyszuru/api.py", line 51, in _check_api_response
  raise SzurubooruHTTPError(msg)
  
pyszuru.api.SzurubooruHTTPError: TagNotFoundError: Tag 'easter_egg' not found.

tag-posts is using working directory as "query" argument

I use Conda on Windows 10 for a virtual Python environment in which I installed szurubooru-toolkit via pip. tag-posts fails due to using the working directory of the terminal as the "query" argument.
After I open CMD I run conda activate booru and then these two commands which fail:
tag-posts.exe --update-implications "id:1"
tag-posts.exe --add-tags "example" "id:1"
Log file:

[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: add_tags = None
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: remove_tags = None
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: update_implications = True
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [tag_posts.parse_args]: query = C:\Users\USER\.conda\envs\booru\Scripts\tag-posts
[DEBUG] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.get_posts]: Getting post from query_url: https://example.booru/api/posts/?query=type%3Aimage%2Canimation+C%3A%5CUsers%5CUSER%5C.conda%5Cenvs%5Cbooru%5CScripts%5Ctag-posts
[CRITICAL] [31.10.2022, 15:47:31 Eastern Daylight Time] [szurubooru.get_posts]: SearchError: Unknown named token: 'c'. Available named tokens: ['ar', 'area', 'aspect-ratio', 'comment', 'comment-count', 'comment-date', 'comment-time', 'content-checksum', 'creation-date', 'creation-time', 'date', 'disliked', 'edit-date', 'edit-time', 'fav', 'fav-count', 'fav-date', 'fav-time', 'feature-count', 'feature-date', 'feature-time', 'file-size', 'flag', 'height', 'id', 'image-ar', 'image-area', 'image-aspect-ratio', 'image-height', 'image-width', 'last-edit-date', 'last-edit-time', 'liked', 'md5', 'note-count', 'note-text', 'pool', 'rating', 'relation-count', 'safety', 'score', 'sha1', 'source', 'submit', 'tag', 'tag-count', 'time', 'type', 'upload', 'uploader', 'width'].
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_user = admin
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_url = https://example.booru
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.__init__]: szuru_api_url = https://example.booru/api
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: add_tags = example
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: remove_tags = None
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: update_implications = False
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [tag_posts.parse_args]: query = C:\Users\USER\.conda\envs\booru\Scripts\tag-posts
[DEBUG] [31.10.2022, 15:48:25 Eastern Daylight Time] [szurubooru.get_posts]: Getting post from query_url: https://example.booru/api/posts/?query=type%3Aimage%2Canimation+C%3A%5CUsers%5CUSER%5C.conda%5Cenvs%5Cbooru%5CScripts%5Ctag-posts
[CRITICAL] [31.10.2022, 15:48:26 Eastern Daylight Time] [szurubooru.get_posts]: SearchError: Unknown named token: 'c'. Available named tokens: ['ar', 'area', 'aspect-ratio', 'comment', 'comment-count', 'comment-date', 'comment-time', 'content-checksum', 'creation-date', 'creation-time', 'date', 'disliked', 'edit-date', 'edit-time', 'fav', 'fav-count', 'fav-date', 'fav-time', 'feature-count', 'feature-date', 'feature-time', 'file-size', 'flag', 'height', 'id', 'image-ar', 'image-area', 'image-aspect-ratio', 'image-height', 'image-width', 'last-edit-date', 'last-edit-time', 'liked', 'md5', 'note-count', 'note-text', 'pool', 'rating', 'relation-count', 'safety', 'score', 'sha1', 'source', 'submit', 'tag', 'tag-count', 'time', 'type', 'upload', 'uploader', 'width'].

Fix Deepbooru related imports if it's not being used

Currently, even if deepbooru_enabled in config.toml is set to false, __init__.py tries to import the deepbooru module and its Tensorflow dependencies.

And even if the Tensorflow package is installed and deepbooru_enabled is set to false, the scripts might fail in case the hosts processor doesn't have the AVX instruction set (common for NAS systems).

Could Upload Media include Tags from Sidecar txt file

Is the currently a way to import images from a folder with sidecars that programs like gallery-dl make with --write-tags instead of having to set the tag to "tagme" and relying on the auto tagger? Additionally, would it be possible to have a boolean in the config so that if the similarity is too high, the tags get copied over anyway like if the same image was downloaded twice, but from two different boorus with two different sets of tags?

Can't delete video posts

I'm unable to use the delete-posts tool to delete video posts. In szurubooru.py, in the get_posts method, which delete-posts uses, videos are excluded from the results:

szurubooru-toolkit/src/szurubooru_toolkit/szurubooru.py

Lines 61 to 65 in 5d94b77

 try: 

 # Ignore mp4 and webms 

 query_params = {'query': f'type:image,animation {query}'} 

 query_url = self.szuru_api_url + '/posts/?' + urllib.parse.urlencode(query_params) 

 logger.debug(f'Getting post from query_url: {query_url}')

Is there a reason videos are explicitly excluded? Could we have an option to include them?

Thanks

Feature Request: Option to keep existing tags when using auto-tagger

When I use auto-tagger to tag a post with Saucenao results, it appears to remove any existing tags and replaces them fully with the new ones.

There are instances where I have custom tags, and it would be good to keep those, but still be able to use auto-tagger. Could this be an option in the config file?

Image/Animation type restriction doesn't seem to be working

The type:image,animation restriction doesn't seem to be working, which results in the list of returned posts to include mp4s and other media, which can cause errors when trying to resize what should only be static images, etc.

I've already fixed this issue on my docker branch here, namely this commit.

However, my docker branch also includes a lot of other minor fixes/changes including:

Catching the image resizing if it errors (just continues with original image)
Continuing on a deepbooru tagging failure instead of failing
Finally, dockerizing into a container with internal cronjobs so it can be run at regular intervals
- Not implemented yet but I've done it before, I could pretty easily set up a github action to build and deploy new builds to docker hub if you'd like!

Would you be interested in any of these changes? I could cherrypick any of the changes out and open a pr if you'd like! Thank you!

Sample cron job needs updating

Unfortunately, it seems like the example command in crontab_sample for the docker image cannot run, and needs to be modified to the following:

*/30 * * * *    cd /szurubooru-toolkit && /usr/local/bin/auto-tagger --add-tags deepbooru "-deepbooru -deepbooru" >/proc/1/fd/1 2>&1

Also, it seems like if you only want to search for a inverted tag (like -deepbooru), you have to append it twice, otherwise it errors with auto-tagger: error: the following arguments are required: query.

Thank you for all your work on this project!

Suggestion: Better logging

I would absolutely love for better logging in these scripts. Particularly the auto-tagger but really all of them could use it. As an example, I keep getting this error and it does nothing but tell me 'image dimensions too small'. While that is nice to know, it doesn't really help me narrow down which image in particular is the one that is too small.

Z:\Misc\szurubooru\szurubooru-scripts-main>python3 auto_tagger.py tag-count:0
Found 117 posts. Start tagging...
  0%|                                                   | 0/117 [00:00<?, ?it/s]image dimensions too small...
Could not get result from SauceNAO with uploaded image: image dimensions too small...
Traceback (most recent call last):
  File "auto_tagger.py", line 129, in <module>
    main()
  File "auto_tagger.py", line 65, in main
    tags, source, rating, limit_short, limit_long = saucenao.get_metadata(post)
  File "C:\Users\shrev\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\syncer.py", line 44, in run
    return asyncio.get_event_loop().run_until_complete(f(*args, **kwargs))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "Z:\Misc\szurubooru\szurubooru-scripts-main\classes\saucenao.py", line 64, in get_metadata
    if results.results:
AttributeError: 'NoneType' object has no attribute 'results'

So some improved logging could really go a long way with these scripts. You may already be working on something but wanted to throw it out there anyways.

Can't get import-from-twitter to work

Hi, I've been trying for a while now to get import-from-twitter working.

I enabled the developer account on twitter, generated the consumer key and secret, and generated the account token and secret, and put them into the config.toml

Whenever I run the command for the same user-ID, I keep getting either "you don't have permission" or a error 403. I've tried regenerating everything several times and still can't get it working. Any help would be greatly appreciated.

Twitter config options still expected

as part of this commit, the [twitter] section in config.toml has been removed, but it seems like the code checking for that option is still there

sorry i am not making a pr because i am also lazy -3-

upload-media defaults

Hello,

Is there a way to change the default setting to safe instead of unsafe when uploading using the upload-media script?

Regards,

Skipping upload on similarity check does not work anymore

Hello,

The skipping that occus when similarity checks occurs is now broken:

[WARNING] [11.12.2023, 03:09:42 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0). Skipping post...
Traceback (most recent call last):
  File "/usr/local/bin/upload-media", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 327, in main
    success, saucenao_limit_reached = upload_post(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable bool object

Thanks

SauceNao autotagger - 'object is not iterable' error

It seems like with certain posts I get this error. Retrying the post doesn't change the error. Other posts can be tagged just fine.

Command is simply: auto-tagger "id:123" or auto-tagger "tag-count:0"

On version 0.9

Fix SauceNAO limit reached

When running the script auto-tagger (or any using it) and the SauceNAO limit has been reached, the script handles it (probably) correctly if the limit is being hit during the run. If the script starts another run, it won't detect that the limit has been reached and runs into a few exceptions.

If limit is reached, pysaucenao return the String Limit reached which we can be used to handle the correct continuation of the script.

Import From Booru only downloads first 100 posts

Downloading from any tag with more than 100 posts won't go above 100, even when using --limit setting.

--limit works for setting it to a lower number, but not above 100.

New script: tag-posts

Implement new script tag-posts with arguments --add-tags=TAGS, --remove-tags=TAGS, --mode=append|overwrite and query.

Basically a manual implementation of the auto-tagger script. While that script already has those functionalities as well, it won't execute unless one automatic option like saucenao_enabled or deepbooru_enabled is set.

With --mode we can specify if existing tags of the post should be kept or overwritten. In that case the --remove-tags argument makes only sense with --mode append.

	try:
	# Ignore mp4 and webms
	query_params = {'query': f'type:image,animation {query}'}
	query_url = self.szuru_api_url + '/posts/?' + urllib.parse.urlencode(query_params)
	logger.debug(f'Getting post from query_url: {query_url}')

reluce / szurubooru-toolkit Goto Github PK

szurubooru-toolkit's People

Contributors

Stargazers

Watchers

Forkers

szurubooru-toolkit's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs