GithubHelp home page GithubHelp logo

waifuset's Introduction

Waifuset

Waifuset 是一个专门用于构建和管理文生图数据集的工具,其设计重点在于提供一个安全、高效、便捷、可以远程访问的用户界面(UI)。

UI

特点

  • 安全:Waifuset 在操作过程中,不会修改数据集中的任何图像文件,确保数据的完整性和安全。
  • 高效:大部分编辑操作在 Waifuset 中都非常高效,能够轻松处理十万乃至百万量级的大型数据集。
  • 易用:UI 设计简洁直观,即使是复杂操作也可以轻松一键完成。
  • 远程访问:通过 Web 服务进行访问,支持在本地或云服务器上部署和访问。
  • UI 交互:Waifuset 的所有操作都通过用户界面完成,即使是没有编程背景的用户也能轻松上手。

局限性

  • 基于标签的标注:Waifuset 假设所有的标注都是以英文逗号分隔的标签形式,而非自然语言描述。所有的标注编辑操作都将严格遵循这一格式。
  • 专注于 Danbooru 标签:Waifuset 的标注优化算法主要面向 Danbooru 标签系统,不支持其他类型的标签体系。
  • 标注专用:该工具主要用于编辑图像标注,而非图像本身。不包含图像编辑和文件系统功能。

适用场景

  1. 数据集标注的构建和清洗:Waifuset 特别适用于构建和维护文生图数据集的标注,提供了一种高效和简便的方式。
  2. 数据集浏览:作为一个数据集浏览器,Waifuset 可以帮助用户有效地查看和管理他们的数据集。

安装指南

为了使用 Waifuset,您需要首先安装项目及其依赖项。请按照以下步骤操作:

基本安装

  1. 打开您的控制台。

  2. 输入以下命令以克隆项目并安装所需的依赖项:

    git clone https://github.com/Eugeoter/waifuset
    cd waifuset
    pip install -r requirements.txt

可选安装

  • 如需使用 WaifuScorer 进行美学评分

    如果您希望使用 WaifuScorer 进行美学评分,您需要安装额外的依赖项。在控制台中输入以下命令:

    pip install torch torchvision pytorch-lightning huggingface-hub git+https://github.com/openai/CLIP.git
  • 如需使用 WaifuTaggerV3 进行图像标注

    若要使用 WaifuTaggerV3 对图像进行标注,您需要安装一些特定的依赖项。请在控制台中输入以下命令:

    pip install torch torchvision onnxruntime huggingface-hub

如何打开控制台

使用指南

快速开始

Waifuset 的启动和使用非常直观。请遵循以下步骤快速开始:

  1. 启动 Waifuset

    在控制台中,输入以下命令以启动 Waifuset:

    python api.py --source path/to/folder --write_to_txt --language cn
    • path/to/folder 应该替换为您数据集的根目录路径。
    • --write_to_txt 参数指示 Waifuset 将标注信息写入 txt 文件。
    • --language cn 参数用于设置界面语言为中文。
  2. 访问 UI 界面

    • 在命令成功执行后,控制台会输出一个本地 URL,例如 Running on local URL: http://xxx.xxx.xxx.xxx:xxxx
    • 打开浏览器并输入显示的 URL(即 http://xxx.xxx.xxx.xxx:xxxx 部分)以访问 Waifuset 的用户界面。
  3. 云端服务器使用

    • 如果您在云端服务器上部署 Waifuset,您可能需要启用 --share 参数以将 Gradio 网页共享到公网。
    • 或者,您可以使用服务器的代理功能访问 UI 界面。

按照这些步骤操作,您可以轻松地开始使用 Waifuset 来管理和编辑您的数据集。

用户手册

点此查看完整用户手册

如何更新

为了更新 Waifuset,请遵循以下步骤:

  1. 打开您的控制台。

  2. 进入项目目录:

    cd path/to/waifuset

    注:请将 path/to/waifuset 替换为实际的项目路径。

  3. 执行以下命令以从远程仓库拉取最新的更新:

    git pull
    • 如果您之前修改过项目源码,例如更改过配置文件,这可能会导致冲突。

    • 若要强制覆盖本地修改并更新项目,请使用:

      git reset --hard
      git pull

安装到 pip

若您希望通过 pip 安装 Waifuset,以便在您的 Python 环境中使用其类和函数,您可以直接从 GitHub 安装。请确保您的环境中已安装 git。然后,运行以下命令:

pip install git+https://github.com/Eugeoter/waifuset.git

或者,在控制台中打开 git clone 后的项目目录,然后运行以下命令:

pip install .

这将允许您在 Python 脚本中导入和使用 Waifuset。

waifuset's People

Contributors

eugeoter avatar

Stargazers

Allen@亮 avatar  avatar Fei avatar  avatar  avatar  avatar Taehoon Kim avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar insp avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar WPC avatar  avatar  avatar  avatar aihao avatar  avatar  avatar  avatar mktn avatar 427 Crystal avatar Samwise Wang avatar  avatar Ilham AJ avatar  avatar Zhangyuyi avatar  avatar GhostInShell avatar xiyong avatar kokonut avatar undefined avatar  avatar  avatar  avatar  avatar  avatar JOKO avatar LinghuC2333 avatar  avatar leanAI avatar  avatar  avatar  avatar codeforker avatar Linch_97 avatar fofo avatar akari avatar qinlige avatar  avatar  avatar hazukiaoi avatar windsing avatar

Watchers

 avatar

waifuset's Issues

无法使用移除操作

The project is very practical, thank you.
BUT after the update, I encountered such a problem: an error occurs when using the “Remove” button, the console information is as follows:
项目非常实用,感谢。
但是当我更新后出现了这样的问题:使用“移除”按钮时报错,控制台信息如下:
remove tags batch processing: 0%| | 0/1556 [21:41<?, ?steps/s]
remove tags batch processing: 0%| | 0/1556 [00:00<?, ?steps/s]Traceback (most recent call last):
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\queueing.py", line 489, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1561, in process_api
result = await self.call_function(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1179, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\utils.py", line 678, in wrapper
response = f(*args, **kwargs)
File "D:\dataset-manager\sd-dataset-manager\waifuset\ui\ui.py", line 1335, in wrapper
results.extend(edit(batch, *args, **kwargs))
File "D:\dataset-manager\sd-dataset-manager\waifuset\utils\log_utils.py", line 9, in inner
res = func(*args, **kwargs)
File "D:\dataset-manager\sd-dataset-manager\waifuset\ui\ui.py", line 1315, in edit
new_img_info = func(img_info.copy(), *args, **extra_kwargs, **kwargs)
TypeError: create_ui..remove_tags() missing 1 required positional argument: 'regex'

ValueError: not enough values to unpack (expected 9, got 8)

Traceback (most recent call last):
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\queueing.py", line 489, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1561, in process_api
result = await self.call_function(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1179, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "C:\Users\56247\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\utils.py", line 678, in wrapper
response = f(*args, **kwargs)
File "D:\dataset-manager\waifuset\waifuset\ui\ui.py", line 963, in load_source
univset, buffer, sample_history, _, subset, _, _, tag_table, tag_feature_table = init_everything()
ValueError: not enough values to unpack (expected 9, got 8)

将 univset, buffer, sample_history, _, subset, _, _, tag_table, tag_feature_table = init_everything()
修改为 univset, buffer, sample_history, subset, _, _, tag_table, tag_feature_table = init_everything() 后正常

UI打不開

PS D:\waifuset> python api.py --source G:\v\modujinbin\0315\1 --write_to_txt --language cn
[ERROR] failed to load priority config:
[ui_dataset] overload: ['G:\v\modujinbin\0315\1'] -> None
reading 1: 100%|████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<?, ?steps/s]
initializing buffer: 0%| | 0/2 [00:00<?, ?steps/s]
Traceback (most recent call last):
File "D:\waifuset\api.py", line 16, in
api(args)
File "D:\waifuset\api.py", line 7, in api
ui: gr.Blocks = create_ui(args)
File "D:\waifuset\waifuset\ui\ui.py", line 184, in create_ui
univset, buffer, sample_history, tag_priority_manager, subset, waifu_tagger, waifu_scorer, tag_table, tag_feature_table = init_everything()
File "D:\waifuset\waifuset\ui\ui.py", line 172, in init_everything
main_dataset = prepare_dataset(univargs)
File "D:\waifuset\waifuset\ui\ui.py", line 86, in prepare_dataset
dataset = UIDataset(
File "D:\waifuset\waifuset\ui\ui_dataset.py", line 342, in init
if img_key not in database:
TypeError: argument of type 'NoneType' is not iterable

new hand

Although, this is a disaster for beginners, I don't even know how to open the ui interface......QAQ

奇怪的报错

佬,这个每次打开会报下错,重装一次又会好

tag table not found in root: /root/sd-dataset-manager/waifuset/waifuset
[ui_dataset] synchronous txts R/W
[ui_dataset] reading biaoqing: 100%|█████| 486/486 [00:00<00:00, 87388.83it/s]
[ui_dataset] info: total=181 | buffer=115 | categories=3
Traceback (most recent call last):
File "/root/sd-dataset-manager/waifuset/api.py", line 16, in
api(args)
File "/root/sd-dataset-manager/waifuset/api.py", line 7, in api
ui: gr.Blocks = create_ui(args)
File "/root/sd-dataset-manager/waifuset/waifuset/ui/ui.py", line 253, in create_ui
query_include_tags = gr.Dropdown(
File "/root/miniconda3/lib/python3.10/site-packages/gradio/components/dropdown.py", line 101, in init
raise ValueError(
ValueError: Custom values are not supported when multiselect is True.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.