GithubHelp home page GithubHelp logo

tencentblueking / bkpaas-python-sdk Goto Github PK

View Code? Open in Web Editor NEW
26.0 12.0 26.0 1.5 MB

蓝鲸 PaaS 平台 Python 工具集

License: MIT License

Python 98.51% Makefile 0.35% Dockerfile 0.07% Shell 0.70% Smarty 0.36%
python sdk devops paas

bkpaas-python-sdk's Issues

bkstorages 在 django 3.2.13 版本使用异常

背景信息:

django 3.2.13 版本, django 默认的Storage 新增了 validate_file_name 方法,该 方法默认接收一个相对路径:
image

image

在最新的 FileSystemStorage 实现中, _save 方法接收一个绝对路径,但是会将路径转换成相对路径,保证下一步的 validate_file_name 不出错:
image

image

paht = "/app/xxxx"
经过_save() 处理之后 会变为 app/xxxx

bkstorages 使用制品库时使用 BKRepoStorage 实例。该实例的 _save() 方法并未将绝对路径转化为相对路径的操作。

image

于是就会引发异常:

image

bug: BKRepoStorage file_overwrite 不生效

提要

首先不知道是不是我对 file_overwrite 的释义有误解,在我的理解上,file_overwrite 表示同名文件会被覆盖
❌ 目前出现的问题: file_overwrite=True 时,同名文件上传后并未覆盖,而是原文件名拼接随机字符串后缀并另存

  • 例如 -> 仓库中已存在文件 1.tar.gz ,最后是多出一个 1_6OFSWbn.tar.gz的文件,期望是覆盖原文件保存

问题定位

Django 提供的 Storage 在 save 时,并不直接使用传入的 name,而是通过 get_available_name 方法,检测重名及文件路径长度,如果重名,该方法会生成随机串拼接到原文件名直到没有文件重复:

def get_available_name(self, name, max_length=None):
    """
    Return a filename that's free on the target storage system and
    available for new content to be written to.
    """
    dir_name, file_name = os.path.split(name)
    file_root, file_ext = os.path.splitext(file_name)
    # If the filename already exists, add an underscore and a random 7
    # character alphanumeric string (before the file extension, if one
    # exists) to the filename until the generated filename doesn't exist.
    # Truncate original name if required, so the new filename does not
    # exceed the max_length.
    while self.exists(name) or (max_length and len(name) > max_length):
        # file_ext includes the dot.
        name = os.path.join(dir_name, "%s_%s%s" % (file_root, get_random_string(7), file_ext))
        if max_length is None:
            continue
        # Truncate file_root if max_length exceeded.
        truncation = len(name) - max_length
        if truncation > 0:
            file_root = file_root[:-truncation]
            # Entire file_root was truncated in attempt to find an available filename.
            if not file_root:
                raise SuspiciousFileOperation(
                    'Storage can not find an available filename for "%s". '
                    'Please make sure that the corresponding file field '
                    'allows sufficient "max_length".' % name
                )
            name = os.path.join(dir_name, "%s_%s%s" % (file_root, get_random_string(7), file_ext))
    return name

如果配置中提供文件覆盖的选项,那我觉得是需要重写该方法,在self.fileoverwrite=True时,对该方法的行为进行改写

因为在 节点管理 -> https://github.com/TencentBlueKing/bk-nodeman 我们对自定义的 Storage 都有覆盖写的需求,所以我的处理方式是直接重写了 get_available_name 方法,加上了 self.fileoverwrite的判断,定义了 StorageFileOverwriteMixin

class StorageFileOverwriteMixin:

  file_overwrite = settings.FILE_OVERWRITE

  def get_available_name(self, name, max_length=None):
      """重写获取文件有效名称函数,支持在 file_overwrite=True 时不随机生成文件名"""

      dir_name, file_name = os.path.split(name)
      file_root, file_ext = os.path.splitext(file_name)

      def _gen_random_name(_file_root) -> str:
          # 在文件名的起始位置添加随机串,源码规则为 "%s_%s%s" % (_file_root, get_random_string(7), file_ext)
          # 上述规则对 .tar.gz 不友好,会在类型后缀中间加随机串,所以改为随机串作为前缀
          return os.path.join(dir_name, "%s_%s%s" % (get_random_string(7), _file_root, file_ext))

      # not self.file_overwrite and self.exists(name) 利用 and 短路特点,如果 file_overwrite=True 就无需校验文件是否存在
      while (not self.file_overwrite and self.exists(name)) or (max_length and len(name) > max_length):
          # file_ext includes the dot.
          name = name if self.file_overwrite else _gen_random_name(file_root)

          if max_length is None:
              continue
          # Truncate file_root if max_length exceeded.
          truncation = len(name) - max_length
          if truncation > 0:
              file_root = file_root[:-truncation]
              # Entire file_root was truncated in attempt to find an available filename.
              if not file_root:
                  raise SuspiciousFileOperation(
                      'Storage can not find an available filename for "%s". '
                      "Please make sure that the corresponding file field "
                      'allows sufficient "max_length".' % name
                  )
              name = name if self.file_overwrite else _gen_random_name(file_root)
      return name

期待讨论该问题是否符合预期,以及是否有更优雅的处理,也很乐意提pr去修复该可能存在的问题~

关于 blue_krill安装出现哈希值不匹配问题

安装 blue_krill出现哈希如下:

Retrieved digest for link blue_krill-1.2.2-py3-none-any.whl(md5:c276504dade988f42f6d0fed21c608b8) not in poetry.lock metadata ['sha256:45a581dfbc3e2d9ee21928cd5e5538e3abad382176800105aee3ccf0f03b7aee', 'sha256:20dac3eed94866cffb4c1771a1eae7ab0f7ed4b877c90d931fb7fc36be8f6c03']

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.