GithubHelp home page GithubHelp logo

tessg22 / lightweight-curator Goto Github PK

View Code? Open in Web Editor NEW
0.0 0.0 5.0 109 KB

Lightweight job to delete old Elasticsearch indices from logging based on percentage

License: Apache License 2.0

Dockerfile 6.61% Makefile 1.19% Python 92.20%

lightweight-curator's People

Contributors

calvix avatar ewoutp avatar glitchcrab avatar hectorj2f avatar marians avatar tessg22 avatar theobrigitte avatar xh3b4sd avatar zeisss avatar

Watchers

 avatar  avatar

Forkers

openshift

lightweight-curator's Issues

Prefixes introduced from OpenShift 4.5

Where:
index_name_prefixes = os.getenv("INDEX_NAME_PREFIXES", "infra-,app-,audit-")

What:
Indices in this example with cluster version 4.4.16 don't have "infra-,app-,audit-" prefix convention.

green  open   project.snow-poker-feature-aloha-potatoes-tree.a41c5e5a1-0c45-4274-a426-7bfc539c6be.2020.09.23                   k8pwmycmRkGnYWrbx2OLMw   3   1         30            0    706.3kb        353.1kb
green  open   project.aloha-cherry.b03e1dfd4-fade-4771-be54-74dd11eb619.2020.09.23                                             5aXlxOh5TYSq51En78YLbA   3   1          7            0      171kb         85.5kb

Breached threshold limit

Where:
IF size_counter < max_allowed_size condition

What:
I tried to write down simple array of sizes [10, 10, 10, 10, 10] with threshold limit lets say 45.
When I go through first 4 items, condition would look like 40 < 45 which is CORRECT.
WRONG is that same condition would allow me to whitelist also last item:

  • Inside of the condition, while adding 10 + 40 == 50 into counter I would have already breached the limit. 

indices_size_counter is not getting incremented

Where:

indices_to_delete = []
indices_size_counter = 0  
for index in sorted(indices, key=lambda x: x.creation_date, reverse=True):  
    indices_smaller_then_max_allowed_size(index, max_allowed_size, indices_size_counter, indices_to_delete)

What:
Value of indices_size_counter is constantly 0 and not getting incremented inside of indices_smaller_then_max_allowed_size function.

Proposed changes:

indices_to_delete = []
indices_size_counter = 0  
for index in sorted(indices, key=lambda x: x.creation_date, reverse=True):  
    indices_to_delete, indices_size_counter = indices_smaller_then_max_allowed_size(index, max_allowed_size, indices_size_counter, indices_to_delete)
def indices_smaller_then_max_allowed_size(index, limit, indices_size_counter, indices_to_delete):
    """
    Returns list of indices which are above threshold limit.
    """
    expected_size = index.size + indices_size_counter

    if indices_size_counter < limit and expected_size < limit:
        indices_size_counter += index.size
        logger.info(f"Do not add into actionable list: {index.name}, summed disk usage is {indices_size_counter} B and disk limit is {limit} B")
    else:
        logger.warning(f"Add into actionable list: {index.name}, summed disk usage is {indices_size_counter} B and disk limit is {limit} B")
        indices_to_delete.append(index.name)

    return indices_to_delete, indices_size_counter

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.