atc0005 / elbow Goto Github PK

By the time the command-line flags are parsed the config object field values have already been "logged", so even when you opt to see those field values at debug level the default logging level of info prevents you from doing so.

refs #7

As of v0.3.1, only one path is supported at a time.

This should probably go in before work on #31 commences.

Based on some reading in https://github.com/golang/go/wiki/Modules, the behavior for go get -u changed to allow more conservative updates of dependencies. The new behavior sounds more natural and is less likely to surprise newcomers, so locking the base behavior to Go 1.13 sounds like a "Good Thing" to do here.

As part of this, I should touch the docs to list Go version 1.13 as the desired Go version. If I learn that this change doesn't impact older (still supported) Go versions, I can update the docs again to include the older version as the minimum required.

Since syslog output is attempted, providing a template for catching and routing the log data could be useful. If providing this snippet, also providing a logrotate snippet would be a good idea.

The idea is to either use hard-coded settings or the standard library to support basic options via positional command-line arguments.

The core concept is an app that can be pointed at a target directory and allowed to recursively prune all files matching specific patterns.

Those patterns can be any of:

• date (modified X days in the past)
• filename (e.g., date/time naming scheme)
• extension (e.g., .tmp)
• ?

The idea is that this tool would be called by a cron job or as needed as a one-shot run to perform cleanup work.

Based on light reading, it doesn't appear that I have to include a NOTICE file, but it appears to be good practice to do so. The intent seems to be to include a quick reference for various licenses attached to the product.

In this project's case, I assume this includes all libraries/packages that are pulled in. I'm not fluent enough with licensing to know whether it applies to any code present in the repo, or just the official releases. In other words, if I have a temporary feature branch that I'm using to test various packages, does the NOTICE file need to be updated for each package that I test or just when an official release is made?

I assume that a good checklist item for tagging releases or (perhaps) even merging a Pull Request is to make sure that licensing is covered?

The v1.2.0 release provides support for configuring defaults directly on the configuration struct. Presumably this also opens up the option for better Help output in order to better communicate those default values to the user.

This is already listed at DEBUG log level, so this option would be enabling reporting of eligible files in some sort of format that is easily read and shared (e.g., logged to a ticket of some kind).

Example/concept usage:

sudo ./elbow --paths /var/log/tomcat7/ --recurse --age 120 --generate-report

Regarding an output format, I'm not sure what would work best. Hopefully something concise for the screen and something else to be saved in a file for later review.

https://github.com/schollz/progressbar

Quite a number of small improvements since the v0.5.0 release. This is a good stopping point before beginning work to implement support for configuration files.

Per the Tools page:

[![GoDoc](https://godoc.org/github.com/atc0005/elbow?status.svg)](https://godoc.org/github.com/atc0005/elbow)

Reasoning: Should make it easier for future readers to quickly view GoDoc documentation for this project (which admittedly is nearly nonexistent at the moment).

See goracle README for inspiration.

Whether through an application loop using one or more timers or a true service/daemon configuration, support having the application repeat its removal process.

Either before or as preparation for this, support could be added to process multiple paths in one invocation. As of v0.3.1, only one path is supported at a time.

Probably worth going ahead and add it now before going much further with the project. I need to look up the templates site that I came across a few years back in order to get a good refresher on the format.

Instead of using a fully-qualified path to the file, the cleanPath() function attempts to remove the file using the bare or "short" name only. When the current working directory is not the same directory where the target file is present, this results in a failed file removal.

This text was left behind:

Placeholder. Add table of all command-line flags here with a very brief
explanation of what they're used for and whether they're required (should be
listed first) or optional (listed last).

Goal

This might be somewhat determined by what packages I decide to pull in, but if not, I'll need to pick something flexible enough that someone would want to use this as a base for their own projects (e.g., someone entirely new to Go like myself).

Packages I'm considering

Resources

• https://choosealicense.com/licenses/

On a whim, I fired up a clean Ubuntu 18.04 LXD container and proceeded with my attempts to build this application using Go version 1.13.1.

• ubuntu@ubuntu-1804-golang-testing:~/elbow$ GOOS=linux go build -o elbow
• ubuntu@ubuntu-1804-golang-testing:~/elbow$ GOOS=linux go build -a -ldflags="-s -w

The first command works without issue. Using the second variant I get a complaint that gcc wasn't found.

Here are the (base) packages which I had to install to make this process work using the provided Makefile:

• make
• upx
• gcc

From there all of the Makefile recipes worked as expected.

Based on light reading, this header should be at the top of each source file so that the license is clearly noted.

This page covers the specifics well, so I should use it as a reference:

https://www.openntf.org/Internal/home.nsf/dx/Applying_Apache_License

Example output from a recent live run against a Tomcat 7 log directory:

INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-14.txt-20190615.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-15.txt-20190616.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-16.txt-20190617.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-17.txt-20190618.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-18.txt-20190619.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-19.txt-20190620.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-20.txt-20190621.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-21.txt-20190622.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-22.txt-20190623.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-23.txt-20190624.bz2 removal_enabled=true
INFO[0000] Removing file                                 file=/var/log/tomcat7/old/localhost_access_log.2019-06-24.txt-20190625.bz2 removal_enabled=true
INFO[0000] 1576 files successfully removed
INFO[0000] localhost_access_log.2015-04-23.txt-20160906.bz2  failed_removal=false
INFO[0000] localhost_access_log.2015-04-24.txt-20160906.bz2  failed_removal=false
INFO[0000] localhost_access_log.2015-04-25.txt-20160906.bz2  failed_removal=false
INFO[0000] localhost_access_log.2015-04-26.txt-20160906.bz2  failed_removal=false

I don't yet know the best way to handle listing/logging the latter items showing what did occur, but logging what was going to occur seems like duplicate information based on the default INFO log level.

A good first start here would be setting the former log messages to DEBUG level so that they're muted by default. The X files successfully removed message would explicitly remain INFO level and the latter messages better evaluated as part of the work on #72.

This should do it once this project releases a stable/normal release:

[![Latest Release](https://img.shields.io/github/release/atc0005/elbow.svg?style=flat-square)](https://github.com/atc0005/elbow/releases/latest)

Example:

https://github.com/peterbourgon/ff/blob/master/README.md

If recursively, we need to create a FileInfo type from each matched file in order to sort by various attributes for later processing.

https://golang.org/pkg/os/#FileInfo

Examples:

• Size
• ModTime

Processing of a specific directory might be easier to implement.

Boolean flag to determine whether recursive processing is used?

While working on #54 and going through the docs/notes for git describe I noticed that only the v0.1.0 tag showed in the git describe output unless I used the --tags option to display both annotated tags (default) as well as lightweight tags.

Current tags:

$ git for-each-ref refs/tags
665a70d2c1cd1b8ead54fea5ba7816a77a74f9eb tag    refs/tags/v0.1.0
1508ae84959b21a79aa09e25f166e6bc6d10a425 commit refs/tags/v0.2.0
1b191cd4ca728f7a40f300406e98e1687c56ecbd commit refs/tags/v0.3.0
91c177c29c196d33177b4f1c3a677297d27cf428 commit refs/tags/v0.3.1
64d55334d64bf11d9e432808e7c0456e2d4f828b commit refs/tags/v0.3.2

It seems that "cutting a release" and having GitHub handle tagging results in lightweight tags only.

Light research indicates that you can convert lightweight tags to annotated tags like so: git tag -a <tag_name> <SHA-1> -f

References:

• https://stackoverflow.com/questions/5002555/can-a-lightweight-tag-be-converted-to-an-annotated-tag
  • https://stackoverflow.com/a/5002886/903870
• https://stackoverflow.com/questions/40479712/how-can-i-tell-if-a-given-git-tag-is-annotated-or-lightweight
  • https://stackoverflow.com/a/40499437/903870
• https://stackoverflow.com/questions/21690298/how-to-create-a-github-release-with-an-annotated-tag
  • https://stackoverflow.com/a/30302668/903870

Ideas here:

• https://gist.github.com/TheHippo/7e4d9ec4b7ed4c0d7a39839e6800cc16
• https://www.reddit.com/r/golang/comments/4cpi2y/question_where_to_keep_the_version_number_of_a_go/
• https://stackoverflow.com/a/15217754/903870

This can be done per file in the logged messages and as a cumulative number in the INFO level message which is most likely to be printed to the console.

This could be particularly useful for summarizing what would be removed if a particular set of options are applied (e.g., what is the size difference with keeping 120 days worth of a type of file vs 60 days).

Example here:

https://github.com/laher/goxc/tree/master/debian

Overview

Brief testing shows that running go against a UPX compressed binary to determine version information doesn't work well. Uncompressed, good results can be obtained via either of:

• go version BINARY_NAME
• go version -m -v BINRARY_NAME

Being able to determine that build information (particularly via automated means) is likely a bigger perk than reducing the initial file size.

Example output

Examining compressed binary

$ go version elbow
elbow: go version not found

$ go version -m -v elbow
elbow: go version not found

Decompress binary

$ upx -d elbow
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94        Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   2887680 <-    876564   30.36%   linux/amd64   elbow

Examine decompressed binary

$ go version elbow
elbow: go1.13.3

$ go version -m -v elbow
elbow: go1.13.3
        path    github.com/atc0005/elbow
        mod     github.com/atc0005/elbow        (devel)
        dep     github.com/alexflint/go-arg     v1.1.0  h1:92ADei0d3TP0mGBdJ/FNcF54X6uFY7BQfhqkrQt3CCE=
        dep     github.com/alexflint/go-scalar  v1.0.0  h1:NGupf1XV/Xb04wXskDFzS0KWOLH632W/EO4fAFi+A70=
        dep     github.com/sirupsen/logrus      v1.4.2  h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
        dep     golang.org/x/sys        v0.0.0-20190422165155-953cdadca894      h1:Cz4ceDQGXuKRnVBDTS23GTn/pU5OE2C0WrNTOYK1Uuc=

This should have gone in with the creation of the file, but better late than never.

refs: #14, #15

For the short term, modify the small shell script called by the Makefile (and the Makefile itself) to allow overriding the default location when using the testenv make option.

The task of updating this method to reflect new fields was evidently missed when working on #10.

While actually trying to use the app myself today, I went to match files based on age and decided I would use the existing README for guidance. I did not find the option listed, aside from further down the page in the "Help Output".

Oops?

refs #10, #27

I've been using the standard library log package and while it works, it seems to be lacking the basic leveled logging features I've come to rely upon over the years (e.g., logger from shell scripts, Python's logging module, ...).

There are a lot of options available, but I'm going to focus first on trying those which are most actively supported and used by some of the bigger teams/projects.

Right now we're keeping X number of files and relying on filename patterns to help limit the files that are removed. To make this tool more useful, it will need to support age-based matching as well. Other criteria can be added later (e.g., size), but this one seems to be the more useful additoin to include first.

Match age in minutes, hours, days, ... ?

This would be useful not only for those reviewing the source, but for GoDoc generated coverage.

Example:

https://github.com/assafmo/joincap/blob/master/main.go

Snippet from the beginning of the file:

// Merge multiple pcap files together, gracefully.
//
//  Usage:
//    joincap [OPTIONS] InFiles...
//
//  Application Options:
//    -v, --verbose  Explain when skipping packets or entire input files
//    -V, --version  Print the version and exit
//    -w=            Sets the output filename. If the name is '-', stdout will be used (default: -)
//
//  Help Options:
//    -h, --help     Show this help message
package main

import (

The existing file is pretty thin; the README doesn't yet include references to any of these files or much detail on the various recent changes (e.g, command-line flags).

As of v0.5.0, godoc does not render any useful text to the reader, either via local godoc binary or godoc.org.

Evidently what I've done thus far isn't sufficient to even provide the overview text to the reader.

Lots of options to choose from. Since this is an Open Source project, funding doesn't appear to be an issue (presumably based on low usage).

This is inspired by #76, but also because I've known for a while that this might eventually be needed. I'm hoping to better isolate code from the main package in the hope/belief that this will make setting up future tests easier and more reliable.

go-flags provides the default:"" struct tag for specifying default values if one is not provided for the specified flag. The current code sets an empty string (explicitly) and Go's default "zero" value for a string is already "", so I'm not sure we really need to spell it out on the struct tag also, but perhaps it could be useful for generation of the help text?

When implemented, the traditional precedence of .env files should be followed.

• https://github.com/bkeepers/dotenv#what-other-env-files-can-i-use
• https://github.com/joho/godotenv#precendence--conventions

This would modify the load order to look something like this:

1. Default values
2. Existing environment variables
3. gotdotenv import of .env files
4. Configuration file
5. Command-line flags

As of this writing we're using the alexflint/go-arg package for flags and environment variable support. My understanding is that by using the godotenv package to load .env files the alexflint/go-arg package will automatically "see" the environment variables and use them. This should make the existing configuration setup seamless.

Minor nit, but worth fixing.

-specific language governing permissions and limitations under the Licens
+specific language governing permissions and limitations under the License.

Not yet sure which package, but this tool will need to be heavily command-line driven.

Standard library

• https://golang.org/pkg/flag/

  • https://gobyexample.com/command-line-flags

• https://gobyexample.com/command-line-arguments

Third-party

• https://github.com/urfave/cli
• https://github.com/peterbourgon/ff
• https://github.com/integrii/flaggy

Implementation / Examples

• https://github.com/tsenart/vegeta/blob/master/main.go
  • looks like they're using the stdlib flag package, quite a bit of validation wrapping

This can either come as part of using another package, or sometime after adding support for config files (#1) and command-line (#2).

At least one set of the build options should include optimized builds for size and speed (to test if nothing else).

CHANGELOG updates, new tag, publish release.

This would be another "pre-release" as the codebase is still not as stable or feature complete as I would like.

Now that the alexflint/go-arg package supports it, update the Config struct to specify defaults settings. The assumption is that this will apply the default settings there which could potentially be used/respected by other configuration packages that reference a default struct tag.

The primary hope/goal is that this will better display default values to the user in the Help output.

This is handled now via syslog + whatever output is chosen (log file, or one of stdout/stderr), but if using an io.MultiWriter the user could choose to log to ALL chosen outputs simultaneously.

Example:

package logging

import (
    "io"
    logging "log"
    "os"

    "github.com/Sirupsen/logrus"
)

var (
    log *logrus.Logger
)

func init() {
    f, err := os.OpenFile("logs/application.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)

    if err != nil {
        logging.Fatalf("error opening file: %v", err)
    }

    log = logrus.New()

    //log.Formatter = &logrus.JSONFormatter{}

    log.SetReportCaller(true)

    mw := io.MultiWriter(os.Stdout, f)
    log.SetOutput(mw)
}

// Info ...
func Info(format string, v ...interface{}) {
    log.Infof(format, v...)
}

// Warn ...
func Warn(format string, v ...interface{}) {
    log.Warnf(format, v...)
}

// Error ...
func Error(format string, v ...interface{}) {
    log.Errorf(format, v...)
}

var (

    // ConfigError ...
    ConfigError = "%v type=config.error"

    // HTTPError ...
    HTTPError = "%v type=http.error"

    // HTTPWarn ...
    HTTPWarn = "%v type=http.warn"

    // HTTPInfo ...
    HTTPInfo = "%v type=http.info"
)

package main

import (

    log "logging"
)

func main() {
    log.Error(log.ConfigError, "Testing the error")
}

Reference:

https://stackoverflow.com/a/55989885/903870

Command that I ran:

ubuntu@TESTBOX:/tmp$ cd /mnt/t/github/elbow; go build && cp -vf elbow /tmp/; cd /tmp/; ./elbow --path /tmp --pattern "reach-masterdev" --keep 1 --recurse --keep-old
 --ignore-errors --age 7 --extension ".tmp" --extension ".war" --log-level info --log-format text

Output:

'elbow' -> '/tmp/elbow'
INFO[0000] Starting evaluation of path                   extensions="[.war]" file_age=7 file_pattern=reach-masterdev path=/tmp

I expected to see something like:

'elbow' -> '/tmp/elbow'
INFO[0000] Starting evaluation of path                   extensions="[.war .tmp]" file_age=7 file_pattern=reach-masterdev path=/tmp

$ golangci-lint run
matches.go:173:23: func `FileMatches.sortByModTimeDesc` is unused (unused)
func (fm FileMatches) sortByModTimeDesc() {
                      ^
config.go:42:26: SA5008: duplicate struct tag "arg" (staticcheck)
        NumFilesToKeep int      `arg:"--keep,env:ELBOW_KEEP" arg:"required" help:"Keep specified number of matching files per provided path."`
                                ^
config.go:51:27: SA5008: duplicate struct tag "arg" (staticcheck)
        Paths           []string `arg:"--paths,env:ELBOW_PATHS" arg:"required" help:"List of comma or space-separated paths to process."`
                                 ^

Need to take a closer look at the struct tags to make sure I've defined them properly.

While this tool will be heavily command-line driven, there are likely to be several good use cases for laying out all desired options via configuration file.

The specific format is yet to be determined.

Potential options:

• YAML
• JSON (not likely)
• TOML (instead of INI)
• flat, space-separated text file