hashicorp / copywrite

Automate copyright headers and license files at scale

copywrite

This repo provides utilities for managing copyright headers and license files across many repos at scale.

You can use it to add or validate copyright headers on source code files, add a LICENSE file to a repo, report on what licenses repos are using, and more.

Getting Started

The easiest way to get started is to use Homebrew:

brew tap hashicorp/tap
brew install hashicorp/tap/copywrite

Installers for Windows, Linux, and MacOS are also available on the releases page.

CLI Usage

This Go app is consumable as a command-line tool. Currently, the following subcommands are available:

❯ copywrite
Copywrite provides utilities for managing copyright headers and license
files in HashiCorp repos.

You can use it to report on what licenses repos are using, add LICENSE files,
and add or validate the presence of copyright headers on source code files.

Usage:
  copywrite [command]

Common Commands:
  headers     Adds missing copyright headers to all source code files
  init        Generates a .copywrite.hcl config for a new project
  license     Validates that a LICENSE file is present and remediates any issues if found

Additional Commands:
  completion  Generate the autocompletion script for the specified shell
  debug       Prints env-specific debug information about copywrite
  dispatch    Dispatches audit jobs for a list of repos
  help        Help about any command
  report      Performs a variety of reporting tasks

Flags:
      --config string   config file (default is .copywrite.hcl in current directory)
  -h, --help            help for copywrite
  -v, --version         version for copywrite

Use "copywrite [command] --help" for more information about a command.

To get started with Copywrite on a new project, run copywrite init, which will interactively help generate a .copywrite.hcl config file to add to Git.

The most common command you will use is copywrite headers, which will automatically scan all files in your repo and copyright headers to any that are missing:

copywrite headers --spdx "MPL-2.0"

You may omit the --spdx flag if you add a .copywrite.hcl config, as outlined here.

--plan Flag

Both the headers and license commands allow you to use a --plan flag, which performs a dry-run and will outline what changes would be made. This flag also returns a non-zero exit code if any changes are needed. As such, it can be used to validate if a repo is in compliance or not.

Config Structure

💡 You can automatically generate a new .copywrite.hcl config with the copywrite init command.

A .copywrite.hcl file can be referenced to provide configuration information for a given project. This file should be specific to each repo and checked into git. If no configuration file is present, default values will be used throughout the copywrite application. An example config structure is shown below:

# (OPTIONAL) Overrides the copywrite config schema version
# Default: 1
schema_version = 1

project {
  # (OPTIONAL) SPDX-compatible license identifier
  # Leave blank if you don't wish to license the project
  # Default: "MPL-2.0"
  license = "MPL-2.0"

  # (OPTIONAL) Represents the copyright holder used in all statements
  # Default: HashiCorp, Inc.
  # copyright_holder = ""

  # (OPTIONAL) Represents the year that the project initially began
  # Default: <the year the repo was first created>
  # copyright_year = 0

  # (OPTIONAL) A list of globs that should not have copyright or license headers .
  # Supports doublestar glob patterns for more flexibility in defining which
  # files or folders should be ignored
  # Default: []
  header_ignore = [
    # "vendor/**",
    # "**autogen**",
  ]

  # (OPTIONAL) Links to an upstream repo for determining repo relationships
  # This is for special cases and should not normally be set.
  # Default: ""
  # upstream = "hashicorp/<REPONAME>"
}

GitHub Authentication

Some commands interact directly with GitHub's API (especially when a .copywrite.hcl config is not present for the project). In order to use these commands successfully, multiple mechanisms are available to provide GitHub credentials and are prioritized in the following order:

• GitHub App credentials can be supplied via the APP_ID, INSTALLATION_ID, and APP_PEM environment variables or a .env file.
• A GITHUB_TOKEN environment variable can be used with a Personal Access Token
• If you use the GitHub CLI, auth information can automatically be used

If none of the above methods work, copywrite will default to using an unauthenticated client.

GitHub credentials are purposely excluded from the .copywrite.hcl config, as that file is meant to be specific to each project and checked in to its repo.

GitHub Action

To make it easier to use copywrite in your own CI jobs (e.g., to add a PR check), you can make use of the hashicorp/setup-copywrite GitHub Action. It automatically installs the binary and adds it to your $PATH so you can call it freely in later steps.

  - name: Setup Copywrite
    uses: hashicorp/setup-copywrite@867a1a2a064a0626db322392806428f7dc59cb3e # v1.1.2

  - name: Check Header Compliance
    run: copywrite headers --plan

💡 Running the copywrite command with the --plan flag will return a non-zero exit code if the repo is out of compliance.

Pre-Commit Hooks

Copywrite can be used as a Pre-Commit Hook for those looking to add copyright headers during local development. A list of supported hooks can be found in here, but the most common use case for adding missing copyright headers can be done by adding the following snippet to your repo's .pre-commit-config.yaml:

  - repo: https://github.com/hashicorp/copywrite
    rev: v0.15.0 # Use any release tag
    hooks:
      - id: copywrite-headers

Debugging

Copywrite supports several built-in features to aid with debugging. The first and most commonly used one is configurable log levels. Copywrite checks the COPYWRITE_LOG_LEVEL environment variable to determine which verbosity to use. The following log levels are supported:

• trace
• debug
• info
• warn
• error
• off (disables logging)

Copywrite also checks for if the RUNNER_DEBUG=1 environment variable is set, which will cause it to default to debug-level logging. This environment variable is set by Github Actions when in debug mode, and can be a useful default. The COPYWRITE_LOG_LEVEL setting takes precedence, however.

It is often useful to introspect information about the state Copywrite finds itself in. The copywrite debug command can print the running configuration, whether or not a config file was loaded, what GitHub auth type is in use, and more. No sensitive information is printed, however.

Development

To maintain a consistent developer experience, this repo comes bundled with VS Code settings. When opening the repo for the first time, you will be asked if you want to install suggested extensions and your workspace will be pre-configured with consistent format-on-save settings.

Before committing code, this repo has been setup to check Go files using pre-commit git hooks. To leverage pre-commit, developers must install pre-commit and associated tools locally:

brew install pre-commit golangci-lint go-critic

Verify install went successfully with:

pre-commit --version

Once you verify pre-commit is installed locally, you can use pre-commit git hooks by installing them in this repo:

pre-commit install

copywrite's Issues

Callout setup-copywrite Usage with Major Version Tag or Commit SHA

Description

HashiCorp repositories with GitHub Actions workflows will soon be enforced to use commit SHA pinning in uses: to curb potential supply chain issues. Currently the README.md section on GitHub Actions points to @main for using setup-copywrite.

Annoying to update (🙁 ) but I'm guessing the preference for the documentation would be to suggest:

uses: hashicorp/setup-copywrite@3ace06ad72e6ec679ea8572457b17dbc3960b8ce # v1.0.0

Or (less securely):

uses: hashicorp/[email protected]

Either which way though, Dependabot should suggest version updates.

Scope of tool

Is the scope of this tool Hashicorp only?

The read me suggests it is intended to be generic, but there non-configurable items such as

copywrite/cmd/license.go

Line 68 in 4cf18ec

copyright := "Copyright (c) " + strconv.Itoa(conf.Project.CopyrightYear) + " HashiCorp, Inc."

which bind it to Hashicorp.

Thanks!

Pre-commit installation fails

It seems like, if i use this tool with pre-commit, it tries to search Go files not in pre-commit .cache directory, but from root path of the repo. Could you please check with some empty git repo to use this pre-commit configuration to get same errors.

After creating .pre-commit-config.yaml and commiting files \ executing pre-commit run

repos:
- repo: https://github.com/hashicorp/copywrite
  rev: v0.16.4
  hooks:
    - id: add-headers
    - id: check-headers

have such error:

Add copyright headers....................................................Failed
- hook id: add-headers
- exit code: 1
go: cannot find main module, but found .git/config in /Users/gabenov/git/sandbox/pre-commit
	to create a module there, run:
	go mod init

or after executing go mod init pre-commit

Validate copyright headers...............................................Failed
- hook id: check-headers
- exit code: 1
no Go files in /Users/gabenov/git/sandbox/pre-commit

version: pre-commit 3.3.2

Allow suppressing "skipping" log lines (by default?)

Context

Currently copywrite headers can be quite chatty on bigger repositories, practically printing out every single file, esp. because it prints out the ones it skips. I can see how this can be useful for debugging!

I would argue though this is far less useful in day2day use, esp. when it's used to just add headers to a handful of new files. The new files just get drowned in the long output.

Proposal

Allow suppressing/hiding the cli: skipping: log lines, or even making that the default option, with a -verbose or -debug flag to enable them.

Pre-Commit Hook

Interesting project. Please consider adding a .pre-commit-hooks.yaml so this can be utilised as a pre-commit hook also.

Reflect nested config files when walking directories

When a repository uses a mix of different licenses for different parts/directories (BUSL or MPL), then it is currently necessary to run copywrite in those nested directories in order for it to notice those nested directories.

For example, Terraform uses the following script to work around this limitation:

https://github.com/hashicorp/terraform/blob/2a5ff48e3d0342c1978adf97c63414fa06c8b9ed/scripts/copyright.sh#L12-L17

directories=$(find . -type f -name '.copywrite.hcl' -execdir pwd \;)

for dir in $directories; do
    cd $dir && go run github.com/hashicorp/copywrite headers
done

It would be great if this could be built into the tool, so we would not have to maintain this shell wrapper.

Prefix copyright text with SPDX tag

Currently the copyright text is provided as a comment.

In the same way the license is prefixed with the SPDX-License-Identifer: tag, should this be prefixed with the SPDX-FileCopyrightText: tag per https://spdx.github.io/spdx-spec/v2.3/file-tags/#h2-file-tags-format ?

Error: API rate limit exceeded

👋 Hi! We're using this GH action in the TFE provider, and with a recent run, I'm seeing the error:

GET /repos/hashicorp/copywrite/releases/latest - 403 in 119ms
Error: API rate limit exceeded for 40.77.45.146. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

copywrite headers break build tags in Golang

Go build tags need to be at the top line of the file to be considered by go compiler.

Right now, running copyright headers in go file with build tags would break the tags as the header will be inserted before the build tags.

The expected result is for copywrite headers to either:

• offer a way to exclude/include files based on build tags
• detect build tags and insert the header after

Add debug or trace output when files with unsupported extensions are found

When you run copywrite on files with unsupported extensions, you get no information that these files are skipped.

Getting a debug output of why the file was skipped would be nice.

The only other way to find out is to look at the source code.

It was discussed during a review: #85 (comment)

Update Copyright year

I want to manage not only copyright header, but also years in it.

For example, my header should look like:
copyright_holder = "HashiCorp, Inc. 2014-2023"

1. So, when i run copywrite headers over file having already created header "HashiCorp, Inc. 2014-2022" will be changed into --> "HashiCorp, Inc. 2014-2023"
2. .copywrite.hcl config file should support variables, so i can template headers like

copyright_holder = "HashiCorp, Inc. ${var.copyright_year}-2023"
copyright_year = 2014

headers plan: command output lists ignored files as missing headers

When running the copywrite headers --plan command within the Nomad repository, the output details files missing headers even though these are ignored via the .copywrite.hcl file and the command exits with status code 0. It is a little confusing to detail these files when they are expected to be ignored.

Example output:

The following files are missing headers:
  
  Using copyright holder: HashiCorp, Inc.
  
  command/asset/*.hcl
  command/agent/bindata_assetfs.go

Example GHA run: https://github.com/hashicorp/nomad/actions/runs/5519757257/jobs/10065442205?pr=17889

Support overriding `ensureCorrectName` (e.g. support `LICENSE.md` instead of `LICENSE`)

Can we please have a feature that lets us turn off this error if the file extension is present, e.g. .md?

copywrite/licensecheck/licensecheck.go

Line 21 in 807a790

desiredPath := filepath.Join(dir, "LICENSE")

Updating scripts without extensions

As I'm testing I've noticed copywrite correctly updates scripts that have a file extension, e.g. .sh, .js etc. but doesn't update scripts that don't.

E.g. it would detect and update check-release.sh but not check-release

Is there a way to update this files, either through explicit configuration or automagically?

YAML files with directives

This is probably more a stylistic thing but if you have a yaml file

%YAML 1.2
# yaml-language-server: $schema=https://json.schemastore.org/pre-commit-config.json
---
default_install_hook_types:
  - pre-commit
...

and update it with copywrite headers you end up with

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

%YAML 1.2
# yaml-language-server: $schema=https://json.schemastore.org/pre-commit-config.json
---
default_install_hook_types:
  - pre-commit
...

This contrasts with the behaviour of shell scripts which maintain the shebang as the first line.

I don't think it particularly matters for TAG directives but in the case of the YAML directive should this be handled similarly?

`header_ignore` pattern not reflected

I tried onboarding this repository https://github.com/hashicorp/vscode-terraform and added testFixture/ into the ignorelist:

schema_version = 1

project {
  license        = "MPL-2.0"
  copyright_year = 2016

  header_ignore = [
    ".vscode-test",
    ".vscode-test-web",
    "out/",
    "node_modules/",
    "syntaxes/",
    "testFixture/",
  ]
}

Despite that copywrite headers attempts to add the header to files under that path:

copywrite headers --plan
Executing in dry-run mode. Rerun without the `--plan` flag to apply changes.

Using license identifier: MPL-2.0

Exempting the following search patterns:
.vscode-test
.vscode-test-web
out/
node_modules/
syntaxes/
testFixture/

The following files are missing headers:
...
2023-01-31T17:22:18.563Z [INFO]  cli: testFixture/actions.tf
2023-01-31T17:22:18.563Z [INFO]  cli: testFixture/modules/sample.tf
2023-01-31T17:22:18.564Z [INFO]  cli: testFixture/sample.tf
Error: missing license header

Is this a bug?

Copyright headers assume copyright assignment to HashiCorp

The bot is creating PRs with headers like

Copyright (c) HashiCorp, Inc.

which appear to assume HashiCorp is the sole copyright holder for all files. How does the bot check that external contributors assigned copyright to HashiCorp?

For example this PR seems definitely incorrect: hashicorp/terraform-provider-azurerm#22657 (comment)

For reference see also https://reuse.software/faq/#copyright-holder-author

Headers Command .headerignore File Support

Description

Hey there 👋 Thanks for this great tool! While looking at some of the configuration file handling for this tool, I noticed:

copywrite/cmd/headers.go

Lines 26 to 28 in 03e9b36

	will automatically be exempted. Any other files or folders should be added to the
	.headerignore file that is expected to be at the root of the directory you execute
	this command in.`,

Which is the help description from the headers command that talks about a .headerignore file. However, in my brief code searching here, I'm not seeing that text appear elsewhere such as code which might load those files.

Is this functionality supported, upcoming, or was it replaced with the "regular" configuration file? Much appreciated!

Incorrect license is not detected/fixed

It appears the copywrite will not detect or attempt to fix wrong license, for example in hashicorp/terraform:

If I attempt to change provider_source.go in root to contain

// SPDX-License-Identifier: MPL-2.0

$ copywrite headers
Using license identifier: BUSL-1.1
Using copyright holder: HashiCorp, Inc.

Exempting the following search patterns:
**/*.tf
**/testdata/**
**/*.pb.go
**/*_string.go
**/mock*.go
docs/plugin-protocol/**
internal/tfplugin*/**

The following files are missing headers:

$ head -2 provider_source.go
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0

autodiscover ignores discovery errors

I ran copywrite init without a LICENSE file and hit a panic. I was able to work around by specifying --spdx flag. I noticed here:

• *data may be nil if an error is returned from Get
• GetLicense() may return nil if no LICENSE is added to the repo yet

vs code settings

The read me makes reference to vs code settings being in the repo to ensure a consistent developer experience however this are not actually present.

s390x Support

In an attempt to build the Vault Secrets Operator for s390x architecture, I ran into issues because copywrite does not support s390x but is required by VSO. Is s390x support possible/on the roadmap?