imgbot / imgbot

An Azure Function solution to crawl through all of your image files in GitHub and losslessly compress them. This will make the file size go down, but leave the dimensions and quality untouched. Once it's done, ImgBot will open a pull request for you to review and merge. [email protected]

Imgbot

Imgbot crawls all your image files in GitHub and submits pull requests after applying a loss less compression. This will make the file size go down, but leave the dimensions and quality just as good.

Configuration

Imgbot supports optional configuration through a .imgbotconfig json file. This is not a required step to using Imgbot and is only for more advanced scenarios. This file should be placed in the root of the repository and set to your liking.

{
    "schedule": "daily", // daily|weekly|monthly
    "ignoredFiles": [
    	"*.jpg",                   // by extension
    	"image1.png",              // by filename
    	"public/special_images/*", // by folderpath
    ],
    "aggressiveCompression": "true", // true|false
    "compressWiki": "true", // true|false
    "minKBReduced": 500 // set reduction threshold (default to 10),
    "prTitle" : "Compressed images", // set pull request title
    // set the pull request body, supports any valid github markdown
    // {optimization_ratio} display a message containing the optimization ratio
    // {optimization_details} display the table containing the optimization details
    "prBody" : " Text before optimization ratio {optimization_ratio} Text after optimization ratio 
                Text before optimization details {optimization_details} Text after optimization details",
    
}

The following are the currently supported parameters. If there are any configuration settings you would like to see changed or supported, please feel free to open an issue here in the repo or shoot an email over to [email protected]

schedule

• Optional
• Accepts: daily|weekly|monthly
• Limits the PRs from Imgbot to once a day, once a week, or once a month respectively
• The default behavior is to receive Imgbot PRs as images require optimization

ignoredFiles

• Optional
• Accepts the syntax for searchPattern on Directory.EnumerateFiles()
• Limits the images optimized by Imgbot by essentially ignoring them
• When ignoring by file name no path is necessary, when ignoring by folder name full path from root is necessary

aggressiveCompression

• Optional
• Accepts: true|false
• Opt in to use lossy compression algorithms
• The default behaviour without this setting is loss less compression

compressWiki

• Optional
• Accepts: true|false
• Opt in to also compress wiki repo
  • Example: https://github.com/YOUR_USERNAME/YOUR_REPOSITORY.wiki.git
• The default behaviour is opt out

minKBReduced

• Optional
• Accepts only numbers as input (e.g. "minKBReduced": 500 for a 500 KB threshold)
• Can be used to limit the frequency of PRs Imgbot will open over time
• The default setting is 10

prTitle

• Optional
• Available only for paid plans
• Accepts only strings as input (e.g. "prTitle": "My title")
• Can be used to display any custom pull request title
• The default setting is "[ImgBot] Optimize images"

prBody

• Optional
• Available only for paid plans
• Accepts only strings as input
• (e.g. "prBody": "Text before {optimization_ratio} Text after"
                         Text before {optimization_details} Text after")
• Can be used to display any custom pull request body, written using github markdown
• Supports two magic tags: {optimization_ratio} //displays the mean optimization ratio for all images
                                         {optimization_details} //display the optimization details for every images
• The default setting generates the body displayed here

Find out more: https://imgbot.net/docs

Contributing

All the code for Imgbot is available on GitHub. We will gladly accept contributions for the service, the website, and the documentation. This is where you can find out how to get set up to run locally as well as detailed information on exactly how Imgbot works.

https://imgbot.net/docs#contributing

imgbot's Issues

Compression not lossless in some cases

see facebook/react#10677

The green got a lot lighter in the first image there.

This may be solvable by offering more configuration, or maybe there is some setting we need to set to be closer to truly lossless.

Inliner error processing after img optimization

See numworks/epsilon#74 (comment)

Some good content to share

It's 2018, your image optimization should be automated

https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/automating-image-optimization/

handle "action": "deleted" in webhook

Request method: POST
content-type: application/json
Expect: 
User-Agent: GitHub-Hookshot/8e9b331
X-GitHub-Delivery: dbe51e2a-8a91-11e7-84d4-43fbb751ead3
X-GitHub-Event: installation

{
  "action": "deleted",
  "installation": {
    "id": 48668,
    "account": {
      "login": "dabutvin",
      "id": 5168796,
      "avatar_url": "https://avatars2.githubusercontent.com/u/5168796?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/dabutvin",
      "html_url": "https://github.com/dabutvin",
      "followers_url": "https://api.github.com/users/dabutvin/followers",
      "following_url": "https://api.github.com/users/dabutvin/following{/other_user}",
      "gists_url": "https://api.github.com/users/dabutvin/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/dabutvin/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/dabutvin/subscriptions",
      "organizations_url": "https://api.github.com/users/dabutvin/orgs",
      "repos_url": "https://api.github.com/users/dabutvin/repos",
      "events_url": "https://api.github.com/users/dabutvin/events{/privacy}",
      "received_events_url": "https://api.github.com/users/dabutvin/received_events",
      "type": "User",
      "site_admin": false
    },
    "repository_selection": "selected",
    "access_tokens_url": "https://api.github.com/installations/48668/access_tokens",
    "repositories_url": "https://api.github.com/installation/repositories",
    "html_url": "https://github.com/settings/installations/48668",
    "app_id": 4706,
    "target_id": 5168796,
    "target_type": "User",
    "permissions": {
      "contents": "write",
      "pull_requests": "write",
      "metadata": "read"
    },
    "events": [

    ],
    "created_at": "2017-08-26T12:01:49-07:00",
    "updated_at": "2017-08-26T12:01:49-07:00",
    "single_file_name": null
  },
  "sender": {
    "login": "dabutvin",
    "id": 5168796,
    "avatar_url": "https://avatars2.githubusercontent.com/u/5168796?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/dabutvin",
    "html_url": "https://github.com/dabutvin",
    "followers_url": "https://api.github.com/users/dabutvin/followers",
    "following_url": "https://api.github.com/users/dabutvin/following{/other_user}",
    "gists_url": "https://api.github.com/users/dabutvin/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/dabutvin/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/dabutvin/subscriptions",
    "organizations_url": "https://api.github.com/users/dabutvin/orgs",
    "repos_url": "https://api.github.com/users/dabutvin/repos",
    "events_url": "https://api.github.com/users/dabutvin/events{/privacy}",
    "received_events_url": "https://api.github.com/users/dabutvin/received_events",
    "type": "User",
    "site_admin": false
  }
}

Duplicate PRs for same branch

This si still happening. Adding a random delay was not good enough..should have known.

Gotta do a little more thinking/research to prevent duplicate prs

Running out of space

This is disk space on the Azure function.

I thought the clones were going to temp directories, but it filled up and then blocked future clones

report compression result in commit message

Invalid paths when using LibGit to clone

examples:

https://github.com/wikirate/wikirate
https://github.com/decko-commons/decko

error: unable to create file db/migrate_cards/data/cards/topic+overview_page+*type_plu_right+*structure: Invalid argument
error: unable to create file db/migrate_cards/data/cards/topic+reference_page+*type_plu_right+*structure: Invalid argument
error: unable to create file db/migrate_cards/data/cards/topic_page_metric_detail+*right+*structure: Invalid argument
error: unable to create file db/migrate_cards/data/cards/verification_email+*html_message: Invalid argument
error: unable to create file db/migrate_cards/data/cards/welcome_email+*html_message: Invalid argument
error: unable to create file db/migrate_cards/data/cards/yearly_variable+*type+*structure: Invalid argument
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.

ImgBot commits should be verified

Can you update ImgBot to use a PGP key for its commits so that its verified commits in our repository records? This would be nice for consistency.

allow for other default branch besides 'master'

We are hardcoded to a branch called master right now

Support alternate locations for .imgbotconfig

see facebook/react-native#16591 (comment)

today we are pinned to the root of the repo, but there's no reason we can't look in other places.

No pull requests

Hi! I'm trying out ImgBot on a private repository but no pull requests are made by ImgBot.
ImgBot has been granted all permissions on this repository. Am I missing a step?

Thanks in advance!

add support for private repo

contact form

convert KB -> MB for 1000KB+ images in commit message

instead of 2,198.30kb we should print 2.2mb

Add totals/averages to the commit message

I wanted to brag for you on Twitter with something like:

On average it compressed our images xx%!

But I don't know the number. Would be great to include this.

Read compression settings from config

Allow consumers to customize:

• filetypes from allowed list http://www.imagemagick.org/script/formats.php#supported
• other options available http://www.imagemagick.org/script/command-line-processing.php#option
• whitelist / blacklist
• timing when bot runs

// .imgbotconfig
{
    "filetypes": ["png", "jpg"],
    "ignoredFiles": ["/img/logo.png", "/img/high-fidelity-images/*.jpg"],
    "schedule": ["1100", "2300"]
}

weird state for image diff

webpack/webpack@master...ImgBotApp:imgbot

theres no diff, but 9 files touched

Sort the commit message image list

Should show the most impactful optimizations first so the major savings are most visible

Is it possible to have ImgBot closes/delete the branch after PR merged?

After we merge a PR fromImgBot, I have to make sure whichever team member it was also deletes the imgbot branch. Otherwise, It prevents imgbot from submitting another PR next time it should. So can it actually handle this instead? Maybe throw it into a configuration setting so it can be turned off but it be on by default?

Decouple the system

Right now we hold a message for the whole process.

Suggestion to start with raising an event that basically says "Pushed a new imgbot branch here"

Then a separate process can handle this event and open the PR.

Imgbot commits as?

hey,

I wanted to exclude some stuff in my travis builds for imgbot by sender name with the jobs. And run only 1 stage for imgbot PR's

This is what I have in my .travis.yml:

stages:
  - name: Test
    if: sender != ImgBotApp
  - name: imgbot
    if: sender = ImgBotApp

Is the name of the user imgbot[bot]? because these rules are not working

Can't Add ImageBot to a Repo I am an admin for

I am a member of the PrimeFaces Extensions team and I want to add ImgBot to our showcase webapp project here:

https://github.com/primefaces-extensions/showcase

But your selection option to start ImgBot is only showing me my personal and forked repos. How can I add the above to ImgBot?

upgrade to latest .net core version

frontend is on 1.1

and backend is on old 4.6.1 :(

Add .imgbotconfig documentation to website

There's a blurb in the readme but we should add the documentation to the landing page somewhere.

New doc section on authorization

After you install ImgBot into your repo, this is registered with GitHub and you are issued an InstallationID. ImgBot itself has an ApplicationID and a private key that is kept secret.
With a combination of the InstallationID, the ApplicationID, and the private key, ImgBot can request an Installation Token from GitHub.

This Installation Token is valid for 10 minutes at a time and provides the permissions agreed to at the time of installation. If the permissions required by ImgBot change you will be notified and have to accept before they can take affect.

ImgBot specifically uses this Installation Token to clone the repo, create a branch, push a commit to the branch, and open pull requests. These actions are performed on behalf of the installation with a username of 'x-access-token' and a password set to the value of the Installation Token.

The author and committer for these acctions are set to ImgBotApp<[email protected]>.

Furthermore, more recently, the commits pushed by ImgBot are digitally signed with a secure and registered PGP private key and password and verified against a public key registered with GitHub. This verification provides an extra level of security to ensure that commits that come into your repo are in fact coming from the authentic ImgBot. And you get the nice green verified badge in the PR.

If at anytime you want to remove ImgBot's ability to interact with your repo, you simply uninstall the application from your settings and all further action's will be immediately blocked with no access.

save installations in a table

Respect Archived repositories

Don't try to clone and optimize the repos that are marked as archived

This is unnecessary work

Case Study section

@dabutvin What do you think about, add section in top of documentation with links and screenshots to already accepted PRs?

update website to state optimization method

people want to know

• What tools do you use to optimize?
• explicitly state if you optimize images using lossless or lossy methods.

Set up commit signature to have valid email settings

Manually re-run imgbot

I see that I can configure a config file to setup a schedule for imgbot but it would be also nice to be able to have e.g. a button in the settings to re-run imgbot either on all or one specific repo as a one-off action

Add support for svg files

add a commit signature

Some repos require this
is there is any drawback to adding this across the board or making it configurable?

Double prs

Getting some instances of 2 prs being created for the same branch comparison at the exact same time

We should be able to do something to make sure there's only 1 being made.

Handle existing imgbot branch

Should we query github to see if branch exists and exit earlier?
Should we open multiple imgbot branches/
Should we force push on top of imgbot branch
Should we start from existing imgbot branch and add additional commit?

Do not require branch 'imgbot' to be deleted if the PR was merged

1. ImgBot opens a new PR master...imgbot
2. User merges imgbot into master branch
3. User does not delete imgbot branch

Today we will see the branch exists and not optimize any images.

We should update to support the scenario where the branch got left behind, but the changes were accepted

favicon

[imgbotconfig] work_in fork vs branch

Allow a configuration to be set to do ImgBot work in forks vs branches.

Some prefer branch while some prefer fork - let's let the user decide.

see 2factorauth/twofactorauth#2809 (comment)

sounds good, does not work

repository size actually grows by the size of images you change minus your 'saved space', because old images are not going anywhere from git history.

Don't reopen a PR if one was rejected

So I enabled the bot for all my repos, and I started to get PR on all of them (duh). The problem is when I close a PR without merging it (because images are not a concern on that repo) I get the PR again and again

Should I disable it manually for each repo I don't want? Or is there a way to not open a PR if I already rejected the previous one.

File paths in pull request comment

Hey there, ImgBot is an awesome idea and nice execution! A minor thing, but I noticed that file paths are a little off in the pull request comment.

First, it looks like it might have trouble with directories starting with underscore. In this PR, \docs_assets\img\logo-nav.png should be \docs\_assets\img\logo-nav.png.

Also wrong slashes are used.

Doesn't affect functionality of the bot at all, but thought you might find it helpful.

Setup ongoing optimizations

The idea is that if you commit to master a new image or images, then you should get a follow up PR with that image or images optimized

Need a landing page for post installation

Should be explicit and state exactly what to expect after hitting install

right now it dumps the user right back to imgbot.net

we should add a page like imgbot.net/installed or some url like that.

Once this page is built, I can update the GitHub app configuration to point to the new page.

Invalid path using `|`

LibGit2Sharp.LibGit2SharpException : cannot checkout to invalid path 'assets/Events | Guest Manager 2018-02-05 17-05-28.png'

@jeffblake FYI getting these failures on your installation

Cool bot, thanks!

Thanks for the PR!

IMGBOT failed to make PR

Hey,

Something must have happened but I noticed that we had a spare branch laying around and it was a branch made by imgbot but no PR had been made. so I made one: acceptbitcoincash/acceptbitcoincash#812

Might want to figure out why it happened though, which is why I have no merged it yet.

Schedule for runs

Please consider somehow adding an ability to schedule compression runs.
For example run it only once a day/week/month/whenever. That way active repos where images often change don't get flooded with commits from this bot.

write tests around commit message

This is getting more complex now with recent additions,
we should pull out of the main pipeline and make a testable class and cover all the cases.

open an issue if there are imgbot updates to make, but the branch is not deleted

If you leave the branch behind by accident this would be a good reminder.
Should be careful not to open multiple issues