andygrunwald / trendinggithub

A twitter bot (@TrendingGithub) to tweet trending repositories and developers from GitHub

@TrendingGithub

A twitter bot (@TrendingGithub) to tweet trending repositories and developers from GitHub.

Follow us at @TrendingGithub.

Important: This is no official GitHub or Twitter product.

Features

• Tweets trending projects every 30 minutes
• Refreshes the configuration of twitters URL shortener t.co every 24 hours
• Blacklisting of repositories for 30 days (to avoid tweeting a project multiple times in a short timeframe)
• Maximum use of 140 chars per tweet to fill up with information
• Debug / development mode
• Multiple storage backends (currently Redis and in memory)

Installation

1. Download the latest release
2. Extract the archive (zip / tar.gz)
3. Start the bot via ./TrendingGithub -debug

For linux this can look like:

curl -L  https://github.com/andygrunwald/TrendingGithub/releases/download/v0.4.0/TrendingGithub-v0.4.0-linux-amd64.tar.gz -o TrendingGithub-v0.4.0-linux-amd64.tar.gz
tar xzvf TrendingGithub-v0.4.0-linux-amd64.tar.gz
cd TrendingGithub-v0.4.0-linux-amd64
./TrendingGithub -debug

Usage

$ ./TrendingGithub -help
Usage of ./TrendingGithub:
  -debug
    	Outputs the tweet instead of tweet it (useful for development). Env var: TRENDINGGITHUB_DEBUG
  -expvar-port int
    	Port which will be used for the expvar TCP server. Env var: TRENDINGGITHUB_EXPVAR_PORT (default 8123)
  -storage-auth string
    	Storage Auth (e.g. myPassword or <empty>). Env var: TRENDINGGITHUB_STORAGE_AUTH
  -storage-url string
    	Storage URL (e.g. 1.2.3.4:6379 or :6379). Env var: TRENDINGGITHUB_STORAGE_URL (default ":6379")
  -twitter-access-token string
    	Twitter-API: Access token. Env var: TRENDINGGITHUB_TWITTER_ACCESS_TOKEN
  -twitter-access-token-secret string
    	Twitter-API: Access token secret. Env var: TRENDINGGITHUB_TWITTER_ACCESS_TOKEN_SECRET
  -twitter-conf-refresh-time duration
    	Twitter: Time interval to refresh the configuration of twitter (e.g. char length for short url). Env var: TRENDINGGITHUB_TWITTER_CONF_REFRESH_TIME (default 24h0m0s)
  -twitter-consumer-key string
    	Twitter-API: Consumer key. Env var: TRENDINGGITHUB_TWITTER_CONSUMER_KEY
  -twitter-consumer-secret string
    	Twitter-API: Consumer secret. Env var: TRENDINGGITHUB_TWITTER_CONSUMER_SECRET
  -twitter-follow-new-person
    	Twitter: Follows a friend of one of our followers. Env var: TRENDINGGITHUB_TWITTER_FOLLOW_NEW_PERSON
  -twitter-follow-new-person-time duration
    	Growth hack: Time interval to search for a new person to follow. Env var: TRENDINGGITHUB_TWITTER_FOLLOW_NEW_PERSON_TIME (default 45m0s)
  -twitter-tweet-time duration
    	Twitter: Time interval to search a new project and tweet it. Env var: TRENDINGGITHUB_TWITTER_TWEET_TIME (default 30m0s)
  -version
    	Outputs the version number and exit. Env var: TRENDINGGITHUB_VERSION

Every parameter can be set by environment variable as well.

Twitter-API settings (twitter-access-token, twitter-access-token-secret, twitter-consumer-key and twitter-consumer-secret) are necessary to use the Twitter API and to set up a tweet by your application. You can get those settings by Twitter's application management.

If you want to play around or develop this bot, use the debug setting. It avoids using the Twitter API for tweet purposes and outputs the tweet on stdout.

The Redis url (storage-url)is the address of the Redis server in format ip:port (e.g. 192.168.0.12:6379). If your server is running on localhost you can use :6379 as a shortcut. storage-auth is the authentication string necessary for your Redis server if you use the Authentication feature.

Storage backends

Why is a storage backend needed at all?

We are looking for popular projects in a regular interval. To avoid tweeting a project or developer multiple times after another we add those records to a blacklist for a specific time.

At the moment there are two backends implemented:

• Memory (used in development)
• Redis (used in production)

Growth hack

We implemented a small growth hack to get a few followers. This hack was suggested by my colleague @mre. It works like described:

• Get all followers from @TrendingGithub
• Choose a random one and get the followers of the choosen person
• Check if this person follows us already
• If yes, repeat
• If no, follow this person

This feature can be activated via the twitter-follow-new-person flag.

Motivation

I love to discover new tools, new projects, new languages, new coding best practices, new exciting ideas and new people who share the same passion like me. I use twitter a lot and have little time to check trending repositories and developers on a daily basis.

Why not combine both to save time and spread favorite projects and developers via tweets?

License

This project is released under the terms of the MIT license.

trendinggithub's Issues

Document the source code

The source code is quite undocumented.
It is hard to get into the code without knowing the thoughts behind it.

Goal of this task: Document the code :)

Hashtags don't work when they have commas in them

Reported by Dimitar Nestorov via Twitter: https://twitter.com/dimitarnestorov/status/958195310702014465

Implement profiling possibilites

See

• http://blog.golang.org/profiling-go-programs
• https://medium.com/@tjholowaychuk/profiling-golang-851db2d9ae24
• http://de.slideshare.net/cloudflare/go-profiling-john-graham-cumming
• https://github.com/bradfitz/talk-yapc-asia-2015/blob/master/talk.md

Twitter supports now 280 chars per tweet

@DimitarNestorov twittered me (https://twitter.com/dimitarnestorov/status/937107847925690368) that twitter now supports 280 chars.

Goal of this task: Make this bot work with this

Extend "Follow people" to follower of @GitHubEng

@TrendingGithub is not tweeting anymore since 2nd November 2016

The issue was reported on twitter by @edasque

[Question] How to detect trend repositories

I love this repository, but I don't see in the documentation the most interesting thing: how to doy determinate a trending Github Repository? Can you explain to me?

I suppose that you are using something like detect the number of new stars in a short period of time. or maybe Github trending have an API or something like that? Are you polling and endpoint?

Definitely, How you do this :-)

Migrate from github.com/garyburd/redigo to github.com/gomodule/redigo

See garyburd/redigo@5dd9313

Searching a new tweet fails if no repository can`t be found

To search a new tweet we scheduling a new function call every 30 minutes.
In this search process we are looking for the first 3 timeframes (week, day, month), and after this every timeframe for every trending (!) language.

If no repository can be found no new search process will be scheduled.

Here is a log:

2015/08/24 00:24:16 Getting projects for timeframe weekly
2015/08/24 00:24:17 Getting projects for timeframe daily
2015/08/24 00:24:17 Getting projects for timeframe monthly
2015/08/24 00:24:18 Getting projects for timeframe weekly and language ruby
2015/08/24 00:24:19 Getting projects for timeframe daily and language ruby
2015/08/24 00:24:19 Getting projects for timeframe monthly and language ruby
2015/08/24 00:24:20 Getting projects for timeframe weekly and language javascript
2015/08/24 00:24:20 Getting projects for timeframe daily and language javascript
2015/08/24 00:24:21 Getting projects for timeframe monthly and language javascript
2015/08/24 00:24:21 Getting projects for timeframe weekly and language php
2015/08/24 00:24:22 Getting projects for timeframe daily and language php
2015/08/24 00:24:22 Getting projects for timeframe monthly and language php
2015/08/24 00:24:23 Getting projects for timeframe weekly and language python
2015/08/24 00:24:23 Getting projects for timeframe daily and language python
2015/08/24 00:24:23 Getting projects for timeframe monthly and language python
2015/08/24 00:24:24 Getting projects for timeframe weekly and language c
2015/08/24 00:24:33 Getting projects for timeframe daily and language c
2015/08/24 00:24:33 Getting projects for timeframe monthly and language c
2015/08/24 00:24:34 Getting projects for timeframe weekly and language unknown
2015/08/24 00:24:35 Getting projects for timeframe daily and language unknown
2015/08/24 00:24:35 Getting projects for timeframe monthly and language unknown
2015/08/24 00:24:36 Getting projects for timeframe weekly and language css
2015/08/24 00:24:36 Getting projects for timeframe daily and language css
2015/08/24 00:24:37 Getting projects for timeframe monthly and language css
2015/08/24 00:24:37 Getting projects for timeframe weekly and language java
2015/08/24 00:24:38 Getting projects for timeframe daily and language java
2015/08/24 00:24:38 Getting projects for timeframe monthly and language java

Add support for metrics via expvar

Some statistics / monitoring might be useful.
The expvar package looks a good candidate.

Checkout this article for a few details: http://www.mikeperham.com/2014/12/17/expvar-metrics-for-golang/

Implement dependency management

Golang got no dependency management. Implement one.
Godep or the experimental one from Go 1.5

Follow @TrendingGithub follower back

To gain the popularity of @TrendingGithub it make sense to follow the @TrendingGithub`s follower back.

Another (additional) idea of @mre is to follow friends of @TrendingGithub`s follower to get new people into the follower line :)
Of course, this should be automated (e.g. every hour)

Greylist repositories - instead of blacklisting

Greylist previously tweeted repositories for 1/2/3 months instead of blacklisting. That way they will reappear if they are trending again.

Implement shutdown due to a SIGTERM

See

• http://en.wikipedia.org/wiki/SIGTERM
• http://12factor.net/disposability

Adjust length of shorten twitter url automatically

Currently we assume 25 letters for the url
The twitters t.co url shortener modifies itselfs (depending on how many urls will be posted).

TODO: Daily GET call to help/configuration to receive max length for URL
Only with this we are able to fill out 140 chars as max as possible.

@link https://dev.twitter.com/overview/t.co
@link https://dev.twitter.com/rest/reference/get/help/configuration

Extend logging to get a few more details on stdout

The logging is quite okay, but it can be better.
At the last "outtage" (#12) it took some time to find out what the root case was.

Have a look at the logging and check if we can improve this.

Add # of watches and # of stars to tweet

As long as you are <140 chars, add useful information like number of watches, number of stars, delta of these, etc