nchah / github-traffic-stats Goto Github PK

This is a great code and I have an idea to use it for other purposes, but I do not see a LICENSE file.

https://developer.github.com/changes/2020-02-14-deprecating-password-auth/

If the token auth can be added, it'd be appreciated.
: D

I already installed requests package.

But, for example, when I executed python github-traffic-stats.py '29988122' 'ALL' 'save_csv' and provided password, I got Bad credentials response.
On the repo traffic page, I can view them without any issue.

Does this mean that I need to give permission to github api or what?
Thank you for your project!

@nchah I am having credential problems with the package. when I run the package as below I get the following response (i have tried manual pw entry and the concatenated version):
pipenv run gts 'sampoearas' 'ALL' 'save_csv'
Password: [Correct Password]
Bad Credentials
Code done.

I also have tried running it with my organization and get the same error
pipenv run gts 'sampoearas' 'ALL' 'save_csv' -o 'araslabs'

After some investigating, I can get the API to pass back a 200 just using my credentials on a /user page, and I can get a direct call to my organization: https://api.github.com/users/araslabs/repos?per_page=100

Any ideas what I'm doing wrong? I can generate a public key if that will help with the debugging.

Thanks for the help,
@sampoearas

I tried reading stats for an org with many repos. The output stops after a chunk of repos. I haven't looked deeper into it. Is this a well-known limit, a bug in the traffic API or with the Python code?

Hello! Only a little detail, in the Readme file the link to psycopg2 isn't working.

Regards

When setting up a personal token for Github access there's a large set of permissions that we could set.

For this script to work, which are the minimal (strictly necessary) permissions we need to provide to the access token?

Is there a difference if a repo is personal or part of an organization?

Hi,

I hope gts is meant to be run also without a end-user ready to type a password at each run.

The syntax documented as username:password no longer works.
The code main() has:

    pw = getpass.getpass('Password:')

I suggest to replace it with:

    if sys.stdin.isatty():
        pw = getpass.getpass('Password:')
    else:
        print 'Password:'
        pw = sys.stdin.readline().rstrip()

This way it'll be possible to use:

{ sleep 1; cat secret.nchah ; } | gts 'nchah' 'github-traffic-stats' 'save_csv'

Otherwise I'm open to any other suggestion to avoid a user typing the password at each request.
Thanks in advance!

I am trying to get traffic info for a repository I manage, and decided to test the code on one of my own repos first. I am trying to run via:
gts dmarshall_bing dad_jokes

I get the following message:

(base) C:\Users\dpm42>gts dmarshall-bing dad_jokes
Password:
Must have push access to repository
Code done.

This is regardless of whether I enter the correct or incorrect password for my account. Is there something wrong with my syntax? I looked into the code and the server is returning a 403 error from the request call.

Thanks!

By default all files are written the local path. It would be nice for an option to specify an output directory. This will also make it easier to work with the merge-csv.sh script.

I can submit a PR for a fix.

export to a database, e.g. sql would be awesome.
someone has done something similar already, but in MS Visual Studio: https://github.com/Microsoft/GitHubTelemetryParsor

Here's what happens when I try to run it:

➜  ~ gts 'tom-doerr' commonroad_batch_processing save_csv
fatal: not a git repository (or any of the parent directories): .git

Is it necessary to run it inside the corresponding git repository?

Hi,

Is there a way to not display the repo that have 0 view/clone?

Thanks in advance,

Hi ,

I have only traffic permission to one of the repisitory but dont have the repository permission, in that case how can i get the traffic-stats for that repository. could you pleae help?

Thanks for your help in advance

@nchah

BRs, Gowrisankar

In the graphs/traffic section there is also a fourth graph with "Popular content" data.
How about adding it in github-traffic-stats?

I get the following error when I run gts 'swaminator' 'nikhil' 'save_csv'

requests.exceptions.SSLError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /repos/swaminator/nikhil/traffic/views (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))

Any idea why?

I'm trying to log GitHub traffic to a dockerized PostgreSQL database with this service definition in docker-compose:

postgres:
  image: postgres
  environment:
    - POSTGRES_DB=github-traffic-stats
    - POSTGRES_USER=postgres
    - POSTGRES_PASSWORD=<magic password>
  volumes:
    - ./volumes/postgres/data:/var/lib/postgresql/data
  restart: always

I'm running github-traffic-stats in a docker container as well, with this service definition in docker-compose:

github-traffic-stats:
  build: ./services/github-traffic-stats
  depends_on:
    - postgres
  environment:
    - GITHUB_USER=gieldemeulemeester
    - GITHUB_TOKEN=<magic token>
    - POSTGRES_HOST=postgres
    - POSTGRES_PORT=5432
    - POSTGRES_DB=github-traffic-stats
    - POSTGRES_USER=postgres
    - POSTGRES_PASSWORD=<magic password>
  restart: unless-stopped

The environment variables above are used for the entrypoint of the container:

python main.py $GITHUB_USER:$GITHUB_TOKEN "ALL" "set_db" --host $POSTGRES_HOST:$POSTGRES_PORT --db-user $POSTGRES_USER:$POSTGRES_PASSWORD --db-name $POSTGRES_DB

However, github-traffic-stats throws an exception relation "repo_overview" does not exist.

Traceback (most recent call last):
File "main.py", line 406, in main
err_msg = repos_json.json().get('message')
AttributeError: 'list' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 467, in
main()
File "main.py", line 436, in main
store_db(db_config, repo, traffic_response, 'views')
File "main.py", line 258, in store_db
__repo_overview_insert(cur, repo, response_type, json_response)
File "main.py", line 273, in __repo_overview_insert
cur.execute(check_count % (repo, response_type))
psycopg2.ProgrammingError: relation "repo_overview" does not exist
LINE 1: SELECT COUNT(*) FROM repo_overview WHERE create_timestamp=DA...

Do I need to initialize the database in some way (create tables, ...)? What am I doing wrong?

It would be awesome if clones and referrers would be tracked as well, as specified in the official api.

I've tried copy-pasting the traffic-sections and modifying them to also track clones, but my understanding of json and the involved datastructures in the py-script seem to be too limited.