GithubHelp home page GithubHelp logo

heycarsten / lcbo-api Goto Github PK

View Code? Open in Web Editor NEW
181.0 181.0 43.0 4.65 MB

A crawler and API server for Liquor Control Board of Ontario retail data

Home Page: https://lcboapi.com

License: GNU General Public License v3.0

Ruby 18.49% JavaScript 53.96% CSS 5.86% HTML 21.67% Shell 0.02%
api api-server big-data docker lcbo learn-to-code learning-by-doing postgresql redis ruby-on-rails

lcbo-api's People

Contributors

gilgen avatar heycarsten avatar minusfive avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lcbo-api's Issues

Help getting an api key for a student project?

Hello, I am a computer programming student at Niagara. I have three weeks to build a mobile app. My proposed app intergrates the listing features of the LCBO and some of the basic social features of untappd in a simple way. Thus, a user could browse their local LCBO for beers and see at a glance which they have tried and which not, as well as the ratings from untappd, and possibly a detail page with more information. The user could make a wish list before they go to the lcbo if they want. My idea is to use barcodes initially to associate LCBO beers with their counterparts on untappd, a slightly inexact process as they don't have the same barcode databases, but good enough to be getting on with for my purposes.

If some one has a server running and would not mind generating me API keys for testing it would save me a lot of time. I have no plans to publish the app at this point and wouldn't share the keys.

Please drop me a note at [email protected] if you would be willing to help.

Thanks,

Erik

Docker-Compose Build not working

I am trying to run the app following your instructions. Not sure if I am doing anything wrong, but Step 8/13 in docker-compose build returns multiple errors in fetching all the necessary files, and then finally returns:
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
ERROR: Service 'app' failed to build: The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get -yqq update && apt-get -yqq install software-properties-common
apt-transport-https build-essential git-core openssl libssl-dev acl zip pv postgresql-client-$POSTGRES_VERSION libpq-dev nodejs yarn' returned a non-zero code: 100

Decompressing lcboapi-2018-12-17.bz2 doesn't work

I tried the tar xzf command recommended in the wiki.

I also tried recommendations from stack overflow
bzip2 -d lcboapi-2018-12-17.bz2
bzip2 -dk lcboapi-2018-12-17.bz2
bunzip2 lcboapi-2018-12-17.bz2

None of these worked either. They returned the error "bzip2: lcboapi-2018-12-17.bz2 is not a bzip2 file."

Localization (fr-CA)

I always wanted to have both English and French Canadian versions of the content and docs on LCBO API, regrettably I'm not bilingual, anyways, if someone is curious to take this on hit me up ⚜️

Centralized crawling

I think it's a bad idea if we all run the crawler for our own purposes, it will DDOS the lcbo website. I'm looking for people to collaborate with, to run one instance of the crawler and share the results. Ideas?

Assess and improve Docker stuff

I've been reading Docker for Rails Developers on the recommendation of my friend and long time colleague Ben Moss, it's excellent and I see that I'm not doing things as well as I could around here, when I have time I'll update stuff with my new found knowledge

Rate Limit Semantics

Yo, currently rate limiting works globally by IP or API Key, this was just a spike and is dumb, going forward rate limiting must meet the following requirements.

For V1 endpoints always rate limit by IP address to 300 requests per hour

  1. When the request is made, increment the hit_count for the request's IP.
  2. If the request has an API Key get its max_hits_per_hour, otherwise use the default for the given context.
  3. TODO: Elaborate more.

Integrate CircleCI

Would be nice to set up continuous integration for LCBO API. It will also make for an easier time bringing on new maintainers. CI is great, CircleCI is free for open source projects, LCBO API should use it πŸ™‚

Request Verification

Users create API keys in order to gain access to LCBO API, this is a small step toward adding visibility to usage of the API. The API is targeted to be used server-side by web apps and backend apps. It's also intended to be used in the browser via CORS or JSON-P. Finally, it's intended to be used from

  • Server -> LCBO API
  • Browser -> LCBO API
  • Mobile App -> LCBO API

Keys have the following attributes:

  • Server keys are rate-limited to 8000 requests per hour, but also do not support CORS or JSON-P, they also don't require an Origin header. Use these when accessing LCBO API from your own web-server or backend.
  • Client keys are rate-limited to 800 requests per hour and support CORS and JSON-P, they require an Origin header to be present and are tied to a domain. Use these when accessing LCBO API from a mobile phone app or browser-based JavaScript application.

Auth Tokens have the following attributes:

  • They are rate-limited to 1000 requests per hour and support CORS but only when accessed from lcboapi.com, when accessed without an Origin they do not support CORS. Don't use auth-tokens for accessing LCBO API data endpoints.

What does the manager app do?

Hi, I'd like to know what the manager app does or was intended to do, its features, and etc. I tried following your instructions to run it, but it does not work - probably because it is outdated. Leaving the errors here, just in case anything can be done. Thanks!

npm install does not build properly:

WARN invalid config loglevel="notice"
Can not download file from https://raw.githubusercontent.com/sass/node-sass-binaries/v2.1.1/win32-x64-node-10.15/binding.node

[email protected] postinstall C:\Users\zeesh\OneDrive\Desktop\Gettit\manager\node_modules\node-sass
node scripts/build.js

internal/modules/cjs/loader.js:583
throw err;
^

Error: Cannot find module 'C:\Users\zeesh\OneDrive\Desktop\Gettit\manager\node_modules\node-sass\node_modules\pangyp\bin\node-gyp'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
at Function.Module._load (internal/modules/cjs/loader.js:507:25)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:283:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)
Build failed
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] had bundled packages that do not match the required version(s). They have been replaced with non-bundled versions.

And then running ember server gives:
internal/modules/cjs/loader.js:583
throw err;
^

Error: Cannot find module 'internal/util/types'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
at Function.Module._load (internal/modules/cjs/loader.js:507:25)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at evalmachine.:44:31
at Object. (C:\Users\zeesh\OneDrive\Desktop\Gettit\manager\node_modules\ember-cli\node_modules\configstore\node_modules\graceful-fs\fs.js:11:1)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)

No Crawler information on running crawler

there is only a brief mention of the crawler but no instructions on how to run the crawler. if you could post the commands to run the crawler id be more then happy to update the read me with the information and a guide on how to use it.

Migrations are pending

Cannot render console from 172.18.0.1! Allowed networks: 127.0.0.1, ::1, 127.0.0.0/127.255.255.255
app_1 | (0.2ms) SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
app_1 | ↳ /usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/log_subscriber.rb:98
app_1 |
app_1 | ActiveRecord::PendingMigrationError (
app_1 |
app_1 | Migrations are pending. To resolve this issue, run:
app_1 |
app_1 | bin/rails db:migrate RAILS_ENV=development
app_1 |
app_1 | ):

Legacy datasets availability

Hello!

Are the legacy datasets still available by chance? I'm specifically looking for the ones above 2451 so I can run some analysis on the last year's worth of data. I'd be happy to look into making new ones available if there's interest.

Thanks! Nice to see this project going open source!

Add Skylight to project

LCBO API was approved for the Skylight open source program! πŸŽ‰ That means the power of Skylight for free!

  • Add Skylight back to the app
  • Add badges to README
  • Add badge to homepage sponsors (tragically, it won't be online after Jan 15th... or will it πŸ€”)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.