openfoodfacts / open-prices Goto Github PK

Story

Admin / moderators could have an easy-to-use interface to make changes (edit or delete price, edit location data...).
Similar to the admin in Django.

Libraries

see below

Story

Similar to #154 , but for labels_tags

Story

We'll need a PostgreSQL database to store data.

See an example of implementation here : https://github.com/openfoodfacts/folksonomy_api/blob/main/folksonomy/api.py

Idea of using peewee for the ORM

Story

Currently the prices are available anonymously. There are filters available, but none on the "owner".

It would be interesting for the authenticated user to fetch prices that we're created by him (but not allow filtering on other owners to respect privacy) owner data is public, we can filter on any owner

We don't do any backup for now.

• perform daily backups of the DB
• generate a daily dump (JSONL) that is available for download to all users.
• update the landing page (prices.openfoodfacts.org) to reflect the existence of this daily dump

Story

Given a proof_id, it is currently not easily possible to get the related prices (and it's price_count).

We could add an extra filter on the PriceFilter, but there is a privacy risk for private proofs.
An alternative is to create a new route /proofs/<id>/prices and make sure that the authenticated user is the owner of the proof

Story

We will want to have each Price linked to a Proof.

• for now it can be only an image
• 1 proof can correspond to multiple prices (ex: receipt)

Currently, we use a single user table to store the associated token. It's not ideal as we can't be logged in on more than one device (error while uploading photos occur).

We should add a session table and migrate the tokens to the new table.

Story

• Price model
• create price
• fetch prices (with filters)

How

The Price model should store the basic fields :

• product_code
• price
• currency
• location_osm_id & location_osm_type
• date

For raw products (fruits, vegetables,...), origins is an information that's important to track (the price is usually different depending on the origin of the country).

The origin can be:

• a country (most common)
• a region

We can use the origin taxonomy (https://static.openfoodfacts.org/data/taxonomies/origins.full.json) and allow users to specific the origin of the product and save it along the price (only for products without barcode).

Story

When fetching prices, we currently get the product_id & location_id.
To avoid having to make additional queries, we would like to return the full Product & Location objects.

Note

Need to optimize queries to avoid N+1 calls to the DB

Story

To better compare products, the category is a good info to store

How ?

We could store the last category known in the categories_hierarchy field ?
Better : the full categories_tags list (what is the difference with categories_hierarchy ?)

Currently, we can't delete a proof once uploaded.

Endpoint: DELETE /api/v1/proofs/:proof_id

This endpoint should deny deleting proofs if:

• some prices are associated with it
• if the user is not the owner of the proof

Story

Currently we consider that a new barcode is necessarily a food product, and we fetch its data from Open Food Facts.

What if a user scans a book ? or a boardgame ? or a deodorant ?

What

Elsewhere

• openfoodfacts/openfoodfacts-dart#831

Story

Following #154

We need to add a new filter capability to find only products of a certain category. Bonus : allow filtering by multiple categories as well.

But I'm not exactly sure how to proceed with the fastapi-filter library. I opened an issue : arthurio/fastapi-filter#556

Or maybe I'm missing something ? 😅

Story

When adding a price, we usually have the product label (e.g. BANANE CRF BIO MH 5 FRUITS IMP) alongside.

This information could be useful, let's store it !

Price.product_label ? --> product_name

Story

There is lots of interesting info in the proof images :

• price tag : price, barcode
• receipt : multiple prices, no barcodes nor EAN but a "product name", date, location
• exif data : date & location ?

What if the server was able to do some image recognition, and return to the frontend all of that info ? 🤩

Challenges

Price tag

Receipt

Without the product EAN or barcode, how to link to the exact product in the OFF database ?

Image examples

Price tag	Multiple price tags	Receipt

Story

Authenticate users via openfoodfacts.org

How ?

See an example of implementation here : https://github.com/openfoodfacts/folksonomy_api/blob/main/folksonomy/api.py

// auth server: https://world.openfoodfacts.org/cgi/auth.pl (pass user_id & password)
// table: auth
// fields: user_id (OFF username), token, last_use

CREATE TABLE auth (token varchar, user_id varchar, last_use timestamp);
CREATE INDEX ON auth (token);

We can't update prices, which starts to become cumbersome to users.

Add a PUT /api/v1/prices endpoint, that allows to update all price information.
The user should only be allowed to update his/her prices.

Story

When a new location is added, we fill a couple of fields in our database.

For the city field it is a bit tricky, we currently look at ["village", "town", "city", "municipality"]

Is it the best way to do it ?

Story

Currently the landing page is available in only 2 languages :

• english : https://prices.openfoodfacts.org/
• french : https://prices.openfoodfacts.org/fr/ : #155
• spanish : #230

Help

See this PR - #155 - for an example (French translation).

Story

Improve typing by adding types and if possible making it mypy compatible (no errors)

Part of

• #31

Story

Some users log in with their e-mail. It is then used as their username. We should fetch their username instead.

From openfoodfacts-explorer I can't query the service, as there is a cross-domain request and the open prices server does not add "openfoodfacts-explorer.vercel.app" Access-Control-Allow-Origin.

Example: https://openfoodfacts-explorer.vercel.app/products/3175681105423

Story

Currently products are added in Open Prices on a per-scan basis, when a price is added.

What about having a list of the top 500 (or more ?) scanned products in Open Food Facts, and importing them in Open Prices.

We could then display them in the frontend, and nudge users to add prices for these products, displaying their price count :)

Story

It would interesting to display the list of contributors, ranked by number of prices added.

For this, we need:

• an endpoint on the User model
• a price_count field that calculates the number of prices added (real-time ? or updated every night ?)

Problem

Part of

• #39

Story

We currently have a price field (mandatory, decimal) for all the items added to the database..

But for products currently discounted, we need an easy way to store the discount-related information.

I see 2 solutions :

1) The price field always stores the current price (full or discounted depending on the item)

• a boolean field price_is_discounted which would be False by default
• an extra decimal field price_without_discount which would be available to fill if the above field is True

2) The price field always stores the full price

• a boolean field price_is_discounted which would be False by default
• an extra decimal field price_discounted which would be available to fill if the above field is True

Story

We have the price (stored in Price.price) and we have the quantity (stored in Product.product_quantity).
It would interesting to compute and return the price per kg or L.

• how to differentiate between kg quantities & L quantities ? an additional product field to get from OFF to help us differentiate beverages ?
• or should we leave the frontend to compute & display this info ?

Story

There seems to be an issue (not sure if a backend or a frontend problem), new prices show up with "1h ago" instead of "1m ago".

I suspect that the timestamps in the backend are stored without timezone.

Story

• how to add a single price
• how to add multiple prices (from a shop shelf)
• how to add multiple prices (from a receipt)

Story

We'll store the price's location info separately.
We'll fetch the data from OpenStreetMap.

Some fields to store :

• osm_id & osm_type
• osm_display_name
• osm_lat
• osm_lon
• other ?

Tasks

• Location model
• on Price create, check if a Location already exists, or create a new on + link it to the Price
• on Location create, fetch metadata from OSM
• api endpoint ?

Story

For each new product we fetch from Open Food Facts its product_quantity, a normalized number in g or mL.

We need to find a way to know if the product is a beverage (mL), or solid (g), so that we can display the information accordingly in the frontend (price per kg or price per L).

Linked to #77

Story

What is the best strategy to efficiently get the last known price of a product ?

Why ?

In the frontend, there are various places where we could display this information to the user :

• in the list of products, along with it's price count - see #123
• when a new price is being added, and the product bar code is scanned, we display information about the product
• ?

Moderators are special users that will be allowed to delete/update prices or proofs of other users.
This is necessary to ensure database quality.

Tasks:

• create a new is_moderator field in the users table : #183
• allow moderators to edit proofs : #235
• allow moderators to delete proofs (without prices) : #197
• allow moderators to edit prices : #259
• allow moderators to delete prices : #260

Story

We use mimetypes.guess_extension() to define the extension of the uploaded file

But image/webp doesn't seem to be managed by the server (nor my local environment - Ubuntu 22.04)

Story

In the frontend, it would be useful for users to fetch prices corresponding to a specific proof

Story

Currently only prices with a EAN code are allowed

Need to add

• a new field category_tag
• also info about the labels
• origins : #72
• and allow filtering on this data

Story

We're starting to get feedback from frontend users who wish to edit or delete their prices (mistake, typos, wrong product...)

What

• Find ways to fill idle time contributing to Open Prices

Why

• The times of the week where you have both a price and the barcode together are scarce.
• We need to exit that paradigm if we want to grow contribution and people coming back daily

How: ideas welcome

• Snap shelves photos, add data later. All 2 tasks can be done at different times
• Split bulk product addition
  • Create a basket in your open prices account by scanning all products (introduce a transient shopping session object)
  • Take a photo of the receipt stored in your open prices account
  • Link both at a later time
  • All three tasks can be done at different times

Story

https://fastapi.tiangolo.com/tutorial/cors/

Story

We'll store the price's product info separately.
We'll fetch the data from OpenFoodFacts.

Some fields to store :

• code
• in_off ?
• off_db (off or opf or obf or opff)
• off_name
• off_image_url
• other ? quantity, categories

Story

Issue to link commits & PRs related to this change

• we had venv & requirements.txt
• we switched to poetry

The requirements.txt file mentionned in the INSTALL.md is wasn't added to the codebase.

Can you add it please ?

Thank you :)

Story

For the v1 we went a bit fast, here is a list of known improvements :

• #221
• schema: add field descriptions (pydantic) & examples
• typing: add return types (especially in crud.py & utils.py)
• background tasks: avoid passing db (see this discussion)
• tests: teardown data between each test, add more tests
• tests: use PostgreSQL instead of SQLite
• documentation: improve it (ex: docker), add docstring

Story

Currently we fetch data from OpenFoodFacts when a price is added and the product is not yet in our database.
But if the product info changes on OFF, or if we add new fields to store, the existing products are not updated.

• this script could run regularly, say every week
• we'll need something similar for OpenStreetMap locations ?

Story

Idea of creating a new field Product.price_count.

Helpful for :

• sorting products by most prices
• displaying the price count in the product details card

How to update :

• incremented synchronously on each new price

Story

When we sync the products from OFF, we currently fetch the raw brands field.

But if we fetched instead the brands_tags we could have better search & filter capabilities for the Frontend.