piccolo-orm / piccolo_api Goto Github PK

We now have the powerful visible_fields filter in PiccoloCRUD.

It would be nice if a consumer of the API could visit the schema endpoint, and see which fields are available (this would include fields in related tables).

For example:

{
    ...
    "fields": ["id", "name", "director.name", "director.id"],
    ...
}

So they can know which fields they can include in the GET param:

/?__visible_fields=name,director.name

Would you please be able to add a license to this repo? Although piccolo-orm/piccolo specifies an MIT license, there is no such licensing on this repo.

We have out of of the box login / logout endpoints, which render nice templates:

The current approach to customising it is to create an entirely new template using template_path (see docs).

Most of the time providing an entirely new template is overkill - we just need to customise some minor things like the background colour, and button colour, to make it more consistent with whichever app we're building.

A nice solution would be for session_login / session_logout to accept a css_variables argument. For example:

session_login(css_variables={'button-color': 'red'})

Noticing that the response from PATH seems to be a text-type instead of json-type response. I'm noticing in the code that the PATH response uses JSONResponse instead of the CustomJSONResponse used by many other endpoints.
As far as I can see, functions either use json response with a dict input:

JSONResponse(self.pydantic_model.schema()

or CustomJSONResponse with a string input:

        json = self.pydantic_model_plural(
            include_readable=include_readable,
            include_columns=tuple(visible_columns),
            nested=nested,
        )(rows=rows).json()
        return CustomJSONResponse(json, headers=headers)

the PATCH response is the only one using JSONResponse with a string param.

I'm guessing this is just a bug (and not a critical one at that) but before I submit a PR to fix I just wanted to make sure.

Based on this discussion:

piccolo-orm/piccolo#185

In some frameworks, such as Django, views are wrapped with transaction handling logic. If an exception is raised, then the transaction is rolled back.

To use transactions is more explicit currently in Piccolo - a context manager has to be used within the view itself.

@app.get("/")
async def my_endpoint():
   try:
        async with MyTable._meta.db.transaction():
            # a bunch of queries
   except Exception:
        return JSONResponse({"message": "something went wrong"})

    return JSONResponse({"message": "success"})

It would be nice to have a decorator which wraps a view with this logic:

@app.get("/")
@piccolo_transaction
async def my_endpoint():
    ...

At version 0.29.0 there seems to be no support for ARRAY types in the CRUD generation.
Code is generated, but an ARRAY of VARCHAR comes out as a single Varchar, and the field cannot be properly used.

In the Piccolo Admin the Arrays of Varchar are handled ok. Are there already plans to support Array types in CRUD?

There's a limitation at the moment with filtering via PiccoloCRUD - you can't specify 'null' for some column types.

Take this as an example:

The user can't type in 'null'. In Piccolo Admin, a solution is to add 'null' as an option in the operator dropdown (similar to 'Equals' etc).

When PiccoloCRUD receives an operator value of 'null' it should ignore the value it receives, and treat it as null instead.

PiccoloCRUD has an option called read_only.

I think it's worth phasing this out in favour of an argument called methods.

It could be something like:

PiccoloCRUD(methods=[HTTP.get])

If read_only is provided in kwargs, then we can just map that to methods=[HTTP.get] and provide a deprecation warning.

It means you have more fine grained control over which methods to expose:

PiccoloCRUD(methods=[HTTP.get, HTTP.post])

Piccolo API has great Pydantic support, but the docs are a bit hidden away under the 'CRUD' section.

It would be good to move them, so 'Serialisation' is a top level topic in the docs, and contains much more examples about how to use create_pydantic_model.

It would be useful to show an app which has all of the session auth components working together (session_login, session_logout and SessionsAuthBackend).

Here's an example:

import datetime
from fastapi import FastAPI
from piccolo_api.csrf.middleware import CSRFMiddleware
from piccolo_api.openapi.endpoints import swagger_ui
from piccolo_api.session_auth.endpoints import session_login, session_logout
from piccolo_api.session_auth.middleware import SessionsAuthBackend
from starlette.middleware import Middleware
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.routing import Route

app = FastAPI()

app.mount(
    "/login/",
    session_login(),
)

private_app = FastAPI(
    routes=[
        Route("/logout/", session_logout()),
    ],
    middleware=[
        Middleware(
            AuthenticationMiddleware,
            backend=SessionsAuthBackend(
                increase_expiry=datetime.timedelta(minutes=30)
            ),
        ),
        Middleware(CSRFMiddleware, allow_form_param=True),
    ],
    docs_url=None,
    redoc_url=None,
)

# The Swagger docs which come with FastAPI don't support CSRF middleware, so we mount
# a custom one which Piccolo provides (accessible at /private/docs):
private_app.mount("/docs/", swagger_ui(schema_url="/private/openapi.json"))

@private_app.get('/my-secret-endpoint/')
def my_endpoint():
    # This is just a normal FastAPI endpoint, and is protected by Session Auth
    pass

app.mount("/private/", private_app)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app)

Hello, sorry if this is a naive assertion, but shouldn't SessionsAuthBackend.authenticate return (AuthCredentials(scopes=[]), UnauthenticatedUser()) if not token but self.allow_unauthenticated == True?

When moving to GitHub Actions from Travis, coveralls had to be disabled as I couldn't get it to work.

It needs adding back.

Based on this discussion:

#81

The session_logout endpoint only works when you send a POST request.

We can modify this endpoint to return a simple logout button when a GET request is received (in a similar way to how session_login works). It will make session_logout easier to use out of the box.

Currently, the __page and __page_size query params aren't documented.

Also, PiccoloCRUD should have a max_page_size argument, to limit abuse of an endpoint.

If the max_page_size is exceeded, an error should be returned. A 403 feels most appropriate, with a body such as The page size limit has been exceeded.

Currently, only offset based pagination is supported in PiccoloCRUD (using the __page and __page_size query params). There are issues with this, as outlined in this article.

Modify PiccoloCRUD so it accepts a __cursor GET parameter. The value will be the ID of a row for now, but encoded as string. Encoding it as a string gives more flexibility (for example, we may want to return a UUID instead of an integer in the future).

The response should then be something like:

{
    "rows": [...],
    "next_cursor: "1234"
}

If both the __page and __cursor GET params are passed to an endpoint, an error should be returned.

I am struggling a bit with Piccolo's authentication systems. I have a FastAPI app and am wrapping it with Starlette's AuthenticationMiddleware, as hinted in the docs, with the joint SessionAuth and SecretTokenAuth providers. The secret token seems to be working alright; my API client won't get results without the correct header-cum-token. However whatever I do on the browser gives me "Auth failed for all backends". I can't get to the login endpoint, though this appears properly configured according to the docs. I tried 'allow unauthenticated' to see if this would loosen up the permissions, but even the FastAPI docs give me this error. Is there any robust example app with SessionAuth to see how everything should be organized?

The code coverage was increased dramatically in this issue: #63

One missing area is a test for the session logout endpoint - would be nice to add this at some point.

We have two great new features for create_pydantic_model (exclude_columns and nested).

I'd like to make sure that they work together. For example:

create_pydantic_model(Band, nested=True, exclude_columns=(Band.manager.id,))

In which case the Manager sub model would omit it's id column.

See: https://www.cockroachlabs.com/docs/stable/timestamp.html

Cockroach always uses Timestamptz (set to +00:00) when Timestamp is specified. This is confusing to piccolo migrations.

gnat@gnat:~/Desktop/piccolo_examples-master/headless_blog_fastapi$ piccolo migrations forwards all

                              BLOG                              
----------------------------------------------------------------
👍 1 migration already complete
🏁 No migrations need to be run

                          SESSION_AUTH                          
----------------------------------------------------------------
👍 1 migration already complete
⏩ 1 migration not yet run
🚀 Running 1 migration:
  - 2019-11-12T20:47:17 [forwards]... The command failed.
expected DEFAULT expression to have type timestamp, but 'current_timestamp() + '01:00:00'' has type timestamptz
For a full stack trace, use --trace

Suggestions? Too new to Piccolo to know what course of action to take in order to make this work smoothly.

Otherwise Cockroach seems to work fairly flawlessly with Piccolo, because of asyncpg.

We're missing a blank link before :param.

Currently, PiccoloCRUD (and hence FastAPIWrapper) allow you to filter by the row ID, but you can't filter by a list of IDs.

For example:

# Currently supported:
DELETE /api/tables/movies?id=1

# Proposed:
DELETE /api/tables/movies?id=1,2,3,4,5

This would be a very useful feature, and would make bulk deletes more efficient.

Now that Python 3.10 has been released, we need to check if Piccolo API works with it, and update the GitHub actions accordingly.

This PR from the main Piccolo repo is a good example of what's required:

piccolo-orm/piccolo#304

Discussed in #174

await MOVIE_POSTER_MEDIA.delete_unused_files()

File ~/.cache/pypoetry/virtualenvs/proj-haD7zQkN-py3.10/lib/python3.10/site-packages/piccolo_api/media/base.py:297, in MediaStorage.delete_unused_files(self, number_shown, auto)
    274 async def delete_unused_files(
    275     self, number_shown: int = 10, auto: bool = False
    276 ):
    277     """
    278     Over time, you will end up with files stored which are no longer
    279     needed. For example, if a row is deleted in the database, which
   (...)
    295 
    296     """
--> 297     unused_file_keys = await self.get_unused_file_keys()
    299     number_unused = len(unused_file_keys)
    301     print(f"There are {number_unused} unused files.")

File ~/.cache/pypoetry/virtualenvs/ducofastapi-haD7zQkN-py3.10/lib/python3.10/site-packages/piccolo_api/media/base.py:272, in MediaStorage.get_unused_file_keys(self)
    266 """
    267 Compares the file keys we have stored, vs what's in the database.
    268 """
    269 db_keys, disk_keys = await asyncio.gather(
    270     self.get_file_keys_from_db(), self.get_file_keys()
    271 )
--> 272 return list(set(disk_keys) - set(db_keys))

TypeError: unhashable type: 'list'

```</div>

Following this:

• Move all the scripts to the scripts folder
• Update github workflows to use the scripts

We added the __visible_fields filter to PiccoloCRUD - but it only works on the bulk GET endpoint, not the detail endpoint.

# Works:
GET /?__visible_fields=name,director.name

# Doesn't currently work:
GET /1/?__visible_fields=name,director.name

As suggested here:

piccolo-orm/piccolo#202

It would be great if create_pydantic_model could generate nested models.

create_pydantic_model(Band, nested_models=[Band.manager])

This would output something equivalent to this:

from pydantic import BaseModel

class ManagerModel:
    name: str    

class BandModel:
    name: str
    manager: ManagerModel

PiccoloCRUD currently has hooks like pre_save, which is used to modify the data before insertion into the database. We should add a post_save too, which is run after insertion into the database. It can be used for logging, sending an email etc.

Having a screen grab showing the generated Swagger docs would be nice on this page:

https://piccolo-api.readthedocs.io/en/latest/fastapi/index.html

The FastAPIWrapper / PiccoloCRUD combo is one of the most powerful things in the Piccolo ecosystem, and a screen grab will make it clearer the functionality it gives you.

That page is also unusual in that it uses Task as an example - we should change it to use Movie, as we can get example code / screen grabs from Piccolo Admin.

For example:

/api/tables/movie?__fields=id,name

Would return:

{
    "rows": [
        {
            "id": 17,
            "name": "The Hobbit: The Battle of the Five Armies",
        },
     ]
}

This is generally useful, but would be great for Piccolo Admin too, as when a user uses a TableConfig to specify that the list page should only show a subset of columns, we aren't over fetching data.

A simple dictionary which the user can define, which maps the English word / phrase to their local language.

For example:

# For Portuguese users:
register(translations={'register': 'registro'})

The only option at the moment is to create your own template. This is definitely easier.

We could probably include a few translations out of the box for some language (just pre-defined dictionaries):

from piccolo_api.shared.auth.translations import register_portuguese

# For Portuguese users:
register(translations=register_portuguese)

Piccolo Admin has a read only mode, which is useful when demoing it online.

This PR updates the register and create_user endpoints to have a read only mode, because we're going to use them in Piccolo Admin.

piccolo-orm/piccolo_admin#168

@dantownsend I've just upgraded my infrastructure to the latest version of FastAPI and I'm now getting a strange error when trying to view the Swagger UI. In the first instance FastAPI is giving me a "failed to load API definition" which is a problem with retrieving the OpenAPI JSON spec. Digging a bit more into the problem however reveals that something is going on when creating the Pydantic response models.

Fetch error
Response status is 500 /api/openapi.json

Digging a bit more into the problem however reveals that something is going on when creating the Pydantic response models. I assume this is soemthing to do with create_pydantic_model. I'm getting an error like the one here, which is the closest I can find to any explanation of the issue:

tiangolo/fastapi#1505

The relevant bit in my case is:

File "/usr/local/lib/python3.7/site-packages/fastapi/openapi/utils.py", line 364, in get_openapi
flat_models=flat_models, model_name_map=model_name_map
File "/usr/local/lib/python3.7/site-packages/fastapi/utils.py", line 28, in get_model_definitions
model_name = model_name_map[model]
KeyError: <class 'pydantic.main.Lemma'>

I'm stumped, as nothing has changed at all in my model / table definitions since the update. Is this likely something due to the interaction of Piccolo - Pydantic - FastAPI or out of Piccolo's scope?

Similar to what we did for Session Auth, maybe we can add a full example app?

There are some outstanding MyPy errors in the codebase, meaning MyPy can't be enabled as part of the build pipeline.

I want to create blockchain backend app, based on piccolo. So I need custom User model (generally, without password field). As suggested in docs, I must implement custom user app. But I also want to use session mechanism. How can I achieve that case?

BTW, Great project! I've struggled with fastapi-sqlalchemy stack and it's back and forth model transitions. I've looked other ORMs, compatible with pydantic and FastAPI (Tortoise, SQLModel, Databases, GINO), but those projects looks too young or unmaintained. And only piccolo had my heart from first look). Thank you and keep doing your great work!

Now we have codecov integrated, it's easier to track code coverage.

There are some areas of the code base which require additional testing.

Related to piccolo-orm/piccolo_admin#167

Given a piccolo model containing an enum:

class CustomerActivationStatus(str, enum.Enum):
    Signup = "Signup"
    Active = "Active"
    Disabled = "Disabled"
    FreeSubscriptionActive = "FreeSubscriptionActive"


class Customer(Table):
    signup_guid = UUID()
    email: str = Varchar()
    created_at: datetime = Timestamptz()
    activation_status = Varchar(
        choices=CustomerActivationStatus,
        default=CustomerActivationStatus.Signup.value
    )

...and using create_pydantic_model to generate a dynamic pydantic class:

CustomerPatchResponseModel = create_pydantic_model(Customer)

..and then using that as a fastapi response model:

@api_v1_router.patch(
    "/customer/guid/{signup_guid}",
    response_model=CustomerPatchResponseModel,
    responses={409: {"model": None}}
)
async def patch_customer_by_guid()
    <the code>

Seems to break fastapi's openapi spec auto-generation:

Traceback (most recent call last):
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 161, in openapi
    return JSONResponse(self.openapi())
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 136, in openapi
    self.openapi_schema = get_openapi(
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/openapi/utils.py", line 387, in get_openapi
    definitions = get_model_definitions(
  File "/projects/customer-api/.venv/lib/python3.9/site-packages/fastapi/utils.py", line 28, in get_model_definitions
    model_name = model_name_map[model]
KeyError: <class 'pydantic.main.Customer'>

If I exclude the enum field, the problem goes away:

CustomerPatchResponseModel = create_pydantic_model(
    Customer,
    exclude_columns=(Customer.activation_status, Customer.email,)
)

I'm pretty sure fastapi is able to deal with regular pydantic models including enums, so I suspect there's something about how create_pydantic_model generates these dynamic models.

Some of the endpoints would benefit from CAPTCHAs for bot protection - e.g. login and register.

As reported here:

piccolo-orm/piccolo#606

PiccoloCRUD is what powers Piccolo Admin. When someone wants to create a new row, we call the 'new' endpoint. For example:

GET /api/tables/movie/new/

{
  "id": null,
  "name": "",
  "rating": 0.0,
  "duration": 0.0,
  "director": null,
  "oscar_nominations": 0,
  "won_oscar": false,
  "description": "",
  "release_date": "2022-09-01T19:14:42.268809",
  "box_office": 0,
  "tags": [],
  "barcode": 0,
  "genre": 0,
  "studio": null
}

This returns the default values for the new row. The problem is with email columns - if the default value is '', then it's not a valid email, so Pydantic won't serialise it, and we get a 500 error.

You can see the relevant code here:

The approach is basically the same as this:

piccolo-orm/piccolo#601

Currently broken by pydantic - waiting for a fix.

../../../virtualenv/python3.9-dev/lib/python3.9/site-packages/fastapi/dependencies/utils.py:265: in get_typed_annotation
    annotation = evaluate_forwardref(annotation, globalns, globalns)
../../../virtualenv/python3.9-dev/lib/python3.9/site-packages/pydantic/typing.py:50: in evaluate_forwardref
    return type_._evaluate(globalns, localns)
E   TypeError: _evaluate() missing 1 required positional argument: 'recursive_guard'

I find a small bug. After the last changes in Piccolo ORM Table.__init__ ignore_missing argument is now _ignore_missing and we have to change this line to row = self.table(_ignore_missing=True) otherwise we get 500 Internal Server Error from get_new method. If you want I can change it.

Now that Piccolo API has a fairly broad set of features, it's important to improve upon the documentation.

I think the following needs adding:

• A 'basics' page - explaining how ASGI middleware works
• Make sure all of the examples are easy to understand
• Improve JWT and Token Auth docs - they don't document the endpoints properly

Example table:

class Recipe(Table):
    name: str = Varchar(length=100)
    content: str = Text()
    author: BaseUser = ForeignKey(references=BaseUser)
    category: str = Varchar(length=50)
    ingredients: List = JSONB()
    extras: Dict = JSON()

Currently, any JSON and JSONB fields from a Piccolo Table is being serialized as strings. It technically works, but parsing it as a string outputs this:

Setting value_type = pydantic.Json for JSON and JSONB fields makes it look a lot better:

It would be good if we modified PiccoloCRUD, so it could accept a media_storage argument (like create_admin does in Piccolo Admin).

PiccoloCRUD(
    Movie,
    media_columns=[
        LocalMediaStorage(Movie.poster, media_path='/srv/media/movie_posters/')
    ]
)

We could then add a new GET parameter called something like __media_urls, which then auto adds the URL to the response for accessing the media.

GET /1/?__media_urls
{
    'id': 1,
    'name': 'Star Wars',
    'poster': 'some-file-abc-123.jpg',
    'poster_url': '/media/some-file-abc-123.jpg',
}

Once this is in place, we can refactor Piccolo Admin slightly, so it passes the media_storage argument to PiccoloCRUD, and lets PiccoloCRUD do all of the heavy lifting.

Related to:

#72

If asyncpg isn't installed, Piccolo API won't work:

As asyncpg is now an optional requirement in Piccolo, we need to work out a way of making Piccolo API handle it gracefully if asyncpg isn't installed.

When using the session_login endpoint, it would be nice to have hooks (basically callback functions) which are called at different parts of the verification process.

There are many situations where this would be valuable. Here are some examples:

1. You only want to let the user login if they have verified their email address.
2. After the user logs in, you want to record this in an external system.
3. Before logging a user in, you want to check if they are on a black list (for example, based on email address or IP address).

There will have to be a minimum of three hook types:

1. Triggered before a login attempt is made.
2. Triggered after the login is successful.
3. Triggered when the login fails.

The session_login function will have a new hooks argument:

def session_login(
    ...,
    hooks: t.List[SessionLoginHook] = []
):
    ...

The hook will accept a request argument, and additional data (such as email, BaserUser instance etc).

If the hook raises an Exception, then the login is stopped, and an error is shown to the user.

It would be great if this functionality was available for all of the auth types that we provide (token auth etc), but session auth is the most important.

We have session_login and session_logout endpoints, which helps with quickly prototyping an app.

It would be great to have a session_signup endpoint too.

It would look very similar to the login form, except with username / email / password / confirm password fields:

Just like we've done with other endpoints. There are many potential use cases:

• If someone keeps getting their password wrong, we may want to log it
• When someone's password is successfully changed, we might want to send an email to that user
• We might want to do some custom validation on the password