GithubHelp home page GithubHelp logo

dahlia / sqlalchemy-imageattach Goto Github PK

View Code? Open in Web Editor NEW
116.0 5.0 25.0 3.14 MB

SQLAlchemy extension for attaching images to entities.

Home Page: https://sqlalchemy-imageattach.readthedocs.io/

License: MIT License

Python 100.00%
python sqlalchemy orm databases

sqlalchemy-imageattach's Introduction

SQLAlchemy-ImageAttach

PyPI

Read the Docs

Build Status

Coverage Status

SQLAlchemy-ImageAttach is a SQLAlchemy extension for attaching images to entity objects. It provides the following features:

Storage backend interface

You can use file system backend on your local development box, and switch it to AWS S3 when it's deployed to the production box. Or you can add a new backend implementation by yourself.

Maintaining multiple image sizes

Any size of thumbnails can be generated from the original size without assuming the fixed set of sizes. You can generate a thumbnail of a particular size if it doesn't exist yet when the size is requested. Use RRS (Reduced Redundancy Storage) for reproducible thumbnails on S3.

Every image has its URL

Attached images can be exposed as a URL.

SQLAlchemy transaction aware

Saved file are removed when the ongoing transaction has been rolled back.

Tested on various environments
  • Python versions: Python 2.7, 3.3 or higher, PyPy
  • DBMS: PostgreSQL, MySQL, SQLite
  • SQLAlchemy: 0.9 or higher (tested on 0.9 to 1.1; see CI as well)

Installation

It's available on PyPI:

$ pip install SQLAlchemy-ImageAttach

sqlalchemy-imageattach's People

Contributors

dahlia avatar kroisse avatar peterlada avatar pylover avatar yoloseem avatar youknowone avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

sqlalchemy-imageattach's Issues

Implicit Contexts with Pyramid

I am running into an issue when attempting to use implicit contexts within a Pyramid application.

Using Pyramid's built-in BeforeRender event I can call the push_store_context function at the beginning of my request/response lifecycle - this works fine.

However, when using either the finished callback or response callbacks to pop the context at the end of the request I encounter an "IndexError: pop from empty list." I have checked my code to make sure all of the "with store_context(store):" implementations have been removed. Nonetheless, the error still occurs. Does this have something to do with how the contexts are managed internally by sqlalchemy-imageattach? Maybe the context has been emptied by the time the pop_store_context function is being called by the finished_callback at the VERY end of the request/response cycle.

imageattach not working with s3 to use AWS4Auth.

I recieve below message when i run 'from_file'.

Message

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message>
<RequestId>D8C3E352D1093AE8</RequestId><HostId>TBW9rtuGORRQ6pvbhTqgw0gb6bmxKqNgV66EcxqhKQcxgU1Xs3ngOm5hJSamZdemWAAZGKnzw/M=</HostId>
</Error>

Question about generating thumbnails (there is no original image yet)

I have a list of images and I would like to insert a Sample entity in the database for each of them.

import uuid

from sqlalchemy import Column, ForeignKey, Unicode
from sqlalchemy.dialects.postgresql import UUID

from sqlalchemy.orm import relationship
from sqlalchemy_imageattach.entity import Image, image_attachment

from .meta import Base

class Sample(Base):
    __tablename__ = 'sample'

    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    name = Column(Unicode)
    picture_id = Column(UUID(as_uuid=True), ForeignKey('picture.id'), nullable=True)
    picture = image_attachment('Picture', foreign_keys=picture_id, single_parent=True)

class Picture(Base, Image):
    __tablename__ = 'picture'

    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, unique=True)
    name = Column(Unicode)

    @property
    def object_id(self):
        return self.id.int

Here is my conftest.py for py.test. I have a fixture that creates the store for the whole testing session, and I have another fixture to create a new SQLAlchemy session for each test.

import pytest

from sqlalchemy import create_engine
from sqlalchemy.orm.session import sessionmaker
from sqlalchemy_imageattach.stores.fs import FileSystemStore

from databasset.models.meta import Base
from databasset import models

DB_URL = 'postgresql://devuser:devuser@localhost:5432/devdatabase'

def database_drop_all(engine):
    from sqlalchemy.engine import reflection
    from sqlalchemy.schema import (
        MetaData,
        Table,
        DropTable,
        ForeignKeyConstraint,
        DropConstraint)

    conn = engine.connect()
    trans = conn.begin()

    inspector = reflection.Inspector.from_engine(engine)

    metadata = MetaData()

    tbs = []
    all_fks = []

    for table_name in inspector.get_table_names():
        fks = []
        for fk in inspector.get_foreign_keys(table_name):
            if not fk['name']:
                continue
            fks.append(ForeignKeyConstraint((), (), name=fk['name']))
        t = Table(table_name, metadata, *fks)
        tbs.append(t)
        all_fks.extend(fks)

    for fkc in all_fks:
        conn.execute(DropConstraint(fkc))

    for table in tbs:
        conn.execute(DropTable(table))

    trans.commit()

def pytest_addoption(parser):
    parser.addoption('--db-url', action='store', default=DB_URL, help='Set the database engine URL')
    parser.addoption('--reset-db', action='store_true', default=False, help='Reset the database content')
    parser.addoption('--fill-db', action='store_true', default=False, help='Fill the database with content')

@pytest.fixture(scope='session')
def engine(request):
    """Session-wide test database."""
    _engine = create_engine(request.config.option.db_url)

    maker = sessionmaker()
    maker.configure(bind=_engine)

    Base.metadata.bind = _engine

    if request.config.option.reset_db:
        Base.metadata.reflect()
        Base.metadata.drop_all()

    Base.metadata.create_all()

    if request.config.option.fill_db:
        _session = maker()
        _session.commit()
        _session.close()

    return _engine

@pytest.yield_fixture(scope='function')
def session(request, engine):
    """Creates a new database session for a test."""
    connection = engine.connect()
    transaction = connection.begin()

    maker = sessionmaker()
    _session = maker(bind=connection)

    yield _session

    transaction.rollback()
    connection.close()
    _session.close()

@pytest.fixture(scope='session')
def store(request):
    _store = FileSystemStore(
        path='/path/to/the/store/images',
        base_url='http://localhost:5000/',
    )
    return _store

Here is my test_example.py which loops over a list of images and tries to insert them in a Sample. Then, I try to create several thumbnails before I commit the session.

import os
from sqlalchemy_imageattach.context import store_context
from databasset.models.sample import Sample

BASE_PATH = os.path.dirname(__file__)
IMG_PATHS = [
    os.path.join(BASE_PATH, 'assets/{}.jpg'.format(i)) for i in range(0, 10)
]

def test_example(session, store):
    with store_context(store):
        for i, img_path in enumerate(IMG_PATHS):
            sample = Sample(name='test image of {}'.format(i))

            with open(img_path, 'rb') as f:
                sample.picture.from_file(f)

            session.add(sample)
            session.flush()
            print(sample.picture.locate())

        samples = session.query(Sample).all()

        for item in samples:
            for width in [100, 200, 300, 400, 500]:
                image = item.picture.generate_thumbnail(width=width)
                session.flush()
                print(image.locate())

        session.commit()

The test fails in generate_thumbnail with the following error message: OSError: there is no original image yet. The original images are all in the folder hierarchy of the store.

What would be the proper way of doing this?

On a side note, I am not sure why I have to specify unique=True on the primary key of the Picture model. I thought a primary key was already a unique constraint. It throws this error without it:

sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) there is no unique constraint matching given keys for referenced table "picture"
[SQL: '\nCREATE TABLE diamond (\n\tid UUID NOT NULL, \n\tname VARCHAR, \n\tpicture_id UUID, \n\tCONSTRAINT pk_diamond PRIMARY KEY (id), \n\tCONSTRAINT fk_diamond_picture_id_picture FOREIGN KEY(picture_id) REFERENCES picture (id)\n)\n\n']

Replacing Wand dependency

I would like to have a general idea of the work required to replace the dependency of wand by something else? My project already requires cv2 and I would like to use it instead of wand in sqlalchemy-imageattach. As I understand it, wand is only used to get some meta data on the image file and to open and save it on the filesystem.

Any suggestions on how I should proceed?

Documentation for transitioning S3Store.

I've been using sqlalchemy-imageattach smoothly on my machine, saving to a local filesystem. But, I'm attempting to switch to an s3 store (with my application running on an EC2 instance setup by Elastic Beanstalk). Now, I'm encountering the following error:

[Fri Jul 22 00:44:12.402173 2016] [:error] [pid 9236] <open file '', mode 'w+b' at 0x7f20b536e1e0>

I'm not sure if this is related to sqlalchemy-imageattach.

When setting up the s3 store do I only need to provide the name of the bucket set up by my Elastic Beanstalk enviroment? No further configuration is required? Must I provided anything else to the S3Store initializer, i.e. some access keys, etc.?

Remove Werkzeug dependency

Currently sqlalchemy-imageattach uses Werkzeug for two reasons:

  • Context locals: we can simply bring it from Werkzeug as like Celery have done.
  • WSGI app that serves static files: itโ€™s only a small part of Werkzeug, so we can independently implement it again from scratch.

Werkzeug is a foundation of web frameworks, and there are similar ones for the same purpose e.g. WebOb. Users of web frameworks based on other foundations (for example, Pyramid is based on WebOb instead of Werkzeug) might get inconvenient by Werkzeug dependency.

Inactive Repository

I was interested in using this repository, but it seems to be no longer maintained and hasn't been touched in roughly 2 years. Is this correct?

Also has anyone found this to have caused any issues if the amazon S3 api has changed in that time?

Remove simples3 dependency

Not every user uses S3 for the backend storage, and some of them might get inconvenient by simples3 dependency.

generate_thumbnail() doesn't work right when using Multiple Image Sets

When using Multiple Image Sets generate_thumbnail() only create a thumbnail to the first image.
This is happening because generate_thumbnail() thinks a thumbnail was already generated and return the thumbnail of the first image.

I'm using Python (2.7.13), SQLAlchemy (1.1.14) and SQLAlchemy-ImageAttach (1.1.0).

One table for multiple images associated with one object/model

Is it possible to use one table for all the images associated with one object/model? It appears that two tables are required as illustrated by the UserPicture and UserFrontCover relationships in the sqlalchemy_imageattach.entity module documentation.

I'd much prefer something like:

class User(Base):
    avatar_image = image_attachment('UserImage')
    background_image = image_attachment('UserImage')

Given the above model, SQLAlchemy can't determine the join condition. When I've come across this before, I generally have the foreign key on the "parent", in this case the User model and looks like this:

class User(Base):
    avatar_image_id = Column(Integer, ForeignKey('user_images.id'))
    avatar_image = relationship('UserImage', foreign_keys=avatar_image_id)
    background_image_id = Column(Integer, ForeignKey('user_images.id'))
    background_image = relationship('UserImage', foreign_keys=background_image_id)

Is there a reason you chose to store the foreign key on the Image model? Is there a way of getting around requiring two tables? Or am I just missing some configuration option to avoid using two tables?

Python 3 support

Python 3 is future, so we also follow them. There are currently three blockers:

  • #1 simples3
  • #2 Werkzeug
  • Wand

Transactions are not multi-session safe

Consider:

a = Session()
b = Session()

add_images_to(a)
add_images_to(b)

a.commit()
b.rollback()

a.commit will commit bs images as well, because _stored_images is global state! It should be session-specific state.

uselist = False produces relationship with uselist=True

image_attachment is overriding the uselist kwargs causing sqlalchemy to still uselist in its representation. This causes code that inspects relationships to mistake the single image for a list of images.

profile_image = image_attachment('UserProfileImage', uselist=False)

I would expect proper behavior to pass the uselist kwargs unchanged.

Error generating thumbnails

Hi!
I am trying to generate thumbnails for my attached images (v.0.8.1).
Unfortunately I am getting 'NotImplementedError: object_id property has to be implemented' when calling .locate() on them. I tried everything in the documentation and debugged through the code, but could not figure out why this happens.

My code looks like this:

class UserPicture(Base, BaseModel, Image):
    user_data_id = Column(Integer, ForeignKey('user_data.id'), primary_key=True)
    user_data = relationship(UserData)
    __tablename__ = 'user_picture'
from sqlalchemy_imageattach.context import store_context

def my_method():
    with store_context(store):
        try:
            thumb = user_data.picture.find_thumbnail(width=150)
        except NoResultFound:
            thumb = user_data.picture.generate_thumbnail(width=150)
        thumb_location = thumb.locate()

I did not change anything on my primary key and it is found in the object_id property, but

pk_value = getattr(self, pk[0])

fails to get a value.

Implement `File` abstraction

This library looks great and I was just thinking that it might be useful to abstract things a bit to allow for generic file attachments to a particular SQLAlchemy model/entity. Perhaps this can be achieved with the current API but I'm having a hard time seeing if its possible given the focus on images.

joinedload (eager loading) support

Relationships defined by image_attachment() cannot be gathered using joinedload(). We seem to need to work something on attribute instrumental implementation.

Storage migration

Migration from a storage to another storage should be possible.

from sqlalchemy_imageattach.migration import migrate
migrate(engine, base, src, dst)

Default images

image_attachment() should be able to set the default image.

wand raises AttributeError in generate_thumbnail with S3Store

I haven't dug into this too terribly much, but I'm having an issue generating thumbnails using S3Store. I'll update later with more info if I can find it. Could be that this is a bug in wand, not sure yet... in any case, it seems like HTTPResponse is expected to be treated like a file like object, but wand seems to be trying to get attributes that HTTPResponse doesn't implement.

Here are the details:

picture.generate_thumbnail(width=320)
File "sqlalchemy_imageattach/entity.py", line 671, in generate_thumbnail
img = WandImage(file=f)
File "wand/image.py", line 1973, in __init__
self.read(file=file, resolution=resolution)
File "wand/image.py", line 2022, in read
fd = libc.fdopen(file.fileno(), file.mode)
AttributeError: 'HTTPResponse' object has no attribute 'mode'
Wand==0.3.8
SQLAlchemy-ImageAttach==0.8.2
Flask-SQLAlchemy==2.0
SQLAlchemy==0.9.8
Pillow==2.6.1
$ python --version                                                                                                                                         (12-17 15:45)
Python 3.4.0

UnboundLocalError: local variable 'e' referenced in S3Store object

There is some unbound error in
sqlalchemy_imageattach.stores.s3.S3Store object

 def upload_file(self, url, data, content_type, rrs, acl='public-read'):
...
        while 1:
            try:
                urllib2.urlopen(request).read()
            except urllib2.HTTPError as e:
                if 400 <= e.code < 500:
                    self.logger.exception(e)
                    self.logger.debug(e.read())
                    raise
                self.logger.debug(e)
                continue
            except IOError:
>               self.logger.debug(e)
E               UnboundLocalError: local variable 'e' referenced before assignment

Failing tests

    def test_print():
        with os.popen('python -m sqlalchemy_imageattach.version') as pipe:
            printed_version = pipe.read().strip()
>           assert printed_version == VERSION
E           assert '' == '0.8.1'
E             + 0.8.1

version_test.py:30: AssertionError

Under python 2.7.5

I do not understand how this should work with (pydantic model) validation

I am trying to use imageattach, and I do not really understand how it should work, how dis/similar it is to just a normal relationship.

Below is a snippet of code, which is mimicking something like a StackOverflow question board, where users could post questions and the questions could have images attached to them. I am following a style quite similar to FastAPI's SQLAlchemy tutorial.

I am able to get users, create users, and get questions. But when I try to create a question (POST a question), I get the following pydantic validation error:

pydantic.error_wrappers.ValidationError: 1 validation error for Question
response -> pictures
  value is not a valid list (type=type_error.list)

I am fairly sure this is an issue with pydantic having trouble understanding the relationship that sqlalchemy-imageattach creates, which is why I am posting here.


schemas.py (Pydantic models)

from typing import List, Optional
from pydantic import BaseModel

class QuestionPictureBase(BaseModel):
    pass

class QuestionPicture(QuestionPictureBase):
    id: int
    question_id: int
    class Config:
        orm_mode = True

class QuestionBase(BaseModel):
    header: str
    details: str

class QuestionCreate(QuestionBase):
    pass

class Question(QuestionBase):
    id: int
    creator_id: int
    pictures: List[QuestionPicture] = []

    class Config:
        orm_mode = True

class UserBase(BaseModel):
    email: Optional[str] = None
    name: str  # This is the nickname they want to go by
    username: str  # This is the unique user name

class UserCreate(UserBase):
    password: str

class User(UserBase):
    id: int
    is_active: bool
    questions: List[Question] = []

    class Config:
        orm_mode = True

Then the SQL ORM models:

models.py

from sqlalchemy import Boolean, Column, ForeignKey, Integer, String, Text
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy_imageattach.entity import Image, image_attachment

from .database import Base

class User(Base):
    __tablename__ = "users"
    id = Column(Integer, primary_key=True, index=True)
    name = Column(String)
    username = Column(String, unique=True, index=True)
    email = Column(String)
    hashed_password = Column(String)
    is_active = Column(Boolean, default=True)
    questions = relationship("Question", back_populates="creator", uselist=True)

class Question(Base):
    __tablename__ = "questions"
    id = Column(Integer, primary_key=True, index=True)
    header = Column(String, index=True)
    details = Column(String, index=True)
    creator_id = Column(Integer, ForeignKey("users.id"))
    pictures = image_attachment("QuestionPicture", uselist=True)
    creator = relationship("User", back_populates="questions")

class QuestionPicture(Base, Image):
    """Model for pictures associated with the question."""
    __tablename__ = "question_pictures"
    id = Column(Integer, primary_key=True)
    question_id = Column(Integer, ForeignKey("questions.id"))
    question = relationship("Question", back_populates="pictures")

Then the CRUD module (edited down to just the function of interest):

crud.py

from sqlalchemy.orm import Session
from . import models, schemas

def create_user_question(db: Session, question: schemas.QuestionCreate, username: str):
    user = get_user_by_username(db=db, username=username)
    db_item = models.Question(**question.dict(), creator_id=user.id)
    db.add(db_item)
    db.commit()
    db.refresh(db_item)
    return db_item

And finally, the main.py module, again edited just to focus upon question creation.

from typing import List
from fastapi import Depends, FastAPI, HTTPException
from sqlalchemy.orm import Session
from . import crud, models, schemas
from .database import SessionLocal, engine

app = FastAPI()

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

@app.post("/users/{username}/questions/", response_model=schemas.Question)
def create_question_for_user(
    username: str, question: schemas.QuestionCreate, db: Session = Depends(get_db)
):
    return crud.create_user_question(db=db, question=question, username=username)

Partitioned storages

Sometimes image files should be spread to two or more physical stores by their object_type. We could implement partitioned storage that implements Store interface e.g.:

PartitionedStore([
    ('user_picture', user_store),
    (re.compile(r'^comic_'), comic_store),
    (['post_attachment', 'comment_attachment'], forum_store)
], default=default_store)

Multiple files attached to an object (Reverse one to many from examples)

Lets say I have a Model Event. An Event can have N attachments represented by EventAttachment model. All the logic seems to be suited to the one case where there is a FK from Event to the file attachment stored on EventAttachment. Obviously i could manage this manually but it seems all the logic for saving and working with files is wrapped up in the ImageSet query class. Am i missing something here?

class Event(db.Model):
    __tablename__ =  "event"
   id = Column(Integer, primary_key=True)


class EventAttachment(db.Model, BaseMixin, Image):
    __tablename__ = 'event_attachment'

    id = Column(Integer, primary_key=True)
    event_id = Column(Integer, ForeignKey('event.id'), nullable=False)

Ideally i want to have N EventAttchement instances with the logic found in the ImageSet class. Ie i can save multiple EventAttachments each with a file directly by doing something like

attachment = EventAttachment(event_id=1)
attachment.from_file(f)

Thanks.

find_thumbnail doesn't work on ImageSubset

I might not use the library as expected, but I could not find a way to use the find_thumbnail method on a ImageSubset object (returned by get_image_set). I get the following error :
AttributeError: 'ImageSubset' object has no attribute 'filter_by'

Looking in the code, it doesn't seem surprising since ImageSubset is not a descendant of BaseImageQuery, and the method filter_by is called on the self variable.

Could you please let me know how this should be done ?
Here is a snippet of my code (Python 3, imageattach version 1.0.0):

class EventPicture(db.Model, Image):
    __tablename__ = 'event_picture'
    event_id = db.Column(db.Integer, db.ForeignKey('event.id'), primary_key=True)
    event = db.relationship("Event")
    order_index = db.Column(db.Integer, primary_key=True)  

    @property
    def object_id(self):
        key = '{0},{1}'.format(self.event_id, self.order_index).encode('ascii')
        return int(sha1(key).hexdigest(), 16)

class Event(db.Model):
    __tablename__ = 'event'
    pictures = image_attachment('EventPicture', uselist=True)

    def get_photo(self, i, width=None, height=None):
    """ get the photo with index i and given width/height """
        with store_context(store):
            image_set = self.pictures.get_image_set(order_index=i)
            image_set.find_thumbnail(width=width,height=height) # This is where the exception occurs
           # find_thumbnail tries to execute filter_by on self, but ImageSubset is not a query, it's query is in it's query attribute.

If I reproduce the code of find_thumbnail but replacing the q variable with image_set.query, then it works smoothly.

Thanks a lot,
Best regards,
Antoine

How to set no photo

As a user of sqlalchemy-imageattach I'd like to set the model's image to blank (like when there is no image) so that I can give my clients a web interface to delete the photo.
#1 I'd like it to be gone from the association with the model (to a state identical to when the association not yet happened).
#2 I'd like the physical image (original and thumbnails) all be deleted as well for security reasons. This can be optional.

Perhaps the feature is there, I'm just not finding the documentation?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.