ponyorm / pony

Pony Object Relational Mapper

Downloads

Pony Object-Relational Mapper

Pony is an advanced object-relational mapper. The most interesting feature of Pony is its ability to write queries to the database using Python generator expressions and lambdas. Pony analyzes the abstract syntax tree of the expression and translates it into a SQL query.

Here is an example query in Pony:

select(p for p in Product if p.name.startswith('A') and p.cost <= 1000)

Pony translates queries to SQL using a specific database dialect. Currently Pony works with SQLite, MySQL, PostgreSQL and Oracle databases.

By providing a Pythonic API, Pony facilitates fast app development. Pony is an easy-to-learn and easy-to-use library. It makes your work more productive and helps to save resources. Pony achieves this ease of use through the following:

• Compact entity definitions
• The concise query language
• Ability to work with Pony interactively in a Python interpreter
• Comprehensive error messages, showing the exact part where an error occurred in the query
• Displaying of the generated SQL in a readable format with indentation

All this helps the developer to focus on implementing the business logic of an application, instead of struggling with a mapper trying to understand how to get the data from the database.

See the example here

Support Pony ORM Development

Pony ORM is Apache 2.0 licensed open source project. If you would like to support Pony ORM development, please consider:

Become a backer or sponsor

Online tool for database design

Pony ORM also has the Entity-Relationship Diagram Editor which is a great tool for prototyping. You can create your database diagram online at https://editor.ponyorm.com, generate the database schema based on the diagram and start working with the database using declarative queries in seconds.

Documentation

Documentation is available at https://docs.ponyorm.org The documentation source is available at https://github.com/ponyorm/pony-doc. Please create new documentation related issues here or make a pull request with your improvements.

License

Pony ORM is released under the Apache 2.0 license.

PonyORM community

Please post your questions on Stack Overflow. Meet the PonyORM team, chat with the community members, and get your questions answered on our community Telegram group. Join our newsletter at ponyorm.org. Reach us on Twitter.

Copyright (c) 2013-2022 Pony ORM. All rights reserved. info (at) ponyorm.org

pony's Issues

pony.orm.dbapiprovider.OperationalError: (1054, "Unknown column 'User.password' in 'field list'")

when i init the database and then

 db.generate_mapping(create_tables=True)

it's ok
but, after i modify the field add a new field
then i use

db.generate_mapping(create_tables=True)

i have the error

OperationalError: 1205 Lock wait timeout exceeded; try restarting transaction

my web frame is Tornado; database is mysql 5.5;
tornado is single process;so in my code i ping the database every 6000ms (ping_db)

@db_session
def main():
    #db.generate_mapping()
    tornado.options.parse_command_line()
    tornado.httpserver.HTTPServer(Application(),
            xheaders=True).listen(options.port)
    print("App started. Listenning on %d" % options.port)
    ping_db = lambda: db.select("name from User where uid = $x", {"x": 5})
    tornado.ioloop.PeriodicCallback(ping_db, 600 * 100).start()

    tornado.ioloop.IOLoop.instance().start()

now i want to save user profile in two ways, the question from the first way to save; as following：

code 1

1 .

    @db_session
    def save(self, user):
        data = self.data #{'birthday': time.time(), 'gender':1}
        user.set(**data)

error detail.

[E 131114 17:47:14 ioloop:816] Error in periodic callback
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 814, in _run
        self.callback()
      File "app.py", line 50, in <lambda>
        ping_db = lambda: db.select("name from User where uid = $x", {"x": 5})
      File "<string>", line 2, in select
      File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 96, in cut_traceback
        return func(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 486, in select
        cursor = database._exec_raw_sql(sql, globals, locals, frame_depth + 2)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 482, in _exec_raw_sql
        return database._exec_sql(adapted_sql, arguments)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 544, in _exec_sql
        if not cache.saving and not cache.optimistic and cache.modified: cache.flush()
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3706, in flush
        if cache.modified: cache.save()
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3720, in save
        obj._save_()
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3591, in _save_
        elif status == 'updated': obj._save_updated_()
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3520, in _save_updated_
        cursor = database._exec_sql(sql, arguments)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 550, in _exec_sql
        try: new_id = provider.execute(cursor, sql, arguments, returning_id)
      File "<string>", line 2, in execute
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/dbapiprovider.py", line 47, in wrap_dbapi_exceptions
        except dbapi_module.OperationalError, e: raise OperationalError(e)
    OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

this is the mysql log

131114 18:15:15   143 Query SELECT `uid`, `name`, `password`, `avatar`, `head_img`, `birthday`, `gender`, `role`, `token`, `created_at`, `modified_at`, `description`, `is_blocked`, `phone`, `phone_active_code`, `email`, `miles`, `time`, `engry`, `collection_count`
FROM `User`
WHERE `uid` = 21
          143 Query UPDATE `User`
SET `birthday` = '1384338245.795'
WHERE `uid` = 21
131114 18:16:06   143 Query rollback
          143 Query rollback
          143 Query select name from User where uid = 5

* i save the data, wait a long time i get the error ( 51833.88ms
) , from the error i can see the error occur when ping_db; but i don't know why ? the way save data will get the error but the second way is ok*

but the second way have no problem code2

    @db_session
    def save(self, user):
        data = self.data  # {'birthday': time.time(), 'gender':1}
        if 'birthday' in data:
             user.birthday = data['birthday']
        if 'gender' in data:
             user.gender = data['gender']
        user.save()

How to disconnect from db and drop all tables?

Hi, currently I am covering flask app with unit tests and I need to initialize database before each test starts and drop database after test is finished. But seems like there is no "easy" way to do it with PonyORM.

What I want is something like how it works with sqlalchemy http://pythonhosted.org/Flask-Testing/:

from flask.ext.testing import TestCase

from myapp import create_app, db

class MyTest(TestCase):

    SQLALCHEMY_DATABASE_URI = "sqlite://"
    TESTING = True

    def create_app(self):

        # pass in test configuration
        return create_app(self)

    def setUp(self):

        db.create_all()

    def tearDown(self):

        db.session.remove()
        db.drop_all()

Please advice: how can I disconnect from db and delete it? Is it possible?

pony.orm.core.ERDiagramError: Mapping is not generated for entity 'Person'

I tried to follow instructions: http://doc.ponyorm.com/firststeps.html

from pony.orm import *

db = Database("mysql", host="192.168.1.117", user="usr", passwd="pwd", db="test")

class Person(db.Entity):
    name = Required(unicode)
    age = Required(int)
    cars = Set("Car")

class Car(db.Entity):
    make = Required(unicode)
    model = Required(unicode)
    owner = Required(Person)

sql_debug(True)
#db.generate_mapping(create_tables=True) # --->>>> I commented this after tables creation otherwise I received pony.orm.core.TransactionError: db_session is required when working with the database

p1 = Person(name='John', age=20)
p2 = Person(name='Mary', age=22)
p3 = Person(name='Bob', age=30)
c1 = Car(make='Toyota', model='Prius', owner=p2)
c2 = Car(make='Ford', model='Explorer', owner=p3)
commit()

exception:
pony.orm.core.ERDiagramError: Mapping is not generated for entity 'Person'

What am I doing wrong? :
Desired tables were generated inside database, but that's all what I can get.

TypeError: an integer is required

When I run a web app with something like

class user(db.Entity):
    joined = Required(datetime)

user1 = user(joined=datetime.now())

I get

  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3546, in new_func
    with self: return old_func(*args, **kwargs)
  File "./app.py", line 76, in create
    user(joined=datetime.now())
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 74, in new_func
    return old_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2746, in __new__
    pkval, avdict = entity._normalize_args_(kwargs, True)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2217, in _normalize_args_
    avdict[attr] = attr.check(val, None, entity, from_db=False)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1123, in check
    val = Attribute.check(attr, val, obj, entity, from_db)  # val may be changed to None here
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 832, in check
    return attr.py_type(val)
TypeError: an integer is required

Changelog

It would be nice to have a changelog in github repo or on the site which has all the changes between versions. I know you post changes in the blog but a single changelog file is a standard practice and is much easier to follow.

Thanks.

"not in"

Hi,

"not in" in a query seems to emit the exact same SQL as "in".

The last 2 outputs from the below script are:

SELECT "b"."id", "b"."name"
FROM "Blah" "b"
WHERE "b"."name" LIKE '%hello%'

It seems in your tests there is

test_declarative_sqltranslator2.py:        q = select(g for g in Group if 'John' not in g.students.name and g.dept == Department[1])

Is there a difference between your test and mine? My test below, which I ran with pony==0.4.9:

from pony.orm import *

sql_debug(True)
db = Database('sqlite', ':memory:')

class Blah(db.Entity):
    name = Required(unicode)

db.generate_mapping(create_tables=True)

with db_session:
    x = Blah(name='hello frank')
    y = Blah(name='hello john')

with db_session:
    print 'No "in"'
    print select(b for b in Blah)[:]

    print '"in"'
    print select(b for b in Blah if 'hello' in b.name)[:]

    print 'not "in"'
    print select(b for b in Blah if 'hello' not in b.name)[:]

Query results cache is not cleared after commit

========more test_pony.py===========
#!/usr/bin/env python
#-- coding: utf-8 --

from pony.orm import Database, Required, sql_debug
from pony.orm import select, commit

db = Database('sqlite', ':memory:')


class User(db.Entity):
    email = Required(unicode)
    password = Required(unicode)

db.generate_mapping(create_tables=True)
sql_debug(True)

for i in range(2):
    user = User(email='1', password='1')
    commit()
    users = list(select(u for u in User))
    print users
    print id(users)

users = list(select(u for u in User))
print users

=======python test_pony.py==========
OPTIMISTIC ROLLBACK

INSERT INTO "User" ("email", "password") VALUES (?, ?)
[u'1', u'1']

COMMIT

SELECT "u"."id", "u"."email", "u"."password"
FROM "User" "u"

[User[1]]
4453021024
OPTIMISTIC ROLLBACK

INSERT INTO "User" ("email", "password") VALUES (?, ?)
[u'1', u'1']

COMMIT

[User[1]]
4453021600
SELECT "u"."id", "u"."email", "u"."password"
FROM "User" "u"

[User[1], User[2]]

select only once if in loop, as it's lazy?

Implement ORDER BY RANDOM()

I was wondering if it was possible to provide support for things like ORDER BY RANDOM(), and if not if it can be implemented.
At the moment I'm just doing random.sample(Database.select()[:], 12) but I was hoping for a better method.

Ambiguous column on inheritance

In pony/orm/examples/inheritance1.py at the end:

for p in select(s for s in Student if s.mentor.salary == 1000):
    print p.name

and I got:

    pony.orm.dbapiprovider.OperationalError: ambiguous column name: classtype

NameError

code tornado web

class GetProfileHandler(BaseHandler):

    @db_session
    @authuser
    def get(self):
        try:
            uid = self.get_argument('uid')
            x = uid
            try:
                user = db.get("uid, name \
                        where uid = $x") # error from
                self.handleres(user)
            except RowNotFound:
                self._res(Error.error['ParameterError'], \
                        'uid not exists', False)
        except HTTPError as e:
            id = self.current_user.uid 
            user = db.get("uid, name, birthday, avatar,\
                    where uid = $x", {"x": id})#ok
            self.handleres(user)

error

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1141, in _when_complete
    callback()
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1162, in _execute_method
    self._when_complete(method(*self.path_args, **self.path_kwargs),
  File "
<string>", line 2, in get
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 331, in new_func
    try: return func(*args, **kwargs)
  File "/home/zh/walkout/utils.py", line 89, in wrapper
    return method(self, *args, **kwargs)
  File "/home/zh/walkout/controllers/user.py", line 248, in get
    where uid = $x")
  File "
    <string>", line 2, in get
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 96, in cut_traceback
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 504, in get
    rows = database.select(sql, globals, locals, 2)
  File "
        <string>", line 2, in select
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 96, in cut_traceback
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 486, in select
    cursor = database._exec_raw_sql(sql, globals, locals, frame_depth + 2)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 481, in _exec_raw_sql
    arguments = eval(code, globals, locals)
  File "
            <?>", line 1, in 
            <module>
NameError: name 'x' is not defined

from the document

the first way get the variable from the current frame(from globals and locals), i don't quite see the word, so i want to know where should put the variable in ?

Query select().orderby().for_update() fails in oracle. Pony v0.4.9

In General

Query select().orderby().for_update() fails in oracle

Two errors have been found:

• pony.orm.dbapiprovider.DatabaseError: ORA-00936: missing expression
• pony.orm.dbapiprovider.DatabaseError: ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.

In Detail

here's my program

class TryOrderByForUpdate(db.Entity):
    _table_ = "TRY_ORDERBY_FORUPDATE"
    t = Required(datetime, default=datetime.now)

@db_session
def select_orderby_forupdate():
    x = select(each for each in TryOrderByForUpdate).for_update().order_by(TryOrderByForUpdate.t)[:3]

When select_orderby_forupdate() is executed, it reports: pony.orm.dbapiprovider.DatabaseError: ORA-00936: missing expression

And I tried to set sql_debug() on, and I found the sql statement is actually not correct:

SELECT * FROM (
    SELECT "each"."ID", "each"."T"
    FROM "TRY_ORDERBY_FORUPDATE" "each"
    ORDER BY "each"."T"
) WHERE ROWNUM <= 3FOR UPDATE

And I tried to modified sqlbuild.py to add extra space before "FOR Update" statement, the first problem gone but a second problem shows: ORA-02014: cannot select FOR UPDATE from view with DISTINCT, GROUP BY, etc.

And this time, the SQL statement becomes

SELECT * FROM (
    SELECT "each"."ID", "each"."T"
    FROM "TRY_ORDERBY_FORUPDATE" "each"
    ORDER BY "each"."T"
) WHERE ROWNUM <= 3 FOR UPDATE

Although it looks right, but shouldn't it be:

    SELECT "each"."ID", "each"."T"
    FROM "TRY_ORDERBY_FORUPDATE" "each"
    WHERE ROWNUM <= 3 FOR UPDATE
    ORDER BY "each"."T"

so, overall, I think there are two problems(although I'm not sure about the second one):

1. "FOR UPDATE" should be seperated from the statements in the front
2. "WHERE ROWNUM <= x" shouldn't be in the outter select but rather in the original one. The outter select is not needed

orm datetime attributeerror

Hey

I am trying to run this line of code but its throwing an attribute error:

models.count(j for j in models.Job if j.email == email and j.created.date() == datetime.datetime.today().date())

I tested and its Job.created that has no attribute 'date'. Is there another way to implement this, or is it a bug or what?

str2datetime(s) has a bug

when datetime str has 'T' ,this will go wrong .
so we use
s = s.strip().lower().replace('t',' ')
instead
s = s.strip().lower()

see detail at
https://gist.github.com/zihuxinyu/903785cf41199d7febd5

AttributeError: 'Query' object has no attribute 'orderby'

AttributeError: 'Query' object has no attribute 'orderby'

Erroneous complaints of "(fieldname) cannot be set to None"

This sample program runs successfully the first time, in which it
initially creates an Account object, but it fails the second time
with...

pony.orm.core.ConstraintError: Attribute Token.account cannot be set
to None

...even though it has successfully looked up an Account object.

import uuid
from datetime import datetime
from pony.orm import *

db_type = 'sqlite'
db = Database('sqlite', 'sample.db', create_db=True)

class Token(db.Entity):
    token   = Required(unicode, index=True)
    created = Required(datetime, default=datetime.utcnow())
    account = Required('Account')
    valid   = Required(bool, default=True)

class Account(db.Entity):
    address  = PrimaryKey(unicode)
    created  = Required(datetime, default=datetime.utcnow())
    token    = Optional(Token)
    password = Optional(unicode)

db.generate_mapping(create_tables=True)

with db_session:
    try:
        acc = Account['[email protected]']
    except ObjectNotFound:
        acc = Account(address='[email protected]')

    assert(acc is not None)

    token = Token(
            account=acc,
            token=uuid.uuid4().hex)

ExprEvalError(src, cause) raised when using lambdas in expression after upgrade to 4.8

job = models.Job.get(lambda j: j.id == id and j.user.id == user.id) or abort(404)

File "", line 2, in get

File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 95, in cut_traceback
return func(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2262, in get
return entity.query_from_lambda(first_arg, globals, locals).get()
File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2574, in query_from_lambda
return Query(code_key, inner_expr, globals, locals)
File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3763, in init
vars, vartypes = extract_vars(extractors, globals, locals)
File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3737, in extract_vars
except Exception, cause: raise ExprEvalError(src, cause)
ExprEvalError: user.id raises NameError: name 'user' is not defined

And here is the relevant code:

user = theuser
job = models.Job.get(lambda j: j.id == id and j.user.id == user.id) or abort(404)

user is a valid orm object. I have had this same bug in a couple places now. I can fix it in code by changing it to something like:
models.select(j for j in models.Job if ... ) or abort(404)

Assertion failed: assert old_dbval is not NOT_LOADED

  File "..../core.py", line 1493, in db_update_reverse
    if new_dbval is not None: reverse.db_set(new_dbval, obj, True)
  File "..../core.py", line 1444, in db_set
    assert old_dbval is not NOT_LOADED
AssertionError

#!/usr/bin/python
from pony.orm import *
db = Database()
class CommonItem(db.Entity):
    url = Required(unicode, unique=False, lazy=False)
    act_item1 = Optional("ActItem1")
class ActItem1(db.Entity):
    regnum = Required(unicode, unique=False, lazy=False)
    common_item = Required("CommonItem")
def main():
    sql_debug(True)
    db.bind('sqlite', ':memory:', create_db=True)
    db.generate_mapping(create_tables=True)
    with db_session:
        c1 = CommonItem(url='http://asd.com')
        a1 = ActItem1(regnum='rrr', common_item=c1)
    with db_session:
        for r in select(r for r in CommonItem if r.act_item1.id):
            print (r.url,r.act_item1.regnum)
if __name__ == '__main__':
    main()

How to use a separate database for testing?

I am trying to integrate Pony with Flask-Testing (unsuccessfully, so far--see #25 )--and I'm wondering how I can use a separate database for testing. In Flask-SQLAlchemy, for example, this would be achieved by setting the SQLALCHEMY_DATABASE_URI config option in the test setup

from flask.ext.testing import TestCase
from app import app
class TestApp(TestCase):
    TESTING = True
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///tmp/test.db'

    def create_app(self):
        app.config.from_object(self)
        return app

I am not sure how to achieve this with Pony, since each Entity class must be declared from a Database that is already configured.

PostgreSQL uuid & hstore types

This project looks fantastic :)

I grepped for uuid in the examples, yet found no instances.

Would love to see support / examples.

Where's the buy now :) ?

PonyORM does not escape _ and % in LIKE queries

So,

select(r for r in ActItem if '_' in r.regnum)

generates

SELECT "r"."id", "r"."regnum", "r"."common_item"
FROM "ActItem" "r"
WHERE "r"."regnum" LIKE '%_%'

effectively selecting ALL records since _ mean ANY symbol in LIKE-queries.

At least, problem in sqlite. You should read this:
http://sqlite.org/lang_expr.html#like

Note, that in some databases syntax is not same. For example, for Firebird:
`SELECT ... FROM ... WHERE last_name LIKE '%A_B%' ESCAPE ''``

Serialize Object to JSON

What is the accepted method to serialize an object to JSON, especially when those objects have nested relationships?

Filtering a query multiple times.

here's a simplified snippet of my code:

query = select(x for x in Model)

filters = [['lambda x: x.status == v', u'pending'], ['lambda x: x.order_group.id == v', u'1']]

for query_str, v in filters:
    query = query.filter(query_str)

the "filters" list is actually generated on the fly. The error I'm getting is:

pony.orm.asttranslation.TranslationError

TranslationError: Meaning of expression v has changed during query translation

I think this is because the conversion is happening lazily, but it seems like you are using this method in: #20 (comment)

Is there a way to force the evaluation, or am I missing something?

Thanks

PS - It seems I can't use filter(**kwargs), as my lambda expressions aren't just "==".

Python 3 support!

UPSERT operator required

The upsert operation is common:

http://en.wikipedia.org/wiki/Upsert
http://www.the-art-of-web.com/sql/upsert/ (postgres)
https://ru.wikipedia.org/wiki/Merge_(SQL)

It will be nice if you implement it in PonyORM:

1. In analogue with с dict.setdefault:
   Table.setdefault(product_name='socks', _defaultfields={price:5}).price += 10
2. In analogue with collect.defaultdict
Table.setdefault(product_name='socks', _create_default=lambda: Table(price=5)).price += 10

or both.

Removing/adding the same object for m2m relation inside one transaction causes integrity error

Database: MySQL
Data model (inherited from django.contrib.auth with help of djony):
User *-* Permission

Test case pre-conditions:

>>> list(u.user_permissions)
[djony_auth_Permission[106]]

Actions sequence and debug output:

>>> p = list(u.user_permissions)[0]
>>> u.user_permissions.remove(p)
>>> u.user_permissions.add(p)
>>> u._database_.commit()
EXECUTEMANY (1)
INSERT INTO `auth_user_user_permissions` (`permission_id`, `user_id`) VALUES (%s, %s)
[106, 2]

ROLLBACK
RELEASE_CONNECTION
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<string>", line 2, in commit
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 96, in cut_traceback
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 463, in commit
    if cache is not None: cache.commit()
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3658, in commit
    cache.save()
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 3724, in save
    attr.add_m2m(added)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1885, in add_m2m
    database._exec_sql(sql, arguments_list)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 550, in _exec_sql
    try: new_id = provider.execute(cursor, sql, arguments, returning_id)
  File "<string>", line 2, in execute
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/dbapiprovider.py", line 46, in wrap_dbapi_exceptions
    except dbapi_module.IntegrityError, e: raise IntegrityError(e)
IntegrityError: (1062, "Duplicate entry '2-106' for key 'user_id'")

The same happens when using .clear() instead of .remove()

The particular use case when it may be necessary:

1. We already have many Permission objects connected to the User, and editing the user.
2. In order to edit, we have sent to the client the User object together with Permission object reference set describing it's permissions.
3. The client changes permissions reference set and returns user and permissions reference set back.
4. The server wants to cleanup user permissions set and fill it in again, to avoid analysing every permission presence separately for this user.

Cannot use memcached? TransactionRolledBack, 'Object belongs to obsolete cache'

When I use memcached for cache, it throw the error:

 File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 871, in __get__
    if not obj._cache_.is_alive: throw(TransactionRolledBack, 'Object belongs to obsolete cache')

How do I use the memcached?

Use table in schema

By default pony generate SQL without schema. For example

select * from users

How add schema to Entity? After add it shuld generate

select * from usr.users

AssertionError

i use pony 0.4.8 with tornado3.1.1! python 2.6.6

when i do

@db_session
def get('/'):
      code body

if code body has error! then i got

Traceback (most recent call last):
      File "/usr/local/lib/python2.6/dist-packages/tornado/web.py", line 1141, in _when_complete
        callback()
      File "/usr/local/lib/python2.6/dist-packages/tornado/web.py", line 1162, in _execute_method
        self._when_complete(method(*self.path_args, **self.path_kwargs),
      File "<string>", line 2, in get
      File "/usr/local/lib/python2.6/dist-packages/pony/orm/core.py", line 3612, in new_func
        with self: return func(*args, **kwargs)
      File "/usr/local/lib/python2.6/dist-packages/pony/orm/core.py", line 3630, in __exit__
        assert isinstance(exc_value, exc_type)
    AssertionError

so i only know i got a error, but i cannot know where the error occur!

i read the source code then i add logging in

orm -> core.py 3625 line

logging.info(exc_value, exc_type, '-=-=-=-=-=-=-=-=')

the log message as follow:

[I 130911 16:54:12 core:3625] Bad message (TypeError('not all arguments converted during string formatting',)): 
{
  'threadName': 'MainThread',
  'name': 'root',
  'thread': 140679896413952,
  'created': 1378889652.9263,
  'process': 12220,
  'processName': 'MainProcess',
  'args': (<type'exceptions.NameError'>,
  '-=-=-=-=-=-     =-=-='),
  'module': 'core',
  'filename': 'core.py',
  'levelno': 20,
  'exc_text': None,
  'pathname': '/usr/local/lib/python2.6/dist-packages/pony/orm/core.py',
  'l     ineno': 3625,
  'msg': "global name 'logging' is not defined",
  'exc_info': None,
  'funcName': '__exit__',
  'relativeCreated': 9760.4219913482666,
  'levelname': '     INFO',
  'msecs': 926.300048828125
}

from the log i can see the error message

global name 'logging' is not defined

so i hope the @db_session can throw the detail error for debug

asyncio + futures support

how to update cache or force recaching

Hi, i'm working on application and apparently there as conflict between the current data and the cached data. so is there away to force reacaching on the new data available ?

Typo in _name_parts_re

https://github.com/ponyorm/pony/blob/orm/pony/utils.py#L120

It appears that this regex is missing a closing bracket on this line. Should it be "[a-z]+" instead?

Error connecting to Oracle without username and password

With Pony if I try:

db = Database('oracle', '/@live')

It gives

---------------------------------------------------------------------------
DatabaseError                             Traceback (most recent call last)
<ipython-input-5-a4c63a535625> in <module>()
----> 1 db = Database('oracle', '/@live')

C:\Python27\lib\site-packages\pony\orm\core.pyc in __init__(self, provider, *args, **kwargs)

C:\Python27\lib\site-packages\pony\utils.pyc in cut_traceback(func, *args, **kwargs)
    111             if tb.tb_frame.f_globals.get('__name__') == 'pony.utils' and tb.tb_frame.f_code.co_name == 'throw':
    112                 raise exc_type, exc, last_pony_tb
--> 113             raise exc  # Set "pony.options.CUT_TRACEBACK = False" to see full traceback
    114         finally:
    115             del tb, last_pony_tb

DatabaseError: ORA-01017: invalid username/password; logon denied

But in cx_Oracle I can do

db_conn = cx_Oracle.connect('/@live')

and it works fine

Multiple commit unsuccessful

In my web app, multiple commit unsuccessful
code is:

            message_box1 = current_user.get_message_box(user=user)
            message_box2 = user.get_message_box(user=current_user)
            if not message_box1:
                message_box1 = MessageBox(sender_id=current_user.id,
                        receiver_id=user.id, status=1).save()
            if not message_box2:
                message_box2 = MessageBox(sender_id=user.id,
                        receiver_id=current_user.id).save()

            message = form.save(user_id=current_user.id,
                    message_box1_id=message_box1.id,
                    message_box2_id=message_box2.id)

Every save() method has the commit() method.
But if the MessageBox().save() in the if block run, the last commit() in form.save() cannot run. no error throw out.
If the MessageBox().save() in the if block not run, the last commit() in the form.save() can run successfully.

UnrepeatableReadError: Object was updated outside of current transaction

I use MySQL, how to solve this problem? thanks

filter() does not work on a plain select()

Consider these two queries:

Model.select()

Model.select(lambda x: x.active)

If filter(lambda x: x.y == 'a') is applied, it will fail with the former query, as "x" has not already been defined.

I realise this is an edge case, but would it be possible in filter to define the value of "x" (or any other variable name that is passed into the lambda) to exist for the purpose of filtering?

If not, I guess people can use Model.select(Lambda x: True) or select(x for x in Model) before doing filtering.

Thanks for your work on pony!

Default value for required attribute ... cannot be empty string

(File: orm/core.py)
This exception is throwing while I am trying to setup a field like Required(unicode,default=u'')

Why? It doesn't correspond to any standard IMHO.

The required attribute is mapped to 'NOT NULL' DDL clause of the SQL. The 'NOT NULL' clause restricts having NULL value of the field, but allows any other value, including empty (0 for integer or empty string for varchar field).

The restriction to not have an empty string in the Required string attribute is the same as to not have a 0 value in the Required integer field. It does not have any meaning for me.

Object %s cannot be used after the database session is over

      File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 81, in new_func 
        return old_func(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 874, in __get__
        if not obj._cache_.is_alive: throw_db_session_is_over(obj)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2699, in throw_db_session_is_over
        throw(TransactionRolledBack, 'Object %s cannot be used after the database session is over' % obj)
      File "/home/cmbuild/github/collipa/models/user.py", line 84, in __str__
        return self.nickname or self.name
      File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 80, in new_func
        if not (pony.MODE == 'INTERACTIVE' and options.CUT_TRACEBACK):
    RuntimeError: maximum recursion depth exceeded in cmp

I got this error after upgrade pony to the latest version. After checking the release notes,
I found even if the read only operation also requires @db_session.

Now Pony eliminates this problem and requires that all database interactions be done 
within db_session,  even if all of the operations performed are read-only. 

But for the free version, there are only 50 simultaneous Python processes. Is that bring the issue?

Any comments are welcome.

Thanks,
laofo

How to filter query inside group statement

Title probably doesn't describe this, wasn't sure how to explain it.

Basically, I want to filter my DB columns via url parameters. With a normal select statement, I have this working fine. However I now need to run the filters on a select statement within another select statement and then call .count() on it, the following example will probably explain more.

def model_with_filters(model, aid, filters):
    query = select(x for x in model)
    for column_name, value in { k: v for k, v in filters.iteritems() if k in model._columns_ and v != ""}.iteritems():
        query = query.filter("lambda x: x.%s == '%s'" % (column_name, value))
    return query.filter("lambda x: x.aff_id== aid")

# This is what I want to do
select((a, model_with_filters(SendPin, a.aff_id, request.args).count) for a in Affiliate)

TypeError: Function 'model_with_filters' cannot be used inside query

However I am not allowed to use a function within the select query. What's the recommended way of being able to apply filters to the count statement if I can't call a function in there?

Also, using the lambdas like above seems really funky, but it's the only way I could get it to work.

Efficient lookup in many-many relationship

I have the following schema, where the relationship between Executable and Symbol is many-to-many.

    class File(db.Entity):
        loc = Required(str, unique=True)
        tim = Optional(datetime)

    class Executable(File):
        sym = Set("Symbol")

    class Symbol(db.Entity):
        sig = Required(str, 5000, encoding='utf-8')
        exe = Set(Executable)

A foreign-key table called Executable_Symbol would be created by Pony to store this relationship, but there seems to be no way to check whether a particular relationship exists via the ORM unless I drop down to raw SQL, i.e.

eid,sid = exe.id,sym.id
db.select('* from Executable_Symbol where executable=$eid and symbol=$sid ')

I figured the best way of doing this is that if I have a Symbol called sym, and an Executable called exe, I can use the expression:

exe in sym.exe

But this seems to be very slow. In comparison, accessing the Executable_Symbol table using raw SQL is much faster, but dropping to raw SQL is not very desirable. My application would check this a few hundred thousand times, so every bit of efficiency would be useful.

Is there a better way to do this?

thanks!

IndexError: list index out of range when One-to-one relation and 'is None'

I just want to select ... from CommonItem where exists(select * from actItem where actitem.common_item=CommonItem.id)

I decide to write this (incorrect):

#!/usr/bin/python
from pony.orm import *
db = Database()
class CommonItem(db.Entity):
    url = Required(unicode, unique=False, lazy=False)
    act_item1 = Optional("ActItem1")
class ActItem1(db.Entity):
    regnum = Required(unicode, unique=False, lazy=False)
    common_item = Required("CommonItem")
def main():
    sql_debug(True)
    db.bind('sqlite', ':memory:', create_db=True)
    db.generate_mapping(create_tables=True)
    with db_session:
        for r in select(r for r in CommonItem if r.act_item1 is not None):
            print r.url
if __name__ == '__main__':
    main()

gives traceback:

.....
  File "...../sqlbuilding.py", line 278, in WHERE
    conditions = flat_conditions(conditions)
  File "...../sqlbuilding.py", line 70, in flat_conditions
    if condition[0] == 'AND':
IndexError: list index out of range

I still not understand how to solve my problem in Pony....

Integer of higher length than 11

I can't seem to find a way for pony to specify a db int field with a length > 11, so if I try and insert something longer than 11 digits, I get:

cannot be stored in the database. DataError: 1264 Out of range value for column 'xxx' at row 1

If I increase the column length manually of the database column, the same problem persists. Any ideas / suggestions?

Using alternative identity in the foreign key

Is it possible to use alternative identity for the foreign key definition?

It is sometimes necessary to declare such a reference to other object, which uses some unique index other than primary key. It mostly useful when the primary index is synthetic, but you would like to use a natural key to define a reference.

For example, you have a User collection, which uses synthetic key. But you have also a User.username field which is a natural key for users.

Then, you have a Document collection, and every Document should refer to User who has created it. You can create a Document.owner = Required(User), but want to have a Document.owner field which refers to User by it's username, rather than id. As a profit, it allows to know username of the Document owner without getting a User record.

It could be achieved as an option using 'reverse_column' or 'reverse_columns' in the attribute, but restricted to use only in symmetric relations now.

Table names are significant when definig one-to_one links (!) CRITICAL BUG.

Example where wrong tables are generated:

from pony.orm import *
db = Database('sqlite', ':memory:', create_db=True)
class Stat(db.Entity):
    icsactivations = Optional("IcsActivation")
    webinarshow = Optional("WebinarShow")
class WebinarShow(db.Entity):
    stats = Required(Stat)
class IcsActivation(db.Entity):
    stats = Required(Stat)
db.generate_mapping(create_tables=True)

Now, just rename Stat to ZStat and see that INVALID column webinarshow DISAPPEAR from table ZStat (!)

I'm disappointed!

Object Message[200] cannot be used after the database session is over

Code just like this:
handler:

from models import Message
class MessageCreateHandler(BaseHandler):
    @db_session
    def post(self):
        message_box_id = int(self.get_argument("message_box_id", 1))
        content = self.get_argument("content", "")
        message = Message(message_box_id=messge_box_id, 
                                 content=content).save()
        return self.write({"content": message.content,
                                "id": message.id})

models:

from models import MessageBox
class Message(db.Entity):
    message_box_id = Required(int)
    content = Required(LongUnicode)

    @property
    def message_box(self):
        return MessageBox[self.message_box_id]

    def save(self):
        self.message_box.status = 0 # When I delete this line, all will be fine.
        commit()
        return self

error message:

File "/home/yetone/web/collipa/controllers/user.py", line 253, in post
        return self.write({"content": message.content, "id": message.id})
      File "<string>", line 2, in __get__
      File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 95, in cut_traceback
        return func(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 888, in __get__
        if not obj._cache_.is_alive: throw_db_session_is_over(obj)
      File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 2737, in throw_db_session_is_over
        throw(TransactionRolledBack, 'Object %s cannot be used after the database session is over' % safe_repr(obj))
      File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 121, in throw
        raise exc
    TransactionRolledBack: Object Message[200] cannot be used after the database session is over

"assert old_dbval is not NOT_LOADED" when getting objects from a table

A simple attempt to search for a single object fails with a cryptic message (some mysterious internal assertion fails). I observe the same behaviour whether I use Postgres or Sqlite (example below) — assertion failed.

I'm using Pony 0.5 obtained from PyPI on Ubuntu 13.10 64-bit.
Python version: “2.7.5+” (from standard Ubuntu package).

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from datetime import datetime
from pony.orm import *

db = Database('sqlite', 'shite.db', create_db=True)


class MyEntity(db.Entity):
    id = PrimaryKey(int, auto=True)
    name = Required(LongUnicode)

@db_session
def show_one():
    got = MyEntity.get(name='one')
    print 'Got', got.name

@db_session
def add_data():
    for new_name in ['one', 'two', 'three']:
        ent = MyEntity(name=new_name)
        print 'Adding', new_name

if __name__ == '__main__':
    #sql_debug(True)
    db.generate_mapping(create_tables=True)
    add_data()
    show_one()

The message I get:

  File "./litebugstr.py", line 29, in 
    show_one()
  File "", line 2, in show_one
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 345, in new_func
    try: return func(*args, **kwargs)
  File "./litebugstr.py", line 17, in show_one
    print 'Got', got.name
  File "", line 2, in __get__
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 97, in cut_traceback
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1339, in __get__
    result = attr.get(obj)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1347, in get
    val = obj._vals_[attr] if attr in obj._vals_ else attr.load(obj)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1332, in load
    attr.db_set(obj, dbval)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 1444, in db_set
    assert old_dbval is not NOT_LOADED
AssertionError

Bad SQL generated when using orm.exists in the filter

Database: MySQL
Data model (Inherited from django.contrib.auth with help of djony):
User *-* Permission

Reproducing:

>>> orm.sql_debug(True)
>>> qq = orm.select(o for o in User.p if orm.exists(p for p in o.user_permissions if p.codename == 'qq'))
>>> orm.show(qq)
SELECT `o`.`id`, `o`.`password`, `o`.`last_login`, `o`.`is_superuser`, `o`.`username`, `o`.`first_name`, `o`.`last_name`, `o`.`email`, `o`.`is_staff`, `o`.`is_active`, `o`.`date_joined`
FROM `auth_user` `o`
WHERE EXISTS (
    SELECT 1
    FROM `auth_user_user_permissions` `t-1`
        ON `o`.`id` = `t-1`.`user_id`, `auth_permission` `p`
    WHERE `p`.`codename` = 'qq'
      AND `t-1`.`permission_id` = `p`.`id`
    )

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<string>", line 2, in show
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 113, in cut_traceback
    raise exc  # Set "pony.options.CUT_TRACEBACK = False" to see full traceback
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ON `o`.`id` = `t-1`.`user_id`, `auth_permission` `p`\n    WHERE `p`.`codename` = ' at line 6")

UPDATE:

The orm.count works fine in the same circumstances:

>>> qq = orm.select(o for o in User.p if orm.count(p for p in o.user_permissions if p.codename == 'qq'))
>>> orm.show(qq)
SELECT `o`.`id`, `o`.`password`, `o`.`last_login`, `o`.`is_superuser`, `o`.`username`, `o`.`first_name`, `o`.`last_name`, `o`.`email`, `o`.`is_staff`, `o`.`is_active`, `o`.`date_joined`
FROM `auth_user` `o`
WHERE (
    SELECT COUNT(DISTINCT `p`.`id`)
    FROM `auth_user_user_permissions` `t-1`, `auth_permission` `p`
    WHERE `o`.`id` = `t-1`.`user_id`
      AND `p`.`codename` = 'qq'
      AND `t-1`.`permission_id` = `p`.`id`
    ) <> 0

id|pas...|las...|is_...|use...|fir...|las...|email|is_...|is_...|dat...|use...
--+------+------+------+------+------+------+-----+------+------+------+------

python3

Are you planning porting ponyorm for python3? Some milestone?

ponyorm migration tool

Hello,

I'm not sure if this is the place to put this question. If there is a better place please let me know.

Is there a DB migration tool for pony orm? Something like alembic for sqlalchemy..

Table `walkout.bodystatus` doesn't exists when from pony0.4.9 to pony0.5-beta

i update pony to 0.5-beta, as usual i run python app.py
and then

Traceback (most recent call last):
  File "app.py", line 60, in <module>
    db.generate_mapping()
  File "<auto generated wrapper of generate_mapping() function>", line 2, in generate_mapping
  File "/usr/local/lib/python2.7/dist-packages/pony/utils.py", line 97, in cut_traceback
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 757, in generate_mapping
    elif check_tables: database.check_tables()
  File "<auto generated wrapper of check_tables() function>", line 2, in check_tables
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 339, in new_func
    try: return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/core.py", line 815, in check_tables
    database.schema.check_tables(database.provider, connection)
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/dbschema.py", line 72, in check_tables
    provider.execute(cursor, sql)
  File "<auto generated wrapper of execute() function>", line 2, in execute
  File "/usr/local/lib/python2.7/dist-packages/pony/orm/dbapiprovider.py", line 45, in wrap_dbapi_exceptions
    except dbapi_module.ProgrammingError, e: raise ProgrammingError(e)
pony.orm.dbapiprovider.ProgrammingError: (1146, "Table 'walkout.bodystatus' doesn't exist")

in fact , i can see my bodystatus Table in database walkout.

mysql> show tables;
+-------------------+
| Tables_in_walkout |
+-------------------+
| BodyStatus   | 

in pony 0.4.9, everything is ok.

this test-pony-orm is my test-script