GithubHelp home page GithubHelp logo

michaelhelmick / django-balancer Goto Github PK

View Code? Open in Web Editor NEW
110.0 110.0 5.0 55 KB

A set of tools for using Django's multi-db feature.

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%
database django python replication

django-balancer's People

Contributors

bkonkle avatar cmheisel avatar michaelhelmick avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

django-balancer's Issues

Can't do subqueries with queries on different DBs.

  File "/path/to/my/code/django/db/models/query.py", line 112, in __nonzero__
    iter(self).next()

  File "/path/to/my/code/django/db/models/query.py", line 106, in _result_iter
    self._fill_cache()

  File "/path/to/my/code/django/db/models/query.py", line 760, in _fill_cache
    self._result_cache.append(self._iter.next())

  File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
    for row in compiler.results_iter():

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
    for rows in self.execute_sql(MULTI):

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 717, in execute_sql
    sql, params = self.as_sql()

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 65, in as_sql
    where, w_params = self.query.where.as_sql(qn=qn, connection=self.connection)

  File "/path/to/my/code/django/db/models/sql/where.py", line 91, in as_sql
    sql, params = child.as_sql(qn=qn, connection=connection)

  File "/path/to/my/code/django/db/models/sql/where.py", line 94, in as_sql
    sql, params = self.make_atom(child, qn, connection)

  File "/path/to/my/code/django/contrib/gis/db/models/sql/where.py", line 50, in make_atom
    return super(GeoWhereNode, self).make_atom(child, qn, connection)

  File "/path/to/my/code/django/db/models/sql/where.py", line 141, in make_atom
    lvalue, params = lvalue.process(lookup_type, params_or_value, connection)

  File "/path/to/my/code/django/db/models/sql/where.py", line 312, in process
    connection=connection, prepared=True)

  File "/path/to/my/code/django/db/models/fields/subclassing.py", line 53, in inner
    return func(*args, **kwargs)

  File "/path/to/my/code/django/db/models/fields/related.py", line 154, in get_db_prep_lookup
    sql, params = value._as_sql(connection=connection)

  File "/path/to/my/code/django/db/models/query.py", line 806, in _as_sql
    raise ValueError("Can't do subqueries with queries on different DBs.")

ValueError: Can't do subqueries with queries on different DBs.

Looks like it's related to the allow_relation method.

InterfaceError: connection already closed

  File "/path/to/my/code/ads/templatetags/ads.py", line 135, in render
    ad_info = AdInfoOverride.objects.get(url=request.path, sites=settings.SITE_ID)

  File "/path/to/my/code/django/db/models/manager.py", line 132, in get
    return self.get_query_set().get(*args, **kwargs)

  File "/path/to/my/code/django/db/models/query.py", line 336, in get
    num = len(clone)

  File "/path/to/my/code/django/db/models/query.py", line 81, in __len__
    self._result_cache = list(self.iterator())

  File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
    for row in compiler.results_iter():

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
    for rows in self.execute_sql(MULTI):

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 726, in execute_sql
    cursor = self.connection.cursor()

  File "/path/to/my/code/django/db/backends/__init__.py", line 75, in cursor
    cursor = self._cursor()

  File "/path/to/my/code/django/db/backends/postgresql_psycopg2/base.py", line 140, in _cursor
    cursor = self.connection.cursor()

InterfaceError: connection already closed

This is probably an issue with my Postgres setup, not django-balancer, but I'm tracking it here just in case.

DatabaseError: canceling statement due to conflict with recovery

  File "/path/to/my/code/app/templatetags/my_templatetags.py", line 320, in my_templatetag
    users = list(User.objects.filter(favoriters | commenters).distinct())

  File "/path/to/my/code/django/db/models/query.py", line 83, in __len__
    self._result_cache.extend(list(self._iter))

  File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
    for row in compiler.results_iter():

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
    for rows in self.execute_sql(MULTI):

  File "/path/to/my/code/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)

  File "/path/to/my/code/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)

DatabaseError: canceling statement due to conflict with recovery
DETAIL:  User query might have needed to see row versions that must be removed.

'WSGIRequest' object has no attribute 'session'

I can't reproduce this in development, but in production I get this error when using PinningSessionMiddleware: 'WSGIRequest' object has no attribute 'session'

For some reason, the request object passed to process_response doesn't have the session data. I suspect this may have something to do with the many custom middleware classes we use, but the confusing part is that everything works fine on our dev/testing server. I'm tracking this issue here in case someone else runs into it. If you have this issue, let me know here so I can help track down the root cause.

Querysets joins

I've got some problems with the balancer. In general, it is working as I need (I have site with +15 000 users, a few PostgreSQL replicated DBs, I prefer to use native Python solution like the balancer than e.g PGPool)

There is a problem with the balancer when the querysets results are joined, namely (simple yet self-describing example):

young_people = Person.objects.filter(age__lt=10)
old_people = Person.objects.filter(age__gt=30)

young_and_old = young_people | old_people

Yes, one could of course give (Q(age__lt=10)|Q(age__gt=30)) to filter but I wanted just to explicitly show what I mean: if You have multiple DB setup, one query (young_people) goes to one DB, then the router can switch to the other DB on next query (with old_people), and Django can not join the two, because they origin from the two different sources.

Any suggestions what to do (beside the trivial answer to try to rethink the task to get the one queryset from single DB, i.e. (Q(age__lt=10)|Q(age__gt=30)) in the example case)? Can one think of a router that would pin the dbs within a transaction (i.e. within a view)?

Thanks!
Sebastian

Take DB downtime into account

When a database in the pool is down, temporarily remove it from the pool but periodically re-check the connection. If it comes back up, add the database back to the pool.

Update pypi

Could you please upload the newest changes to pypi?

Slave connection not present

I've found a small issue in the balancer.

I've got the following configuration:

  • Two DB servers (postgres replicated with streaming) -- let's call them master and slave
  • Three seperated instances of the same projects running on three diferent machines

And the DB connections are defined as:

  • First instance of my app is balanced over 2 DB servers
  • Second instance is connected to master DB server
  • Third instance is connected to master DB server

When ussing pinning routers it looks like the Second and Third instances are trying to use slave connection, while thre's no info about it in the settings.py I gues that this is stored somewhere with the session (I've been using PinningSessionMiddleware)... Anyway, I'm having the ConnectionDoesNotExist: The connection slave doesn't exist error.

I've switched to .WeightedMasterSlaveRouter and we'll see if the problem still occurs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.