michaelhelmick / django-balancer Goto Github PK
View Code? Open in Web Editor NEWA set of tools for using Django's multi-db feature.
License: BSD 3-Clause "New" or "Revised" License
A set of tools for using Django's multi-db feature.
License: BSD 3-Clause "New" or "Revised" License
Just as the title says, as well as prepping for Django 1.9
File "/path/to/my/code/django/db/models/query.py", line 112, in __nonzero__
iter(self).next()
File "/path/to/my/code/django/db/models/query.py", line 106, in _result_iter
self._fill_cache()
File "/path/to/my/code/django/db/models/query.py", line 760, in _fill_cache
self._result_cache.append(self._iter.next())
File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
for row in compiler.results_iter():
File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
for rows in self.execute_sql(MULTI):
File "/path/to/my/code/django/db/models/sql/compiler.py", line 717, in execute_sql
sql, params = self.as_sql()
File "/path/to/my/code/django/db/models/sql/compiler.py", line 65, in as_sql
where, w_params = self.query.where.as_sql(qn=qn, connection=self.connection)
File "/path/to/my/code/django/db/models/sql/where.py", line 91, in as_sql
sql, params = child.as_sql(qn=qn, connection=connection)
File "/path/to/my/code/django/db/models/sql/where.py", line 94, in as_sql
sql, params = self.make_atom(child, qn, connection)
File "/path/to/my/code/django/contrib/gis/db/models/sql/where.py", line 50, in make_atom
return super(GeoWhereNode, self).make_atom(child, qn, connection)
File "/path/to/my/code/django/db/models/sql/where.py", line 141, in make_atom
lvalue, params = lvalue.process(lookup_type, params_or_value, connection)
File "/path/to/my/code/django/db/models/sql/where.py", line 312, in process
connection=connection, prepared=True)
File "/path/to/my/code/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/path/to/my/code/django/db/models/fields/related.py", line 154, in get_db_prep_lookup
sql, params = value._as_sql(connection=connection)
File "/path/to/my/code/django/db/models/query.py", line 806, in _as_sql
raise ValueError("Can't do subqueries with queries on different DBs.")
ValueError: Can't do subqueries with queries on different DBs.
Looks like it's related to the allow_relation
method.
File "/path/to/my/code/ads/templatetags/ads.py", line 135, in render
ad_info = AdInfoOverride.objects.get(url=request.path, sites=settings.SITE_ID)
File "/path/to/my/code/django/db/models/manager.py", line 132, in get
return self.get_query_set().get(*args, **kwargs)
File "/path/to/my/code/django/db/models/query.py", line 336, in get
num = len(clone)
File "/path/to/my/code/django/db/models/query.py", line 81, in __len__
self._result_cache = list(self.iterator())
File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
for row in compiler.results_iter():
File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
for rows in self.execute_sql(MULTI):
File "/path/to/my/code/django/db/models/sql/compiler.py", line 726, in execute_sql
cursor = self.connection.cursor()
File "/path/to/my/code/django/db/backends/__init__.py", line 75, in cursor
cursor = self._cursor()
File "/path/to/my/code/django/db/backends/postgresql_psycopg2/base.py", line 140, in _cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed
This is probably an issue with my Postgres setup, not django-balancer, but I'm tracking it here just in case.
It seems that mutating a global (setting) is asking for trouble. Perhaps BasePoolRouter should copy the thing that it assigns to self.pool.
https://github.com/bkonkle/django-balancer/blob/develop/balancer/routers.py#L81
File "/path/to/my/code/app/templatetags/my_templatetags.py", line 320, in my_templatetag
users = list(User.objects.filter(favoriters | commenters).distinct())
File "/path/to/my/code/django/db/models/query.py", line 83, in __len__
self._result_cache.extend(list(self._iter))
File "/path/to/my/code/django/db/models/query.py", line 269, in iterator
for row in compiler.results_iter():
File "/path/to/my/code/django/db/models/sql/compiler.py", line 672, in results_iter
for rows in self.execute_sql(MULTI):
File "/path/to/my/code/django/db/models/sql/compiler.py", line 727, in execute_sql
cursor.execute(sql, params)
File "/path/to/my/code/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
DatabaseError: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
I can't reproduce this in development, but in production I get this error when using PinningSessionMiddleware: 'WSGIRequest' object has no attribute 'session'
For some reason, the request object passed to process_response
doesn't have the session data. I suspect this may have something to do with the many custom middleware classes we use, but the confusing part is that everything works fine on our dev/testing server. I'm tracking this issue here in case someone else runs into it. If you have this issue, let me know here so I can help track down the root cause.
I've got some problems with the balancer. In general, it is working as I need (I have site with +15 000 users, a few PostgreSQL replicated DBs, I prefer to use native Python solution like the balancer than e.g PGPool)
There is a problem with the balancer when the querysets results are joined, namely (simple yet self-describing example):
young_people = Person.objects.filter(age__lt=10)
old_people = Person.objects.filter(age__gt=30)
young_and_old = young_people | old_people
Yes, one could of course give (Q(age__lt=10)|Q(age__gt=30)) to filter but I wanted just to explicitly show what I mean: if You have multiple DB setup, one query (young_people) goes to one DB, then the router can switch to the other DB on next query (with old_people), and Django can not join the two, because they origin from the two different sources.
Any suggestions what to do (beside the trivial answer to try to rethink the task to get the one queryset from single DB, i.e. (Q(age__lt=10)|Q(age__gt=30)) in the example case)? Can one think of a router that would pin the dbs within a transaction (i.e. within a view)?
Thanks!
Sebastian
When a database in the pool is down, temporarily remove it from the pool but periodically re-check the connection. If it comes back up, add the database back to the pool.
Could you please upload the newest changes to pypi?
I've found a small issue in the balancer.
I've got the following configuration:
And the DB connections are defined as:
When ussing pinning routers it looks like the Second and Third instances are trying to use slave connection, while thre's no info about it in the settings.py I gues that this is stored somewhere with the session (I've been using PinningSessionMiddleware)... Anyway, I'm having the ConnectionDoesNotExist: The connection slave doesn't exist
error.
I've switched to .WeightedMasterSlaveRouter and we'll see if the problem still occurs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.