How to further fix to purge_old_records task (?) about tokenserver

[jrgm]

Maybe option #1, the sleep, and just going into a hover pattern is good enough (combined with increased limit/offset to search deeper). But doesn't really feel right.

[rfk]

Are these downed nodes ever going to come back, or are they permanently decommissioned? We want to make sure we don't abandon them permanently in a state where we'll never try to delete the data, but will leave the data hanging around on a node somewhere. If the node is never coming back, we should delete the old records in the db.

I think what you propose here seems fine, don't select them in the first place if we're going to ignore them. We'd have to double-check for other callers of this method to ensure they won't be impacted, but I think it's mostly tests and we can adjust as needed.

[rfk]

IIUC the sleep option might result in just selecting the same set of downed rows over and over again, until some new rows get marked as replaced and become available for cleanup. This in turn would risk ignoring older replaced rows that happen to be "hidden behind" the downed ones.

[jrgm]

Are these downed nodes ever going to come back, or are they permanently decommissioned?

I don't know right now, but was going to check tomorrow with Erik. It was all the same node.

[jrgm]

We'd have to double-check for other callers of this method to ensure they won't be impacted, but I think it's mostly tests and we can adjust as needed.

Yeah, I haven't actually written this as a patch and I'll probably have to adjust tests. Question: are there "external" users of this module?

[jrgm]

IIUC the sleep option might result in just selecting the same set of downed rows over and over again, until some new rows get marked as replaced and become available for cleanup. This in turn would risk ignoring older replaced rows that happen to be "hidden behind" the downed ones.

Yeah, good point. It could get in a state where it was always working (of punting) on the head of the result set, and never reaching into the backlog.

[rfk]

Question: are there "external" users of this module?

Note that I know of.

[jrgm]

Hrm. How does one use with_hint in our SQLAlchemy sql.py code? Because I can't just make the use index change I suggested above (tests fail on that syntax).

[jrgm]

So, I tried this as a simple test of using mysql hints:

from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, BigInteger, Index
from sqlalchemy.sql import select, text as sqltext

engine = create_engine('mysql://root@localhost/test')
conn = engine.connect()
meta = MetaData()

users = Table(
  'users',
  meta,
  Column('uid', BigInteger, primary_key = True),
  Column('service', Integer),
  Column('email', String(255)),
  Column('replaced_at', BigInteger),
)
Index('replaced_at_idx', users.c.service, users.c.replaced_at)
meta.create_all(engine)

query = sqltext('uid, service, replaced_at from users')
stmt = select([query])
stmt = stmt.with_statement_hint('use index (replaced_at_idx)', dialect_name='mysql')
stmt = stmt.where(sqltext('service = 1'))

res = conn.execute(stmt).fetchall() 
for row in res:
  print row

and I get:

sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError) (1064, "You have an 
error in your SQL syntax; check the manual that corresponds to your MySQL server version for 
the right syntax to use near 'use index (replaced_at_idx)' at line 2")
[SQL: SELECT uid, service, replaced_at from users 
WHERE service = 1 use index (replaced_at_idx)]

Notice that it simply appends the use index to the end of the query. But I don't grok the documentation for how to use this (and possibly this feature just doesn't work since the sqlalchemy tests do not test for a select with a where clause).

[rfk]

Using with_hint seems to do the job for me, e.g.:

select([users.c.uid, users.c.service, users.c.replaced_at]).with_hint(users, "USE INDEX (replaced_at_idx)", dialect_name="mysql")).where(users.c.service == 1)

IIUC you have to use this variant which accepts the users table as a additional argument, so that it knows to attach the hint to the appropriate table in the select clause.

[jrgm]

Thanks. Hmm, I had tried with_hint, but not with the users.c.* syntax. I'll work with that to create a patch.