pgq / python-skytools

Scripting utilities for Python

Generic Queue for PostgreSQL

PgQ is PostgreSQL extension that provides generic, high-performance lockless queue with simple API based on SQL functions.

• Supported PostgreSQL versions: 10..16
• Deprecated but might work: 9.x

python-skytools's Issues

PG10 Compatibility Issue: pg_sequence

Per https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1753b1b027035029c2a2a1649065762fafbf63f3, PostgreSQL 10 'separates the metadata from the sequence data'. SELECT * FROM seqname no longer returns the metadata columns.

We ran into an error when attempting londiste INI add-table against a PG10 DB.

2018-10-01 16:36:04,019 69946 ERROR Job name crashed: 'increment_by'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/skytools/scripting.py", line 578, in run_func_safely
    r = func()
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/skytools/adminscript.py", line 62, in work
    fn(*cmdargs)
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/londiste/setup.py", line 193, in cmd_add_table
    self.add_table(src_db, dst_db, tbl, create_flags, src_tbls)
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/londiste/setup.py", line 230, in add_table
    s = skytools.TableStruct(src_curs, src_dest_table)
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct.py", line 643, in __init__
    self.seq_list += self._load_elem(curs, col.seqname, seq_args, TSeq)
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct.py", line 556, in _load_elem
    elem_list.append(eclass(name, row))
  File "/usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct.py", line 491, in __init__
    if row['increment_by'] != 1:
  File "/usr/lib/python2.7/dist-packages/psycopg2/extras.py", line 144, in __getitem__
    x = self._index[x]
KeyError: 'increment_by'

To my surprise, I was not able to find a solution or any mentioning about the issue anywhere, considering the fact that PG10 has been out for over a year. I decided to hack the code to add pg_sequence into the SQL query for sequence metadata, as follows:

$ diff -u /usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct*.py
--- /usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct_ORIG.py     2014-11-05 07:00:53.000000000 -0800
+++ /usr/lib/python2.7/dist-packages/skytools-3.0/skytools/dbstruct.py  2018-10-02 17:26:09.053156743 -0700
@@ -483,7 +483,19 @@
 class TSeq(TElem):
     """Info about sequence."""
     type = T_SEQUENCE
-    SQL = """SELECT *, %(owner)s as "owner" from %(fqname)s """
+    SQL = """SELECT '%(fqname)s'::name AS sequence_name
+  , last_value
+  , seqstart AS start_value
+  , seqincrement AS increment_by
+  , seqmax AS max_value
+  , seqmin AS min_value
+  , seqcache AS cache_value
+  , log_cnt
+  , CASE WHEN seqcycle THEN 'yes' ELSE 'no' END AS is_cycled
+  , is_called
+  , %(owner)s as owner
+FROM pg_catalog.pg_sequence, %(fqname)s
+WHERE seqrelid = '%(fqname)s'::regclass::oid"""
     def __init__(self, seq_name, row):
         self.name = seq_name
         defn = ''

Please consider putting the PG10+ support to the code in the next release.

Debian packages:

• skytools3 3.2.6-4
• skytools3-ticker 3.3-1.pgdg16.04+1
• python-skytools3 3.2.6-4
• postgresql-10 10.5-1.pgdg16.04+1

londiste - adding table partition to replication

Hello,

Partitioned table stats_replicate.wb_ads_blocks is being partitioned using partman (declarative partioning, PARTITION BY RANGE ).
I am trying to add table partitions to replication like this:
londiste skytools_devel.ini add-table stats_replicate.wb_ads_blocks_default

Table partition is added but merge_state than states as in-copy and there is an error on leaf node:
LOG: statement: ALTER TABLE ONLY stats_replicate.wb_ads_blocks_default NO INHERIT stats_replicate.wb_ads_blocks
ERROR: cannot change inheritance of a partition

This check returns parent table and than follows alter table only no inherit :

SELECT n.nspname||'.'||c.relname AS name
FROM pg_inherits i
JOIN pg_class c ON i.inhparent = c.oid
JOIN pg_namespace n ON c.relnamespace = n.oid
WHERE i.inhrelid = 33073131

But, it seems that this check works the same for declarative partitioning as for partitioning using inheritance.

Version being used:
londiste, Skytools version 3.6.1

Could you, please, advise if there is workaround to use it like in this case?

scriptmgr.py deprecation in non-legacy skytools?

First of all, greetings, thank you for all the hard work on these projects and my apologies in front if this is an incorrect place to ask these questions, alas I found no other reasonable spot.
I would like to ask whether scriptmgr.py has been deprecated in modern python-skytools implementations? I am only able to find it in the legacy-skytools package and am wondering has the functionality maybe been replaced with something different or has it been simply dropped?

Thank you and regards,
Kalle