`opentaxii-delete-blocks` not deleting records from `collection_to_content_block` about opentaxii

[arcsector]

Could you please separate your error using line breaks? It makes it very hard to discern what's going on.

[arcsector]

Thanks, much better. It sort of looks like it's a sqlite3 error (evidenced by the sqlite3.IntegrityError). Could you try doing this on a mysql backend or postgresql (whichever you're more comfortable with)? That may effect the behavior of the sql engine. If that doesn't work, in the meantime you could just use your own sqlalchemy or sqlsoup script to clear the database of all blocks.

Also keep in mind that the opentaxii-delete-blocks command is in the latest version, not the stable release, so it isn't 100% guaranteed to work.

[dhkron]

I've tested it in PostgreSQL - the error occurs already in the deletion.

Traceback (most recent call last):
  File "/home/gil/.local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/gil/.local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: update or delete on table "content_blocks" violates foreign key constraint "collection_to_content_block_content_block_id_fkey" on table "collection_to_content_block"
DETAIL:  Key (id)=(1) is still referenced from table "collection_to_content_block".

[arcsector]

This isn't an error with Opentaxii or sqlalchemy; it's a SQL error. If you go in and try to execute a DELETE statement on collection_to_content_block, you will get the same response.

I believe the way to correct this is to delete the content block first, then the collection, then the collection_to_content_block.

[traut]

@dhkron by default sqllite3 does not enable foreign keys. I suspect that's why CASCADEs do not work. Could you please verify?

[traut]

regarding postgresql error -- it is strange, cascades are in place. Could you provide a description of that table in your DB?

[dhkron]

This is the output of \d+ collection_to_content_block in psql.

                  Table "public.collection_to_content_block"
      Column      |  Type   | Modifiers | Storage | Stats target | Description 
------------------+---------+-----------+---------+--------------+-------------
 collection_id    | integer | not null  | plain   |              | 
 content_block_id | integer | not null  | plain   |              | 
Indexes:
    "collection_to_content_block_pkey" PRIMARY KEY, btree (collection_id, content_block_id)
    "ix_collection_to_content_block_content_block_id" btree (content_block_id)
Foreign-key constraints:
    "collection_to_content_block_collection_id_fkey" FOREIGN KEY (collection_id) REFERENCES data_collections(id)
    "collection_to_content_block_content_block_id_fkey" FOREIGN KEY (content_block_id) REFERENCES content_blocks(id)

Setup (all python is python3)

1. Start two PostgreSQL servers.
2. Start opentaxii machine
3. Add services, collections & a user (opentaxii-create-services etc)
4. Sanity test to services with taxii-discovery
5. Push data with a minimal cabby script & verify with taxii-poll
6. Then opentaxii-delete-blocks and boom!

sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) update or delete on table "content_blocks" violates foreign key constraint "collection_to_content_block_content_block_id_fkey" on table "collection_to_content_block"
DETAIL:  Key (id)=(1) is still referenced from table "collection_to_content_block".
 [SQL: 'DELETE FROM content_blocks WHERE content_blocks.id IN (SELECT content_blocks.id \nFROM data_collections JOIN collection_to_content_block AS collection_to_content_block_1 ON data_collections.id = collection_to_content_block_1.collection_id JOIN content_blocks ON content_blocks.id = collection_to_content_block_1.content_block_id \nWHERE data_collections.id = %(id_1)s AND content_blocks.timestamp_label > %(timestamp_label_1)s AND content_blocks.timestamp_label <= %(timestamp_label_2)s)'] [parameters: {'id_1': 1, 'timestamp_label_1': '1970-01-01', 'timestamp_label_2': '2017-12-10'}]

For completeness of setup, this is my opentaxii.yml file:

---
domain: "agent-taxii:9000"
support_basic_auth: yes
save_raw_inbox_messages: yes

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: postgresql://user:[email protected]:9001/opentaxii
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: postgresql://user1:[email protected]:9002/opentaxii1
    create_tables: yes
    secret: secret


logging:
  opentaxii: debug
  root: debug

xml_parser_supports_huge_tree: no

hooks:

[dhkron]

I think the problems is related to the creation of the delete-cascade.
When I created the delete-cascade manually, it worked.

ALTER TABLE collection_to_content_block
DROP CONSTRAINT collection_to_content_block_content_block_id_fkey,
ADD CONSTRAINT collection_to_content_block_content_block_id_fkey
    FOREIGN KEY (content_block_id)
    REFERENCES content_blocks(id)
    ON DELETE CASCADE;

[traut]

@dhkron thanks for the extensive info on the issue. It seems cascades are not created automatically, which is strange, since sqlalchemy should take care of that. I'll look into it

[traut]

closing this since I can not reproduce it. please comment if you can reproduce it in the latest release 0.1.10.