Issue after merge about ghostwriter

[bneg]

Similar error on clean install with git clone

ubuntu@[redacted]:~/Ghostwriter$ sudo docker-compose -f local.yml run --rm django /seed_data                                                                                                                                                                               Starting ghostwriter_postgres_1 ... done                                                                                                                                                                                                                                        PostgreSQL is available                                                                                                                                                                                                                                                         Loading /app/ghostwriter/reporting/fixtures/initial.json...
Installed 12 object(s) from 1 fixture(s)
Loading /app/ghostwriter/shepherd/fixtures/initial.json...
Installed 26 object(s) from 1 fixture(s)
Loading /app/ghostwriter/rolodex/fixtures/initial.json...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "rolodex_objectivestatus" does not exist
LINE 1: UPDATE "rolodex_objectivestatus" SET "objective_status" = 'A...
               ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 30, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "/usr/local/lib/python3.7/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 851, in _save_table
    forced_update)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 900, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 760, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1429, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 99, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: Problem installing fixture '/app/ghostwriter/rolodex/fixtures/initial.json': Could not load rolodex.ObjectiveStatus(pk=1): relation "rolodex_objectivestatus" does not exist
LINE 1: UPDATE "rolodex_objectivestatus" SET "objective_status" = 'A...
               ^

[optum-shadow]

It appears it is having a problem with
SELECT "reporting_reportfindinglink"."id", "reporting_reportfindinglink"."title", "reporting_reportfindinglink"."position", "reporting_reportfindinglink"."affected_entities", "reporting_reportfindinglink"."description", "reporting_reportfindinglink"."impact", "reporting_reportfindinglink"."mitigation", "reporting_reportfindinglink"."replication_steps", "reporting_reportfindinglink"."host_detection_techniques", "reporting_reportfindinglink"."network_detection_techniques", "reporting_reportfindinglink"."references", "reporting_reportfindinglink"."finding_guidance", "reporting_reportfindinglink"."complete", "reporting_reportfindinglink"."severity_id", "reporting_reportfindinglink"."finding_type_id", "reporting_reportfindinglink"."report_id", "reporting_reportfindinglink"."assigned_to_id", "reporting_report"."id", "reporting_report"."title", "reporting_report"."creation", "reporting_report"."last_update", "reporting_report"."complete", "reporting_report"."archived", "reporting_report"."project_id", "reporting_report"."created_by_id", "reporting_report"."delivered", "rolodex_project"."id", "rolodex_project"."codename", "rolodex_project"."start_date", "rolodex_project"."end_date", "rolodex_project"."note", "rolodex_project"."slack_channel", "rolodex_project"."complete", "rolodex_project"."client_id", "rolodex_project"."operator_id", "rolodex_project"."project_type_id" FROM "reporting_reportfindinglink" INNER JOIN "reporting_report" ON ("reporting_reportfindinglink"."report_id" = "reporting_report"."id") LEFT OUTER JOIN "rolodex_project" ON ("reporting_report"."project_id" = "rolodex_project"."id") WHERE ("reporting_reportfindinglink"."assigned_to_id" = 1 AND "reporting_report"."complete" = false AND "reporting_reportfindinglink"."complete" = false) ORDER BY "rolodex_project"."end_date" ASC LIMIT 10

[chrismaddalena]

You are missing the ObjectiveStatus values in the database. Run this command to load those values and other new values you are probably missing after the latest update:

docker-compose -f local.yml run --rm django /seed_data

That will load the ObjectiveStatus model with "Active," "On Hold," and "Complete" values to get started.

[optum-shadow]

That is when this error happens..
docker-compose -f local.yml run --rm django /seed_data
OUTPUT
Starting ghostwriter_postgres_1 ... done
PostgreSQL is available
Loading /app/ghostwriter/reporting/fixtures/initial.json...
Installed 12 object(s) from 1 fixture(s)
Loading /app/ghostwriter/shepherd/fixtures/initial.json...
Installed 28 object(s) from 1 fixture(s)
Loading /app/ghostwriter/rolodex/fixtures/initial.json...
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "rolodex_objectivestatus" does not exist
LINE 1: UPDATE "rolodex_objectivestatus" SET "objective_status" = 'A...
^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "manage.py", line 30, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.7/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
obj.save(using=self.using)
File "/usr/local/lib/python3.7/site-packages/django/core/serializers/base.py", line 223, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 779, in save_base
force_update, using, update_fields,
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 851, in _save_table
forced_update)
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 900, in _do_update
return filtered._update(values) > 0
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 760, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1429, in execute_sql
cursor = super().execute_sql(result_type)
File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 99, in execute
return super().execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: Problem installing fixture '/app/ghostwriter/rolodex/fixtures/initial.json': Could not load rolodex.ObjectiveStatus(pk=1): relation "rolodex_objectivestatus" does not exist
LINE 1: UPDATE "rolodex_objectivestatus" SET "objective_status" = 'A...

[fastlorenzo]

I think the issue comes from the fact that ./ghostwriter/rolodex/migrations/ doesn't include the changes to the database (ObjectiveStatus table is not created).

@chrismaddalena : Did you forget to include a migration file?

[fastlorenzo]

The following command worked for me:

sudo docker-compose -f production.yml run --rm django sh -c "/app/manage.py makemigrations && /app/manage.py migrate"

It creates the migration files for the database and applies it.

Then you can run the following command to include the new initial data:

docker-compose -f local.yml run --rm django /seed_data

[optum-shadow]

So those commands worked for me also. I am very new to django so should i run docker-compose -f production.yml run --rm django sh -c "/app/manage.py makemigrations && /app/manage.py migrate" before every update?

[chrismaddalena]

I've had issues with migrations between Django instances, so I did not include a migration file. @fastlorenzo is correct, the new models and model changes need to be created with a migration. The commands are documented here:

https://ghostwriter.wiki/development/modifying-code

I'll look into how best to avoid this in the future. I'm glad it solved the problem!

[Freakazoidile]

Following the install instructions on the wiki on two fresh install, separate VMs, encountered the same issue as OP, tried fixing it for a few hours before finding this closed issue. Tried @fastlorenzo 's recommendation but encountered the error below. I then tried starting on a clean slate a third time and running the migrate command before running the database seeding and received the below error. It's obvious what the issue here is but I'm not familiar enough with the inner workings of everything to figure this out, looks like a quick fix for someone more familiar with where to make changes to the migration.

Happy to provide more info to help get this resolved so others who want to check out the tool don't give up.

    return self.cursor.execute(sql, params)
psycopg2.errors.DuplicateColumn: column "delivered" of relation "reporting_report" already exists


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/manage.py", line 30, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 234, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.7/site-packages/django/db/migrations/operations/fields.py", line 112, in database_forwards
    field,
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 447, in add_field
    self.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 137, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column "delivered" of relation "reporting_report" already exists