edumfa-schema-upgrade: Unknown column 'policy.edumfanode' in 'field list' about edumfa

[fritterhoff]

Mhm. Ok seems like you were not running the latest privacyIDEA version 3.9.2? Did you run the schema migration ever before?

[ikreb7]

Mhm. Ok seems like you were not running the latest privacyIDEA version 3.9.2? Did you run the schema migration ever before?

No, this is not true. I had privacyIDEA==3.9.2. Once, but there were the mysqlclient package missing. But this shouldn't be a problem.

[ikreb7]

This are the logs for the first run:

(venv) edumfa@netz-mfa-db01:~$ edumfa-schema-upgrade eduMFA/migrations/
Can not import grpc modules.

               _       __  __ ______      
              | |     |  \/  |  ____/\    
       ___  __| |_   _| \  / | |__ /  \   
      / _ \/ _` | | | | |\/| |  __/ /\ \  
     |  __/ (_| | |_| | |  | | | / ____ \ 
      \___|\__,_|\__,_|_|  |_|_|/_/    \_          v1.4.0
        
Traceback (most recent call last):
  File "/opt/edumfa/venv/bin/edumfa-manage", line 1718, in <module>
    manager.run()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 371, in current
    command.current(config, verbose=verbose)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/command.py", line 625, in current
    script.run_env()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/edumfa/eduMFA/migrations/env.py", line 110, in <module>
    run_migrations_online()
  File "/opt/edumfa/eduMFA/migrations/env.py", line 86, in run_migrations_online
    engine = engine_from_config(
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 739, in engine_from_config
    return create_engine(url, **options)
  File "<string>", line 2, in create_engine
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
    return fn(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 150, in dbapi
    return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
++ Stamping DB to 4f32a4e1bf33
Can not import grpc modules.

               _       __  __ ______      
              | |     |  \/  |  ____/\    
       ___  __| |_   _| \  / | |__ /  \   
      / _ \/ _` | | | | |\/| |  __/ /\ \  
     |  __/ (_| | |_| | |  | | | / ____ \ 
      \___|\__,_|\__,_|_|  |_|_|/_/    \_          v1.4.0
        
Traceback (most recent call last):
  File "/opt/edumfa/venv/bin/edumfa-manage", line 1718, in <module>
    manager.run()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 389, in stamp
    command.stamp(config, revision, sql=sql, tag=tag)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/command.py", line 690, in stamp
    script.run_env()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/edumfa/eduMFA/migrations/env.py", line 110, in <module>
    run_migrations_online()
  File "/opt/edumfa/eduMFA/migrations/env.py", line 86, in run_migrations_online
    engine = engine_from_config(
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 739, in engine_from_config
    return create_engine(url, **options)
  File "<string>", line 2, in create_engine
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
    return fn(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 150, in dbapi
    return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
++ Upgrading DB schema.
Can not import grpc modules.

               _       __  __ ______      
              | |     |  \/  |  ____/\    
       ___  __| |_   _| \  / | |__ /  \   
      / _ \/ _` | | | | |\/| |  __/ /\ \  
     |  __/ (_| | |_| | |  | | | / ____ \ 
      \___|\__,_|\__,_|_|  |_|_|/_/    \_          v1.4.0
        
Running online
Traceback (most recent call last):
  File "/opt/edumfa/venv/bin/edumfa-manage", line 1718, in <module>
    manager.run()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/flask_migrate/__init__.py", line 271, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/edumfa/eduMFA/migrations/env.py", line 110, in <module>
    run_migrations_online()
  File "/opt/edumfa/eduMFA/migrations/env.py", line 86, in run_migrations_online
    engine = engine_from_config(
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 739, in engine_from_config
    return create_engine(url, **options)
  File "<string>", line 2, in create_engine
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
    return fn(*args, **kwargs)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/opt/edumfa/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 150, in dbapi
    return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'

[fritterhoff]

ModuleNotFoundError: No module named 'MySQLdb' is normally caused by a broken connection string. Are you using mysql+pymysql?

Would you mind to run a edumfa-manage db current -d edumfa/migrations? I'm wondering why the script tries to execute outdated migrations.

[ikreb7]

Yes, I fixed this already. See Issue #40.

If I run edumfa-manage db current -d edumfa/migrations is all fine:

Can not import grpc modules.

               _       __  __ ______      
              | |     |  \/  |  ____/\    
       ___  __| |_   _| \  / | |__ /  \   
      / _ \/ _` | | | | |\/| |  __/ /\ \  
     |  __/ (_| | |_| | |  | | | / ____ \ 
      \___|\__,_|\__,_|_|  |_|_|/_/    \_          v1.4.0
        

[fritterhoff]

Mhm actually the output should tell you something else.
e.g.

edumfa@mfa01-dev:~$ edumfa-manage db current -d src/edumfa/migrations/
Can not import grpc modules.

          _       __  __ ______
         | |     |  \/  |  ____/\
  ___  __| |_   _| \  / | |__ /  \
 / _ \/ _` | | | | |\/| |  __/ /\ \
|  __/ (_| | |_| | |  | | | / ____ \
 \___|\__,_|\__,_|_|  |_|_|/_/    \_          v1.4.0

Running online
4a0aec37e7cf

[ikreb7]

No, the last two lines are missing :/

I tried the migration with my second node and get the error again with the missing edumfanode :(

[ikreb7]

There are a migration error pinode and edumfanode, if I rename the column to edumfanode, run the migration, rename back to pinode, run migration, and rename back, it works then. I think the order is broken.

But I don't have time today the go through all alembic scripts.

[fritterhoff]

First of all thanks a lot for the bug report. Seems like the logging of the migrate scripts is broken ... :/

As a first quick workaround you could change the logging locally in the migrate package so it simply writes to console. Depending on your installation you should find a /usr/local/lib/python3.10/site-packages/flask_migrate/init.py (or maybe in your venv)

That should contain:

@wraps(f)
    def wrapped(*args, **kwargs):
        try:
            f(*args, **kwargs)
        except (CommandError, RuntimeError) as exc:
            log.error('Error: ' + str(exc))
            sys.exit(1)
    return wrapped

Would be great if you temporary replace the log.error with a print and try the edumfa-manage db current -d edumfa/migrations again.

I also will check why the logging does not print anything at all ...

[fritterhoff]

Ok... so the logging configuration is broken - sorry for that.

You must change the alembic.ini to

[alembic]
# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s

# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false


[loggers]
keys = root,sqlalchemy,alembic,flask_migrate

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console
qualname =

[logger_flask_migrate]
handlers=console
qualname=flask_migrate
level=INFO


[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

[fritterhoff]

The default alembic.ini gets changed with #41

[fritterhoff]

Hi! Do you need further assistance?

[fritterhoff]

Any update on this issue?

[ikreb7]

Hey,

sorry I was ill. I updated to v2.0.1 and this helps. Thank you for your support!

[fritterhoff]

Great thanks for the feedback!