In the event that the first task doesn't return a command for some reason, eg. the prompt was too complex and just required a discussion on designing something, we can end up causing an error in the celery worker as follows:
[2023-04-15 16:11:59,948: INFO/MainProcess] Task ix.task_log.tasks.agent_runner.start_agent_loop[a1473004-be68-4afd-b9a4-b31d6d3c92a5] received
[2023-04-15 16:11:59,951: INFO/ForkPoolWorker-4] AgentProcess initializing task_id=4
[2023-04-15 16:12:00,091: INFO/ForkPoolWorker-4] intialized command registry
[2023-04-15 16:12:00,106: INFO/ForkPoolWorker-4] AgentProcess loaded n=0 chat messages from persistence
[2023-04-15 16:12:00,119: INFO/ForkPoolWorker-4] AgentProcess initialized
[2023-04-15 16:12:00,119: INFO/ForkPoolWorker-4] starting process loop task_id=4
[2023-04-15 16:12:00,121: INFO/ForkPoolWorker-4] first tick for task_id=4
[2023-04-15 16:12:00,122: INFO/ForkPoolWorker-4] ticking task_id=4
[2023-04-15 16:12:17,475: WARNING/ForkPoolWorker-4] --- Logging error ---
[2023-04-15 16:12:17,483: WARNING/ForkPoolWorker-4] Traceback (most recent call last):
[2023-04-15 16:12:17,484: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,484: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,485: WARNING/ForkPoolWorker-4] File "/var/app/ix/task_log/tasks/agent_runner.py", line 13, in start_agent_loop
return process.start()
^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,486: WARNING/ForkPoolWorker-4] File "/var/app/ix/agents/process.py", line 188, in start
self.loop(n=authorized_for, tick_input=tick_input)
[2023-04-15 16:12:17,486: WARNING/ForkPoolWorker-4] File "/var/app/ix/agents/process.py", line 193, in loop
self.tick(execute=execute)
[2023-04-15 16:12:17,486: WARNING/ForkPoolWorker-4] File "/var/app/ix/agents/process.py", line 213, in tick
command = self.command_registry.get(command_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,486: WARNING/ForkPoolWorker-4] File "/var/app/ix/commands/registry.py", line 73, in get
return self.commands[name]
~~~~~~~~~~~~~^^^^^^
[2023-04-15 16:12:17,487: WARNING/ForkPoolWorker-4] KeyError: ''
[2023-04-15 16:12:17,487: WARNING/ForkPoolWorker-4]
During handling of the above exception, another exception occurred:
[2023-04-15 16:12:17,487: WARNING/ForkPoolWorker-4] Traceback (most recent call last):
[2023-04-15 16:12:17,487: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,487: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/utils/log.py", line 146, in format
msg = super().format(record)
^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/logging/__init__.py", line 695, in format
record.exc_text = self.formatException(record.exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/utils/log.py", line 142, in formatException
r = super().formatException(ei)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/logging/__init__.py", line 645, in formatException
traceback.print_exception(ei[0], ei[1], tb, None, sio)
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/traceback.py", line 124, in print_exception
te = TracebackException(type(value), value, tb, limit=limit, compact=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/traceback.py", line 690, in __init__
self.stack = StackSummary._extract_from_extended_frame_gen(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/traceback.py", line 416, in _extract_from_extended_frame_gen
for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/traceback.py", line 353, in _walk_tb_with_full_positions
positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,488: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/traceback.py", line 366, in _get_code_position
positions_gen = code.co_positions()
^^^^^^^^^^^^^^^^^
[2023-04-15 16:12:17,489: WARNING/ForkPoolWorker-4] AttributeError: '_Code' object has no attribute 'co_positions'
[2023-04-15 16:12:17,489: WARNING/ForkPoolWorker-4] Call stack:
[2023-04-15 16:12:17,491: WARNING/ForkPoolWorker-4] File "/usr/local/bin/celery", line 8, in <module>
sys.exit(main())
[2023-04-15 16:12:17,491: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
[2023-04-15 16:12:17,491: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 217, in main
return celery(auto_envvar_prefix="CELERY")
[2023-04-15 16:12:17,491: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/bin/base.py", line 134, in caller
return f(ctx, *args, **kwargs)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/bin/worker.py", line 351, in worker
worker.start()
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/concurrency/base.py", line 129, in start
self.on_start()
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/concurrency/prefork.py", line 109, in on_start
P = self._pool = Pool(processes=self.limit,
[2023-04-15 16:12:17,492: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 463, in __init__
super().__init__(processes, *args, **kwargs)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 1046, in __init__
self._create_worker_process(i)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/concurrency/asynpool.py", line 480, in _create_worker_process
return super()._create_worker_process(i)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 1158, in _create_worker_process
w.start()
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/process.py", line 124, in start
self._popen = self._Popen(self)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/context.py", line 333, in _Popen
return Popen(process_obj)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/popen_fork.py", line 24, in __init__
self._launch(process_obj)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/popen_fork.py", line 79, in _launch
code = process_obj._bootstrap()
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/process.py", line 327, in _bootstrap
self.run()
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
[2023-04-15 16:12:17,493: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 292, in __call__
sys.exit(self.workloop(pid=pid))
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 362, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 649, in fast_trace_task
R, I, T, Rstr = tasks[task].__trace__(
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 468, in trace_task
I, R, state, retval = on_error(task_request, exc, uuid)
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 379, in on_error
R = I.handle_error_state(
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 178, in handle_error_state
return {
[2023-04-15 16:12:17,494: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 237, in handle_failure
self._log_error(task, req, einfo)
[2023-04-15 16:12:17,495: WARNING/ForkPoolWorker-4] File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 265, in _log_error
logger.log(policy.severity, policy.format.strip(), context,
[2023-04-15 16:12:17,495: WARNING/ForkPoolWorker-4] Message: 'Task %(name)s[%(id)s] %(description)s: %(exc)s'
Arguments: {'hostname': 'celery@576889d96cb7', 'id': 'a1473004-be68-4afd-b9a4-b31d6d3c92a5', 'name': 'ix.task_log.tasks.agent_runner.start_agent_loop', 'exc': "KeyError('')", 'traceback': 'Traceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 451, in trace_task\n R = retval = fun(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 734, in __protected_call__\n return self.run(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/app/ix/task_log/tasks/agent_runner.py", line 13, in start_agent_loop\n return process.start()\n ^^^^^^^^^^^^^^^\n File "/var/app/ix/agents/process.py", line 188, in start\n self.loop(n=authorized_for, tick_input=tick_input)\n File "/var/app/ix/agents/process.py", line 193, in loop\n self.tick(execute=execute)\n File "/var/app/ix/agents/process.py", line 213, in tick\n command = self.command_registry.get(command_name)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/var/app/ix/commands/registry.py", line 73, in get\n return self.commands[name]\n ~~~~~~~~~~~~~^^^^^^\nKeyError: \'\'\n', 'args': '[]', 'kwargs': "{'task_id': 4}", 'description': 'raised unexpected', 'internal': False}