A web application to validate and upload target lists with pointing simulations for PFS observation
Home Page: https://pfs-etc.naoj.hawaii.edu/uploader/
License: MIT License
It would be nice to save the PPP plots as html.
related to #64
The distributions of PPCs for an input list vary slightly, though the number of pointings are same. I think it's the nature of Gurobi optimization and making it more reproducible increase the computational time. We may just need to document it to let users know it.
The result shown in the "Time estimate" is very confusing.
In the documentation, it says that the warning will be raised in the following cases:
I think the first one can be treated as a warning (but with yellow, not red).
But it should not say "Please reduce the time".
The second one should be shown as an ERROR, not a warning.
Describe the bug
KeyError happens and it seems to freeze the app.
To Reproduce
I don't know. Need to be explored.
Expected behavior
No error.
Additional context
I found the following lines in a log file.
[I 231004 15:45:11 pn_app:40] S24B-QN009.csv is selected.
[I 231004 15:45:11 utils:392] [STAGE 1] Checking column names
[I 231004 15:45:11 utils:209] Required key `obj_id` is found
[I 231004 15:45:11 utils:209] Required key `ob_code` is found
[I 231004 15:45:11 utils:209] Required key `ra` is found
[I 231004 15:45:11 utils:209] Required key `dec` is found
[I 231004 15:45:11 utils:209] Required key `equinox` is found
[I 231004 15:45:11 utils:209] Required key `priority` is found
[I 231004 15:45:11 utils:209] Required key `exptime` is found
[I 231004 15:45:11 utils:209] Required key `resolution` is found
[W 231004 15:45:11 utils:226] Optional key `pmra` is missing
[W 231004 15:45:11 utils:226] Optional key `pmdec` is missing
[W 231004 15:45:11 utils:226] Optional key `parallax` is missing
[W 231004 15:45:11 utils:226] Optional key `tract` is missing
[W 231004 15:45:11 utils:226] Optional key `patch` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_g` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_r` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_i` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_z` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_y` is missing
[W 231004 15:45:11 utils:226] Optional key `filter_j` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_g` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_r` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_i` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_z` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_y` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_j` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_g` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_r` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_i` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_z` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_y` is missing
[W 231004 15:45:11 utils:226] Optional key `flux_error_j` is missing
[I 231004 15:45:11 utils:394] [STAGE 1] required_keys status: True (Success if True)
[I 231004 15:45:11 utils:397] [STAGE 1] optional_keys status: True (Success if True)
[I 231004 15:45:11 utils:410] [STAGE 2] Checking string values
[I 231004 15:45:11 utils:412] [STAGE 2] status: True (Success if True)
[I 231004 15:45:11 utils:420] [STAGE 3] Checking wheter values are in allowed ranges
[I 231004 15:45:11 utils:422] [STAGE 3] status: True (Success if True)
[I 231004 15:45:11 utils:429] [STAGE 4] Checking whether all ob_code are unique
[I 231004 15:45:11 utils:372] All 'ob_code' are unique.
[I 231004 15:45:11 utils:431] [STAGE 4] status: True (Success if True)
2023-10-04 15:45:11,435 error handling message
message: Message 'PATCH-DOC' content: {'events': [{'kind': 'MessageSent', 'msg_type': 'bokeh_event', 'msg_data': {'type
': 'event', 'name': 'button_click', 'values': {'type': 'map', 'entries': [['model', {'id': 'p2194'}]]}}}]}
error: KeyError('priority')
Traceback (most recent call last):
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/protocol_handler.py", line 97, in handle
work = await handler(message, connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/session.py", line 94, in _needs_document_lock_wrapper
result = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/session.py", line 286, in _handle_patch
message.apply_to_document(self.document, self)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/protocol/messages/patch_
doc.py", line 104, in apply_to_document
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.payload, setter=setter))
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py",
line 443, in invoke_with_curdoc
return f()
^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/protocol/messages/patch_
doc.py", line 104, in <lambda>
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.payload, setter=setter))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/document.py", l
ine 391, in apply_json_patch
DocumentPatchedEvent.handle_event(self, event, setter)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/events.py", lin
e 245, in handle_event
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/events.py", lin
e 280, in _handle_event
cb(event.msg_data)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py",
line 390, in trigger_event
model._trigger_event(event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/util/callback_manager.py
", line 113, in _trigger_event
self.document.callbacks.notify_event(cast(Model, self), event, invoke)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py",
line 260, in notify_event
invoke_with_curdoc(doc, callback_invoker)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py",
line 443, in invoke_with_curdoc
return f()
^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/util/callback_manager.py
", line 109, in invoke
cast(EventCallbackWithEvent, callback)(event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 491,
in _server_event
self._comm_event(doc, event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 478,
in _comm_event
state._handle_exception(e)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/io/state.py", line 436,
in _handle_exception
raise exception
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 476,
in _comm_event
self._process_bokeh_event(doc, event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 413,
in _process_bokeh_event
self._process_event(event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/button.py", line
243, in _process_event
self.clicks += 1
^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
367, in _f
instance_param.__set__(obj, val)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
369, in _f
return f(self, obj, val)
^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/__init__.py", line 625,
in __set__
super(Dynamic,self).__set__(obj,val)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
369, in _f
return f(self, obj, val)
^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
1252, in __set__
obj.param._call_watcher(watcher, event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
2043, in _call_watcher
self_._execute_watcher(watcher, (event,))
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
2025, in _execute_watcher
watcher.fn(*args, **kwargs)
File "/work/monodera/gh/monodera/pfs_target_uploader/src/pfs_target_uploader/pn_app.py", line 167, in cb_validate
panel_targets.show_results(df_input)
File "/work/monodera/gh/monodera/pfs_target_uploader/src/pfs_target_uploader/widgets.py", line 335, in show_results
self.table_all.value = pd.DataFrame()
^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
367, in _f
instance_param.__set__(obj, val)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
369, in _f
return f(self, obj, val)
^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
1252, in __set__
obj.param._call_watcher(watcher, event)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
2043, in _call_watcher
self_._execute_watcher(watcher, (event,))
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line
2025, in _execute_watcher
watcher.fn(*args, **kwargs)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/tables.py", line
1494, in _update_cds
super()._update_cds(*events)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/util/__init__.py", line
391, in wrapped
fn(self, *args, **kwargs)
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/tables.py", line
277, in _update_cds
self._processed, data = self._get_data()
^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/tables.py", line
1294, in _get_data
df = self._sort_df(df)
^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/tables.py", line
375, in _sort_df
df_sorted = df.sort_values(fields, ascending=ascending, kind='mergesort',
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/pandas/core/frame.py", line 69
30, in sort_values
keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/pandas/core/frame.py", line 69
30, in <listcomp>
keys = [self._get_label_or_level_values(x, axis=axis) for x in by]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/monodera/gh/monodera/pfs_target_uploader/.venv/lib/python3.11/site-packages/pandas/core/generic.py", line
1844, in _get_label_or_level_values
raise KeyError(key)
KeyError: 'priority'
Is your feature request related to a problem? Please describe.
Currently, a target is considered as visible when the total visible time exceeds the requested exposure time. This would not be correct as a unit exposure time is 900s and the counts need to be quantized.
Describe the solution you'd like
Visibility is judged by a floor-ed number of unit exposure time in an observing period.
Is your feature request related to a problem? Please describe.
There is no authentication now. It is possible for random people to submit a relatively large file every seconds, though it's not likely.
Also, it's impossible for us to know who submitted a particular target list and we can communicate with users only by upload IDs.
Describe the solution you'd like
Add an authentication mechanism using PROMS and STARS IDs at least for submission. Then we can send an email to the person who made the submission.
Additional context
#64
Is your feature request related to a problem? Please describe.
User cannot know the exact content of the pointings after submission.
Describe the solution you'd like
I'd like to propose to make a zip (or other archive) file containing outputs (target list, pointing lists, and other diagnostic plots if any) and make it downloadable for users.
Describe alternatives you've considered
PPP result page can contain such information so that users can know the output and play with it in their local environment for further investigation (or possibly social debugging).
Additional context
If paired with authentication, we can make a link to files and tell them by a confirmation email.
I was just wondering what is "None" at the bottom of the summary paragraph.
Describe the bug
PPP crashes when the number of pointings is larger than 3999 by the limitation of hvplot.
Expected behavior
PPP return error message.
Error output
2023-10-23 10:04:40,166 error handling message
message: Message 'PATCH-DOC' content: {'events': [{'kind': 'MessageSent', 'msg_type': 'bokeh_event', 'msg_data': {'type': 'event', 'name': 'button_click', 'values': {'type': 'map', 'entries': [['model', {'id': 'p12670'}]]}}}]}
error: ValueError('Argument must be between 1 and 3999')
Traceback (most recent call last):
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/protocol_handler.py", line 97, in handle
work = await handler(message, connection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/session.py", line 94, in _needs_document_lock_wrapper
result = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/server/session.py", line 286, in _handle_patch
message.apply_to_document(self.document, self)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/protocol/messages/patch_doc.py", line 104, in apply_to_document
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.payload, setter=setter))
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 443, in invoke_with_curdoc
return f()
^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/protocol/messages/patch_doc.py", line 104, in <lambda>
invoke_with_curdoc(doc, lambda: doc.apply_json_patch(self.payload, setter=setter))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/document.py", line 391, in apply_json_patch
DocumentPatchedEvent.handle_event(self, event, setter)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/events.py", line 245, in handle_event
event_cls._handle_event(doc, event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/events.py", line 280, in _handle_event
cb(event.msg_data)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 390, in trigger_event
model._trigger_event(event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/util/callback_manager.py", line 113, in _trigger_event
self.document.callbacks.notify_event(cast(Model, self), event, invoke)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 260, in notify_event
invoke_with_curdoc(doc, callback_invoker)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/document/callbacks.py", line 443, in invoke_with_curdoc
return f()
^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/bokeh/util/callback_manager.py", line 109, in invoke
cast(EventCallbackWithEvent, callback)(event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 491, in _server_event
self._comm_event(doc, event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 478, in _comm_event
state._handle_exception(e)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/io/state.py", line 436, in _handle_exception
raise exception
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 476, in _comm_event
self._process_bokeh_event(doc, event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/reactive.py", line 413, in _process_bokeh_event
self._process_event(event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/widgets/button.py", line 243, in _process_event
self.clicks += 1
^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 367, in _f
instance_param.__set__(obj, val)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 369, in _f
return f(self, obj, val)
^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/__init__.py", line 625, in __set__
super(Dynamic,self).__set__(obj,val)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 369, in _f
return f(self, obj, val)
^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 1252, in __set__
obj.param._call_watcher(watcher, event)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 2043, in _call_watcher
self_._execute_watcher(watcher, (event,))
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 2025, in _execute_watcher
watcher.fn(*args, **kwargs)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/src/pfs_target_uploader/pn_app.py", line 227, in cb_PPP
res_mode_, nppc, p_result_fig, p_result_ppc, p_result_tab_ = ppp_result(
^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/src/pfs_target_uploader/utils/ppp.py", line 1180, in ppp_result
return nppc, p_result_fig, p_result_tab, p_result_ppc
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/src/pfs_target_uploader/utils/ppp.py", line 1137, in ppp_plotFig
obj_alloc = obj_allo1[:nppc_fin]
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/layout/base.py", line 782, in __init__
params['objects'] = [panel(pane) for pane in objects]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/layout/base.py", line 782, in <listcomp>
params['objects'] = [panel(pane) for pane in objects]
^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/pane/base.py", line 87, in panel
pane = PaneBase.get_pane_type(obj, **kwargs)(obj, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/param.py", line 788, in __init__
self._replace_pane()
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/param.py", line 854, in _replace_pane
new_object = self.eval(self.object)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/param.py", line 813, in eval
return eval_function(function)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/util/__init__.py", line 328, in eval_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/param/parameterized.py", line 407, in _depends
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/panel/depends.py", line 249, in wrapped
return eval_fn()(*combined_args, **combined_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/src/pfs_target_uploader/utils/ppp.py", line 1010, in plot_ppc
for ii in ppc_coord
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/src/pfs_target_uploader/utils/ppp.py", line 1008, in PFS_FoV_plot
ppc_tot_plot = [
^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/overlay.py", line 178, in __init__
super().__init__(items, **params)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/dimension.py", line 1321, in __init__
items = self._process_items(items)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/dimension.py", line 1342, in _process_items
cls._unpack_paths(vals, items, counts)
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/dimension.py", line 1395, in _unpack_paths
new_path = util.make_path_unique(path, counts, new)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/util.py", line 1907, in make_path_unique
path = path + (int_to_roman(count),)
^^^^^^^^^^^^^^^^^^^
File "/Users/monodera/Dropbox/NAOJ/PFS/pfs_target_uploader/.venv/lib/python3.11/site-packages/holoviews/core/util.py", line 1162, in int_to_roman
raise ValueError("Argument must be between 1 and 3999")
ValueError: Argument must be between 1 and 3999
If applicable, add screenshots to help explain your problem.
Computer Environment:
Additional context
It's not realistic, but.
Buttons move depending on the validation status and progress. I'd like to somehow fix the positions of buttons so that one does not need to scroll the side pane.
According to the document, the flux of the first filter (i.e. on the left-most column) will be used if more than one filters were used for given targets. However, the result looks different. Here is a screenshot of target table after validation.
Information on the semester (A or B) is required to check the visibility. The semester identifier is hard-coded in the current version, but would be nice to move this to the environmental variable.
This still implies that we only run PPP for one semster, rather than multi semesters. The latter case may be favored by intensive and filler proposers, but I'm not sure if we explicitly allow to set the time interval. Maybe it is possible to make them as hidden, but editable parameters.
Currently, the output data are saved in directories structured as follows.
${OUTPUT_DIR_PREFIX}/
├── ${OUTPUT_DIR_data}/ # target list
├── ${OUTPUT_DIR_ppc}/ # PPP pointing centers
└── ${OUTPUT_DIR_ppp}/ # PPP summary
Each folder contains a corresponding file for each upload_id.
I'm wondering if all lists are organized for each upload_id as follows.
${OUTPUT_DIR_PREFIX}/
├── ${upload_id_1}/ # for upload_id 1
├── ${upload_id_2}/ # for upload_id 2
└── ${upload_id_3}/ # for upload_id 3
In this case, the review might be easier, but the number of directories increases as time goes. Maybe we can make directories for semesters.
I don't know which is better, but it's worth considering.
Is your feature request related to a problem? Please describe.
ROT and other information recorded in the pointing list is the numbers of pointings in L and M modes shown at the sliders in the main panel. It is somewhat unclear for users which numbers will be sent to the observatory.
Describe the solution you'd like
Near the section Step 3, we could show the current ROT (or number of pointings) synced from the sliders and/or tables.
Describe alternatives you've considered
All pointing are saved regardless of the slider values.
In contrast to the pointing lists, the target list would contain objects considering all pointings assigned by the online ppp.
Here is collection of comments on documentation page.
https://drive.google.com/drive/folders/1kh9RbItKxcQVEOQVVub51uC82Ppye45T?usp=sharing
We put comments on screenshot of each page.
Is your feature request related to a problem? Please describe.
Since equinox is fixed to be "J2000.0", the equinox column is not important and ignored even if it exists in the input list.
Describe the solution you'd like
equinox can be removed from the input list. Instead describe it in the documentation like "ra (deg) corresponding to J2000.0 or in ICRS at the epoch of 2000.0)".
When I uploaded the target list with a wrong filter name, it failed at stage 3', no details were shown.
This is a screenshot of resultant page.
Is your feature request related to a problem? Please describe.
Visiblity check can be done at Step 2.
Describe the solution you'd like
Additional context
This is very minor, but it might be better if we do not show
Error
Detected errors are listed below. Please fix them.
in the red box if there are no errors. The current red box can be misleading.
The same thing can be said for the "Warning" part.
When some of ob_codes are invisible, the following message show up:
Some ob_codes are not visible in the input observing period.
It will be helpful to show the number of such ob_codes.
upload_id is generated by secrets.token_hex(8) with Python (e.g, 315428a98335c916). In the documentation of secrets, I find the following description.
As of 2015, it is believed that 32 bytes (256 bits) of randomness is sufficient for the typical use-case expected for the [secrets](https://docs.python.org/3/library/secrets.html#module-secrets) module.
https://docs.python.org/3/library/secrets.html#how-many-bytes-should-tokens-use
A 32 bytes string looks like a0079371f017e22e21128ec11f80fbf4f9a80c43a1b01bca05ef57309f85b30f and it seems a bit too long.
I'd like to hear some opinion if it is sufficiently long for our purpose.
One can also look up past upload IDs for duplication and regenerate another ID.
It would be nice to put an example query for the case of HSC-SSP to make a target list that is valid for PFS.
Details of the validation for flux columns are not in the document. While I'm planning to change the look of the status indicators, at some point it needs to be consistent.
Probably it is planned to make a document on uploader site itself, it will be nice to add a README file to the zipped file, just explaining what the each file is. I think a fixed file should be fine. For example, a file listing
- ppc_{upload_id}.ecsv : list of all simulated pointing center
- psl_{upload_id}.ecsv : summary table of PPP simulaition
...
or something like this..
I'm wondering how we ask users to put flux and filter information. I can think of two possibilities.
nJy. (endorsed by Tanaka-san)targetdbThe targetdb has columns like filter_{grizyj} and psf_flux_{grizyj} and psf_flux_error_{grizyj}. We can add total_flux* columns in the targetdb and alias to flux_* in the input list. Then, we can ask users to use them as columns of the input list. The flux-related columns looks like the following.
obj_id,filter_r,flux_r,filter_j,flux_j,flux_error_j
1,rp_gaia,1000000,,,
2,bp_gaia,100000,,,
3,i2_hsc,10000,j_mko,20000,200
bp_gaia in filter_r (the above example), while it's entirely valid and has no problem downstream, I suppose.The other possibility is that users use the pre-defined filter name in the column header in the input list. The example list in the case 1 looks like the following in this case.
obj_id,rp_gaia,bp_gaia,i2_hsc,j_mko,j_mko_error
1,1000000,,,,
2,,100000,,,
3,,,10000,20000,200
targetdb.r2_hsc and r_ps1 are regarded as _r filters, and cannot co-exist if we set filter categories.To judge feasibility of programs, it is helpful to see achieved S/N on the uploaded targets.
For example, a plot of exposure time vs flux of the targets could be an option. If the plot also shows model lines for a few sets of S/N, it will be nice.
Is your feature request related to a problem? Please describe.
In the admin page, the PPC list is shown by clicking the magnify icon. I feel it is too wide after opening the list.
Describe the solution you'd like
It may look simple and perhaps more beautiful if the PPC list is shown as a floating panel.
Describe alternatives you've considered
As it is.
The following request is based on my personal thoughts, so I think we can decide which should be needed:
1) A plot showing exposure time vs flux -> Issue#43
2) Filter the programs by range. for example, filter the target list with N(object) > 1000, or filter the program with Time_tot_L > 50 [h].
3) I guess N(object) on "target list" tab means "entry, i.e., the number of the lines... the header should be modified
64-bit integers after a certain number are not correctly displayed in the target list. This related to a Bokeh datamodel, so it would be difficult to fix. We may need to consider to use a different widget if those can handle them properly exists.
Input list:
| ob_code | obj_id | ra | dec | equinox | exptime | priority | resolution |
|---|---|---|---|---|---|---|---|
| obj0_eet900 | 109471469256015134 | 146.925596 | 1.228884 | J2000 | 900 | 9 | M |
| obj1_eet900 | 110931472826814909 | 147.282679 | 2.445355 | J2000 | 900 | 7 | M |
| obj2_eet900 | 109481470008251431 | 147.000815 | 1.234118 | J2000 | 900 | 8 | M |
| obj3_eet900 | 109891465704899386 | 146.570465 | 1.582479 | J2000 | 900 | 4 | M |
| obj4_eet900 | 109891465718875969 | 146.57187 | 1.579617 | J2000 | 900 | 6 | M |
| obj5_eet900 | 109921478271454299 | 147.82713 | 1.603211 | J2000 | 900 | 2 | M |
| obj6_eet900 | 110871474784583379 | 147.478434 | 2.394059 | J2000 | 900 | 3 | M |
| obj7_eet900 | 109471464933044585 | 146.493297 | 1.228462 | J2000 | 900 | 4 | M |
| obj8_eet900 | 109731462327942764 | 146.232787 | 1.443574 | J2000 | 900 | 1 | M |
Output table:
(I'm not sure if it is related with another Issue, but) when I uploaded one of the mock-up catalogue (S24B-QN009.csv) and pushed "validated" button, the list looked strange. (The phenomena didn't not always occur..)
Is your feature request related to a problem? Please describe.
Visibility check takes long time for a large catalog.
Describe the solution you'd like
Maybe parallelize the visibility check?
Is your feature request related to a problem? Please describe.
When the target list is large (e.g., >1000), computational time becomes recognizable.
Describe the solution you'd like
It would be nice if we can vectorize the process.
Describe alternatives you've considered
Just as it is.
I guess it will be better to raise an error if the End Date in the config tab is set to a date earlier than the Start date.
In the current version, it does not raise an error and it pretends like it's calculating (but no results will come back, of course...).
Perhaps it worth considering to show a loading spinner or bar for validation and ppp.
Coordinate transformation from sky to pfi plane only accepts J2000. based on agreement with the AG camera code. I think it is users responsibility to convert the targets. (If target uploader accepts other equinox, we need some code to convert sky coordinates when you run PPP and SFA which use pfs_utils.
I'm wondering whether the colors used to differentiate the status, plots, etc. are colorblind-friendly or not. For example, it's not colorblind-friendly to use red and green together, but they are used in the status indicator. It's not easily possible to use arbitrary colors in the BooleanStatus widget, while there are choices.
Also, it may be worth checking similar things for other places such as plots.
When I selected and ran validation a new target list after validating (and running PPC) another target list, validation status remains.
I used a catalog containing 100 objects with random positions at -30<dec<85 and online PPP won't return and eventually the connection is closed. I'm wondering if it's due to the catalog, PPP, Panel app (bokeh server), or Nginx.
When I tried clicking the legend of the low-resolution mode plot, in some cases the plot becomes weird like below.
So far I have tried uploading two target lists and it happened on both of them.
I tried the reload button on the bokeh plot menu bar, but could not solve the issue.
This does not happen in the "Distribution of the targets & PPC plot".
I feel that this might be a bug in the bokeh plot, so I am not so sure if we can debug this.
Is your feature request related to a problem? Please describe.
It's not clear how the ROT is calculated.
Describe the solution you'd like
In the Config tab, we can list the numbers for the unit exposure time and assumed overheads for calibration, fiber configuration, etc. as non-editable boxes.
Describe alternatives you've considered
List these parameter in the documentation.
When I used a target list which has a wrong format (I didn't mean it though), the validation process never finished.
This is a target list
S24B-QN009.mod.test.csv
ob_code obj0_eet900 (the first one) is missing resolution.
After the list is corrected by adding resolution, validation process finishes in a moment.
Currently, a night is defined from 18:30 to 5:30. We may need to consider changing it to twilight to twilight, sunset-to-sunrise, etc.
The graph showing sky distribution of the targets and PPCs seems to fix the y-maximum as 0 when the list has only the targets with dec < 0.
It is trivial, but it would be more convenient to if the y-limit is adjusted.
Currently, non-Gecko-based web browsers such as Chrome, Safari, Brave, Vivaldi, etc. cannot handle the situation when the same file with modified content is re-selected in the FileInput widget.
There are some posts in the forums of Panel and Bokeh packages, but so far no clear answer.
I want to have a CLI tool to run the online PPP calculation.
Describe alternatives you've considered
It's okay if it does not exist.
Describe the bug
When the link to the "example" target list is clicked, the connection is closed immediately.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
One should be able to click the "validate" button again, for example and see the results.
Computer Environment:
Additional context
Maybe upstream problem. I'll ask Panel community with a simple reproducible example.
It looks that one can press buttons such as 'Validate' and 'Run PPP' before finishing to upload the input file. For a small list uploading file content is nearly instant, but for a large file the issue become visible. I'd like to lock buttons until the upload finishes and keep googling...
Output on validation disappeared when I clicked "step 3" button.
It it is not so difficult, it will be helpful to keep (at least) warnings/info.
Is there something like an “available time duration” for the slider in the “Result of PPP” tab?
What I experienced is below:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.