robertrosca / jaxtam.jl

Just Another X-ray Timing Analysis Module

Hi 👋, I'm Robert Rosca

jaxtam.jl's Issues

TODO: Speed up generation of append table

Making the append table takes a (relatively) long time since it has to read through thousands of logs sequentially, this was not really the best way I could have handled the logging

Solutions:

1. Minor, probably, changes to some of the base logging functions, such that each observation goes to a master log file, with nested dictionaries where the first key is the obsid. This abomination of nested dictionaries can then be parsed to create a DataFrame

2. Something with distributed arrays where the for loop over all observations is parallelised

3. Check the last edit time for the append file, and the last edit time of the log file, that way it can be skipped if it has not been updated recently

4. Ignore the problem since it's running on a server at night so it taking two minutes to update a table doesn't matter too much