looselab / minotourcli Goto Github PK

Here's the error:

No Fastq Seen
FastQ Upload Status:
Files queued/processed/skipped/time/per file:0/0/0/1604944861.2268634/0
New reads seen/uploaded/skipped:0/0/0
Traceback (most recent call last):
  File "/home/grid/GIT/minotourcli/test_env/bin/minFQ", line 11, in <module>
    load_entry_point('minFQ', 'console_scripts', 'minFQ')()
  File "/home/grid/GIT/minotourcli/minFQ/minFQ.py", line 505, in main
    sys.stdout.write(", ".join(args.WATCHLIST) + "\n")
TypeError: sequence item 0: expected str instance, NoneType found

See title.

At the moment a -w won't throw a warning.

When Minions are created all the scripts for that position are sent to Minotour one at a time in individual post requests.

Would it be possible to send them in one request in a bulk JSON object. Would be way less hassle and some reduced overhead for the MinoTour database manager.

Currently ONT-fast5-api is a prerequisite - but not clear why?

Do we use it?

There are some more streaming rpcs available.

Needs fixing!

As in subject.

A platform QC does not generate a histogram and so MinFQ has a bit of a nervous breakdown.

If we don't have any other file to process, and we reach EOF we should post, otherwise roll over!

Crashing during platform QC

Method whitelist is no longer a parameter - pin the versions of things in setup.py and add a new wheel

There appears to be an issue with grpcio on apple silicon such that it doesn't correctly compile.

Installation can be fixed by following these instructions.

https://stackoverflow.com/questions/72620996/apple-m1-symbol-not-found-cfrelease-while-running-python-app

Not sure how this can be resolved normally.

Port choice should be default at port 80.

Loading data with a flowcell name is a problem. Can we somehow determine flowcell name from another file?

If starting an artic task (Job type 16) require a reference to be provided

Currently read data are uploaded by default. Consider making this just metrics only.

Occasionally I need to monitor a single sequencing position rather than all connected sequencing positions. Currently to accomplish this (monitoring MinKNOW and FASTQ) requires starting two minFQ instances that each upload only one aspect, e.g:

# Upload only FASTQ
minFQ ... -w /path/to/folder -nm

# Upload only live data
minFQ ... -nf

Ideally there should be a switch/flags for including or excluding named sequencing devices. e.g:

# Upload only device X1 and X2, exclude all other devices
minFQ ... --include X1 X2

# Exclude device X1 only include all other devices
minFQ ... --exclude X1

provided id to reference that didn't exist, just uploaded the data but didn't create the task

Provide a task flag where you can create a job master entry in the minotour app straight from the get go

/data/barcode_mapping2
Somehow they all use the same unblocked read_ids file - need to use the run_id to make sure the file is correct

Could show you branch and version number, so we can know if we're using the right one.

If you restart the server for example by making changes, you get this error ->
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Could it reconnect or at least close gracefully?

Check that the field names sent are correct, and capture read_length_type (Event or estimated bases), and wells_per_channel (Cheers flongle)

Currently we gzip all payloads, even if they are too small to really gain any speed via compression -> post -> decompression even things like 20kb are compressed so look at size limits before compressing and adding Content-Encoding to the request headers

Data that is posted to the server should be compressed before sending, and decompressed on receipt by the server. Would offer a suspected decrease in data transfer times