markmac99 / pi-meteortools

A collection of scripts and code to analyse meteor and aurora data

README for pi-meteortools

Pi

This folder contains scripts that I use post the RMS and UKMON processing, to manage uploads of data to youtube and send me an email about how many detections there were.

Radio

Scripts and config used on my windows-based Radio meteor detector.

Windows

Scripts deployed to my windows desktop to collect data from my pis and review it using CMN_Binviewer, then generate various outputs for my website. This includes some scripts that do nice things like generate multicamera radiant maps create all-month stacks per camera. There are also a lot of legacy scripts i need to remove.

Wpserver

Scripts deployed to my AWS server where further processing is done to generate nice graphs for my website.

Infrastructure

Terraform to create the AWS assets used by the above.

pi-meteortools's Issues

Create single stacks page

Single page showing all four camera's nightly stacks (or trackstacks), so its easy to quickly look for fireballs.
Should also be able to look back at historic dates so we can easily review something from a while back.

latest-page job doesn't reset to 0900

After being rescheduled, the job doesn't move back to the correct start time

Separately store ML rejects

For future training / analysis

Check and restart allsky if it gets stuck

Check the log's datestamp
if more than 5 minutes old
check if allsky running (in case i have manually stopped it)
if its running, then restart it.

if its not running, check syslog for errors maybe?

email woes

1. mail from weatherpi fails with
   <<< 550-5.7.26 This mail is unauthenticated, which poses a security risk to the
   <<< 550-5.7.26 sender and Gmail users, and has been blocked. The sender must
   <<< 550-5.7.26 authenticate with at least one of SPF or DKIM. For this message,
   <<< 550-5.7.26 DKIM checks did not pass and SPF check for [weatherpi.localdomain]
   <<< 550-5.7.26 did not pass with ip: [217.43.43.250]. The sender should visit
   <<< 550-5.7.26 https://support.google.com/mail/answer/81126#authentication for
   <<< 550 5.7.26 instructions on setting up authentication. y2-20020a5d4ac2000000b00317e08e4881si3543772wrs.676 - gsmtp
   554 5.0.0 Service unavailable

Sender is marked as [email protected]

Crop allsky image

Crop the image square to save real-estate and video size.
Also would look neater.

Upload auroracam timelapse to YT

Should be able to do this now as i haev dropped one of the allsky vids.

Add cloudfront logic

mjmm-data is exposed through cloudfront, so it can serve out the stacks and videos. Add this logic to terraform

Adjust auroracam brightness with daylight

At present, auroracam switches mode abruptly at dawn. It'd be better to gradually change the sensitivity so that the overall brightness remains roughly constant. Maybe the camera has an auto-exposure mode? Worth checking. Otherwise, i could determine the end of astronomical dark and gradually adjust the gain down till full daylight.

Better housekeeping on auroracam

• upload intersting tarballs to server
• delete uninteresting ones till 3GB free space
• compress folders older than 1 week

Create historical single-page views

Using the js file i'm already creating, create an historical version of cameradata

Upload new allsky to youtube

new allsky isn't uploadedto YT. Figure out how to do that.

Add match count to the MQ/OpenHab data

would be nice to see - could also add a summary to the cameradata page maybe ?

create single page with all four nightly stacks for quick checking

in addition to the monthly stack, display the trackstacks and nightly stacks on one page (and maybe allsky).

Disconnect the s3fuse filesystem

Its expensive to use and causes regular movements back out of IA storage.

Network resilience in YT and AWS uploads - at least retry a bit!

Recent issues w/ connection to YT and AWS failing. Need some resilience - back off and retry at a minimum

Monthly stacks include many aircraft

Improve the process that cleans them
and provide a manual process

Move auroracam to astro2

Astro2 is unused, move auroracam to it
Rename to auroracam to simplify everything
will need some space management as it only has a small disk (32GB) and no working SSD slot
but test the slot!

Check cloudwatch metrics in MM account

Why 40 cents per day from 07/oct ? What changed then?

Ensure auroracam restarts use existing folder

Intra-night restarts create a new folder.
Check for and use any existing folder instead.

Create default "no trackstack" image

if there aren't enough meteors, trackstack will fail to create a file.
In this case create a default from the stack file and overlay "NO TRACKSTACK TODAY" on it.

Remove hard-coded details in terraform

account, bucket names etc