Comments (17)
After talking with @fostermh
Since the present method @fostermh use is using the appache log on top of the container. We will attempt to log usage by having a second container running in parallel that access the nginx log of the caprover erddap app.
The initial method was using appache, Matt will review if this method is compatible with NGINX which is used by caprover.
from hakai-datasets.
In consequence, we will ignore this present issue while developing #166
from hakai-datasets.
sorted. the host_url environment variable must contain the protocol, which I had forgotten to include in the telegraf setup. Thanks for noticing.
from hakai-datasets.
expanding on the details Jessy has already posted.
We will attempt to use telegraf running in a docker container to parse the nginx logs from caprover. This mirrors the current production setup in which erddap is running behind a proxy (apache) and telegraf is used to scrape the apache logs and report back on erddap usage.
TODO:
- export nginx logs from caprover nginx container so other containers can access them. See the following for details:
- caprover/caprover#880
- https://caprover.com/docs/nginx-customization.html
In short, update nginx config, via caprover setup page, to send access logs to/nginx-shared/
- mount nginx logs in telegraf container (
/captain/data/nginx-shared/
) and adjust log scraper settings as needed - setup plausible page and add key to telegraf container config
- setup sentry project and add key to telegraf container config (for cron job like monitoring)
- adjust sentry cron timeout so we don't get spammed and hate matt
- profit.
https://github.com/cioos-siooc/cwatch-telegraf
from hakai-datasets.
from hakai-datasets.
Just to add to present thread. Looks like ERDDAP itself is also suggesting to use the Tomcat/Apache/(NGINX?) log to track statistics and usage.
https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#tomcatLogs
@fostermh not sure if you started developing this after reading the ERDDAP docs but this is reassuring to see :)
from hakai-datasets.
I had not read the docs but yes nice to see that we independently arrived at the same solution. Hopefully we are on the correct track. :-)
from hakai-datasets.
@fostermh Thanks, I'll give this a shot on the development instance next week.
from hakai-datasets.
Other options I've used to parse NGINX - out of the box - in the past.
These need to be ruled out before a custom setup is attempted.
from hakai-datasets.
https://github.com/HakaiInstitute/erddap-goaccess
There is an example report.html in there we will need to check against requirements.
from hakai-datasets.
telegraf is scraping logs for the development version of erddap
- telegraf container has been setup https://captain.erddap.hakai.app/#/apps/details/telegraf
- pushing to plausible https://plausible.server.hakai.app/development.erddap.hakai.app
- and is checking in with sentry https://hakai-institute.sentry.io/crons/hakai-telegraf-erddap/hakai-erddap-telegraf-checkin/
from hakai-datasets.
Amazing "scrapping".
At the next ERDDAP 2.0 meeting lets compare and contrast and figure out what next or if this is oh so done.
from hakai-datasets.
haha spelling corrected. Yes would be good to contrast, sounds good.
from hakai-datasets.
I just had a quick look at the plausible and for some reasons the urls it refers to have a section doubled:
https://development.erddap.hakai.appdevelopment.erddap.hakai.app/erddap/tabledap/HakaiBamfieldBoL5min.htmlTable
Must be something related to the setup with plausible.
from hakai-datasets.
I just ran sucessfully on development.erddap.hakai.app/erddap the cde harvester. You should now
have logged a number of different csv querries to the erddap. :)
I think we should have all we need now
from hakai-datasets.
Nice! Lets meet at the end of this week and review these two analytics quick.
from hakai-datasets.
- @steviewanders Needs to add this to an analytics section of the resulting documentation & diagram but basically:
- https://github.com/allinurl/goaccess works to see all requests NGINX serves, which allows us to sort between monitoring bots, our own API requests (CDE et al.), and normal users of the HTML pages
- The specific setup for the above is here https://github.com/HakaiInstitute/erddap-goaccess
- It relies upon a NGINX
access.log
being persisted outside the ERDDAP container to the EC2 file system, which could benefit from a simple backup in case of $something_bad - Need to add a weekly cron job to generate a report and dump it and the log to S3
- Plausible can and has been added to the HTML template for ERDDAP so it runs as a client side Javascript tracker just GA
- Combined, these two address our two usage questions around ERDDAP.
from hakai-datasets.
Related Issues (20)
- Hakai ERDDAP is sending notifications to info at cioospacific.ca
- New ERDDAP Dataset: Hakai HPLC dataset Research dataset
- ...
- Algae Explorer Sentinel datasets daily and 8 day datasets HOT 1
- BamfieldBoL remote station new dataset HOT 6
- Decide which repo will be base repo for Hakai Data Mob. tasks and issues HOT 5
- [Issue]: SewardBoL view is not available and is breaking the creation of the different erddap views. HOT 1
- KC Buoy DO data is missing HOT 2
- Hakai Goose server is unstable HOT 18
- Add Hakai ERDDAP uptime check link HOT 1
- Deploy ERDDAP through Caprover HOT 4
- Rename Hakai ERDDAP repository? HOT 5
- Which email address should Hakai ERDDAP admin points to HOT 21
- Maintain Datasets Files between the different servers HOT 2
- Vulnerability with postgresql driver HOT 7
- ERDDAP with NGINX HOT 1
- new erddap.hakai.app server requires sudo docker
- hakai server catalogue.hakai.org docker v1.40 HOT 2
- HAKAI Nature Trust CTD Research dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hakai-datasets.