fujio-turner / sg-log-reader-demo Goto Github PK

Create the opposite chart to have more detailed Sync Errors to see if it effects sync.

I see how many connections are made but need to know by how many users. Can probably do a simple JS count while in the loop and update some HTML.

People do crazy things against the HTTP _changes feed that can effect SG. So need to track and record all the starts around all the HTTP calls like version 1.x had.

put the trouble makes on the y-axis

Sometimes the graph gets messy and data overlays over each other making it hard to read. Need to have a way to dynamically remove lines from graph.

Use http://www.chartjs.org/ to be able to show or hide different lines in the graph

Alot of the Line parsing and REG_EX things break from one version of SG to another. So need some basic unit test to at least the reg work as expected forwards and backwards.

Looks like windows build of the logs put spaces in different place in the HTTP making it harder to find _changes and other calls in the log scraper.

Here is the docs of the Streaming End Point: https://docs.couchbase.com/cloud/app-services/monitoring/configure-log-streaming-app-endpoint.html

Currently the log reader is a one shot of processing a sg_info.log file.

Want to create an option run it as a live service to take the HTTP PUTS from the above service and insert them into a CB cluster.

Weird crash on line 448 to 450 in sg-log-reader.py around calculating the time delta

Need to update the print "something" commands to print("something") for Python 3

The processing of the wsID LIST can be done by different threads to speed up the processing of a sg_log file.

Some people put extra time stamps in front of the Timestamps in that SG normally puts.
Need to put an string offset to be able to get to that SG timestamp

Need to count errors in changes feed when there is a "revocation" error

Sync Gateway Process logs put _time= in front of the string and break the script.

_time=2018-03-14T09:41:30.507-07:00 _level=INFO _msg=go-couchbase: call to ViewCustom("sync_gateway", "channels") in github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromView took 6.215245811s

Looks like as the sg_log_reader.py get to the end of the file it keep trying to read items past the end of the file that don't exist.

I think the solution is to as it process near the end of the file to tell the script only go so far so that you don't go past the end of the file.

The Orphanes have UUID for Keys ... need to make them predictable and eventually have a way to convert them back to a wsID based key.

graph is showing n/a instead of (0:00:000) when the event happens in the same time EX GET request & response happens in the same millisecond.

Many time there are go error not related to IMPORT , QUERY or DCP. So need to start adding those in the the docType:sgErrors

Yes the web socket Ids are unique but if you run multiple sg from different sync gateway all behind the same load balancer I cann't tell which logs is who's

The adding of sg process errors changes the x-axis value making them not the same as the channel cache values.
The work around is to click/remove the sg process , but the better method is to also change the channel cache chart x-axis too.

Need to be able to decode BLIP and WebSocket based replication.

Currently the tool only show completed
HTTP: & HTTP+:

There are many time HTTP: with no HTTP+: due to timeouts or failures.

Read up on passing Data List to multiple cores.

https://superfastpython.com/multiprocessing-share-large-data-between-processes/

Process Import.
by a per second
make the key un-unique on re-process of log file

Its maxing out the single core that the script is operating on.
Since it process a list of all the unique WebSocket ID and go through the IN-Memory version of the SG log file.
I beat we can pass the in memory list to multiple instantiation of this script and cut of that big list into 4 smaller list and have each object do work on multiple cores.

Right now in the begging and end of a sg_info.log there are Web-socket Ids that are incomplete.
But If I got a big sg_collect with a bunch of sg_info.log one right after another I need a way to process the orphans / incomplete WebSocket Calls.

The docs don't need to live forever to process an issue. maybe 48-72 hours. enough to load them in the morning on friday and still see them back on Monday afternoon. else reprocess them.