acaudwell / logstalgia Goto Github PK

Hi i'm wondering how i create a custom log format for Java's Jersey "Web Server"
An example of log entry is:

[2016-11-02 03:36:47,434]   INFO - erver.RequestLog.bundleProcess - 172.16.28.6 - - [02/Nov/2016:05:36:47 +0000] "GET /hub/api/rest/oauth2/target HTTP/1.1" 200 9 "-" "Jersey/2.22.2 (Apache HttpClient 4.5)" 0 

How i adapt Logstalgia for that?

Hi,

Could you please help with parsing the following format?
11.223.33.13 - 2016-09-29 12:32:58 0.165 POST /some URI/URI 200

#Fields: c-ip auth_user date time time-taken cs-method cs-uri sc-status

Can help me with this error, thanks

./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking how to run the C++ preprocessor... g++ -E
checking for X... libraries , headers
checking for a sed that does not truncate output... /bin/sed
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether we are using the Microsoft C compiler... no
checking windows.h usability... no
checking windows.h presence... no
checking for windows.h... no
checking for GL/gl.h... yes
checking for OpenGL/gl.h... no
checking for OpenGL library... -lGL
checking for GL/glu.h... yes
checking for OpenGL/glu.h... no
checking for OpenGL Utility library... -lGLU
checking for varargs GLU tesselator callback function type... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for FT2... yes
checking for PCRE... yes
checking for GLEW... yes
checking for SDL2... configure: error: Package requirements (sdl2 SDL2_image) were not met:

No package 'sdl2' found
No package 'SDL2_image' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables SDL2_CFLAGS
and SDL2_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

Its installed already
sdl2
SDL2_image

centos
CentOS release 6.10 (Final)

PKG CONFIG

/usr/lib64/pkgconfig
/usr/lib64/pkgconfig/com_err.pc
/usr/lib64/pkgconfig/fontutil.pc
/usr/lib64/pkgconfig/form.pc
/usr/lib64/pkgconfig/formw.pc
/usr/lib64/pkgconfig/gnome-python-desktop-2.0.pc
/usr/lib64/pkgconfig/gnome-python-extras-2.0.pc
/usr/lib64/pkgconfig/gnome-screensaver.pc
/usr/lib64/pkgconfig/libargon2.pc
/usr/lib64/pkgconfig/libcrypto.pc
/usr/lib64/pkgconfig/libedit.pc
/usr/lib64/pkgconfig/libpcre.pc
/usr/lib64/pkgconfig/libpcrecpp.pc
/usr/lib64/pkgconfig/libselinux.pc
/usr/lib64/pkgconfig/libsepol.pc
/usr/lib64/pkgconfig/libssl.pc
/usr/lib64/pkgconfig/libxml-2.0.pc
/usr/lib64/pkgconfig/menu.pc
/usr/lib64/pkgconfig/menuw.pc
/usr/lib64/pkgconfig/ncurses.pc
/usr/lib64/pkgconfig/ncursesw.pc
/usr/lib64/pkgconfig/netapi.pc
/usr/lib64/pkgconfig/notify-python.pc
/usr/lib64/pkgconfig/openssl.pc
/usr/lib64/pkgconfig/panel.pc
/usr/lib64/pkgconfig/panelw.pc
/usr/lib64/pkgconfig/tic.pc
/usr/lib64/pkgconfig/tinfo.pc
/usr/lib64/pkgconfig/zlib.pc
/usr/share/pkgconfig
/usr/share/doc/pkgconfig-0.23
/usr/share/doc/pkgconfig-0.23/AUTHORS
/usr/share/doc/pkgconfig-0.23/COPYING
/usr/share/doc/pkgconfig-0.23/NEWS
/usr/share/doc/pkgconfig-0.23/README
/usr/share/gtksourceview-2.0/language-specs/pkgconfig.lang
/usr/share/pkgconfig/gnome-default-applications.pc
/usr/share/pkgconfig/gnome-doc-utils.pc
/usr/share/pkgconfig/gnome-icon-theme.pc
/usr/share/pkgconfig/gnome-keybindings.pc
/usr/share/pkgconfig/shared-mime-info.pc
/usr/share/pkgconfig/usbutils.pc
/usr/share/pkgconfig/xml2po.pc
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/checksum_data
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/checksum_type
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/from_repo
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/from_repo_revision
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/from_repo_timestamp
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/installed_by
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/reason
/var/lib/yum/yumdb/p/481188174c707a1a3cfa09696240917b7c3ef8b7-pkgconfig-0.23-9.1.el6-x86_64/releasever

I try also

export PKG_CONFIG_PATH=/usr/bin/pkgconfig

nothing

Now the logstalgia can support various log format, but still many users may have various formsts.

Why not support the customization?

Request to add log format customization like https://github.com/allinurl/goaccess does.

logstalgia is amazing!
I'm curious about it！

how look count the number of requests in window?

src/ncsa.cpp 161

never used variable 'code'

Hello,
Absolutely love this software, so well done.

However, I like the idea of having more info show when hovering over a request, but whatever I try has zero effect.

Tried with quotes..
ssh root@myserver "tail -f /var/log/access.log" | logstalgia --display-fields "path,hostname,response_size,response_code,method,protocol,timestamp,referrer,user_agent,vhost" --sync

Tried without quotes..
ssh root@myserver "tail -f /var/log/access.log" | logstalgia --display-fields path,hostname,response_size,response_code,method,protocol,timestamp,referrer,user_agent,vhost --sync

Tried fewer fields..
ssh root@myserver "tail -f /var/log/access.log" | logstalgia --display-fields path,user_agent --sync

Using logstalgia-1.1.0.

Whatever I set, hovering over a request only shows something like "001 /home/".
Hovering over an ip address on the left only ever shows something like "001 157.55.39-".

Please can you post a working example?

Thanks

I tend to run several instances of logstalgia on my monitors while I work. One for each subsystem. It would be super helpful if you could add a configuration param to set the window title... or add a title somewhere.

Great project! I use this a whole lot to visualize all our nginx connected systems.

Hi Andrew,

I'm enjoying using Logstalgia to view server logs, it's great, thank you! I'm wondering how to change things around so that on the left hand side of the screen it shows the user agent, rather than the source IP address? There are some poorly behaved user agents out that use a range of source IP addresses to quickly index a website. I'm interested in using Logstalgia identifying and displaying this behaviour.

Regards,

Warren.

Is it possible to run this headless and just stream to youtube or file?

terminate called after throwing an instance of 'SDLInitException'
  what():  No available video device
[1]    19736 abort (core dumped)  logstalgia -1280x720 -o - /var/log/nginx/access.log

on Mac, it's much quicker to brew install this tool than manually install all the deps required to build.. but it seems like the brew recipe is out of date - is it maintained by you? are you able to update it?

So, we're running a live view of our server traffic on our screens, and we're currently sitting at 400000000 requests and counting. At the moment, the last digit is not showing and overflowing to the right. It'd be nice if it moved to the left as the number of digits increased to accommodate for large amount of requests.

Make proper sorting of ip addresses on the left side of the screen. Not to mixed up the address, starting for example at 13.xx.xx.xx and 130.xx.xx.xx. The sequence of addresses should be from 1 to 255 at each category of the address.

Looks like timestamps are converted to longs. At high request rates (100+ QPS), this causes batches of balls to get sent every second instead of as a stream. This not only ruins the effect, but also causes lag as hundreds of balls are rendered in batches.

hello

is there a way to exploit JSON based logs with ??

thx

Compiling 1.0.8 on FreeBSD produces the following error:

src/FileWatcher/FileWatcher.cpp:43:15: error: unknown type name 'FILEWATCHER_IMPL'
                mImpl = new FILEWATCHER_IMPL();
                            ^
1 error generated.
gmake[1]: *** [Makefile:986: src/FileWatcher/logstalgia-FileWatcher.o] Error 1

Some services generate logs which include the URL scheme (e.g. AWS ELB logs). The existing regular expression used with --hide-url-prefix does not account for this, resulting in URLs with the prefix still included.

I've create PR #18 to account for this edge case.

Example:

AWS Application ELB Log

h2 2017-04-30T00:55:00.366836Z app/router-prod/7a09fea154b03892 1.2.3.4:38273 x.x.x.x:80 0.000 0.700 0.000 200 200 247 60150 "GET https://xxx.xxx.xxx:443/re/recommended-for-you?collectionId=46 HTTP/2.0" "type/Windows.Xbox make/Microsoft model/Xbox One osVersion/10.0.15063.2019" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:954365800015:targetgroup/app/00000000000 "Root=1-0000000000000"

Logstalgia Custom Log

1493524502|1.2.3.4|https://xxx.xxx.xxx:443/re/recommended-for-you?collectionId=46|200|60150|1

Output

$ logstalgia --hide-url-prefix custom.log

I am getting the following error when running on macos.

$ logstalgia - 
libc++abi.dylib: terminating with uncaught exception of type SDLInitException: Failed creating OpenGL pixel format
[1]    5362 abort      logstalgia

For streaming video need hide window, example:

$ logstalgia --quiet --output-ppm-stream - access_log \
| ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libvpx -b 10000K logstalgia.webm

To stream into webpage as web administration but no need the graphical window in the server. you may not use the OS user interface? example, in CentOS with ffserver stream, in this way the user does not have to be installed logstalgia to observe traffic.

2d2006b fails to build on FreeBSD. Full build log.