intaro / pinboard

Realtime PHP monitoring system which aggregates and displays Pinba data.

Intaro Pinboard

Intaro Pinboard (Pinba Board) is a realtime PHP monitoring system which aggregates and displays pinba data.

Developed on Silex framework and works with PHP 5.3.3 or later.

Installation

Before Pinboard installation you should already be installed pinba. Since the version 1.5 Intaro Pinboard requires Pinba 1.1 or higher.

1. Download application:

    $ git clone git://github.com/intaro/pinboard.git --branch=v1.5.2
   

2. Download composer:

    $ curl -sS https://getcomposer.org/installer | php
   

3. Install dependency libraries through composer (and enter the parameters of connection to Pinba database):

    $ php composer.phar install
   

4. Initialize app (command will create additional tables and define crontab task):

    $ ./console migrations:migrate
    $ ./console register-crontab
   

5. Point the document root of your webserver or virtual host to the web/ directory. Read more in Silex documentation. Example for nginx + php-fpm:

    server {
        listen 80;
        server_name pinboard.site.ru;
        root /var/www/pinboard/web;
   
        #site root is redirected to the app boot script
        location = / {
            try_files @site @site;
        }
   
        #all other locations try other files first and go to our front controller if none of them exists
        location / {
            try_files $uri $uri/ @site;
        }
   
        #return 404 for all php files as we do have a front controller
        location ~ \.php$ {
           return 404;
        }
   
        location @site {
            fastcgi_pass unix:/tmp/php-fpm.sock;
            include fastcgi_params;
            fastcgi_param  SCRIPT_FILENAME $document_root/index.php;
            fastcgi_param HTTPS $https if_not_empty;
        }
   
        location ~ /\.(ht|svn|git) {
            deny  all;
        }
    }
   

More details in section Installation of the documentation.

Update

Update from 0.1 to 1.0

Branch 1.0 brings migrations machinery which allows to update Pinboard easy when it requires database schema transformation.

Switch to branch 1.0

$ git fetch
$ git checkout v1.0

Update vendors

$ php composer.phar update

Register migration

$ ./console migrations:version --add 20131013132150

Update between 1.x versions

Switch to branch 1.x

$ git fetch
$ git checkout v1.x

Update vendors

$ php composer.phar update

Apply changes to database

$ ./console migrations:migrate

Add to parameters.yml new options from parameters.yml.dist.

More Information

Documentation in Wiki.

License

Intaro Pinboard is licensed under the MIT license.

pinboard's Issues

Slow query on main page for user with full access

Снизить частоту рассылок писем "drawdown of indicators"

Может не стоит так часто рассылать письма "drawdown of indicators" где статус даты [OK]? Если все ОК - то кого это должно волновать?

Add sort on page overview

• Add sorting by js
• Example http://tablesorter.com/docs/#Demo

Не удаляется файл блокировки (падает процесс аггрегации) .../AggregateCommand.php.lock file

After some time cant see data in web.
After try start Crontab task manually get error:

Cannot run data aggregating: the another instance of this script is already executing. Otherwise, remove /var/www/pinboard/src/Pinboard/Command/AggregateCommand.php.lock file

but nothing running at this time.
to resolve this issue, to the end of aggregate command in crontab add
&& rm -rf /%PATH_to_lockFIle%/AggregateCommand.php.lock

pinboard v.1.1

Remove old rows from ending of table on update of page Live

Every 5 seconds page Live throw ajax requests fresh data from server and adds them to top of table. Table size grows and after a time can make page not responding.

Total values on the homepage

Show total values in table on the homepage.

Agregation can be run few times in paralel

If agregation function not finished in 5 minutes interval. New instance will be run by cron which will overload db with heavy commands running in parallel. And possible will not have much sence. Possible need to add some lock to avoid this.

SQL Exception on latest sources

www # /var/www/pinboard/console aggregate

[PDOException]
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

aggregate

Can it work without APC?

Try to start, getting exception:

2013/07/22 18:08:09 [error] 12904#0: *1135 FastCGI sent in stderr: "PHP message: PHP Fatal error: Call to undefined function Doctrine\Common\Cache\apc_fetch() in /var/www/pinboard/vendor/doctrine/common/lib/Doctrine/Common/Cache/ApcCache.php on line 52" while reading response header from upstream, client:XX.XX.XX.XX, server: somesite.shalb.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "somesite.shalb.com"

PHP-FPM and secure does not work

When using the secure option with PHP-FPM, the user can not login.

The fix, found in Pydio documentation is to use theses lines in .htaccess:

Following lines seem to be necessary if PHP is working

with apache as CGI or FCGI. Just remove the

See http://doc.tiki.org/WebDAV#Note_about_Apache_with_PHP_as_fcgi_or_cgi

RewriteCond %{HTTP:Authorization} ^(.)
RewriteRule ^(.) - [E=HTTP_AUTHORIZATION:%1]

Add indexes to pinboard tables

Add necessary indexes to pinboard tables.

Display http status for requests in Live tab

Remove old records from pinboard tables

• Add to console aggregate removing old record from pinboard tables.
• Add parameter to config/parameters.yml with life time of records.

Embed migration machinery

1. Embed migration machinery for future Pinboard development and updating
2. Remove unused tables

Diagram Req/sec is broken when hostname contains non-word symbols

Add notifying about error statuses

• Add association between sites and email for notifying in config/parameters.yml throw regexp
• Add notifying to email about 5xx statuses in console aggregate command

String-like sorting fields at the main overview

At the main overview page sorting by values Requests / Sec and Pages with errors performs as values are string, number-like sorting is needed.

Show hostname for error pages in email notification

For error pages in email notification display hostname. It's useful if site runs on several servers.

Add security check throw sites

• Add parameter in config/parameters.yml for turn on/off authentication
• Add section in config/parameters.yml for user/pass settings (example http://silex.sensiolabs.org/doc/providers/security.html#securing-a-path-with-http-authentication)
• Add section in config/parameters.yml for access settings for users to sites with regexp
• Realize auth checking in index.php and access checking in controllers
• Add console command for generating config string by username and password. May be realize adding of generated config string to parameters.yml

Duplicate records in "CPU peak usage" tab

Example of duplication:

Whoops, looks like something went wrong.

2/2Twig_Error_Runtime: An exception has been thrown during the rendering of a template ("Parameter "serverName" for route "server" must match "[^/]++" ("ct=/yandsearch
Sec" given) to generate a corresponding URL.") in "index.html.twig" at line 24.

Add notification about increasing of 90th and 95th percentiles of request time higher than a border value

Add notification about increasing of 90th and 95th percentiles of request time higher than a border value.

Show request distribution by hostname on chart "Requests per sec"

Condition:

• Page Overview
• Mode "All hosts"
• Chart "Requests per sec"

Now this chart display all requests per sec to website. In mode "All hosts" add figures for requests/sec per hostname.

Error on empty server name

Hi!

I found that if I have empty server name on pinba stats (I know that it's wrong but I don't think it's a good reason to crush total pinboard because of it) pinboard will crush.

Twig_Error_Runtime: An exception has been thrown during the rendering of a template ("Parameter "serverName" for route "server" must match "[^/]++" ("" given) to generate a corresponding URL.") in "index.html.twig" at line 18.

Error during Agregation

/var/www/pinboard/src/Pinboard/Command/../../../console aggregate
[PDOException]
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

aggregate

Combine error pages in email notifications

Email notiication instead of:

example.com

* HTTP status 503: example.com/ajax/search/autocomplete.php
* HTTP status 503: example.com/ajax/search/autocomplete.php
* HTTP status 503: example.com/ajax/search/autocomplete.php
* HTTP status 503: example.com/ajax/search/autocomplete.php
* HTTP status 503: example.com/ajax/search/autocomplete.php
* HTTP status 503: example.com/index.php
* HTTP status 503: example.com/product.php

must form:

example.com

* HTTP status 503: example.com/ajax/search/autocomplete.php (5)
* HTTP status 503: example.com/index.php
* HTTP status 503: example.com/product.php

Wrong chart rendering in "Overview" tab for servers with several hosts

If server has several hosts charts in "Overview" tab contain duplicated values for each time position which quantity equals hosts quantity.

Add pagination in pages Request Time, Memory peak usage, Statuses

• Display with pagination all records for a last day in pages Request Time, Memory peak usage, Statuses.
• Add parameter to config/parameters.yml for records count on page in pagination.

Absolute urls in base.html.twig

When the pinboard installed not in the root, html resouces (js, css, etc.) won't load beacause absolute URL paths are used in the base.html.twig. Suggest to change them for relative paths.

Broken dependency in the current symphony/security dev-master version

Please, consider updating composer.json, the line of the version of symphony/security component as its dev-master version currently has a call to event->isMasterRequest() method, which doesn't exist in the current 2.3. Or possibly add the line to include the current dev-master of http-kernel component.
The problem is that piboard won't start throwing fatal error.

./console init failed during install

I did all steps before like on documentation but receive this error then:
./console init
Creating database tables...

[PDOException]
SQLSTATE[HY000] [2002] No such file or directory

init [--no-tables] [--no-indexes] [--no-crontab]

I checked my db settings from console (mysql -u pinboard -p -h 127.0.0.1) and they woking fine

First version of API

1. Make small REST JSON API for server overview:

• URL: /server/{serverName}/{host}/overview.json?apiKey={apiKey}
• Return json with 4 arrays equal data in charts on /server/{host} (request time, memory peak usage, req/sec, errors) for a last day
• {host} — optional parameter, equals all by default

2. Add config section for api keys. Key can contain limitation for access for server information by regexp.

Sometimes Live tab doesn't load new requests on ajax

Ajax loading broken when ID of requests in request table achieves value of 999999.

aggregate locking after smtp exception

Sometimes smtp is not accessable thats why aggregate command throws exception

  [Swift_IoException]                            
  Connection to smtp.ht-systems.ru:25 Timed Out  



aggregate


PHP Fatal error:  Uncaught exception 'Swift_IoException' with message 'Connection to smtp.ht-systems.ru:25 Timed Out' in /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/StreamBuffer.php:174
Stack trace:
#0 /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(438): Swift_Transport_StreamBuffer->readLine(14)
#1 /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(301): Swift_Transport_AbstractSmtpTransport->_getFullResponse(14)
#2 /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/EsmtpTransport.php(224): Swift_Transport_AbstractSmtpTransport->executeCommand('QUIT??', Array, Array)
#3 /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(237): Swift_Transport_EsmtpTransport->executeCommand('QUIT??', Array)
#4 /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/Ab in /home/webmaster/pinboard/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/StreamBuffer.php on line 174

Add handling of this situation.

Problem during installation

i`m receiving error when run : php composer.phar install
Loading composer repositories with package information
Installing dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

Problem 1
- Installation request for symfony/security dev-master -> satisfiable by symfony/security[dev-master].
- symfony/security dev-master requires symfony/http-foundation ~2.4 -> no matching package found.

Potential causes:

• A typo in the package name
• The package is not available in a stable-enough version according to your minimum-stability setting
  see https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion for more details.

Read http://getcomposer.org/doc/articles/troubleshooting.md for further common problems.

Please help me

Add play/pause in Live tab

• Add play/pause buttons in Live tab
• Fix layout of Live tab in Safari

Password encoding

Hi,
I want encode my password like describe in the documentation.

I have encode my paasmord to sha512 with 5000 iterations and re-encode this hash to base64.

I have launch all operation in PHP script.

echo base64_encode(crypt('myPasswordToEncode', '$6$rounds=5000$usesomesillystringforsalt$'));

The hash is not matching when I try to log in.

A suggestion ?

If agregate function was not run for some reason - pinboard show empty screen

Will be good to show some error to let user know where to look

Need to add tool to create password hash for config

Now i need to be a programmer to write it.

PDOException

Добрый день. После установки pinboard и запуска получаю такую штуку

/usr/local/src/pinboard/src/Pinboard/Command/../../../console -vvv aggregate

[PDOException]
SQLSTATE[42000]: Syntax error or access violation: 1065 Query was empty

Exception trace:
() at /usr/local/src/pinboard/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:716
PDO->query() at n/a:n/a
call_user_func_array() at /usr/local/src/pinboard/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:716
Doctrine\DBAL\Connection->query() at /usr/local/src/pinboard/src/Pinboard/Command/AggregateCommand.php:202
Pinboard\Command\AggregateCommand->execute() at /usr/local/src/pinboard/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:245
Symfony\Component\Console\Command\Command->run() at /usr/local/src/pinboard/vendor/symfony/console/Symfony/Component/Console/Application.php:880
Symfony\Component\Console\Application->doRunCommand() at /usr/local/src/pinboard/vendor/symfony/console/Symfony/Component/Console/Application.php:194
Symfony\Component\Console\Application->doRun() at /usr/local/src/pinboard/vendor/symfony/console/Symfony/Component/Console/Application.php:124
Symfony\Component\Console\Application->run() at /usr/local/src/pinboard/console:15

aggregate

Подскажите, пожалуйста, что с этим можно сделать.

show tables;
| info |
| ipm_info |
| ipm_mem_peak_usage_details |
| ipm_report_2_by_hostname_and_server |
| ipm_report_by_hostname |
| ipm_report_by_hostname_and_script |
| ipm_report_by_hostname_and_server |
| ipm_report_by_hostname_server_and_script |
| ipm_report_by_script_name |
| ipm_report_by_server_and_script |
| ipm_report_by_server_name |
| ipm_report_status |
| ipm_req_time_details |
| ipm_status_details |
| report_by_hostname |
| report_by_hostname_and_script |
| report_by_hostname_and_server |
| report_by_hostname_server_and_script |
| report_by_script_name |
| report_by_server_and_script |
| report_by_server_name |
| request |
| tag |
| timer |
| timertag |

PHP 5.4.17 (cli) (built: Jul 23 2013 13:57:49)
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies
with XCache v3.0.3, Copyright (c) 2005-2013, by mOo
with XCache Cacher v3.0.3, Copyright (c) 2005-2013, by mOo

На Xcache переключил - тут ошибок не всплывает.

Опции сборки php
./configure --enable-fpm --enable-libxml --with-mcrypt --enable-mbstring --with-gd --with-mysql-sock --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-mysql=mysqlnd --enable-sockets --with-iconv --with-gettext --with-zlib --with-freetype-dir=/usr --with-jpeg-dir=/usr --prefix=/usr/local/php-fpm --with-config-file-path=/usr/local/php-fpm/etc --with-config-file-scan-dir=/usr/local/php-fpm/etc/conf.d --with-fpm-user=www-data --with-fpm-group=www-data --disable-simplexml --disable-xmlreader --disable-xmlwriter --disable-tokenizer --without-sqlite3 --without-pdo-sqlite --with-curl

Спасибо.

./console aggregate return PDOException

Выдается такая ошибка:
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1.
Насколько я понял, это вызвано разницей полей в таблицах ipm и pinba. При этом я собирал pinba_engine из ветки master:

git remote show origin
* remote origin
  Fetch URL: https://github.com/tony2001/pinba_engine.git
  Push  URL: https://github.com/tony2001/pinba_engine.git
  HEAD branch: master
  Remote branches:
    devel                     tracked
    devel_ntag                tracked
    devel_percentiles         tracked
    devel_read_by_pos         tracked
    devel_schema              tracked
    devel_schema_request_tags tracked
    generator                 tracked
    gh-pages                  tracked
    master                    tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Array of hosts for user access in secure section of parameters.yml

Add ability to define array of hosts for user access settings. Example for testuser3:

secure:
    enable: false
    users:
        testuser1:
            password: my_password1
        testuser2:
            password: my_password2
            hosts: 'example\.com$'
        testuser3:
            password: my_password3
            hosts:
               - 'example\.com$'
               - 'some\-domain\.net$'

Is it possible to display the data for a month?

Pinboard displays statistic for a day. Can it show data for various period?

Add tag support

Requirements are in process.

Add option to ignore some hosts for notification

In config/parameters.yml add option notification/ignore:

notification:
    ignore:
        - (foo|otp)\.example\.com$
        - second-example\.ru$

In command aggregate don't send emails about 5xx statuses for this hosts.

404 Error Going to Root Intaro Pinboard Page

When I go to...

http://intaro.github.io/pinboard/

and click the logo in the upper left I get...

404 There isn't a GitHub Page here.

Thanks.

Sort domains in top menu alphabetically

Empty Dashboard

I received empty dashboard with message "Whoops, looks like something went wrong" after some time (service works from 30 minutes to 1 day).
How can i fix this?

enable debug: Whoops, looks like something went wrong.

2/2 Twig_Error_Runtime: An exception has been thrown during the rendering of a template ("Parameter "serverName" for route "server" must match "[^/]++" ("" given) to generate a corresponding URL.") in "_menu.html.twig" at line 10.
in /mnt/web/pinboard/vendor/twig/twig/lib/Twig/Template.php line 291 .....

1/2 InvalidParameterException: Parameter "serverName" for route "server" must match "[^/]++" ("" given) to generate a corresponding URL.
in /mnt/web/pinboard/vendor/symfony/routing/Symfony/Component/Routing/Generator/UrlGenerator.php line 167 .....

MySql non-standart port

I had an issue with the installation on mysql with non-standart port (3307 in my case). The solution is to add two lines of code:

1. /src/init.php file:

$app->register(new Silex\Provider\DoctrineServiceProvider(), array(
    'db.options' => array(
...
        'port'     => $app['params']['db']['port'],
...
    )
));

2. /config/parameters.yml:

db:
...
    port: 3307
...

Don't display 404 on pages Statuses, Request time, Memory peak usage when no data