I might be confused but it seemed like when I ran examples/run-tests that the procedures would only be compiled if there was a detected change. This no longer seems to be happening and the procedures are always being complied.

I'm thinking it might be nice to have an additional command line parameter for trace / untrace / report that would allow you specify a pattern to not include a function or functions in. For instance, if I want to trace the entire database that is great but if I want to execute a whole schema or function there isn't a way at the moment. The command line param might take a list of files as well as the pattern. Thoughts???

This is an old project, it's meant to be a simple and utilitarian: do one thing and do it okay enough. However the UI has always been ugly, even by 2010 standards.

If anyone out there wants to put a new coat of paint up on the reports, pick better fonts, maybe narrow down the 12 different font sizes to 2-3, make the little "percent complete" indicators look better, I would gladly accept the PR!

A function that looks like this:

create or replace function foo()
returns void
language plpgsql
as $$
DECLARE
   DECLARE mycur CURSOR FOR SELECT * from pg_database;
BEGIN
   return;
END;
$$;

Results in:

        ****
        Error compiling procedure public.foo
        Source: /home/user/piggly/cache/Dumper/511efb81e82810ddd8b9359d7c6a7758.plpgsql
        Exception Message:
        Expected one of [ \t\n\v\f\r], 'as', 'not', ':=', '=', '(', '[', [a-z\200-\377_0-9$%], '/*', '--', ';' at line 2, column 36 (byte 44) after declare

declare mycur cursor for select
****

On the other hand, changing it to say "... SELECT 1..." instead of "...SELECT *..." works (though of course it's not very useful to have a query that does effectively nothing).

seems like config.dry_run was removed. I commented out lines 27-29 in trace.rb:

        elsif config.dry_run?
          puts procedures.map{|p| p.signature }
          exit 0

and similar code from lines 20-22 in untrace.rb, and lines 29-31 in report.rb.
to get around this. without doing this, I get the following error:

$ piggly trace
C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/command/trace.rb:27:in `main': undefined method `dry_run?' for #<Piggly::Config:0x32fac30> (NoMethodError)
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/command/base.rb:15:in `main'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/bin/piggly:8:in `<top (required)>'
        from C:/Ruby22/bin/piggly:23:in `load'
        from C:/Ruby22/bin/piggly:23:in `<main>'

IF (CASE
WHEN x IS NULL THEN FALSE
WHEN x = 'abc' THEN FALSE
ELSE TRUE
END)
THEN
RAISE EXCEPTION 'error message';
END IF;

The parser seem to have issues with FALSE. It appears that the parser is assuming there needs to be a assignment that occurs after the FALSE.

A function which declares a bigint is throwing an error when executing the trace command.
Env: Docker postgres:12.1-alpine

Extract from problematic function
CREATE OR REPLACE FUNCTION test.piggly ( param1 INT default 0 ) RETURNS TABLE ( total_rows bigint ) ...

And the output showing the error...
Traceback (most recent call last):
10: from /usr/bin/piggly:23:in <main>
9: from /usr/bin/piggly:23:in 'load'
8: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/bin/piggly:8:in '<top (required)>'
7: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/command/base.rb:15:in 'main'
6: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/command/trace.rb:33:in 'main'
4: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/installer.rb:14:in 'install'all'
2: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/installer.rb:16:in 'block in install'
1: from /usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/installer.rb:50:in 'trace'
/usr/lib/ruby/gems/2.6.0/gems/piggly-2.3.1/lib/piggly/installer.rb:50:in 'exec': ERROR: syntax error at or near ""int8"" (PG::SyntaxError)
LINE 1: ...param1" "int4" default 0, t "total_rows" "int8")

in the case of a structure like this:

FOR i in 1..4 LOOP
/*magic happens*/
END LOOP;

piggly reports 'loop always iterates more than once' . this hard coded loop was put there by design, since it is for calculating a thing exactly 4 times. There is no conditional here, so I fail to see the lack of coverage.

We only really need PGConn, but if ActiveRecord is available and part of the test suite, go ahead and use it's active connection.

The parser incorrectly requires a space before THEN. But PostgreSQL accepts IF(cond)THEN, because there's a parenthesis to separate the keyword.

The grammar node that needs to change is expressiontUntilThen. There are probably similar bugs in the expressiontUntil... nodes.

Piggly calls fork in several places, but this call fails on Windows. The win32 gem doesn't emulate the call sufficiently, so the best option is to avoid forking on Windows.

Parser might be simplified by changing each keyword rule to have a negative look-ahead assertion that the next character is not alpha-numeric. This could improve parsing rules for statements like RAISE; and RAISE WARNING; and RAISE INFO 'message'; which have complicated whitespace rules (to ensure tokens are separated)

Hi,

With the most recent release a technique that I used to use to filter out only certain procedures for running piggly seems to have gotten broken. I used to do something like this:

piggly trace -s myschema.someproc -s myschema.some_other_proc

and it used to process only those procs. now it seems it only processes the first one. any ideas?

Since I no longer write PL/pgSQL during the course of my work, it takes me some time to setup a test environment when making changes to this project. Now that Docker is widely used, it makes sense to use it for this purpose. Now that GitHub can execute Docker containers, we can write automated integration tests that talk to an actual PostgreSQL server.

If someone is interested in contributing to this, I'll happily merge a PR. Otherwise, this will be worked on in my spare time.

I made a script to run piggly:

#!/usr/bin/env bash

piggly trace -s '/cas\./'

pg_prove --ext .sql -h postgres -q -S client_min_messages='warning' 2> coverage.txt

echo "#### COVERAGE"
cat coverage.txt
echo "####"

piggly report -s '/cas\./' -f coverage.txt

piggly untrace -s '/cas\./'

But it gives me the following error:

compiling 1 procedures
ProcessQueue running concurrently
Compiling cas.update_timestamp
tracing 1 procedures
Compiling cas.update_timestamp
t/user.sql .. ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.02 cusr  0.04 csys =  0.10 CPU)
Result: PASS
#### COVERAGE
psql:t/user.sql:10: WARNING:  PIGGLY 9602c614b41623a5
psql:t/user.sql:10: WARNING:  PIGGLY 4a573c5ae5aa22f9
####
Compiling cas.update_timestamp
clearing previous coverage
/var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/profile.rb:25:in `[]': No tag with id 9602c614b41623a5 (RuntimeError)
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/profile.rb:35:in `ping'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/profile.rb:100:in `block in notice_processor'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/command/report.rb:68:in `block in read_profile'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/command/report.rb:68:in `each'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/command/report.rb:68:in `read_profile'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/command/report.rb:37:in `main'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/command/base.rb:15:in `main'
        from /var/lib/gems/2.3.0/gems/piggly-2.2.4/bin/piggly:8:in `<top (required)>'
        from /usr/local/bin/piggly:22:in `load'
        from /usr/local/bin/piggly:22:in `<main>'
restoring 1 procedures

The generated Trace code is:

BEGIN perform public.piggly_branch($PIGGLY$2a293b29b5a03a84$PIGGLY$);NEW.updated_at = now(); perform public.piggly_branch($PIGGLY$dba45fbf3b18e0d4$PIGGLY$);RETURN NEW; END;

So it seems like the IDs indeed don't match! The piggly trace output looks suspicious because it's saying Compiling cas.update_timestamp twice, maybe that's the reason? But why?

CREATE OR REPLACE FUNCTION public.my_func()
RETURNS VOID AS
$BODY$
DECLARE
schema TEXT = 'pg_catalog';
r RECORD;
BEGIN
FOR r IN EXECUTE 'SELECT * FROM ' || quote_ident(schema) || 'pg_user;'
LOOP
END LOOP;
END;
$BODY$
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;

Doesn't seem to like this FOR-IN-EXECUTE statement.

Missing require 'set' in the else clause below.

in /piggly/lib/piggly/command/base.rb:69

@return [Enumerable]

def filter(config, index)
if config.filters.empty?
index.procedures
else
require 'set' # Add this require statement
config.filters.inject(Set.new){|s, f| s | index.procedures.select(&f) }
end
end

I have a function:

CREATE FUNCTION cas.user_full_name(u cas.user)
			RETURNS VARCHAR AS $$ BEGIN
				RETURN u.first_name || ' ' || u.last_name;
			END; $$ LANGUAGE plpgsql IMMUTABLE;

And piggly trace -s '/cas\./' outputs:

postgres@8df3dca48d04:~$ piggly trace -s '/cas\./'
compiling 2 procedures
ProcessQueue running concurrently
tracing 2 procedures
/var/lib/gems/2.3.0/gems/piggly-2.2.4/lib/piggly/installer.rb:50:in `exec': ERROR:  zero-length delimited identifier at or near """" (PG::SyntaxError)
LINE 1: ...lace function "cas"."user_full_name" (in "u" "cas".""user"")
                                                              ^

Error installing traced procedure cas.user_full_name from /home/postgres/piggly/cache/Dumper/53c863ad9a8bf9d1ebf0372d02c797f2.plpgsql

Probably because user is a keyword and an identifier?

Hi, I'm very interested in this library, but I would like to integrate it into existing code coverage tools. Is there any way to export the code coverage to an ubiquitous intermediate format, such as gcov, that can be read in by services such as Coveralls?

It seems that I am covering new ground trying to unit-test some PL/pgSQL code that will be used in production.

Thanks in advance.

let's assume for a moment that the following procedure exists:

CREATE OR REPLACE FUNCTION cat.func_potato_fish(p_dog_cat_fish_id integer)
  RETURNS text AS
$BODY$
DECLARE
  v_dog_id INTEGER;
  v_state   TEXT;
  v_message TEXT;
BEGIN
  IF p_dog_cat_fish_id IS NULL THEN
    RAISE EXCEPTION 'Can not recalculate potato cat for fish if fish id passed is NULL';
  END IF;

  SELECT fr.dog_id
  INTO v_dog_id
  FROM cat.dog_cat fr
  INNER JOIN cat.dog_cat_fish fra
    ON fr.dog_cat_id = fra.dog_cat_id
  WHERE fra.dog_cat_fish_id = p_dog_cat_fish_id;

  PERFORM cat.func_potato_exct_nrmld_score(p_dog_cat_fish_id);
  PERFORM cat.func_potato_exct_dog_likelihood(v_dog_id);
  PERFORM cat.func_cmpst_exct_dog_score(v_dog_id);
  RETURN '';

  EXCEPTION
  WHEN OTHERS
  THEN
    GET STACKED DIAGNOSTICS v_message = MESSAGE_TEXT,
    v_state = RETURNED_SQLSTATE;
    RAISE WARNING 'Exception - % -ERROR- %', v_state, v_message;
    RETURN v_message;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE
  COST 100;

let's further assume that a suite of pg_tap tests have been built to test this procedure, These tests overwrite the definitions of the dependent procedures, cat.func_potato_exct_nrmld_score, cat.func_potato_exct_dog_likelihood, cat.func_cmpst_exct_dog_score:

CREATE TEMPORARY TABLE calls_to__func_potato_exct_nrmld_score
(
  p_dog_cat_fish_id INTEGER
);

CREATE OR REPLACE FUNCTION cat.func_potato_exct_nrmld_score(p_dog_cat_fish_id INTEGER DEFAULT NULL)
  RETURNS INTEGER
LANGUAGE plpgsql
AS $$
BEGIN
  INSERT INTO calls_to__func_potato_exct_nrmld_score
  VALUES (p_dog_cat_fish_id);
  RETURN 0;
END;
$$;

This is done in order to isolate the procedure from its dependencies, and to capture call to the procedures for the purpose of validating that the calls were made with the appropriate inputs. the overwritten procedures capture the called parameter values in a table, so that they can be asserted against in later parts of the test. When tests are written this way, piggly apparently is unable to accurately capture the fact that the call:

PERFORM cat.func_potato_exct_nrmld_score(p_dog_cat_fish_id);

was actually made. Not sure how easy this will be to fix.

With large test suites, it would be beneficial to run piggly using rcov to avoid having to run the suite once for PL/pgSQL coverage and another time for the Ruby coverage.

The error appears to be related to this:

raise '{some string}'
using errorcode = {some number}

Here is the unparsable code:

CREATE OR REPLACE FUNCTION bar.foo_start_sp(
    OUT po_out integer,
    OUT po_status text)
  RETURNS record AS
$BODY$
DECLARE
  v_cnt     INTEGER;
  v_state   TEXT;
  v_message TEXT;
BEGIN
  RAISE NOTICE 'Started foo_START_SP';
  
  LOCK TABLE foo_lock IN ACCESS EXCLUSIVE MODE NOWAIT;

  
  SELECT count(*)
  INTO v_cnt
  FROM foo_lock l;
  IF v_cnt > 0
  THEN
    RAISE 'Concurrent execution'
    USING ERRCODE = 23054;
  END IF;

  INSERT INTO bar.foo_lock (crtn_ts, updt_ts, stage_nm)
  VALUES (current_timestamp, current_timestamp, 'boo boo processing started');

  
  UPDATE bar.event_trggr
  SET prcsd_st = 0
  WHERE prcsd_st IS NULL;

  po_out := 1;
  RAISE NOTICE 'Completed foo_START_SP';
  EXCEPTION
  WHEN SQLSTATE '23054'
    THEN
      po_status := 'Exception: Concurrent processing going';
      po_out := -1;
      RAISE WARNING 'Exception: Concurrent processing going';
  WHEN OTHERS
    THEN
      GET STACKED DIAGNOSTICS v_message = MESSAGE_TEXT,
      v_state = RETURNED_SQLSTATE;
      RAISE WARNING 'Exception - % -ERROR- %', v_state, v_message;
      po_out := -2;
      po_status := v_message;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE
  COST 100;

Here's the error:

Compiling riskstg.foo_start_sp
            ****
            Error compiling procedure bar.foo_start_sp
            Source: C:/Users/K26962/RAW/comp-workspace/src/test/resources/sql/pigg    ly/piggly/cache/Dumper/8f65cd90164efdc6ff80a6a102acac8a.plpgsql
            Exception Message:
            Expected one of /*, --, exception, warning, notice, info, log, debug a    t line 16, column 11 (byte 279) after declare
  v_cnt     integer;
  v_state   text;
  v_message text;
begin
  raise notice 'started foo_start_sp';

  lock table foo_lock in access exclusive mode nowait;


  select count(*)
  into v_cnt
  from foo_lock l;
  if v_cnt > 0
  then
    raise
            ****

Ubuntu 12.04.5 (x86_64)
Ruby 2.2.4p230 (x86_64)

Fresh git checkout, bundle install ran OK,
bundle exec rake spec fails with 3 test failures:

Failures:

  1) Piggly::Dumper::Index when cache file exists when the cache index file has two entries has two procedures
     Failure/Error: @first  = Dumper::ReifiedProcedure.from_hash \
     NoMethodError:
       undefined method `split' for nil:NilClass
     # ./lib/piggly/dumper/reified_procedure.rb:66:in `defaults'
     # ./lib/piggly/dumper/reified_procedure.rb:137:in `from_hash'
     # ./spec/examples/dumper/index_spec.rb:35:in `block (4 levels) in <module:Piggly>'

  2) Piggly::Dumper::Index when cache file exists when the cache index file has two entries is indexed by identifier
     Failure/Error: @first  = Dumper::ReifiedProcedure.from_hash \
     NoMethodError:
       undefined method `split' for nil:NilClass
     # ./lib/piggly/dumper/reified_procedure.rb:66:in `defaults'
     # ./lib/piggly/dumper/reified_procedure.rb:137:in `from_hash'
     # ./spec/examples/dumper/index_spec.rb:35:in `block (4 levels) in <module:Piggly>'

  3) Piggly::Dumper::Index when cache file exists when the cache index file has two entries reads each procedure's source_path
     Failure/Error: @first  = Dumper::ReifiedProcedure.from_hash \
     NoMethodError:
       undefined method `split' for nil:NilClass
     # ./lib/piggly/dumper/reified_procedure.rb:66:in `defaults'
     # ./lib/piggly/dumper/reified_procedure.rb:137:in `from_hash'
     # ./spec/examples/dumper/index_spec.rb:35:in `block (4 levels) in <module:Piggly>'

Finished in 1.21 seconds
291 examples, 3 failures, 60 pending

Failed examples:

rspec ./spec/examples/dumper/index_spec.rb:55 # Piggly::Dumper::Index when cache file exists when the cache index file has two entries has two procedures
rspec ./spec/examples/dumper/index_spec.rb:59 # Piggly::Dumper::Index when cache file exists when the cache index file has two entries is indexed by identifier
rspec ./spec/examples/dumper/index_spec.rb:64 # Piggly::Dumper::Index when cache file exists when the cache index file has two entries reads each procedure's source_path

Example:
CREATE OR REPLACE FUNCTION my_schema.my_proc
(
) RETURNS SETOF my_schema.my_table AS $$
BEGIN
RETURN QUERY
SELECT *
FROM my_schema.my_table;
END;
$$ LANGUAGE plpgsql VOLATILE STRICT SECURITY INVOKER;

Appears the parser is not schema aware in this case. The return type generated by the trace drops the schema and postgres is unable to find this type in the public schema.

x TEXT :=E'\002schema\003';

Appears there needs to be a space after the := for the parser to handle this, however Postgres does not require it.

here's the proc that caused the issue (with names changed to protect the innocent) the proc takes in an array of a record type, the unnests it, and joins it to a table, and finally inserts the result into another table, captures the count and reports it in a message

CREATE OR REPLACE FUNCTION foo.bar_store(p_baz foo.type_baz[])
  RETURNS integer AS
$BODY$
DECLARE
  v_cnt INTEGER;
BEGIN
  INSERT INTO foo.foo_table(bar, barbar_fl, foofoo_fl, bazbaz_ts)
    SELECT
      a.zoorzoor_id,
      b.zap_fl,
      b.zip_fl,
      current_timestamp
    FROM unnest(p_baz) a
      INNER JOIN foo.zimzam b
        ON a.zoor_id = b.zoor_id
  ON CONFLICT (bar)
    DO UPDATE SET barbar_fl= excluded.barbar_fl, foofoo_fl= excluded.foofoo_fl, last_updt_ts=current_timestamp;
  GET DIAGNOSTICS v_cnt = ROW_COUNT;
  RAISE NOTICE '[risk.bar_store] Updated % bar stuffs', v_cnt;
  RETURN v_cnt;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE
  COST 100;

this results in the following error:

            ****
            Error compiling procedure foo.bar_store
            Source: C:/Users/K26962/RAW/comp-workspace/src/test/resources/sql/piggly/cache/Dumper/4cde0c9accf97c38a7592321d00d388e.plpgsql
            Exception Message:
            Expected one of /*, --, stacked at line 18, column 7 (byte 582) after declare

/piggly/piggly/lib/piggly/installer.rb:50:in `exec': ERROR: type "pg_catalog.int" does not exist (PGError)

Error installing traced procedure my_schema.my_proc from /piggly/piggly/3530/piggly/cache/Dumper/6bd1ffffba0c2638792631166a10e1df.plpgsql
from /piggly/piggly/lib/piggly/installer.rb:50:in trace' from /piggly/piggly/lib/piggly/installer.rb:16:ininstall'
from /piggly/piggly/lib/piggly/installer.rb:14:in each' from /piggly/piggly/lib/piggly/installer.rb:14:ininstall'
from /piggly/piggly/lib/piggly/command/trace.rb:55:in install' from /piggly/piggly/lib/piggly/command/trace.rb:30:inmain'
from /piggly/piggly/lib/piggly/command/base.rb:15:in `main'
from /piggly/piggly/bin/piggly:8
restoring 576 procedures

snippet from index.yml

name: !ruby/object:Piggly::Dumper::QualifiedName
names:
- my_schema
- my_proc
oid: "48986"
secdef: false
setof: false
strict: true
type: !ruby/object:Piggly::Dumper::QualifiedName
names:
- pg_catalog
- int
volatility: volatile

• !ruby/object:Piggly::Dumper::SkeletonProcedure
  arg_modes: []

I think postgres is complaining that it doesn't like the schema qualified datatype.

piggly/example/run-specs and piggly/example/run-tests has the following comment.

echo "OK, view $EXAMPLE/piggly/reports/index.html"

reports should not be plural. change to

echo "OK, view $EXAMPLE/piggly/report/index.html"

Got the error below.
Initially thought I resolved by placing a space between "--" and "transfer" in the comment "--transfer last to past" but its just that piggly compiles the functions in different orders on different executions, and the error remains.
Might be due to "%rowtype" datatype. See below

C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/parser.rb:23:in 'block in parse': Expected one of /*, --, :=, = at line 6, column 8 (byte 144) after declare new_rev     int; (Piggly::Parser::Failure)
        past_record past_statement_value%rowtype;
begin
  --transfer last to past
  with
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/util/thunk.rb:23:in 'call'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/util/thunk.rb:23:in 'force!'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/compiler/trace_compiler.rb:27:in 'compile'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/command/trace.rb:49:in 'block (2 levels) in trace'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/util/process_queue.rb:59:in 'call'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/util/process_queue.rb:59:in 'serially'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/util/process_queue.rb:49:in 'execute'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/command/trace.rb:53:in 'trace'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/command/trace.rb:32:in 'main'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/lib/piggly/command/base.rb:15:in 'main'
        from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/piggly-2.0.0/bin/piggly:8:in ''
        from C:/Ruby22-x64/bin/piggly:23:in 'load'
        from C:/Ruby22-x64/bin/piggly:23:in ''

this doesn't appear to have ill effects (yet) but figured Id report it;

$ gem install piggly
unable to convert "\xA3" from ASCII-8BIT to UTF-8 for lib/piggly/reporter/resour                                                   ces/sortable.js, skipping
Successfully installed piggly-2.2.2
Parsing documentation for piggly-2.2.2
Installing ri documentation for piggly-2.2.2
Done installing documentation for piggly after 2 seconds
1 gem installed

piggly seems to have problems with procedures that return table types. A procedure with this signature:

CREATE OR REPLACE FUNCTION risk.func_agrt_firm_params(IN p_firm_id integer DEFAULT NULL::integer)
  RETURNS TABLE(firm_id integer, shared_fl boolean, district_id integer, finop_bus_act_fl boolean, sp_bus_act_fl boolean, underwriting_fl boolean, net_capital numeric) AS
$BODY$
BEGIN
-- there be dragons here
END;
$BODY$
  LANGUAGE plpgsql VOLATILE

produces a cache index like this:

- !ruby/object:Piggly::Dumper::SkeletonProcedure
  arg_defaults:
  - NULL::integer
  arg_types:
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: integer
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: integer
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: integer
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: numeric
    array: ''
  arg_names:
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_firm_id
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: firm_id
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: shared_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: district_id
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: finop_bus_act_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: sp_bus_act_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: underwriting_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: net_capital
  arg_modes:
  - in
  - t
  - t
  - t
  - t
  - t
  - t
  - t
  setof: true
  volatility: volatile
  type: !ruby/object:Piggly::Dumper::QualifiedType
    schema: pg_catalog
    name: record
    array: ''
  secdef: false
  strict: false
  name: !ruby/object:Piggly::Dumper::QualifiedName
    schema: risk
    name: func_agrt_firm_params
  oid: '808889'
  identifier: fe2d4b76b3d641208f28a6c0cea52d80
- !ruby/object:Piggly::Dumper::SkeletonProcedure
  arg_defaults:
  - 
  - 
  - 
  - 
  - 
  - 
  - 
  arg_types:
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: integer
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: integer
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: boolean
    array: ''
  - !ruby/object:Piggly::Dumper::QualifiedType
    schema: 
    name: numeric
    array: ''
  arg_names:
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_firm_id
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_shared_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_district_id
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_finop_bus_act_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_sp_bus_act_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_underwriting_fl
  - !ruby/object:Piggly::Dumper::QualifiedName
    schema: 
    name: p_net_capital
  arg_modes:
  - in
  - in
  - in
  - in
  - in
  - in
  - in
  setof: false
  volatility: volatile
  type: !ruby/object:Piggly::Dumper::QualifiedType
    schema: risk
    name: type_agrt_firm_oprnl_aplbl
    array: ''
  secdef: false
  strict: false
  name: !ruby/object:Piggly::Dumper::QualifiedName
    schema: risk
    name: func_agrt_calc_firm_aplbl
  oid: '808890'
  identifier: 101ba1ab688d27f31a2abcaf24e9028b

piggly trace produces this error:

C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/installer.rb:50:in `exec': ERROR:  syntax error at or near ""int4"" (PG::SyntaxError)
LINE 1: ...irm_id" "int4" DEFAULT NULL::integer, t "firm_id" "int4", t ...
                                                             ^

Error installing traced procedure risk.func_agrt_firm_params from C:/Users/K26962/RAW/comp-workspace/src/test/resources/sql/piggly/cache/Dumper/fe2d4b76b3d641208f28a6c0cea52d80.plpgsql
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/installer.rb:50:in `trace'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/installer.rb:16:in `block in install'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/installer.rb:14:in `each'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/installer.rb:14:in `install'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/command/trace.rb:58:in `install'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/command/trace.rb:33:in `main'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/lib/piggly/command/base.rb:15:in `main'
        from C:/Ruby22/lib/ruby/gems/2.2.0/gems/piggly-2.2.1/bin/piggly:8:in `<top (required)>'
        from C:/Ruby22/bin/piggly:23:in `load'
        from C:/Ruby22/bin/piggly:23:in `<main>'

It seems like the rewriting is putting one of the table columns in as another input parameter instead of creating a table parameter.

The README file and probably kputnam.github.com/piggly have outdated instructions.

Hi, I think (not sure) piggly is working fine. But maybe our SQL is too complex. You can find the sql here: https://github.com/wolbodo/pms/blob/master/database/db-logic.sql

dexter@zaphod:~/projects/wolbodo/pms$ piggly trace -d piggly.yml 
/var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:72:in `defaults': Couldn't parse default arguments (RuntimeError)
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:137:in `from_hash'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:80:in `block in all'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:80:in `each'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:80:in `map'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/dumper/reified_procedure.rb:80:in `all'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/command/trace.rb:39:in `dump'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/command/trace.rb:17:in `main'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/lib/piggly/command/base.rb:15:in `main'
        from /var/lib/gems/2.3.0/gems/piggly-2.0.0/bin/piggly:8:in `<top (required)>'
        from /usr/local/bin/piggly:23:in `load'
        from /usr/local/bin/piggly:23:in `<main>'

hi, running "piggly trace" gives me the below error message repeatedly.
is it because i am using $a$ in some postgresql functions?

ProcessQueue running concurrently
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/compiler/cache_dir.rb:20: invalid multibyte escape: /[\000-\010\016-\037\177-\300]/
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/compiler/trace_compiler.rb:22:in compile' /mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/command/trace.rb:49:inblock (2 levels) in trace'
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/util/process_queue.rb:79:in call' /mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/util/process_queue.rb:79:inblock in concurrently'
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/util/process_queue.rb:77:in fork' /mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/util/process_queue.rb:77:inconcurrently'
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/util/process_queue.rb:47:in execute' /mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/command/trace.rb:53:intrace'
/mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/command/trace.rb:32:in main' /mnt/hgfs/g/kputnam-piggly-7560eb8/lib/piggly/command/base.rb:15:inmain'
/mnt/hgfs/g/kputnam-piggly-7560eb8/bin/piggly:8:in `

TRUNCATE / NOTIFY keywords do not appear to be supported in the grammar parser.

lib/piggly/dumper/prodcedure.rb around line 50 needs to be modified to work with strictness and security keywords.

before:
# Returns source SQL function definition statement
# @return [String]
def definition(body)
[%[create or replace function "#{@namespace}"."#{@name}" (#{arguments})],
%[ #{strictness} #{security} returns #{type} as $PIGGLY$],
body,
%[$PIGGLY$ language plpgsql #{@volatility}]].join("\n")
end

after:
# Returns source SQL function definition statement
# @return [String]
def definition(body)
[%[create or replace function "#{@namespace}"."#{@name}" (#{arguments})],
%[ returns #{type} as $PIGGLY$],
body,
%[$PIGGLY$ language plpgsql #{strictness} #{security} #{@volatility}]].join("\n")
end

Hi,

I am trying to run piggly trace on my existing postgresql database.

I get the following error:

`defaults': Couldn't parse default arguments

/var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:84:in defaults': Couldn't parse default arguments (RuntimeError) from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:150:in from_hash'
from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:92:in block in all' from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:92:in each'
from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:92:in map' from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/dumper/reified_procedure.rb:92:in all'
from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/command/trace.rb:39:in dump' from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/command/trace.rb:17:in main'
from /var/lib/gems/3.0.0/gems/piggly-2.3.1/lib/piggly/command/base.rb:15:in main' from /var/lib/gems/3.0.0/gems/piggly-2.3.1/bin/piggly:8:in <top (required)>'
from /usr/local/bin/piggly:25:in load' from /usr/local/bin/piggly:25:in

I am wondering if there is anything obvious that I am / have done wrong here. I did install using gem install piggly.

Thank you

The procedures that show up on the report web page are not schema qualified.

FOR r IN EXECUTE 'SELECT * FROM pg_user;'
LOOP
END LOOP;

Appears the FOR-IN-EXECUTE statement is not supported by the grammar parser.

The current method of replacing an existing function with an instrumented uses CREATE OR REPLACE FUNCTION. Since we don't save all the metadata like COST or other attributes, those can be removed on the instrumented version, and won't be restored with piggly untrace. This approach also requires recording parameter names, defaults, and other parts of the function signature so that we can properly redefine the function.

It seems like there's a simpler way to do this, which avoids those problems.

UPDATE pg_proc SET prosrc = '...' WHERE oid = 'snippets(int, int)'::regprocedure;

This preserves any parameter names, cost annotations, and other attributes and might also avoid other problems. Some work is needed to ensure this won't cause other issues. For example, if we only store the OID and the source, what do we do if the UPDATE statement doesn't update any rows (presumably the user replaced the proc since we last looked)?

Stored procedure written using CREATE PROCEDURE are not working. Why trying to trace it gives error - "DETAIL: "total_score" is a procedure."

1. Is there anyone facing the same error?
2. Do we have support for this in piggly or does it only supports CREATE FUNCTION?

Example:
CREATE OR REPLACE FUNCTION public.my_proc(OUT "timestamp" bigint)
RETURNS SETOF record AS
$BODY$

This function contains a param name that just so happens to also be a type. If the name is not quoted the traced procedure will not be able to be uploaded to the database. This issue would most likely occur if there is a space in the name of parameter as well.

Luke noted it can be difficult to tell which code is being described by the bottom-right TOC links, and he suggested using JavaScript to highlight the region of code. Great idea, Luke!