GithubHelp home page GithubHelp logo

istore's Issues

istore precludes parallel queries

There are some aggregates that are still not parallel-safe (namely, isagg). They rely on the AVL-tree as an internal state, which makes it slightly more challenging to implement parallel behavior, however, it should pay off if the aggregate Is the last step to the long table scan that can potentially be parallelized.

aggregate with floor

We should have an aggregate with a floor that one can sum istores together with a floor value. The idea is to filter out, for example, invalid negative numbers.

Proposed invocation:

sum_floor(istorecol, int)

Example:

GIven the following istores in a column called myval:

 {1 => 3, 2 => -2, 3 => 4 }
 {1 => -3, 2 => 5, 3 => 15}
 {1 => 2, 2 => -3, 3 => -1, 4 => -3}

Then:

 sum_floor(myval, 0)

would generate

 {1 => 5, 2 => 5, 3 => 19}

potential int overflow in clamp_below

It looks like clamp_below runs into an int overflow:

 SELECT clamp_below('"-3"=>"42", "0"=>"2235078024"'::bigistore,0);
    clamp_below
--------------------
 "0"=>"-2059889230"
(1 row)

istore/src/bigistore.c

Lines 1248 to 1270 in 08ee021

bigistore_clamp_pass(BigIStore *is, int32 clamp_key, int delta_dir)
{
BigIStore * result_is;
BigIStorePair *pairs;
BigIStorePairs creator;
int32 clamp_sum = 0;
int index = 0, count = 0, delta_buflen = 0;
/* short circuit out of the funciton if there is nothing to clamp */
if (delta_dir > 0 && FIRST_PAIR(is, BigIStorePair)->key >= clamp_key)
return is;
if (delta_dir < 0 && LAST_PAIR(is, BigIStorePair)->key <= clamp_key)
return is;
pairs = FIRST_PAIR(is, BigIStorePair);
index = delta_dir > 0 ? 0 : is->len - 1;
while (((delta_dir > 0) && (index < is->len && pairs[index].key <= clamp_key)) ||
((delta_dir < 0) && (index >= 0 && pairs[index].key >= clamp_key)))
{
INTPL(pairs[index].val, clamp_sum, clamp_sum);
delta_buflen += bigis_pair_buf_len(pairs + index);
index += delta_dir, count++;
}

I guess clamp_sum should be an int64 actually

postgres 9.2

Hello!

Are there any plans to support version 9.2?

Thanks,
Ibrahim

cistore

our new data type cistore (combined integer store) should save the key and values as integer

SELECT 'de::phone::ios=>25'::cistore;
         cistore
------------------------
 "de::phone::ios"=>25

basically the first byte represents country, the second the device_type and the third byte should be interpreted as os_name

We also want a fourth byte which is a plain integer

SELECT 'de::phone::ios=>25'::cistore;
         cistore
------------------------
 "de::phone::ios::0"=>25

this one is later some cohort place or punchcard hour, whatever we need.

remove compiler warnings

i get

src/device_type.c:45:1: warning: no previous prototype for function 'device_type_recv' [-Wmissing-prototypes]
device_type_recv(PG_FUNCTION_ARGS)
^
src/device_type.c:55:1: warning: no previous prototype for function 'device_type_send' [-Wmissing-prototypes]
device_type_send(PG_FUNCTION_ARGS)
^
src/device_type.c:148:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
src/device_type.c:168:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
4 warnings generated.
src/istore.c:19:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
src/istore.c:23:1: warning: no previous prototype for function 'istore_sum_up' [-Wmissing-prototypes]
istore_sum_up(PG_FUNCTION_ARGS)
^
src/istore.c:66:1: warning: no previous prototype for function 'is_exist' [-Wmissing-prototypes]
is_exist(PG_FUNCTION_ARGS)
^
src/istore.c:73:5: warning: variable 'key' is used uninitialized whenever switch default is taken
      [-Wsometimes-uninitialized]
    GET_KEYARG_BY_TYPE(in, key);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
src/istore.h:255:13: note: expanded from macro 'GET_KEYARG_BY_TYPE'
            default:                                          \
            ^~~~~~~
src/istore.c:74:21: note: uninitialized use occurs here
    if (is_find(in, key))
                    ^~~
src/istore.c:69:16: note: initialize the variable 'key' to silence this warning
    int     key;
               ^
                = 0
src/istore.c:83:1: warning: no previous prototype for function 'is_fetchval' [-Wmissing-prototypes]
is_fetchval(PG_FUNCTION_ARGS)
^
src/istore.c:90:5: warning: variable 'key' is used uninitialized whenever switch default is taken
      [-Wsometimes-uninitialized]
    GET_KEYARG_BY_TYPE(in, key);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~
src/istore.h:255:13: note: expanded from macro 'GET_KEYARG_BY_TYPE'
            default:                                          \
            ^~~~~~~
src/istore.c:91:29: note: uninitialized use occurs here
    if ((pair = is_find(in, key)) == NULL )
                            ^~~
src/istore.c:86:16: note: initialize the variable 'key' to silence this warning
    int     key;
               ^
                = 0
src/istore.c:99:1: warning: no previous prototype for function 'is_add' [-Wmissing-prototypes]
is_add(PG_FUNCTION_ARGS)
^
src/istore.c:167:1: warning: no previous prototype for function 'is_add_integer' [-Wmissing-prototypes]
is_add_integer(PG_FUNCTION_ARGS)
^
src/istore.c:196:1: warning: no previous prototype for function 'is_subtract' [-Wmissing-prototypes]
is_subtract(PG_FUNCTION_ARGS)
^
src/istore.c:263:1: warning: no previous prototype for function 'is_subtract_integer' [-Wmissing-prototypes]
is_subtract_integer(PG_FUNCTION_ARGS)
^
src/istore.c:292:1: warning: no previous prototype for function 'is_multiply' [-Wmissing-prototypes]
is_multiply(PG_FUNCTION_ARGS)
^
src/istore.c:361:1: warning: no previous prototype for function 'is_multiply_integer' [-Wmissing-prototypes]
is_multiply_integer(PG_FUNCTION_ARGS)
^
src/istore.c:545:1: warning: no previous prototype for function 'istore_from_array' [-Wmissing-prototypes]
istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:557:1: warning: no previous prototype for function 'device_istore_from_array' [-Wmissing-prototypes]
device_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:575:1: warning: no previous prototype for function 'country_istore_from_array' [-Wmissing-prototypes]
country_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:593:1: warning: no previous prototype for function 'os_name_istore_from_array' [-Wmissing-prototypes]
os_name_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:654:1: warning: no previous prototype for function 'istore_agg' [-Wmissing-prototypes]
istore_agg(PG_FUNCTION_ARGS)
^
src/istore.c:698:1: warning: no previous prototype for function 'istore_agg_finalfn' [-Wmissing-prototypes]
istore_agg_finalfn(PG_FUNCTION_ARGS)
^
src/istore.c:719:1: warning: no previous prototype for function 'istore_array_add' [-Wmissing-prototypes]
istore_array_add(PG_FUNCTION_ARGS)
^
19 warnings generated.
src/istore_io.c:244:1: warning: no previous prototype for function 'type_istore_to_istore' [-Wmissing-prototypes]
type_istore_to_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:269:1: warning: no previous prototype for function 'istore_to_device_istore' [-Wmissing-prototypes]
istore_to_device_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:292:1: warning: no previous prototype for function 'istore_to_country_istore' [-Wmissing-prototypes]
istore_to_country_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:315:1: warning: no previous prototype for function 'istore_to_os_name_istore' [-Wmissing-prototypes]
istore_to_os_name_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:338:1: warning: no previous prototype for function 'istore_out' [-Wmissing-prototypes]
istore_out(PG_FUNCTION_ARGS)
^
src/istore_io.c:346:1: warning: no previous prototype for function 'istore_in' [-Wmissing-prototypes]
istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:356:1: warning: no previous prototype for function 'istore_recv' [-Wmissing-prototypes]
istore_recv(PG_FUNCTION_ARGS)
^
src/istore_io.c:381:1: warning: no previous prototype for function 'istore_send' [-Wmissing-prototypes]
istore_send(PG_FUNCTION_ARGS)
^
src/istore_io.c:403:1: warning: no previous prototype for function 'cistore_out' [-Wmissing-prototypes]
cistore_out(PG_FUNCTION_ARGS)
^
src/istore_io.c:411:1: warning: no previous prototype for function 'cistore_in' [-Wmissing-prototypes]
cistore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:421:1: warning: no previous prototype for function 'device_istore_in' [-Wmissing-prototypes]
device_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:431:1: warning: no previous prototype for function 'country_istore_in' [-Wmissing-prototypes]
country_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:441:1: warning: no previous prototype for function 'os_name_istore_in' [-Wmissing-prototypes]
os_name_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:451:1: warning: no previous prototype for function 'cistore_from_types' [-Wmissing-prototypes]
cistore_from_types(PG_FUNCTION_ARGS)
^
src/istore_io.c:486:1: warning: no previous prototype for function 'cistore_cohort_from_types' [-Wmissing-prototypes]
cistore_cohort_from_types(PG_FUNCTION_ARGS)
^
15 warnings generated.
src/os_name.c:31:1: warning: no previous prototype for function 'os_name_recv' [-Wmissing-prototypes]
os_name_recv(PG_FUNCTION_ARGS)
^
src/os_name.c:41:1: warning: no previous prototype for function 'os_name_send' [-Wmissing-prototypes]
os_name_send(PG_FUNCTION_ARGS)
^
src/os_name.c:141:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^

on mac os

build failure with postgresql-12

version 0.1.6 doesn't seem to appreciate pg-12:

src/bigistore.c:580:58: error: unknown type name �FunctionCallInfoData�; did you mean �FunctionCallInfoBaseData�?
  580 | setup_firstcall(FuncCallContext *funcctx, BigIStore *is, FunctionCallInfoData *fcinfo)
      |                                                          ^~~~~~~~~~~~~~~~~~~~
      |                                                          FunctionCallInfoBaseData
src/istore.c:579:55: error: unknown type name �FunctionCallInfoData�; did you mean �FunctionCallInfoBaseData�?
  579 | setup_firstcall(FuncCallContext *funcctx, IStore *is, FunctionCallInfoData *fcinfo)
      |                                                       ^~~~~~~~~~~~~~~~~~~~
      |                                                       FunctionCallInfoBaseData
src/bigistore.c: In function �bigistore_each�:
src/bigistore.c:621:9: warning: implicit declaration of function �setup_firstcall� [-Wimplicit-function-declaration]
  621 |         setup_firstcall(funcctx, is, fcinfo);
      |         ^~~~~~~~~~~~~~~
src/istore.c: In function �istore_each�:
src/istore.c:620:9: warning: implicit declaration of function �setup_firstcall� [-Wimplicit-function-declaration]
  620 |         setup_firstcall(funcctx, is, fcinfo);
      |         ^~~~~~~~~~~~~~~
make: *** [<builtin>: src/bigistore.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [<builtin>: src/istore.o] Error

missing equality operator

there is no = operator for istore / bigistore because of that certain sql features like NULLIF or EXCEPT can not be used

would be nice to have the ability to compare for equality

Istore multiplication is not very friendly

One can do:

select istore'1 => 10, 2 => 20' * 2;

but not

select 2 * istore'1 => 10, 2 => 20';

They should be equivalent in my opinion and latter is much more readable if used in conjunction with istore() function.

scount

for the numhstore the array_agg step in statments like

SELECT array_count(array_agg(val)) FROM (VALUES('de'),('us'),('de'),('en'))t(val);
           array_count
---------------------------------
 "de"=>"2", "en"=>"1", "us"=>"1"
(1 row)

seems to be the slowest part.

An aggregate function scount (set count) that counts directly without the memory consuming array_agg step might help and could be faster.

Introduce functions to control the upper and lower boundaries of the values in istore

Namely,
in_range(istore, lower, upper) to return true iff all values in the map are in the given range;
less_than(istore, upper) - returns true iff all values are lower than the given one.
greater_than(istore, lower) - true iff all values are greater the given one.
also, less_or_equal version of all.

In addition,
floor(istore, floor_value) to set all values lower than the floor_value to that value
ceiling(istore, ceiling_value) to set all values higher than the ceiling_value to that value.

That would be kind of coalesce values, together with the functions to check if we need to coalesce (for cases when there are values in the cohorts we need to correct, i.e. negatives).

Operator istore - int[]

Hello!
Could you add a new subtract operator?

istore - integer[]

Description:
subtract keys from istore
Example:
'1=>4,2=>5'::istore - '{2,4}'::integer[]
Result:
"1"=>"4"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.