adjust / istore Goto Github PK
View Code? Open in Web Editor NEWdevelopment repo for integer hstore replacement in postgres
License: MIT License
development repo for integer hstore replacement in postgres
License: MIT License
There are some aggregates that are still not parallel-safe (namely, isagg
). They rely on the AVL-tree as an internal state, which makes it slightly more challenging to implement parallel behavior, however, it should pay off if the aggregate Is the last step to the long table scan that can potentially be parallelized.
SELECT '0=>4000000000'::bigistore -> 0;
?column?
------------
-294967296
(1 row)
We should have an aggregate with a floor that one can sum istores together with a floor value. The idea is to filter out, for example, invalid negative numbers.
Proposed invocation:
sum_floor(istorecol, int)
Example:
GIven the following istores in a column called myval:
{1 => 3, 2 => -2, 3 => 4 }
{1 => -3, 2 => 5, 3 => 15}
{1 => 2, 2 => -3, 3 => -1, 4 => -3}
Then:
sum_floor(myval, 0)
would generate
{1 => 5, 2 => 5, 3 => 19}
It looks like clamp_below
runs into an int overflow:
SELECT clamp_below('"-3"=>"42", "0"=>"2235078024"'::bigistore,0);
clamp_below
--------------------
"0"=>"-2059889230"
(1 row)
Lines 1248 to 1270 in 08ee021
I guess clamp_sum
should be an int64
actually
Hello!
Are there any plans to support version 9.2?
Thanks,
Ibrahim
our new data type cistore
(combined integer store) should save the key and values as integer
SELECT 'de::phone::ios=>25'::cistore;
cistore
------------------------
"de::phone::ios"=>25
basically the first byte represents country
, the second the device_type
and the third byte should be interpreted as os_name
We also want a fourth byte which is a plain integer
SELECT 'de::phone::ios=>25'::cistore;
cistore
------------------------
"de::phone::ios::0"=>25
this one is later some cohort place or punchcard hour, whatever we need.
i get
src/device_type.c:45:1: warning: no previous prototype for function 'device_type_recv' [-Wmissing-prototypes]
device_type_recv(PG_FUNCTION_ARGS)
^
src/device_type.c:55:1: warning: no previous prototype for function 'device_type_send' [-Wmissing-prototypes]
device_type_send(PG_FUNCTION_ARGS)
^
src/device_type.c:148:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
src/device_type.c:168:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
4 warnings generated.
src/istore.c:19:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
src/istore.c:23:1: warning: no previous prototype for function 'istore_sum_up' [-Wmissing-prototypes]
istore_sum_up(PG_FUNCTION_ARGS)
^
src/istore.c:66:1: warning: no previous prototype for function 'is_exist' [-Wmissing-prototypes]
is_exist(PG_FUNCTION_ARGS)
^
src/istore.c:73:5: warning: variable 'key' is used uninitialized whenever switch default is taken
[-Wsometimes-uninitialized]
GET_KEYARG_BY_TYPE(in, key);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
src/istore.h:255:13: note: expanded from macro 'GET_KEYARG_BY_TYPE'
default: \
^~~~~~~
src/istore.c:74:21: note: uninitialized use occurs here
if (is_find(in, key))
^~~
src/istore.c:69:16: note: initialize the variable 'key' to silence this warning
int key;
^
= 0
src/istore.c:83:1: warning: no previous prototype for function 'is_fetchval' [-Wmissing-prototypes]
is_fetchval(PG_FUNCTION_ARGS)
^
src/istore.c:90:5: warning: variable 'key' is used uninitialized whenever switch default is taken
[-Wsometimes-uninitialized]
GET_KEYARG_BY_TYPE(in, key);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
src/istore.h:255:13: note: expanded from macro 'GET_KEYARG_BY_TYPE'
default: \
^~~~~~~
src/istore.c:91:29: note: uninitialized use occurs here
if ((pair = is_find(in, key)) == NULL )
^~~
src/istore.c:86:16: note: initialize the variable 'key' to silence this warning
int key;
^
= 0
src/istore.c:99:1: warning: no previous prototype for function 'is_add' [-Wmissing-prototypes]
is_add(PG_FUNCTION_ARGS)
^
src/istore.c:167:1: warning: no previous prototype for function 'is_add_integer' [-Wmissing-prototypes]
is_add_integer(PG_FUNCTION_ARGS)
^
src/istore.c:196:1: warning: no previous prototype for function 'is_subtract' [-Wmissing-prototypes]
is_subtract(PG_FUNCTION_ARGS)
^
src/istore.c:263:1: warning: no previous prototype for function 'is_subtract_integer' [-Wmissing-prototypes]
is_subtract_integer(PG_FUNCTION_ARGS)
^
src/istore.c:292:1: warning: no previous prototype for function 'is_multiply' [-Wmissing-prototypes]
is_multiply(PG_FUNCTION_ARGS)
^
src/istore.c:361:1: warning: no previous prototype for function 'is_multiply_integer' [-Wmissing-prototypes]
is_multiply_integer(PG_FUNCTION_ARGS)
^
src/istore.c:545:1: warning: no previous prototype for function 'istore_from_array' [-Wmissing-prototypes]
istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:557:1: warning: no previous prototype for function 'device_istore_from_array' [-Wmissing-prototypes]
device_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:575:1: warning: no previous prototype for function 'country_istore_from_array' [-Wmissing-prototypes]
country_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:593:1: warning: no previous prototype for function 'os_name_istore_from_array' [-Wmissing-prototypes]
os_name_istore_from_array(PG_FUNCTION_ARGS)
^
src/istore.c:654:1: warning: no previous prototype for function 'istore_agg' [-Wmissing-prototypes]
istore_agg(PG_FUNCTION_ARGS)
^
src/istore.c:698:1: warning: no previous prototype for function 'istore_agg_finalfn' [-Wmissing-prototypes]
istore_agg_finalfn(PG_FUNCTION_ARGS)
^
src/istore.c:719:1: warning: no previous prototype for function 'istore_array_add' [-Wmissing-prototypes]
istore_array_add(PG_FUNCTION_ARGS)
^
19 warnings generated.
src/istore_io.c:244:1: warning: no previous prototype for function 'type_istore_to_istore' [-Wmissing-prototypes]
type_istore_to_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:269:1: warning: no previous prototype for function 'istore_to_device_istore' [-Wmissing-prototypes]
istore_to_device_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:292:1: warning: no previous prototype for function 'istore_to_country_istore' [-Wmissing-prototypes]
istore_to_country_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:315:1: warning: no previous prototype for function 'istore_to_os_name_istore' [-Wmissing-prototypes]
istore_to_os_name_istore(PG_FUNCTION_ARGS)
^
src/istore_io.c:338:1: warning: no previous prototype for function 'istore_out' [-Wmissing-prototypes]
istore_out(PG_FUNCTION_ARGS)
^
src/istore_io.c:346:1: warning: no previous prototype for function 'istore_in' [-Wmissing-prototypes]
istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:356:1: warning: no previous prototype for function 'istore_recv' [-Wmissing-prototypes]
istore_recv(PG_FUNCTION_ARGS)
^
src/istore_io.c:381:1: warning: no previous prototype for function 'istore_send' [-Wmissing-prototypes]
istore_send(PG_FUNCTION_ARGS)
^
src/istore_io.c:403:1: warning: no previous prototype for function 'cistore_out' [-Wmissing-prototypes]
cistore_out(PG_FUNCTION_ARGS)
^
src/istore_io.c:411:1: warning: no previous prototype for function 'cistore_in' [-Wmissing-prototypes]
cistore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:421:1: warning: no previous prototype for function 'device_istore_in' [-Wmissing-prototypes]
device_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:431:1: warning: no previous prototype for function 'country_istore_in' [-Wmissing-prototypes]
country_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:441:1: warning: no previous prototype for function 'os_name_istore_in' [-Wmissing-prototypes]
os_name_istore_in(PG_FUNCTION_ARGS)
^
src/istore_io.c:451:1: warning: no previous prototype for function 'cistore_from_types' [-Wmissing-prototypes]
cistore_from_types(PG_FUNCTION_ARGS)
^
src/istore_io.c:486:1: warning: no previous prototype for function 'cistore_cohort_from_types' [-Wmissing-prototypes]
cistore_cohort_from_types(PG_FUNCTION_ARGS)
^
15 warnings generated.
src/os_name.c:31:1: warning: no previous prototype for function 'os_name_recv' [-Wmissing-prototypes]
os_name_recv(PG_FUNCTION_ARGS)
^
src/os_name.c:41:1: warning: no previous prototype for function 'os_name_send' [-Wmissing-prototypes]
os_name_send(PG_FUNCTION_ARGS)
^
src/os_name.c:141:1: warning: control may reach end of non-void function [-Wreturn-type]
}
^
on mac os
version 0.1.6 doesn't seem to appreciate pg-12:
src/bigistore.c:580:58: error: unknown type name �FunctionCallInfoData�; did you mean �FunctionCallInfoBaseData�?
580 | setup_firstcall(FuncCallContext *funcctx, BigIStore *is, FunctionCallInfoData *fcinfo)
| ^~~~~~~~~~~~~~~~~~~~
| FunctionCallInfoBaseData
src/istore.c:579:55: error: unknown type name �FunctionCallInfoData�; did you mean �FunctionCallInfoBaseData�?
579 | setup_firstcall(FuncCallContext *funcctx, IStore *is, FunctionCallInfoData *fcinfo)
| ^~~~~~~~~~~~~~~~~~~~
| FunctionCallInfoBaseData
src/bigistore.c: In function �bigistore_each�:
src/bigistore.c:621:9: warning: implicit declaration of function �setup_firstcall� [-Wimplicit-function-declaration]
621 | setup_firstcall(funcctx, is, fcinfo);
| ^~~~~~~~~~~~~~~
src/istore.c: In function �istore_each�:
src/istore.c:620:9: warning: implicit declaration of function �setup_firstcall� [-Wimplicit-function-declaration]
620 | setup_firstcall(funcctx, is, fcinfo);
| ^~~~~~~~~~~~~~~
make: *** [<builtin>: src/bigistore.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [<builtin>: src/istore.o] Error
there is no =
operator for istore
/ bigistore
because of that certain sql features like NULLIF
or EXCEPT
can not be used
would be nice to have the ability to compare for equality
One can do:
select istore'1 => 10, 2 => 20' * 2;
but not
select 2 * istore'1 => 10, 2 => 20';
They should be equivalent in my opinion and latter is much more readable if used in conjunction with istore()
function.
for the numhstore the array_agg
step in statments like
SELECT array_count(array_agg(val)) FROM (VALUES('de'),('us'),('de'),('en'))t(val);
array_count
---------------------------------
"de"=>"2", "en"=>"1", "us"=>"1"
(1 row)
seems to be the slowest part.
An aggregate function scount
(set count) that counts directly without the memory consuming array_agg
step might help and could be faster.
Namely,
in_range(istore, lower, upper)
to return true iff all values in the map are in the given range;
less_than(istore, upper)
- returns true iff all values are lower than the given one.
greater_than(istore, lower)
- true iff all values are greater the given one.
also, less_or_equal version of all.
In addition,
floor(istore, floor_value)
to set all values lower than the floor_value to that value
ceiling(istore, ceiling_value)
to set all values higher than the ceiling_value to that value.
That would be kind of coalesce values, together with the functions to check if we need to coalesce (for cases when there are values in the cohorts we need to correct, i.e. negatives).
Hello!
Could you add a new subtract operator?
istore - integer[]
Description:
subtract keys from istore
Example:
'1=>4,2=>5'::istore - '{2,4}'::integer[]
Result:
"1"=>"4"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.