PGroonga is a PostgreSQL extension to use Groonga as index. PGroonga makes PostgreSQL fast full text search platform for all languages!
Home Page: https://pgroonga.github.io/
License: Other
PGroonga makes PostgreSQL fast full text search platform for all languages!
See Web site for more details:
PGroonga is released under PostgreSQL license that is similar to BSD license and MIT license.
See COPYING file for details such as copyright holders.
Bundled software is released under different license. Here are bundled software and license information:
Copyright (c) 2012-2021, Yann Colletvendor/xxHash/LICENSE
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Hi
We try following 2 statements :
select * from csp_appdata.product product where name0 @@ '沙琪玛';
select *, pgroonga.score(product) from csp_appdata.product product where name0 @@ '沙琪玛';
The former takes 152 ms.
And the latter takes much longer, with pgroonga-1.1.4, it takes around 11-12 seconds, since pgroonga-1.1.5, the speedup is almost 2X, now it takes 5027 ms, around 33X slower if compared with the former(152 ms).
Any way to improve the speed of pgroonga.score further?
FYI, There are 49152 matching rows out of 245760 rows, and the values of name0 field are like following:
' 香港 蓝欧 蛋酥味沙琪玛 408g 20盒408g '
' 香港 蓝欧 黑糖味沙琪玛 408g 20盒408g '
' 香港 蓝欧 芝麻味沙琪玛 408g 20盒408g '
' 香港 蓝欧 芝麻味沙琪玛 568g 12袋568g '
' 香港 蓝欧 黑糖味沙琪玛 568g 12袋568g '
' 香港 蓝欧 蛋酥味沙琪玛 568g 12袋568g '
' 香港 蓝欧 蛋酥味沙琪玛 408g 20盒*408g '
...
Thanks a lot.
--Yongxiang
Hello!
Thanks for this amazing library, I just want to know if its possible do what the title says (at least on varchar), I want to search on a column with name titles (not long) but if a user make a mistake in the spelling him cant get any result, example:
mal_api_dev=# SELECT title_romaji, mal_id FROM animes_anime WHERE title_romaji %% 'evangelion';
title_romaji | mal_id
------------------------------------------------+--------
Neon Genesis Evangelion | 30
Neon Genesis Evangelion: Death & Rebirth | 31
Neon Genesis Evangelion: The End of Evangelion | 32
Evangelion: 1.0 You Are (Not) Alone | 2759
Evangelion: 2.0 You Can (Not) Advance | 3784
Evangelion: 3.0 You Can (Not) Redo | 3785
Evangelion: 3.0+1.0 | 3786
Petit Eva: Evangelion@School | 4130
Schick x Evangelion | 31115
(9 rows)
mal_api_dev=# SELECT title_romaji, mal_id FROM animes_anime WHERE title_romaji %% 'evanelion';
Empty set
mal_api_dev=# SELECT title_romaji, mal_id FROM animes_anime WHERE title_romaji %% 'evangrlion';
Empty set
Greetings!
https://github.com/pgroonga/pgroonga/blob/master/sql/regexp/text/regexp/partial/indexscan.sql
It is possible to execute create index .. using pgroonga(... pgroonga.text_regexp_ops) command by non-administrator user?
CREATE INDEX grnindex ON memos USING pgroonga (content pgroonga.text_regexp_ops);
ERROR: permission denied for schema pgroonga
User can't execute create index command after grant command.
grant ALL on ALL TABLES IN SCHEMA pgroonga to user;
grant ALL on SCHEMA pgronga to user;
ERROR: schema "pgronga" does not exist
Thanks.
According to the postgres wiki,
authors are not supposed to explicitly create a schema where to live in, because that means users will have to think about adapting their search_path, which can get pretty hairy sometimes
Most extensions just install to public so that there is no need to grant special permissions (as in #8 and #9) and the functions are available without qualification or changes to search_path.
I made a build on my machine that installed to public and was able to go through the tutorial without problems.
I find IN operator not working sometimes, however, after I recreate pgroonga composite index, it works, so I guess the index is out of sync from data.
common-test=# select * from pg_indexes where tablename = 'customer';
schemaname | tablename | indexname | tablespace | indexdef
-------------+-----------+------------------------+------------+----------------------------------------------------------------------------------------------
csp_appdata | customer | customer_id_pkey | | CREATE UNIQUE INDEX customer_id_pkey ON csp_appdata.customer USING btree (id)
csp_appdata | customer | customer_customerid0_u | | CREATE UNIQUE INDEX customer_customerid0_u ON csp_appdata.customer USING btree (customerid0)
csp_appdata | customer | customer_webpage0_i | | CREATE INDEX customer_webpage0_i ON csp_appdata.customer USING pgroonga (id, webpage0)
(3 rows)
common-test=# set enable_seqscan=off;
SET
common-test=# set enable_indexscan=on;
SET
common-test=# set enable_bitmapscan=on;
SET
common-test=# explain select id from csp_appdata.customer where id in (10001, 10002);
QUERY PLAN
----------------------------------------------------------------------------------
Bitmap Heap Scan on customer (cost=0.00..6.97 rows=2 width=8)
Recheck Cond: (id = ANY ('{10001,10002}'::bigint[]))
-> Bitmap Index Scan on customer_webpage0_i (cost=0.00..0.00 rows=2 width=0)
Index Cond: (id = ANY ('{10001,10002}'::bigint[]))
(4 rows)
common-test=# select id from csp_appdata.customer where id in (10001, 10002);
id
-------
10002
(1 row)
common-test=# explain select id from csp_appdata.customer where id in (10001);
QUERY PLAN
--------------------------------------------------------------------------------------
Index Only Scan using customer_id_pkey on customer (cost=0.28..8.29 rows=1 width=8)
Index Cond: (id = 10001)
(2 rows)
common-test=# select id from csp_appdata.customer where id in (10001);
id
-------
10001
(1 row)
common-test=# drop index csp_appdata.customer_webpage0_i;
DROP INDEX
common-test=# select * from pg_indexes where tablename = 'customer';
schemaname | tablename | indexname | tablespace | indexdef
-------------+-----------+------------------------+------------+----------------------------------------------------------------------------------------------
csp_appdata | customer | customer_id_pkey | | CREATE UNIQUE INDEX customer_id_pkey ON csp_appdata.customer USING btree (id)
csp_appdata | customer | customer_customerid0_u | | CREATE UNIQUE INDEX customer_customerid0_u ON csp_appdata.customer USING btree (customerid0)
(2 rows)
common-test=# CREATE INDEX customer_webpage0_i ON csp_appdata.customer USING pgroonga (id, webpage0);
CREATE INDEX
common-test=# explain select id from csp_appdata.customer where id in (10001, 10002);
QUERY PLAN
----------------------------------------------------------------------------------
Bitmap Heap Scan on customer (cost=0.00..6.97 rows=2 width=8)
Recheck Cond: (id = ANY ('{10001,10002}'::bigint[]))
-> Bitmap Index Scan on customer_webpage0_i (cost=0.00..0.00 rows=2 width=0)
Index Cond: (id = ANY ('{10001,10002}'::bigint[]))
(4 rows)
common-test=# select id from csp_appdata.customer where id in (10001, 10002);
id
-------
10001
10002
(2 rows)
Is possible search in many columns in a table at the same time?
In this case in the same table I have some titles in romaji, japanese and english, is there a way to search in the tree columns at the same time?
Greetings!
The instructions provide for assigning a different Tokenizer and Normalizer, but don't say whether it is possible to use token filters, as explained @ http://groonga.org/docs/reference/token_filters.html
I notice in my indices, if I search for a plural word (e.g. "bananas"), I won't find any cases of the singular word (e.g. "banana"), though if I only search for singular words, I will find the plurals. I'd very much like to fix this.
Hello!
Is it possible to search for the whole word only? Is there any suitable operator or tokenizer?
The following query helps to check it (no FALSE in IS_OK column)
drop table if exists memos;
CREATE TABLE memos (id integer, content text);
INSERT INTO memos VALUES (1, 'PostgreSQL is a relational database management system.');
INSERT INTO memos VALUES (2, 'Groonga is a fast full text search engine that supports all languages.');
INSERT INTO memos VALUES (3, 'PGroonga is a PostgreSQL extension that uses Groonga as index.');
INSERT INTO memos VALUES (4, 'There is groonga command 3.6V');
drop index if exists pgroonga_content_index;
CREATE INDEX pgroonga_content_index ON memos USING pgroonga (content pgroonga.text_full_text_search_ops_v2) WITH (tokenizer='TokenDelimit');
SET enable_seqscan = off;
SELECT '@@' as oper, true as "is_ok", 'relational| fast', * FROM memos WHERE content @@ 'relational | fast'
union all
SELECT '@@' as oper, false, 'databK & fast', * FROM memos WHERE content @@ 'databK & fast'
union all
SELECT '@@' as oper, true, 'database & fast', * FROM memos WHERE content @@ 'database & fast'
union all
SELECT '@@' as oper, true, 'database', * FROM memos WHERE content @@ 'database'
union all
SELECT '@@' as oper, false, 'databa', * FROM memos WHERE content @@ 'databa'
union all
SELECT '@@' as oper, false, 'databa_', * FROM memos WHERE content @@ 'databa '
union all
SELECT '@@' as oper, false, '_databa', * FROM memos WHERE content @@ ' databa'
union all
SELECT '@@' as oper, true,'_engine_', * FROM memos WHERE content @@ ' engine '
union all
SELECT '@@' as oper, true, '3.6V engine1', * FROM memos WHERE content @@ '3.6V or engine'
union all
SELECT '&@' as oper, true, '3.6V engine', * FROM memos WHERE content &@> ARRAY['3.6V', 'engine']
union all
SELECT '&@' as oper, false, '_3.6_ _engine_', * FROM memos WHERE content &@> ARRAY[' 3.6 ', ' engin ']
union all
SELECT '&@' as oper, false, '3.6 engine', * FROM memos WHERE content &@> ARRAY['3.6', 'engin']
union all
SELECT '&@' as oper, false, '3.6', * FROM memos WHERE content &@> ARRAY['"fast"']
union all
SELECT '&@' as oper, false, '36', * FROM memos WHERE content &@> ARRAY['36']
union all
SELECT '&@' as oper, false, 'extens', * FROM memos WHERE content &@> ARRAY['extens']
union all
SELECT '&@' as oper, false, '_extension_', * FROM memos WHERE content &@> ARRAY[' extension ']
union all
SELECT '&@' as oper, false, 'extens_', * FROM memos WHERE content &@> ARRAY['extens ']
union all
SELECT '&`' as oper, false, 'groong', * FROM memos WHERE content &` 'content @ "groong"'::text
union all
SELECT '&`' as oper, false, 'roonga', * FROM memos WHERE content &` 'content @ "roonga"'::text
union all
SELECT '&`' as oper, false, 'roonga_', * FROM memos WHERE content &` 'content @ "roonga "'::text
union all
SELECT '&`' as oper, false, '_roonga', * FROM memos WHERE content &` 'content @ " roonga "'::text
Thank you!
pgroonga/pgroonga.github.io#9 より
varcharにはインデックスを使ったLIKE/ILIKEを使えないので、pgroonga.varchar_regexp_opsを使っているときはLIKE/ILIKEは必ずシーケンシャルサーチになります。
という説明を http://pgroonga.github.io/ja/reference/operators/like.html に記載する
誤字や誤訳、未訳を見つけたらお気軽にプルリクエストを送ってください！
https://pgroonga.github.io/ja/install/os-x.html
Wrong
https://github.com/pgroonga/pgroonga.github.io/blob/master/ja/ja/install/os-x.md
Correct
https://github.com/pgroonga/pgroonga.github.io/blob/master/ja/install/os-x.md
ja/ja -> ja
Since the last update on 2017-05-01, pgroonga / groonga crashes regulary.
Basic Information:
pgroonga.log:
2017-05-04 13:51:30.916892|i|8313: hits=40
2017-05-04 13:51:30.920866|i|8378: [object][search][index][key][exact] <Lexicon11521365_0.index>
2017-05-04 13:51:30.920884|i|8378: grn_ii_sel > (181956)
2017-05-04 13:51:30.920925|i|8378: n=4 (181956)
2017-05-04 13:51:30.923152|d|8378: [ii][cursor][min] skip: 0x5630de53dde0: min(1222078->1303320): chunk(33->36): chunk-used(true->true)
2017-05-04 13:51:30.923530|d|8378: [ii][cursor][min] skip: 0x5630de53dde0: min(1479703->1501695): chunk(43->44): chunk-used(true->true)
2017-05-04 13:51:30.925069|i|8378: exact: 40
2017-05-04 13:51:30.925079|i|8378: hits=40
2017-05-04 13:51:30.925221|i|8378: [object][search][index][key][exact] <Lexicon11521366_0.index>
2017-05-04 13:51:30.925227|i|8378: grn_ii_sel > (181956)
2017-05-04 13:51:30.925274|i|8378: n=4 (181956)
2017-05-04 13:51:30.931706|i|8378: exact: 53
2017-05-04 13:51:30.931726|i|8378: hits=53
2017-05-04 13:51:41.947700|n|11095: grn_init: <7.0.2>
2017-05-04 13:51:41.947824|n|11095: pgroonga: initialize: <1.2.0>
...
2017-05-04 13:52:32.148899|i|11194: <0x7f115e4822a0:0> expired
i=0x5630df1984d0 max=625 (3/3)
2017-05-04 13:52:32.190654|d|11194: flushing a[0]=460828
seg=7(0x7f114e1d4000) free=24
2017-05-04 13:52:32.203047|d|11194: flushed a[0]=460828
seg=7(0x7f114b6d4000) free=24->247712 nterms=901 v=1
2017-05-04 13:52:32.203176|i|11194: <0x7f115e4822a0:0> expired
i=0x5630df1984d0 max=625 (3/3)
2017-05-04 13:52:35.206077|C|8378: -- CRASHED!!! --
2017-05-04 13:52:35.213660|C|8378:
/usr/lib/x86_64-linux-gnu/libgroonga.so.0(+0xa0e9e) [0x7f115d85be9e]
2017-05-04 13:52:35.213688|C|8378:
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f138aa0bcb0]
2017-05-04 13:52:35.213693|C|8378:
/usr/lib/postgresql/9.5/lib/pgroonga.so(+0x7e01) [0x7f115e05ee01]
2017-05-04 13:52:35.213696|C|8378:
/usr/lib/postgresql/9.5/lib/pgroonga.so(+0x7ec7) [0x7f115e05eec7]
2017-05-04 13:52:35.213699|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(+0x2d98a1) [0x5630db6918a1]
2017-05-04 13:52:35.213703|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(proc_exit+0x10) [0x5630db6918e0]
2017-05-04 13:52:35.213706|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(PostgresMain+0x41f) [0x5630db6b01ef]
2017-05-04 13:52:35.213710|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(+0x9d678) [0x5630db455678]
2017-05-04 13:52:35.213713|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(PostmasterMain+0x104b) [0x5630db65742b]
2017-05-04 13:52:35.213716|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(main+0x842) [0x5630db456372]
2017-05-04 13:52:35.213719|C|8378:
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f138a9f6f45]
2017-05-04 13:52:35.213722|C|8378: postgres: postgres hostname
127.0.0.1(57248) idle(+0x9e3dd) [0x5630db4563dd]
2017-05-04 13:52:35.213726|C|8378: ----------------
2017-05-04 13:52:35.218067|n|8313: grn_fin (0)
2017-05-04 13:52:35.223029|n|8283: grn_fin (0)
2017-05-04 13:52:57.134642|n|25104: grn_init: <7.0.2>
2017-05-04 13:52:57.134779|n|25104: pgroonga: initialize: <1.2.0>
From postgresql.log:
2017-05-04 13:52:35 CEST [1920-39] LOG: Serverprozess (PID 8378) wurde
von Signal 6 beendet: Aborted
2017-05-04 13:52:35 CEST [1920-40] LOG: aktive Serverprozesse werden
abgebrochen
2017-05-04 13:52:35 CEST [11095-3] postgres@hostname WARNUNG: breche
Verbindung ab wegen Absturz eines anderen Serverprozesses
2017-05-04 13:52:35 CEST [11095-4] postgres@hostname DETAIL: Der
Postmaster hat diesen Serverprozess angewiesen, die aktuelle Transaktion
zurückzurollen und die Sitzung zu beenden, weil ein anderer
Serverprozess abnormal beendet wurde und möglicherweise das Shared
Memory verfälscht hat.
2017-05-04 13:52:35 CEST [11095-5] postgres@hostname TIPP: In einem
Moment sollten Sie wieder mit der Datenbank verbinden und Ihren Befehl
wiederholen können.
Example:
CREATE INDEX pgroonga_content_index
ON memos
USING pgroonga (content)
WITH (token_filters='TokenFilterStem',
plugins='token_filters/stem');If plugin information is included in CREATE_INDEX, we can dump and restore it.
PostgreSQL 9.4.5 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit
Hi,
The index size is reported as zero when using the function pg_relation_size
relpages is reported as 0 too in pg_class
-[ RECORD 2 ]--+------------------
relname | blog_pgroonga_idx
relnamespace | 2200
reltype | 0
reloftype | 0
relowner | 16384
relam | 559313
relfilenode | 656024
reltablespace | 103240
relpages | 0
reltuples | 500000
relallvisible | 0
reltoastrelid | 0
relhasindex | f
relisshared | f
relpersistence | p
relkind | i
relnatts | 1
relchecks | 0
relhasoids | f
relhaspkey | f
relhasrules | f
relhastriggers | f
relhassubclass | f
relispopulated | t
relreplident | n
relfrozenxid | 0
relminmxid | 0
relacl |
reloptions |
Is it possible to use pgroonga.score by using an alternate key instead of primary key?
My table uses a composite primary key so I added a unique alternate key for pgroonga. But the pgroonga.score is always 0.
create index ix_rand_string_groonga_test20151220
on groonga_test_20151220 using pgroonga(rand_string)
with(tokenizer='TokenBigram',normalizer='none');
ERROR: pgroonga: nonexistent normalizer: <none>
Time: 4.475 ms
normalizer='' worked.
create index ix_rand_string_groonga_test20151220
on groonga_test_20151220 using pgroonga(rand_string)
with(tokenizer='TokenBigram',normalizer='');
CREATE INDEX
Time: 8.971 ms
PostgreSQL 9.6 has just been released. Would the new generic WAL interface in 9.6 allow pgroonga to support streaming replication?
PostgreSQL 9.6.2 installed
groonga-release-1.2.0-1.noarch.rpm installed
Master-Slave Streaming Replication configured according to this replication replication doc
After initial configuration is done, the following 2 observations are found:
psql (9.6.2)
Type "help" for help.
devqa=# select id,title from "test".contact where title %% '履歴書';
id | title
------------------+--------
0030000UJgFfAAL1 | 履歴書
(1 row)
devqa=# update "test".contact set title = '履歴書 - 1' where sfid = '0030000UJgFfAAL1';
UPDATE 1
devqa=#
devqa=#
devqa=# select id,title from "test".contact where title %% '履歴書';
ERROR: operator does not exist: text %% unknown
LINE 1: ...tle from "test".contact where title %% '履歴書...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
devqa=# This error is gone only when I reconnect to Postgres server again OR after server restart.
devqa=# select id,title from "test".contact where title %% '履歴書';
ERROR: pgroonga: object isn't found: <Sources6034694>My question here what am doing wrong here? and how can it be fixed?
I encountered a bug in my system with pgroonga indexes on SSD:
When i make update of data on one table, all the indexes in the tablespace, placed in that SSD are being corrupted. I try to use "io_flush" when updating the table and the indexes of the table are ok, but the indexes of the other tables on the same drive(TABLESPACE) stay corrupted. If i change the place of indexes on the SAS drive, all is ok.
My system is ubuntu with version:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
PostgreSQL version is 9.5.
It's useful to confirm libgroonga version.
Hello.
Can't use groonga index when I use multiple @. operator with index?
Can I use multiple @
Thanks.
Use single @~ operator.
SELECT id, content
FROM memos
WHERE content @~ 'groonga'; 2015-12-23 17:46:31.344849|n| grn_init: <5.1.0>
2015-12-23 17:46:31.367846|i| [object][search][index][key][regexp] <Lexicon76407_0.index>
2015-12-23 17:46:31.367866|i| grn_ii_sel > (groonga)
2015-12-23 17:46:31.367911|i| n=4 (groonga)
2015-12-23 17:46:31.368037|i| exact: 2
2015-12-23 17:46:31.368041|i| hits=2
2015-12-23 17:46:31.380598|n| grn_fin (0)
use multiple @~ operator.
SELECT id, content
FROM memos
WHERE content @~ 'groonga' or content @~ 'rdbms';it does not output grn_ii_sel entry, so It does not use groonga index.
Is this correct?
2015-12-23 17:48:46.226355|n| grn_init: <5.1.0>
2015-12-23 17:48:46.250727|n| grn_fin (0)
\timing
CREATE TABLE memos (
id integer,
content text
);
INSERT INTO memos VALUES (1, 'PostgreSQL is a RDBMS.');
INSERT INTO memos VALUES (2, 'Groonga is fast full text search engine.');
INSERT INTO memos VALUES (3, 'PGroonga is a PostgreSQL extension that uses Groonga.');
CREATE INDEX grnindex ON memos USING pgroonga (content pgroonga.text_regexp_ops);
SET enable_seqscan = off;
SET enable_indexscan = on;
SET enable_bitmapscan = off;
set pgroonga.log_level = 'debug';
SELECT id, content
FROM memos
WHERE content @~ 'groonga' or content @~ 'rdbms';
--set pgroonga.log_level = 'debug';
--SELECT id, content
-- FROM memos
-- WHERE content @~ 'groonga';
DROP TABLE memos;Using
Postgresql 9.6.3 and pgroonga-1.2.3-postgresql-9.6.3-2-x64.zip on Windows
If the index is created using the primary key column (for pgroonga.score) and the PK column is not the first column score_memos(text,id), queries using the index will throw an error.
CREATE TABLE score_memos
(
text TEXT,
id SERIAL NOT NULL
CONSTRAINT score_memos_pkey
PRIMARY KEY
);
CREATE INDEX pgroonga_score_memos_content_index
ON score_memos
USING pgroonga (id, TEXT );
INSERT INTO score_memos VALUES ('PostgreSQL is a relational database management system.');
INSERT INTO score_memos VALUES ('Groonga is a fast full text search engine that supports all languages.');
INSERT INTO score_memos VALUES ('PGroonga is a PostgreSQL extension that uses Groonga as index.');
INSERT INTO score_memos VALUES ('There is groonga command.');
INSERT INTO score_memos VALUES ('There truck is groonga command.');
SET enable_seqscan = OFF;
SET enable_indexscan = ON;
SET enable_bitmapscan = ON;
SELECT
*,
pgroonga.score(score_memos)
FROM score_memos
WHERE text &@ 'groonga' OR text &@ 'pg';
If the table is created in this column order score_memos(id,text), then pgroonga works correctly.
CREATE TABLE score_memos
(
id SERIAL NOT NULL
CONSTRAINT score_memos_pkey
PRIMARY KEY,
text TEXT
);
The most recent postgresql-9.5-pgroonga packages for Ubuntu 16.04 Xenial (1.1.9-2) appear to have out of date dependency information for libgroonga0. They show libgroonga0 (>= 6.0.7) instead of libgroonga0 (>= 6.1.1) as indicated here.
This means that the pgroonga extension is broken unless the user does apt-get upgrade libgroonga0 manually.
$ apt-cache show postgresql-9.5-pgroonga
Package: postgresql-9.5-pgroonga
Source: pgroonga
Priority: optional
Section: database
Installed-Size: 176
Maintainer: PGroonga Project <[email protected]>
Architecture: amd64
Version: 1.1.9-2~xenial1
Depends: libc6 (>= 2.14), libgroonga0 (>= 6.0.7), postgresql-9.5
Filename: pool/main/p/pgroonga/postgresql-9.5-pgroonga_1.1.9-2~xenial1_amd64.deb
Size: 53798
MD5sum: 7201bf605b6c44e91b556e068dfd8b75
SHA1: 02ca934ec1d0f32e21e794bc8369b851d380cbd2
SHA256: f334b07e7f41ae9de1dd369092f71eaf39c00fc4e90f5daa9d741391530e6d24
Description-en: Fast full-text search plugin for PostgreSQL based on Groonga
PGroonga is a PostgreSQL plugin. It provides fast full-text search feature.
It is based on Groonga full-text search engine.
Description-md5: 33bd35692abaff6780c6a0fe63ee0b51
Just in case my system somehow had an old version of the package, I downloaded the .deb directly and confirmed it also has the wrong dependency info:
$ dpkg-deb -I postgresql-9.5-pgroonga_1.1.9-2~xenial1_amd64.deb
new debian package, version 2.0.
size 53798 bytes: control archive=853 bytes.
506 bytes, 13 lines control
2246 bytes, 24 lines md5sums
Package: postgresql-9.5-pgroonga
Source: pgroonga
Version: 1.1.9-2~xenial1
Architecture: amd64
Maintainer: PGroonga Project <[email protected]>
Installed-Size: 176
Depends: libc6 (>= 2.14), libgroonga0 (>= 6.0.7), postgresql-9.5
Section: database
Priority: optional
Homepage: http://pgroonga.github.io/
Description: Fast full-text search plugin for PostgreSQL based on Groonga
PGroonga is a PostgreSQL plugin. It provides fast full-text search feature.
It is based on Groonga full-text search engine.
Groonga 5.0.6
PGroonga master (63b36b2)
の環境でsnippet_htmlを試していたのですが、検索結果がヒットする場合とヒットせず0件になってしまう場合がありました。
試行錯誤したところ、確実に再現するケースを作れたので共有します。
INSERT INTOでデータを登録する前にTRUNCATEを実行すると0件になります。
TRUCATE無しではヒットします。
Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
setup_script = <<SCRIPT
# install groonga with the development package
sudo apt-get -y install software-properties-common
sudo add-apt-repository -y universe
sudo add-apt-repository -y ppa:groonga/ppa
sudo apt-get update
sudo apt-get -y install groonga libgroonga-dev
sudo apt-get -y install groonga-tokenizer-mecab
sudo apt-get -y install groonga-token-filter-stem
# install postgresql server with the development package
echo 'deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main' | sudo sh -c 'cat > /etc/apt/sources.list.d/postgresql.list'
curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-9.4 postgresql-server-dev-9.4
# build and install pgroonga
sudo apt-get -y install git make gcc
git clone https://github.com/pgroonga/pgroonga
cd pgroonga
make
sudo make install
# create test script
cat <<'EOS' > ~/fail.sh
echo "=== $0 start ==="
db_name=pgroonga_snippet_html_test
sudo -u postgres dropdb $db_name || :
sudo -u postgres createdb $db_name
sudo -u postgres psql $db_name <<'EOF'
CREATE EXTENSION pgroonga;
CREATE TABLE memos (
id integer,
content text
);
CREATE INDEX pgroonga_content_index ON memos USING pgroonga (content);
TRUNCATE memos;
INSERT INTO memos VALUES (1, 'PostgreSQLはリレーショナル・データベース管理システムです。');
INSERT INTO memos VALUES (2, 'Groongaは日本語対応の高速な全文検索エンジンです。');
INSERT INTO memos VALUES (3, 'PGroongaはインデックスとしてGroongaを使うためのPostgreSQLの拡張機能です。');
INSERT INTO memos VALUES (4, 'groongaコマンドがあります。');
SET enable_seqscan = off;
SELECT * FROM memos WHERE content like '%groonga%';
EXPLAIN SELECT * FROM memos WHERE content like '%groonga%';
SELECT * FROM json_array_elements(pgroonga.command('select ' || pgroonga.table_name('pgroonga_content_index') || ' --command_version 2 --output_columns "snippet_html(content)" --match_columns "content" --query ''groonga''')::json->1->0);
EXPLAIN SELECT * FROM json_array_elements(pgroonga.command('select ' || pgroonga.table_name('pgroonga_content_index') || ' --command_version 2 --output_columns "snippet_html(content)" --match_columns "content" --query ''groonga''')::json->1->0);
EOF
echo "=== $0 exit ==="
EOS
chmod +x ~/fail.sh
sed '/^TRUNCATE memos;$/d' ~/fail.sh > ~/success.sh
chmod +x ~/success.sh
SCRIPT
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.provision "shell", inline: setup_script
config.vm.provision "shell", inline: "~/fail.sh"
config.vm.provision "shell", inline: "~/success.sh"
end
vagrant upの実行結果抜粋
==> default: === /root/fail.sh start ===
==> default: dropdb: database removal failed: ERROR: database "pgroonga_snippet_html_test" does not exist
==> default: CREATE EXTENSION
==> default: CREATE TABLE
==> default: CREATE INDEX
==> default: TRUNCATE TABLE
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: SET
==> default: id | content
==> default: ----+---------------------------------------------------------------------------
==> default: 2 | Groongaは日本語対応の高速な全文検索エンジンです。
==> default: 3 | PGroongaはインデックスとしてGroongaを使うためのPostgreSQLの拡張機能です。
==> default: 4 | groongaコマンドがあります。
==> default: (3 rows)
==> default: QUERY PLAN
==> default: -------------------------------------------------------------------------------------
==> default: Index Scan using pgroonga_content_index on memos (cost=0.14..8.16 rows=1 width=36)
==> default: Index Cond: (content ~~ '%groonga%'::text)
==> default: (2 rows)
==> default: value
==> default: ---------------------------
==> default: [0]
==> default: [["snippet_html","null"]]
==> default: (2 rows)
==> default: QUERY PLAN
==> default: ---------------------------------------------------------------------------
==> default: Function Scan on json_array_elements (cost=0.03..1.03 rows=100 width=32)
==> default: (1 row)
==> default: === /root/fail.sh exit ===
==> default: Running provisioner: shell...
default: Running: inline script
==> default: stdin: is not a tty
==> default: === /root/success.sh start ===
==> default: CREATE EXTENSION
==> default: CREATE TABLE
==> default: CREATE INDEX
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: INSERT 0 1
==> default: SET
==> default: id | content
==> default: ----+---------------------------------------------------------------------------
==> default: 2 | Groongaは日本語対応の高速な全文検索エンジンです。
==> default: 3 | PGroongaはインデックスとしてGroongaを使うためのPostgreSQLの拡張機能です。
==> default: 4 | groongaコマンドがあります。
==> default: (3 rows)
==> default: QUERY PLAN
==> default: -------------------------------------------------------------------------------------
==> default: Index Scan using pgroonga_content_index on memos (cost=0.14..8.16 rows=1 width=36)
==> default: Index Cond: (content ~~ '%groonga%'::text)
==> default: (2 rows)
==> default: value
==> default: -----------------------------------------------------------------------------------------------------------------------------------------------
==> default: [3]
==> default: [["snippet_html","null"]]
==> default: [["<span class=\"keyword\">Groonga</span>は日本語対応の高速な全文検索エンジンです。"]]
==> default: [["P<span class=\"keyword\">Groonga</span>はインデックスとして<span class=\"keyword\">Groonga</span>を使うためのPostgreSQLの拡張機能です。"]]
==> default: [["<span class=\"keyword\">groonga</span>コマンドがあります。"]]
==> default: (5 rows)
==> default: QUERY PLAN
==> default: ---------------------------------------------------------------------------
==> default: Function Scan on json_array_elements (cost=0.03..1.03 rows=100 width=32)
==> default: (1 row)
==> default: === /root/success.sh exit ===
./fail.shの実行結果ではsnippet_htmlのところが0件になってしまっています。
補足：
ただし、元々試していたデータベースではTRUCATEはたぶん実行したことがないので、
別の原因でもsnippet_htmlを使うとヒットしなくなる場合があるかと思います。
残念ながら、このデータベースは社外秘の文書が入っているので共有できないです。
diff --git a/pgroonga.sql b/pgroonga.sql
index 05de77d..358bc34 100644
--- a/pgroonga.sql
+++ b/pgroonga.sql
@@ -430,12 +430,25 @@ CREATE FUNCTION pgroonga.match_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.match_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_match_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &@ (
PROCEDURE = pgroonga.match_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &@ (
+ PROCEDURE = pgroonga.match_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.query_text(text, text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_query_text'
@@ -443,12 +456,25 @@ CREATE FUNCTION pgroonga.query_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.query_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_query_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &? (
PROCEDURE = pgroonga.query_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &? (
+ PROCEDURE = pgroonga.query_text_array,
+ LEFTARG = text[[,
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.similar_text(text, text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_similar_text'
@@ -456,12 +482,25 @@ CREATE FUNCTION pgroonga.similar_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.similar_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_similar_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &~? (
PROCEDURE = pgroonga.similar_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &~? (
+ PROCEDURE = pgroonga.similar_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.prefix_text(text, text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_prefix_text'
@@ -469,12 +508,25 @@ CREATE FUNCTION pgroonga.prefix_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.prefix_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_prefix_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &^ (
PROCEDURE = pgroonga.prefix_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &^ (
+ PROCEDURE = pgroonga.prefix_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.prefix_rk_text(text, text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_prefix_rk_text'
@@ -482,12 +534,25 @@ CREATE FUNCTION pgroonga.prefix_rk_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.prefix_rk_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_prefix_rk_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &^~ (
PROCEDURE = pgroonga.prefix_rk_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &^~ (
+ PROCEDURE = pgroonga.prefix_rk_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.script_text(text, text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_script_text'
@@ -495,12 +560,25 @@ CREATE FUNCTION pgroonga.script_text(text, text)
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.script_text_array(text[], text)
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_script_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &` (
PROCEDURE = pgroonga.script_text,
LEFTARG = text,
RIGHTARG = text
);
+CREATE OPERATOR &` (
+ PROCEDURE = pgroonga.script_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text
+);
+
CREATE FUNCTION pgroonga.match_contain_text(text, text[])
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_match_contain_text'
@@ -508,12 +586,25 @@ CREATE FUNCTION pgroonga.match_contain_text(text, text[])
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.match_contain_text_array(text[], text[])
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_match_contain_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &@> (
PROCEDURE = pgroonga.match_contain_text,
LEFTARG = text,
RIGHTARG = text[]
);
+CREATE OPERATOR &@> (
+ PROCEDURE = pgroonga.match_contain_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text[]
+);
+
CREATE FUNCTION pgroonga.query_contain_text(text, text[])
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_query_contain_text'
@@ -521,12 +612,25 @@ CREATE FUNCTION pgroonga.query_contain_text(text, text[])
IMMUTABLE
STRICT;
+CREATE FUNCTION pgroonga.query_contain_text_array(text[], text[])
+ RETURNS bool
+ AS 'MODULE_PATHNAME', 'pgroonga_query_contain_text_array'
+ LANGUAGE C
+ IMMUTABLE
+ STRICT;
+
CREATE OPERATOR &?> (
PROCEDURE = pgroonga.query_contain_text,
LEFTARG = text,
RIGHTARG = text[]
);
+CREATE OPERATOR &?> (
+ PROCEDURE = pgroonga.query_contain_text_array,
+ LEFTARG = text[],
+ RIGHTARG = text[]
+);
+
CREATE FUNCTION pgroonga.prefix_contain_text_array(text[], text)
RETURNS bool
AS 'MODULE_PATHNAME', 'pgroonga_prefix_contain_text_array'
@@ -564,6 +668,15 @@ CREATE OPERATOR CLASS pgroonga.text_full_text_search_ops_v2 FOR TYPE text
OPERATOR 18 &@> (text, text[]),
OPERATOR 19 &?> (text, text[]);
+CREATE OPERATOR CLASS pgroonga.text_array_full_text_search_ops_v2 FOR TYPE text[]
+ USING pgroonga AS
+ OPERATOR 12 &@ (text[], text),
+ OPERATOR 13 &? (text[], text),
+ OPERATOR 14 &~? (text[], text),
+ OPERATOR 15 &` (text[], text),
+ OPERATOR 18 &@> (text[], text[]),
+ OPERATOR 19 &?> (text[], text[]);
+
CREATE OPERATOR CLASS pgroonga.text_term_search_ops_v2 FOR TYPE text
USING pgroonga AS
OPERATOR 16 &^,Hi, i see groonga support nested different table column to one index:
see:
http://groonga.org/docs/tutorial/match_columns.html#nested-index-search
http://groonga.org/ja/docs/tutorial/match_columns.html#nested-index-search
but , how can i defined an index like this in PGroonga?
I tried the way like this but not working:
CREATE INDEX pgroonga_nested_index ON table1, table2 USING pgroonga (table1.col, table2.col);
Thanks for any advises.
On MacBook, I see follow error since pgroonga-1.1.8 when I make a mvn build of a module called "common-test"
ERROR: pgroonga: object isn't found: <Sources366583664>
CONTEXT: automatic analyze of table "common-test.csp_appdata.customer"
An interesting fact is that, this error shows up about 1 minute after the mvn build finishes.
I was trying to install pgroonga-1.2.0 form source. However according to instruction, I installed groonga and libgroonga-dev packages too. But unfortunately getting error:
In file included from src/pgroonga.c:5:0:
src/pgrn-command-escape-value.h:3:21: fatal error: groonga.h: No such file or directory
compilation terminated.
: recipe for target 'src/pgroonga.o' failed
make[1]: *** [src/pgroonga.o] Error 1
I have passed PG_CONFIG with the make command and also confirmed pkg-config --list-all includes groonga.
THanks in advance.
See also: zulip/zulip#5603
tsearch's @@ is case insensitive, but PGroonga's @@ is case sensitive. And, now @@ is deprecated in PGroonga and announced to migrate to new &? operator.
For application developers familiar to PostgreSQL, tsearch compatible operator seems useful and it will reduce barriers to start using of PGroonga.
Hi,
PGroonga probably caused a deadlock. I believe its strongly connected to (P)Groonga, because the Deadlock happened again (or still?) after restarting postgres a few hours later ( sudo service postgresql restart) and running the same transaction.
After dropping the PGroonga indices, this transaction went through without problems.
We had the same problem on another machine a few month ago too, but with PGroonga 1.1.9 (as
Postgresql 9.5.4 on Ubuntu 14.04 LTS / Trusty
PGroonga 1.2.1 & libgroonga0 7.0.3
Please ask if you need any more information.
Indizes:
create index zga on sucheintrag using pgroonga(volltextgeraeta) WITH (tokenizer='TokenTrigram', normalizer='');
create index zva on sucheintrag using pgroonga(volltexta) WITH (tokenizer='TokenTrigram', normalizer='');
postgresql.log (excerpt)
2017-07-06 04:25:38 CEST [25119-1] postgres@kris LOG: duration: 14564.392 ms statement: vacuum analyze verbose public.artikel
2017-07-06 04:25:49 CEST [27458-1] postgres@kris LOG: duration: 10906.223 ms statement: vacuum analyze verbose public.stueckliste
2017-07-06 04:26:05 CEST [27672-1] postgres@kris LOG: duration: 15643.526 ms statement: vacuum analyze verbose public.produkt
*** Error in `postgres: postgres kris [local] VACUUM': munmap_chunk(): invalid pointer: 0x000055c29292d250 ***
...
2017-07-06 04:48:44 CEST [22593-94] LOG: server process (PID 28253) was terminated by signal 6: Aborted
2017-07-06 04:48:44 CEST [22593-95] DETAIL: Failed process was running: vacuum analyze verbose public.sucheintrag
2017-07-06 04:48:44 CEST [22593-96] LOG: terminating any other active server processes
2017-07-06 04:48:44 CEST [17103-1] postgres@kris WARNING: terminating connection because of crash of another server process
2017-07-06 04:48:44 CEST [17103-2] postgres@kris DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
... (repeated)
2017-07-06 04:48:44 CEST [15112-2] WARNING: terminating connection because of crash of another server process
2017-07-06 04:48:44 CEST [15112-3] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-07-06 04:48:44 CEST [15112-4] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-07-06 04:48:44 CEST [24489-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:44 CEST [24500-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:44 CEST [24510-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:44 CEST [24520-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:44 CEST [24530-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:44 CEST [22593-97] LOG: all server processes terminated; reinitializing
2017-07-06 04:48:45 CEST [24570-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:45 CEST [24569-1] LOG: database system was interrupted; last known up at 2017-07-06 04:41:47 CEST
2017-07-06 04:48:46 CEST [24573-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:46 CEST [24575-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:46 CEST [24577-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:47 CEST [24595-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:49 CEST [24596-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:50 CEST [24597-1] postgres@kris FATAL: the database system is in recovery mode
2017-07-06 04:48:52 CEST [24569-2] LOG: database system was not properly shut down; automatic recovery in progress
2017-07-06 04:48:52 CEST [24569-3] LOG: redo starts at 169/672A6CB8
2017-07-06 04:48:52 CEST [24569-4] LOG: invalid record length at 169/672DB9D8
2017-07-06 04:48:52 CEST [24569-5] LOG: redo done at 169/672DB9B0
2017-07-06 04:48:52 CEST [24569-6] LOG: last completed transaction was at log time 2017-07-06 04:47:17.730385+02
2017-07-06 04:48:52 CEST [24569-7] LOG: MultiXact member wraparound protections are now enabled
2017-07-06 04:48:53 CEST [22593-98] LOG: database system is ready to accept connections
2017-07-06 04:48:53 CEST [24603-1] LOG: autovacuum launcher started
... (the following appears every 21 Minutes in the log)
2017-07-06 05:09:54 CEST [30644-1] postgres@kris ERROR: pgroonga: failed to set column value: grn_io_lock failed
2017-07-06 05:09:54 CEST [30644-2] postgres@kris STATEMENT: UPDATE "sucheintrag" SET "crea"=$1,"lmod"=$2,"ldel"=$3,"bot"=$4,"mandant"=$5,"volltextn"=$6,"volltexta"=$7,"volltextgeraetn"=$8,"volltextgeraeta"=$9,"suchklasse"=$10,"dirtyvollt
ext"=$11,"produkt"=$12 WHERE "id"=$13
pgroonga.log (default log level, excerpt):
2017-07-06 04:13:29.282776|n|31354: grn_init: <7.0.3>
2017-07-06 04:13:29.282933|n|31354: pgroonga: initialize: <1.2.1>
2017-07-06 04:16:26.250258|n|31354: grn_fin (0)
2017-07-06 04:20:27.747063|n|30326: grn_init: <7.0.3>
2017-07-06 04:20:27.747214|n|30326: pgroonga: initialize: <1.2.1>
2017-07-06 04:23:27.452161|n|30326: grn_fin (0)
2017-07-06 04:26:05.430571|n|28253: grn_init: <7.0.3>
2017-07-06 04:26:05.430721|n|28253: pgroonga: initialize: <1.2.1>
2017-07-06 04:27:32.852909|n|13483: grn_init: <7.0.3>
2017-07-06 04:27:32.853081|n|13483: pgroonga: initialize: <1.2.1>
2017-07-06 04:28:26.695134|n|13483: grn_fin (0)
2017-07-06 04:34:46.642268|n|19672: grn_init: <7.0.3>
2017-07-06 04:34:46.642419|n|19672: pgroonga: initialize: <1.2.1>
2017-07-06 04:34:49.945606|n|19672: io(base/721070/pgrn.000011D) collisions(1000/10): lock failed 1000 times
2017-07-06 04:36:28.040982|n|21138: grn_init: <7.0.3>
2017-07-06 04:36:28.041150|n|21138: pgroonga: initialize: <1.2.1>
2017-07-06 04:37:41.835086|n|21170: grn_init: <7.0.3>
2017-07-06 04:37:41.835249|n|21170: pgroonga: initialize: <1.2.1>
2017-07-06 04:40:49.826158|n|21138: grn_fin (0)
2017-07-06 04:40:54.061475|n|21170: grn_fin (0)
2017-07-06 04:46:53.549713|n|11897: grn_init: <7.0.3>
2017-07-06 04:46:53.549872|n|11897: pgroonga: initialize: <1.2.1>
2017-07-06 04:47:16.739324|n|17103: grn_init: <7.0.3>
2017-07-06 04:47:16.739483|n|17103: pgroonga: initialize: <1.2.1>
2017-07-06 04:48:38.585298|n|11897: grn_fin (0)
2017-07-06 04:53:57.987123|n|30644: grn_init: <7.0.3>
2017-07-06 04:53:57.987280|n|30644: pgroonga: initialize: <1.2.1>
2017-07-06 04:53:59.548606|n|30644: io(base/721070/pgrn.000011D) collisions(1000/6): lock failed 1000 times
2017-07-06 04:57:26.163223|n|827: grn_init: <7.0.3>
2017-07-06 04:57:26.163384|n|827: pgroonga: initialize: <1.2.1>
2017-07-06 04:58:46.705532|n|827: grn_fin (0)
2017-07-06 05:03:37.361773|n|19505: grn_init: <7.0.3>
2017-07-06 05:03:37.361932|n|19505: pgroonga: initialize: <1.2.1>
2017-07-06 05:03:50.407183|n|19505: grn_fin (0)
2017-07-06 05:04:29.389968|n|5995: grn_init: <7.0.3>
2017-07-06 05:04:29.390124|n|5995: pgroonga: initialize: <1.2.1>
2017-07-06 05:08:27.088960|n|5995: grn_fin (0)
2017-07-06 05:09:54.252436|w|30644: [DB Locked] time out(900000): io(base/721070/pgrn.000011D) collisions(900000/6)
2017-07-06 05:09:54.252470|e|30644: grn_io_lock failed
2017-07-06 05:09:54.284455|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_io_lock+0x28e) [0x7fb2b15a970e]
2017-07-06 05:09:54.284467|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_ii_column_update+0x15c) [0x7fb2b158c3ec]
2017-07-06 05:09:54.284471|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_obj_default_set_value_hook+0x81) [0x7fb2b1466ca1]
2017-07-06 05:09:54.284474|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_obj_set_value+0x1593) [0x7fb2b1479583]
2017-07-06 05:09:54.284477|e|30644: /usr/lib/postgresql/9.5/lib/pgroonga.so(+0xa0d9) [0x7fb2b1c520d9]
2017-07-06 05:09:54.284479|e|30644: /usr/lib/postgresql/9.5/lib/pgroonga.so(pgroonga_insert+0x5d) [0x7fb2b1c57cbd]
2017-07-06 05:09:54.284482|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(FunctionCall6Coll+0xa6) [0x55c2911f6336]
2017-07-06 05:09:54.284485|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(index_insert+0xa5) [0x55c290edd7e5]
2017-07-06 05:09:54.284488|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecInsertIndexTuples+0x2cc) [0x55c290ff254c]
2017-07-06 05:09:54.284491|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x20bc28) [0x55c29100dc28]
2017-07-06 05:09:54.284493|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecModifyTable+0x3ff) [0x55c29100e34f]
2017-07-06 05:09:54.284496|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecProcNode+0x188) [0x55c290ff6308]
2017-07-06 05:09:54.284499|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(standard_ExecutorRun+0x132) [0x55c290ff31f2]
2017-07-06 05:09:54.284502|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x2fb509) [0x55c2910fd509]
2017-07-06 05:09:54.284504|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x2fb751) [0x55c2910fd751]
2017-07-06 05:09:54.284507|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(PortalRun+0x1c5) [0x55c2910fe365]
2017-07-06 05:09:54.681939|n|30644: grn_fin (0)
2017-07-06 05:14:58.105577|n|6287: grn_init: <7.0.3>
2017-07-06 05:14:58.105733|n|6287: pgroonga: initialize: <1.2.1>
2017-07-06 05:14:59.705018|n|6287: io(base/721070/pgrn.000011D) collisions(1000/6): lock failed 1000 times
...
2017-07-06 04:53:57.987123|n|30644: grn_init: <7.0.3>
2017-07-06 04:53:57.987280|n|30644: pgroonga: initialize: <1.2.1>
2017-07-06 04:53:59.548606|n|30644: io(base/721070/pgrn.000011D) collisions(1000/6): lock failed 1000 times
...
2017-07-06 05:09:54.252436|w|30644: [DB Locked] time out(900000): io(base/721070/pgrn.000011D) collisions(900000/6)
2017-07-06 05:09:54.252470|e|30644: grn_io_lock failed
2017-07-06 05:09:54.284455|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_io_lock+0x28e) [0x7fb2b15a970e]
2017-07-06 05:09:54.284467|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_ii_column_update+0x15c) [0x7fb2b158c3ec]
2017-07-06 05:09:54.284471|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_obj_default_set_value_hook+0x81) [0x7fb2b1466ca1]
2017-07-06 05:09:54.284474|e|30644: /usr/lib/x86_64-linux-gnu/libgroonga.so.0(grn_obj_set_value+0x1593) [0x7fb2b1479583]
2017-07-06 05:09:54.284477|e|30644: /usr/lib/postgresql/9.5/lib/pgroonga.so(+0xa0d9) [0x7fb2b1c520d9]
2017-07-06 05:09:54.284479|e|30644: /usr/lib/postgresql/9.5/lib/pgroonga.so(pgroonga_insert+0x5d) [0x7fb2b1c57cbd]
2017-07-06 05:09:54.284482|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(FunctionCall6Coll+0xa6) [0x55c2911f6336]
2017-07-06 05:09:54.284485|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(index_insert+0xa5) [0x55c290edd7e5]
2017-07-06 05:09:54.284488|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecInsertIndexTuples+0x2cc) [0x55c290ff254c]
2017-07-06 05:09:54.284491|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x20bc28) [0x55c29100dc28]
2017-07-06 05:09:54.284493|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecModifyTable+0x3ff) [0x55c29100e34f]
2017-07-06 05:09:54.284496|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(ExecProcNode+0x188) [0x55c290ff6308]
2017-07-06 05:09:54.284499|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(standard_ExecutorRun+0x132) [0x55c290ff31f2]
2017-07-06 05:09:54.284502|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x2fb509) [0x55c2910fd509]
2017-07-06 05:09:54.284504|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(+0x2fb751) [0x55c2910fd751]
2017-07-06 05:09:54.284507|e|30644: postgres: postgres kris 127.0.0.1(47088) UPDATE(PortalRun+0x1c5) [0x55c2910fe365]
2017-07-06 05:09:54.681939|n|30644: grn_fin (0)
OK cases:
\App\Post::whereRaw('body @@ ?', $query);
\App\Post::whereRaw('body @@ :query', ['query'=>$query]);
\App\Post::whereRaw('body @@ \'Groonga OR PGroonga\'');
NG case:
\App\Post::whereRaw('body &? ?', $query);
\App\Post::whereRaw('body &? :query', ['query'=>$query]);
\App\Post::whereRaw('body &? \'Groonga OR PGroonga\'');
\App\Post::whereRaw('body &?? ?', $query);
\App\Post::whereRaw('body &\? ?', $query);
\App\Post::whereRaw('body &\\? ?', $query);
\App\Post::whereRaw('body &\\\? ?', $query);
\App\Post::whereRaw('body &\\\\? ?', $query);
Error message example, for \App\Post::whereRaw('body &? :query', ['query'=>$query]); case:
SQLSTATE[HY093]: Invalid parameter number: mixed named and positional parameters (SQL: select * from "posts" where body &与えられたクエリ文字列 :query order by "id" desc)
Basically PDO (PHP Data Object) cannot treat ? as non-placeholder character.
(See also: https://stackoverflow.com/questions/16311939/how-to-prevent-pdo-from-interpreting-a-question-mark-as-a-placeholder and https://github.com/php/php-src/blob/master/ext/pdo/pdo_sql_parser.c#L54 )
Larabel is built on the PDO so it is impossible to use &? operator on Larabel.
We need alternative operator &@@ or something without the character ?.
If we use the the test case from tutorial:
CREATE TABLE score_memos (
id integer PRIMARY KEY,
content text
);
CREATE INDEX pgroonga_score_memos_content_index
ON score_memos
USING pgroonga (id, content);
INSERT INTO score_memos VALUES (1, 'PostgreSQL is a relational database management system.');
INSERT INTO score_memos VALUES (2, 'Groonga is a fast full text search engine that supports all languages.');
INSERT INTO score_memos VALUES (3, 'PGroonga is a PostgreSQL extension that uses Groonga as index.');
INSERT INTO score_memos VALUES (4, 'There is groonga command.');
SET enable_seqscan = off;
SELECT * FROM score_memos WHERE id in (1, 2, 3, 4);
We will see 0 row returns
I see follow error(I see it with all versions since 1.1.4, and I didn't check earlier versions):
ERROR: could not read block 0 in file "base/37488798/37492018": read only 0 of 8192 bytes
CONTEXT: SQL statement "SELECT unicode0 as unicode, pinyin0 as pinyin FROM csp_appdata.csputfpinyinmapping WHERE unicode0 IN (SELECT unnest(str_arr))"
PL/pgSQL function public.csp_convert_to_pinyin(text) line 14 at FOR over SELECT rows
STATEMENT: CREATE INDEX csputfpinyinmapping_createdbyentityid_f ON csputfpinyinmapping USING btree (public.csp_convert_to_pinyin(createdbyentityid));
ERROR: could not read block 0 in file "base/37488798/37492026": read only 0 of 8192 bytes
CONTEXT: SQL statement "SELECT unicode0 as unicode, pinyin0 as pinyin FROM csp_appdata.csputfpinyinmapping WHERE unicode0 IN (SELECT unnest(str_arr))"
PL/pgSQL function public.csp_convert_to_pinyin(text) line 14 at FOR over SELECT rows
STATEMENT: CREATE INDEX csputfpinyinmapping_unicode0_f ON csputfpinyinmapping USING btree (public.csp_convert_to_pinyin(unicode0));
Really, they are NOT there:
yongxiang-MacBook-Pro:csp_platform_gyx-master-1214 yongxiang.gao$ ls /usr/local/var/postgres/base/37488798/3749201
37492010 37492011 37492012 37492013 37492014 37492015 37492016 37492017 37492019
yongxiang-MacBook-Pro:csp_platform_gyx-master-1214 yongxiang.gao$ ls /usr/local/var/postgres/base/37488798/3749202
37492020 37492021 37492022 37492023 37492024 37492025 37492027 37492028 37492029
When I query url @~ 'http://'
It does not ouput any debug message.
table create
create table urls(id serial,url text);
\copy urls(url) from '/path/to/url_list'
create index ix_url_urls on urls using pgroonga(url) with(tokenizer='TokenRegexp');
output debug message(OK)
set pgroonga.log_level = 'debug';
SET enable_seqscan = off;
select * from urls where url like 'http://%';
2015-12-16 17:32:03.388254|n| grn_init: <5.1.0>
2015-12-16 17:32:03.388394|n| vm.overcommit_memory kernel parameter should be 1: <0>: See INFO level log to resolve this
2015-12-16 17:32:03.388406|i| Some processings with vm.overcommit_memory != 1 may break DB under low memory condition.
2015-12-16 17:32:03.388411|i| To set vm.overcommit_memory to 1
2015-12-16 17:32:03.388415|i| add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and restart your system or
2015-12-16 17:32:03.388419|i| run 'sudo /sbin/sysctl vm.overcommit_memory=1' command.
2015-12-16 17:32:03.408179|i| [object][search][index][key][exact] <Lexicon76021_0.index>
2015-12-16 17:32:03.408210|i| grn_ii_sel > (http://)
2015-12-16 17:32:03.408315|i| n=4 (http://)
2015-12-16 17:32:03.408626|i| exact: 66
2015-12-16 17:32:03.408643|i| hits=66
2015-12-16 17:32:03.417420|n| grn_fin (0)
output debug message(NG)
set pgroonga.log_level = 'debug';
SET enable_seqscan = off;
select * from urls where url @~ 'http://';
2015-12-16 17:32:17.338293|n| grn_init: <5.1.0>
2015-12-16 17:32:17.338348|n| vm.overcommit_memory kernel parameter should be 1: <0>: See INFO level log to resolve this
2015-12-16 17:32:17.338352|i| Some processings with vm.overcommit_memory != 1 may break DB under low memory condition.
2015-12-16 17:32:17.338354|i| To set vm.overcommit_memory to 1
2015-12-16 17:32:17.338356|i| add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and restart your system or
2015-12-16 17:32:17.338365|i| run 'sudo /sbin/sysctl vm.overcommit_memory=1' command.
2015-12-16 17:32:17.352145|n| grn_fin (0)
CentOS 6
PostgreSQL 9.3.17
PGroonga 1.2.3
Groonga 1.3.0-1
groonga-devel, groonga-libs 7.0.4-1.el6
上記の環境で、PRIMARY KEYがTEXTのみのとき、pgroonga.score()が常に0を返します。
簡単に試した限りでは、検索結果そのものには影響がないようでした。
Verified environments:
Steps to reproduce:
vagrant up for a clean environment.
Install PostgreSQL from PostgreSQL's repository. Go to https://www.postgresql.org/download/linux/redhat/ and choose PostgreSQL 9.6, CentOS 6, i386 (or x86_64).
# yum install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-centos96-9.6-3.noarch.rpm
# yum install postgresql96
# yum install postgresql96-server
# service postgresql-9.6 initdb
# chkconfig postgresql-9.6 on
# service postgresql-9.6 start
Install sources: yum install postgresql96-devel
Install Groonga: http://groonga.org/docs/install/centos.html#centos-6
Build PGroonga: http://pgroonga.github.io/install/source.html
Expected result, on x86_64:
PATH=/usr/pgsql-9.6/bin:$PATH make finishes successfully.
Actual result, on i386:
PATH=/usr/pgsql-9.6/bin:$PATH make failes with the error message:
[root@localhost pgroonga-1.2.3]# PATH=/usr/pgsql-9.6/bin:$PATH make
make -f pgroonga.mk all
make[1]: ディレクトリ `/home/vagrant/pgroonga-1.2.3' に入ります
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -DLINUX_OOM_SCORE_ADJ=0 -Ivendor/xxHash -DPGRN_VERSION="\"1.2.3\"" -fpic -I/usr/include/groonga -I. -I./ -I/usr/pgsql-9.6/include/server -I/usr/pgsql-9.6/include/internal -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include -c -o src/pgroonga.o src/pgroonga.c
/usr/pgsql-9.6/include/server/postgres.h:47 から include されたファイル中,
src/pgroonga.h:3 から,
src/pgroonga.c:1 から:
/usr/pgsql-9.6/include/server/c.h:53:23: error: pg_config.h: そのようなファイルやディレクトリはありません
Hi, I use postgres 9.6beta3 and get error:
pgroonga: failed to create database: <pg_tblspc/0/PG_9.6_201607071/0/pgrn>: system call error: No such file or directory: failed to open file info path: <pg_tblspc/0/PG_9.6_201607071/0/pgrn>
CONTEXT: parallel worker
ERROR: server conn crashed?
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
)
my postgres settings
max_parallel_workers_per_gather = 4
parallel_tuple_cost = 0
Seems like this bug dependent to new parallel sequence scan feature introduced in 9.6.
my docker-compose.yml
postgresqlmaster:
image: postgres:9.6-beta3
privileged: true
environment:
- POSTGRES_PASSWORD=database
- POSTGRES_USER=database
- POSTGRES_DB=database
ports:
- "5432:5432"
With postges 9.5 all is ok.
Hello.
I was wondering if you could provide us.
Windows 64bit + PostgreSQL 9.6.2-3
Regular expression pattern should use normalized characters. Normalization rule has Groonga. It's better that the same normalization logic is used both in user program and Groonga.
PGroonga should provide pgroonga_normalize() function that normalizes the given text by Groonga's normalization rule. User programs should use the function to get normalized text.
次の説明をREADMEに追加するとわかりやすくて良い。( 関連 #8 )
create extension pgroongaを実行した場合
\dnで確認できる。grant ALL on SCHEMA pgroonga to username;で指定したユーザにスキーマのアクセス権限を付与できる。
CREATE INDEX grnindex ON memos USING pgroonga (content pgroonga.text_regexp_ops);のようなコマンドを実行できる。pgroonga_test=# create extension pgroonga;
CREATE EXTENSION
pgroonga_test=# \dn
List of schemas
Name | Owner
----------+----------
pgroonga | postgres
public | postgres
(2 rows)
pgroonga_test=# grant ALL on SCHEMA pgroonga to user;
GRANT余談
create database groonga_test owner xxxとして所有者を変更しているが、所有者に対してもgrant allしないとスキーマの権限が付与できないNeed this URL link on pgroonga pages.
http://groonga.org/docs/reference/tuning.html
I found this log in pgroonga.log
2015-12-15 15:28:05.117397|n| vm.overcommit_memory kernel parameter should be 1: <0>: See INFO level log to resolve this
When trying to create extension pgroonga on Ubuntu Trusty using the official PPA, I get this error:
ERROR: could not load library "/usr/lib/postgresql/9.3/lib/pgroonga.so": /usr/lib/postgresql/9.3/lib/pgroonga.so: undefined symbol: PGrnJSONBLookupPathsTable
ERROR: schema "pgroonga" does not exist
@kou this is breaking the Zulip CI -- is this something you can do a release to fix in the next day? Otherwise we'll probably have to remove pgroonga support from Zulip.
(As a sidenote, I encourage you folks to setup CI installing your packages on the various Ubuntu releases you support to avoid regressions like this)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.