apsaradb / polardb-for-postgresql Goto Github PK

Industry: General
Function Expectations: statement-level timeout hints, for example
/*+ Set(statement_timeout "100ms") */ select * from abc where info like '%abcde%';

What problem is solved: avalanche prevention, specifying an upper limit on the execution time of some particular SQL
Alternative: Due to implementation issues, pg_hint_plan is currently unable to set a timeout on statement_timeout.

Update | delete the skip locked, improve batch concurrent operations (the current need to select for update the skip locked, then update. How many interactions are needed)

Global index of partitioned table

if can support helm to install k8s, we can fast experience PolarDB

I'm presuming that you may want to add the copyright and year as part of the LICENSE file, line 190.

Industry: Internet industry
Function Expectations:
The ability to limit flow based on SQL characteristic rules.
Implement at the hook level.
Timeout is only supported at user, db, and global statement levels. Large granularity.
Hope to be able to configure SQL, QPS to achieve.
SQL1: QPS < 10000,
SQL2: QPS < 1000

Scene:
Prevent avalanches.
Prevent some businesses or individuals from exhausting resources by submitting some SQL.

Logical replication supports DDLs

I followed the README, but pgxc_ctl binary was not generated.
So I took advice from others, run "make install to generate pgxc_ctl binary file in PGXC_CTL_HOME".

It does generate pgxc_ctl binary.
But if you use it, you will get an error:

/usr/bin/bash
/home/postgres/polardb/polardbhome/bin/pgxc_ctl is not directory. Check your configurfation

Industry: Internet
Expectations: PG_BASEBACKUP is capable of multi-threaded backups as well as backups in standby libraries
What problems to solve: First of all, multi-threaded backup can play a very good role in speeding up large databases, which can effectively reduce the burden of the primary database and reduce the occurrence of exceptions by performing backups from the standby database
Expected function rating: there must be use
Database pain point: now with more than 2T instances, single-threaded backups can take several hours and can only be done in the primary database, greatly increasing the risk of the primary database. After checking the current mainstream backup tools (Barman, bgbackreset), they are all based on pg_basebackup for development and upgrade. The principle is basically the same, need to execute pg_start_backup and pg_stop_backup, these two commands can only be executed in the main database.

Index scan sample method, corresponding to table sample

There is a configure error : header file <Python.h> is required for Python.

But if run " ./configure --prefix=/home/postgres/polardb/polardbhome ", it's ok.

I tried to install python, it didn't work.

https://github.com/alibaba/PolarDB-for-PostgreSQL/blob/master/doc/polardb/arch.md

请问目前是否有交流社区或者交流群。我在搭建过程中，使用pgxc_ctl -c $HOME/polardb/polardb_paxos.conf init all初始化集群时出现以下问题，是我需要自己安装pg服务吗？

在拉取了最新代码，执行./build.sh进行编译，然而并不能正常编译通过，我不知道你们是怎么编译通过的。
gcc version 10.2.0
cmake version 3.19.6

细节请看：

https://asciinema.org/a/GUXq3STFScbiUDTPy4avaMODk

类似
/root/PolarDB-for-PostgreSQL/src/backend/polar_dma/libconsensus/dependency/easy/src/thread/easy_uthread.c:266:17: error: ‘SIG_BLOCK’ undeclared (first use in this function)
这种，应该是你们编码错误，定义 SIG_BLOCK 的头文件<signal.h>都没 include 吧

这个错误我猜应该是用到 SIG_BLOCK 的源文件中没 include，而是隐藏在其它3方库里include了，而我这里恰恰没有这个3方库。但是不得不说，这是一个不好的编码习惯。比如
#include <iostream>
之后用到了std::string，不包含 <string> 也是能用 std::string 的，但这不是个好习惯，不能依赖间接包含，何况你这是间接依赖了一个3方库中的 include。

出错的地方有点多，麻烦看bash录制的视频。

希望你们能弄的健壮一点，像pg那样一次编译通过，无需折腾

SQL syntax support delete | update limit, is suitable for batch operation

Undo table access method engine, reduce bloat

Industry: Traditional

Functional Expectations: After the logon failure exceeds the limits, lock the user

What issues to solve: Improve security, prevent brute force attacks, and be compatible with Oracle

Alternative: auth_delay

Development Tip: Login Hook?

SQL syntax: Support to bypass the shared buffer SELECT and INSERT. Advantage: Bursting large table queries does not affect hot data in the shared buffer

When initializing the database, starting the database reported an error that the file could not be accessed.But the file does exist, and the user is authorized.

如下是路径所在目录：

The "main" branch generated by Github is useless since you have "master" branch.

38/5000
Index Skip Scan Optimizer, which currently requires CTE recursive simulation

File level migration capability, support DATAFILE + WAL incremental migration, improve migration speed.
Products currently supported on the market:
postgrespro: pg_transport
aws: pg_transport

Rotate Table support, by specifying the table size upper limit, the number of records upper limit, LILO, FIFO and other configable data aging algorithm, automatically aging data.

postgresql社区不缺分布式集群部署机制。

If you open-source a project, then ensure that everyone is able to participate.
English is spoken by everyone in the open-source communities.
Either accept this fact and ensure that every communication is in English AND NOT in Chinese.

如果你開源一個項目，那麼確保每個人都能參與。
開源社區中的每個人都說英語。
要么接受這個事實，並確保每一次交流都是用英語而不是中文，要么把你的項目移到不同的地方。

like the difference between polardb and pg....

I am trying to compile a fresh checkout on Ubuntu 20.10 and get this error while linking:

/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `ENGINE_cleanup'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `SSL_get_ex_new_index'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `SSLv23_method'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `ERR_free_strings'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `SSL_load_error_strings'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `SSL_library_init'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `OPENSSL_add_all_algorithms_noconf'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `EVP_cleanup'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `CRYPTO_set_id_callback'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `CRYPTO_set_locking_callback'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `CRYPTO_num_locks'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `CRYPTO_cleanup_all_ex_data'
/usr/bin/ld: ../../src/backend/polar_dma/lib//libpolar_consensus_wrapper.so: undefined reference to `CRYPTO_mem_leaks_fp'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:71: postgres] Error 1
make[2]: Leaving directory '/home/ajung/src/PolarDB-for-PostgreSQL/src/backend'
make[1]: *** [Makefile:42: all-backend-recurse] Error 2
make[1]: Leaving directory '/home/ajung/src/PolarDB-for-PostgreSQL/src'
make: *** [GNUmakefile:11: all-src-recurse] Error 2

Industry: General
Expected function :pg_stat_statements provides metrics for P99, P95
What problem to solve: Monitor the execution stability of PG statements as a reference for business metrics.

Multi-tenancy function, time based dynamic configuration quota.

Storage quota function (table level, user level, schema level, database level)
Resource isolation (session level, user level, DB level, schema level)

for exp:
08:00-23:00 for oltp resource management configure.
23:00-08:00 for OLAP resource management configure.

Divide WAL into Database and implement Database Pluggable function in the future

I've tried to build PolarDB on MacOS but failed because of the linking problems with pre-compiled DMA libraries (for Linux I suppose). Can you replace it with the source code and fix the Makefile (at the moment it simply copies the binaries)?

Wal log writing in parallel, currently a large amount of data import, there may be two performance bottlenecks, datalbock extend exclusive lock, or Wal insert exclusive lock

Online DDL support, some DDL need rewrite table, support PG_REPACK 2-stage File Replace capability, reduce the lock table time of such DDL operations.

环境：centos 7.6.1810
问题：编译安装执行 ./onekey.sh all时，报错如下：

/home/polardb/PolarDB-for-PostgreSQL-master/src/backend/polar_dma/libconsensus/polar_wrapper/polar_consensus_log.cc:21:19: fatal error: paxos.h: No such file or directory
 #include "paxos.h"

环境：centos 7.6.1810
问题：编译安装执行 ./onekey.sh all时，报错如下：

checking which semaphore API to use... unnamed POSIX
checking for /dev/urandom... yes
checking which random number source to use... /dev/urandom
checking Python.h usability... no
checking Python.h presence... no
checking for Python.h... no
configure: error: header file <Python.h> is required for Python

Index partition (not partition table index)

Does PolarDB support failover slot such as Aliyun RDS for PostgreSQL?

Materialized view based on mv log, At present, MV does not support MV Log. To refresh MV requires a join comparison between MV Log and the base table, which may cost a lot when refreshing

Industry: General
Function Expectations:
Database, table, schema-level snapshot, time point clone, time point flash back function.

Scene:
In the enterprise PaaS service scenario, a different table or schema or database for each database corresponds to an enterprise in the software service, and when the enterprise's data needs to be rolled back, the entire database instance does not need to be rolled back.

Block level compression

The Global Cache, which has a large number of objects in a large number of partitioned tables or PaaS user scenarios, currently PG uses the session-level Catalog Cache, which can result in a large amount of object metadata memory footprint, resulting in a large amount of wasted memory.

Industry: General
Expectations: Configurable connection life cycles, such as a connection that lives for a few minutes and is released immediately when it is idle (actively disconnecting from the client)
In some PaaS services, the longer a connection lives, the more PG metadata accessed, the more local memory in the session (will not be freed), and the more connections will cause OOM. Global Cache can also solve this kind of problem.

大概看了一下相对熟悉的几个folder的code，除了distributed_txn，polar_datamax，polar_dma这三个目录，其它的目录是不是基本就是pg的code，没有改动？

I want to build an HA cluster with three physical servers. Is the configuration correct?DDB1 dDB2 DDB3 is the host map configured to /etc/hosts.
polardb_paxos.txt