zonkyio / embedded-postgres-binaries Goto Github PK

View Code? Open in Web Editor NEW

126.0 126.0 25.0 402 KB

Lightweight bundles of PostgreSQL binaries with reduced size intended for testing purposes.

License: Apache License 2.0

Shell 100.00%

embedded-postgresql postgres postgresql postgresql-database

embedded-postgres-binaries's People

Contributors

Stargazers

Watchers

embedded-postgres-binaries's Issues

building an embedded binary with postgis - for Mac

Hi,

Wanted to ask whether you are planning to release binaries of embedded postgres with postgis extension? (following this PR)

I have tried to build it manually following the instructions on the project's README, however encountered an error I couldn't resolve.
Only then I noticed you mention on the README that "this option is not (yet) available for Windows and Mac OS platforms" - which probably explained why my manual build failed on my mac.

I really want to use your great embedded postgres library for my projects JUnit tests, however my projects rely heavily on postgis, and without having a proper binary I won't be able to utilize it :(

Can you assist/guide me how to solve it?

Thanks in advance!

Include psql and pg_restore executables

Would it be possible to include the psql and pg_restore executables in the bundle?

We are looking to put together some schema/data migration tests from know database states and are struggling to figure out how to seed the embedded database without the tools.

Thanks!

repack-postgres.sh does not use https

The repack-postgres.sh script downloads postgres binaries from http://get.enterprisedb.com, instead of https. This doesn't inspire much confidence in the integrity of the binary I'm running in my machine. Should probably be changed to use https.

releasing newer versions

Hi,

Do you have plans to release newer versions of the postgres binaries (10.19, 11.14, 12.9, 13.5, 14.1)? Especially 14.1 would be great as this contains 'fat' binaries for MacOS (IAW they run natively on the M1).

-h

Support PostgreSQL 13

https://www.postgresql.org/about/news/postgresql-13-released-2077

Support PostgreSQL 15

Please add support for PostgreSQL 15 as it's released already in stable version

https://www.postgresql.org/about/news/postgresql-15-released-2526/

Support Apple Silicon

I'm trying to use embedded-postgres-binaries via Rust's pg-embed crate on an Apple Silicon laptop. It looks like this requires a build for darwin/arm64v8 which does seem to exist. Would be great to have this!

need for pgcrypto in linux alpine binaries

I use embedded postgres to test my application and it works well on mac but not on linux alpine because pgcrypto is not available in this configuration.
What can I do to be able to use pgcrypto even in linux alpine ?

embedded postgres fail to detect m1 system or distribution then exits

macOs Monterey M1 pro
using arm64v8 platform deps

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //payments-rest/src/test:java/com/hellobrigit/payments/paymentmethods/PaymentMethodsFixturesStorageParityTest
-----------------------------------------------------------------------------
16:03:54.102 [main] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - Detected a Darwin x86_64 system
16:03:54.152 [main] INFO  i.z.t.d.p.e.DefaultPostgresBinaryResolver - Detected distribution: 'Unknown'
16:03:54.157 [main] INFO  i.z.t.d.p.e.DefaultPostgresBinaryResolver - System specific postgres binaries found: 'postgres-darwin-x86_64.txz'
16:03:54.342 [main] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - Extracting Postgres...
16:03:58.887 [main] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - Postgres binaries at /private/var/tmp/_bazel_emadjamal/9b9231e7f015c4d40b7cc7ea76c3f426/sandbox/darwin-sandbox/3722/execroot/__main__/_tmp/52a14f9afbb1f742b57dc3053da7964c/embedded-pg/PG-d95d5c5fccc1e1ef45de6533fd8e6d0a
16:04:03.099 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - The files belonging to this database system will be owned by user "emadjamal".
16:04:03.100 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - This user must also own the server process.
16:04:03.100 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 
16:04:03.100 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - The database cluster will be initialized with locale "C".
16:04:03.101 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - The default text search configuration will be set to "english".
16:04:03.101 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 
16:04:03.101 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - Data page checksums are disabled.
16:04:03.101 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 
16:04:03.101 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - fixing permissions on existing directory /private/var/tmp/_bazel_emadjamal/9b9231e7f015c4d40b7cc7ea76c3f426/sandbox/darwin-sandbox/3722/execroot/__main__/_tmp/52a14f9afbb1f742b57dc3053da7964c/epg2192929978199786820 ... ok
16:04:03.102 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - creating subdirectories ... ok
16:04:03.102 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - selecting dynamic shared memory implementation ... posix
16:04:03.255 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - selecting default max_connections ... 20
16:04:03.879 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - selecting default shared_buffers ... 400kB
16:04:03.879 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - selecting default time zone ... UTC
16:04:03.881 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - creating configuration files ... ok
16:04:03.917 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - running bootstrap script ... 2022-03-24 16:04:03.917 UTC [23667] FATAL:  could not create shared memory segment: Cannot allocate memory
16:04:03.917 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 2022-03-24 16:04:03.917 UTC [23667] DETAIL:  Failed system call was shmget(key=17634247, size=56, 03600).
16:04:03.917 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 2022-03-24 16:04:03.917 UTC [23667] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMALL parameter.  You might need to reconfigure the kernel with larger SHMALL.
16:04:03.917 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - 	The PostgreSQL documentation contains more information about shared memory configuration.
16:04:03.926 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - child process exited with exit code 1
16:04:03.926 [initdb:pid(23284)] INFO  i.z.t.d.p.embedded.EmbeddedPostgres - initdb: removing contents of data directory "/private/var/tmp/_bazel_emadjamal/9b9231e7f015c4d40b7cc7ea76c3f426/sandbox/darwin-sandbox/3722/execroot/__main__/_tmp/52a14f9afbb1f742b57dc3053da7964c/epg2192929978199786820"
java.lang.ExceptionInInitializerError
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at com.google.testing.junit.runner.PersistentTestRunner.getTestClass(PersistentTestRunner.java:206)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:142)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:85)
Caused by: java.lang.IllegalStateException: Process [/private/var/tmp/_bazel_emadjamal/9b9231e7f015c4d40b7cc7ea76c3f426/sandbox/darwin-sandbox/3722/execroot/__main__/_tmp/52a14f9afbb1f742b57dc3053da7964c/embedded-pg/PG-d95d5c5fccc1e1ef45de6533fd8e6d0a/bin/initdb, -A, trust, -U, postgres, -D, /private/var/tmp/_bazel_emadjamal/9b9231e7f015c4d40b7cc7ea76c3f426/sandbox/darwin-sandbox/3722/execroot/__main__/_tmp/52a14f9afbb1f742b57dc3053da7964c/epg2192929978199786820, -E, UTF-8] failed
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.system(EmbeddedPostgres.java:632)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.initdb(EmbeddedPostgres.java:252)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres.<init>(EmbeddedPostgres.java:159)
	at io.zonky.test.db.postgres.embedded.EmbeddedPostgres$Builder.start(EmbeddedPostgres.java:583)
	at io.zonky.test.db.postgres.embedded.PreparedDbProvider.createOrFindPreparer(PreparedDbProvider.java:84)
	at io.zonky.test.db.postgres.embedded.PreparedDbProvider.<init>(PreparedDbProvider.java:64)
	at io.zonky.test.db.postgres.embedded.PreparedDbProvider.forPreparer(PreparedDbProvider.java:59)
	at com.hellobrigit.storage.rest.backends.sql.BrigitDatabaseProvider.lambda$new$1(BrigitDatabaseProvider.java:37)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at com.hellobrigit.storage.rest.backends.sql.BrigitDatabaseProvider.<init>(BrigitDatabaseProvider.java:33)
	at com.hellobrigit.payments.paymentmethods.PaymentMethodsFixturesStorageParityTest.<clinit>(PaymentMethodsFixturesStorageParityTest.kt:672)
	... 5 more

BazelTestRunner exiting with a return value of 1
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2022-03-24 16:04:03 --

Postgres binary of version 14.10.0 does not work on Mac

As version 14.10.0 is the new default one since zonkyio/embedded-postgres#123, we noticed this issue.

The following error occurs:

/var/folders/s1/1jm7mb_j6jqgtq78f24h5b280000gp/T/embedded-pg/PG-f15761a1c59ff5f8349c12c01b3af539/bin/initdb -A trust -U postgres -D /var/folders/s1/1jm7mb_j6jqgtq78f24h5b280000gp/T/epg4423307102381004501 -E UTF-8
dyld[2295]: Library not loaded: @loader_path/../lib/libintl.8.dylib
  Referenced from: <2023A287-FF50-3D03-8AF9-D2783EDBED04> /private/var/folders/s1/1jm7mb_j6jqgtq78f24h5b280000gp/T/embedded-pg/PG-f15761a1c59ff5f8349c12c01b3af539/lib/libgssapi_krb5.2.2.dylib
  Reason: tried: '/private/var/folders/s1/1jm7mb_j6jqgtq78f24h5b280000gp/T/embedded-pg/PG-f15761a1c59ff5f8349c12c01b3af539/lib/../lib/libintl.8.dylib' (no such file), '/usr/lib/libintl.8.dylib' (no such file, not in dyld cache)

I've tried out versions 14.8.0, 14.9.0, 15.0.0, and 16.0.0 as well. Those seem to work fine.

Missing llvmjit_types.bc file

In our project we were trying to upgrade the binaries to the 12.1.0 version (the later ones don't work well on MacOS, I've seen this issue is already reported). Unfortunately we've encountered an error when running tests saying that llvmjit_types.bc file is missing:

Caused by: org.postgresql.util.PSQLException: ERROR: LLVMCreateMemoryBufferWithContentsOfFile(/private/var/folders/md/3xl8q9351yv675vqkxxz0jdrjmgpd2/T/embedded-pg/PG-abc4d7798f0045b30e0a18facc5799d0/lib/postgresql/llvmjit_types.bc) failed: No such file or directory at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)

The error is present only on MacOS. When we run the build on Linux (both locally and on CI) it doesn't happen. When we use the default version of the binaries from embedded-database-spring-test everything works just fine on all operating systems.

Our other dependencies are as following:

io.zonky.test:embedded-database-spring-test:1.5.5
io.zonky.test:embedded-postgres:1.2.8

Provide embedded postgres+postgis binaries

Would there be any interest in generating packages with the postgis extensions available?

I started investigating myself, but the current Gradle and Docker patterns doen't seem to lend themselves easily to extension. Ideally I'd expect additional artifacts to be built and extended with the postgis classifier, but as far as I can tell there's no clear way to reach this point without rewriting large swathes of the scripts.

I'd be happy to raise a PR for this if I could get a suggestion as to the desired approach!

MacOS arm64 binaries for 10, 11, 12, 13 do not contain arm64 binaries and require rosetta.

The following embedded-postgres-binaries-darwin-arm64v8-x.y.z.jar archives do not actually contain arm64v8 binaries:

(tested with

jar xf embedded-postgres-binaries-darwin-arm64v8-10.20.0.jar
tar xjf postgres-darwin-arm_64.txz
file bin/postgres

)

10.20.0 (contains x86_64 and i386)
11.15.0 (contains x86_64)
12.10.0 (contains x86_64)
13.6.0 (contains x86_64)

all released 14.x versions are ok (contain both x86_64 and arm64)

as a result, none of these binaries actually work on an arm64 mac unless rosetta is installed.

macOS Catalina and Mojave fails due to hardened rules

Since the last release, 9.6.17, i'm no longer able to use embedded-postgres. I get an IllegalStateException saying initDb failed:

27.03.2020 11:37:39.309 [main] WARN  o.s.w.c.s.GenericWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: 
org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: 
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: 
Failed to instantiate [liquibase.integration.spring.SpringLiquibase]: 
Factory method 'liquibase' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'dataSource': 
FactoryBean threw exception on object creation; nested exception is com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.IllegalStateException: 
Process [/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/epg17954755376373960693, -E, UTF-8] failed

Some more investigation reveals that initDb is indeed unable to run:

❯ /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb -A trust -U postgres -D /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/epg17954755376373960693 -E UTF-8
dyld: Library not loaded: @loader_path/../lib/libz.1.dylib
  Referenced from: /private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/lib/libxml2.2.dylib
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs
no data was returned by command ""/private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb".
Check your installation.

It seems like this issue: #17 seems to cause this problem since going back to 9.6.16 fixes the issue.

I'm running on MacOS 10.15.2 and java 11:

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

Add PostGIS 3.2 support

I know the documentation says only versions 2.5.2+, 2.4.7+, 2.3.9+ but I wanted to build with 3.2.4. When building with the -PpostgisVersion=3.2.4 flag, my build is failing with the following error:

configure: error: unable to find protobuf-c/protobuf-c.h using CPPFLAGS. You can disable MVT and Geobuf support using --without-protobuf

After doing some digging, it appears the following libraries are missing from the Docker image: protobuf-c-compiler libprotobuf-c-dev libprotobuf-dev. Adding these to the apt-get install and apk add lines in the build-postgres-debian.sh and build-postgres-alpine.sh seem to resolve this issue. I am able to successfully build PostGIS 3.2.4 with these changes.

Migrate from Travis CI to GitHub Actions

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

Make Darwin bundle fully self-contained by including libpq

I don't use your fork but it looks nice. I prepared a PR for the upstream but it looks like your fork has the same issue. Fortunately it's a one line fix.

Full details at opentable/otj-pg-embedded#135 but the gist is that bin/initdb links to the PG C client libpq but the repack scripts don't include that library. It mostly works because OSX comes with some version of libpq but the results aren't as embedded as you would like!

Missing libstdc++ and libgcc libraries in alpine binaries

The start of initdb process fails with the following error:

Error loading shared library libstdc++.so.6: No such file or directory (needed by /usr/local/pg-test/bin/../lib/libicui18n.so.58)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /usr/local/pg-test/bin/../lib/libicui18n.so.58)
Error loading shared library libstdc++.so.6: No such file or directory (needed by /usr/local/pg-test/bin/../lib/libicuuc.so.58)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /usr/local/pg-test/bin/../lib/libicuuc.so.58)
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating /usr/local/pg-test/bin/../lib/libicui18n.so.58: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found

Note that only alpine non-lite 10+ binaries are affected.

postgis support

Do you have plans to publish binaries with postgis embedded?

Working with uuid extension issue

Hello!

I've stuck with uuid extension issue. Adding uuid extension is not allowed. If your script has line like this
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
You've got the error. This is because there're no libossp-uuid.so.16 library in the bundled postgresql. Please, add this and it will work fine.

Include `pg_dump` and `pg_restore` into the packaged archives

As postgresql advances through major versions, it is impossible to use the data-dir from a previous version in the new version. Including the pg_dump and pg_restore will help in transferring data to the new version

Mac os 12.2 M1 Pro embedded version 14.2

Zonky fails to find the postgres command running under Rosetta 2:

:06.139 INFO [main] i.z.t.d.p.e.DefaultPostgresBinaryResolver System specific postgres binaries found: 'postgres-darwin-x86_64.txz'
22:37:06.340 INFO [main] i.z.t.d.p.e.EmbeddedPostgres Postgres binaries at /var/folders/3j/7r1wlvfx0pz4czr1bny8llg80000gp/T/embedded-pg/PG-a2a9bc65661eac6f108fc920268a87b3
22:37:06.414 INFO [initdb:pid(68833)] i.z.t.d.p.e.EmbeddedPostgres no data was returned by command ""/private/var/folders/3j/7r1wlvfx0pz4czr1bny8llg80000gp/T/embedded-pg/PG-a2a9bc65661eac6f108fc920268a87b3/bin/postgres" -V"
22:37:06.414 INFO [initdb:pid(68833)] i.z.t.d.p.e.EmbeddedPostgres The program "postgres" is needed by initdb but was not found in the
22:37:06.414 INFO [initdb:pid(68833)] i.z.t.d.p.e.EmbeddedPostgres same directory as "/private/var/folders/3j/7r1wlvfx0pz4czr1bny8llg80000gp/T/embedded-pg/PG-a2a9bc65661eac6f108fc920268a87b3/bin/initdb".
22:37:06.414 INFO [initdb:pid(68833)] i.z.t.d.p.e.EmbeddedPostgres Check your installation.
Exception

Support PostgreSQL 16

Upgrade to Gradle 6.8 (due to Bintray is shutting down)

More info: https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

zonkyio / embedded-postgres-binaries Goto Github PK

embedded-postgres-binaries's People

Contributors

Stargazers

Watchers

Forkers

embedded-postgres-binaries's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs