A list of PrestoSQL/Trino resources
Unmerged Connectors (data sources):
Generic JDBC trinodb/trino#3105
Flexible (csv/excel/txt/raw/html/json/xml/word doc/powerpoint/pdf/outlook email/zip/gzip/bzip2 file from websites or local disk) snowlift/trino-storage#25
HTTP Rest API sources https://github.com/prestosql-rocks/presto-rest / https://github.com/cecoppinger/presto-rest / https://github.com/nineinchnick/trino-rest
Sybase (SAP) ASE & IQ trinodb/trino#3462 / trinodb/trino#2976
Teradata prestodb/presto#12078 / https://github.com/jmrozanec/presto-teradata-connector
DB2 https://github.com/IBM/presto-db2
Snowflake trinodb/trino#2551 / https://github.com/rahulbsw/trino-snowflake/tree/main/src/main/java/io/trino/snowflake / https://github.com/awslabs/aws-athena-query-federation/pull/454/files
Hbase https://github.com/harbby/presto-connectors/tree/master/presto-hbase / https://github.com/analysys/presto-hbase-connector / https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-hbase
Riak https://github.com/kuenishi/presto-riak
Vertica trinodb/trino#6134 / https://github.com/alexsumin/presto-vertica-connector / https://github.com/lev4ik/presto-vertica-connector / https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-vertica
Carbondata https://github.com/apache/carbondata/tree/master/integration/presto
SAP Hana https://github.com/qq5132834/presto-0.233-hana-connector
Salesforce trinodb/trino#2548
DynamoDB https://github.com/buremba/presto-dynamodb / https://github.com/Teradata/presto/tree/c125c3413c19bc6bb69e5f16e4c6f769062ff1cf/presto-dynamo / https://github.com/mugglmenzel/presto/tree/master/presto-dynamo / https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-dynamodb
Athena https://github.com/kutny/presto-athena/tree/master/src/main/java/com/facebook/presto/plugin/athena
DocumentDB https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-docdb
Timestream https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-timestream
Neptune https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-neptune
Sagemaker https://github.com/aws-samples/amazon-athena-train-amazon-sagemaker
S3 without Hive https://github.com/EMCECS/presto-s3-connector
Azure CosmosDB (supported by Mongo connector)
Azure SQL DB (supported by SQLServer connector)
Netezza https://github.com/aakashnand/trino-netezza / https://github.com/combineads/plugin-presto-netezza
Lucene https://github.com/totticarter/presto-lucene
Ignite trinodb/trino#8323 / https://github.com/prabhuom1/presto-ignite-connector-plugin / https://github.com/emhlbmc/presto-ignite
Databricks Spark Delta lake prestodb/presto#16843 / https://docs.delta.io/0.7.0/presto-integration.html
Azure Synapse https://docs.starburstdata.com/latest/connector/starburst-synapse.html
Pulsar trinodb/trino#8020 / https://github.com/apache/pulsar/tree/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto (inc. protobuf - https://github.com/apache/pulsar/pull/9841/files)
Ethereum https://github.com/xiaoyao1991/presto-ethereum
Cockroach trinodb/trino#8317
Excel https://github.com/phillip2019/trino-plugins/tree/main/spreadsheet-storage-handler
MapD https://github.com/NVIDIA/presto-mapd-connector
Arrow https://github.com/Praveen2112/presto/tree/arrow_connector/presto-arrow-flight/src/main/java/io/prestosql/plugin/arrow / https://github.com/Parth-Brahmbhatt/presto-1/tree/arrow-flight/presto-arrowflight/src/main/java/io/prestosql/plugin/arrowflight / https://github.com/koralium/Koralium/tree/master/presto/trino-connector-arrowflight
kdb https://github.com/sand-stone/dataswitch/tree/master/presto-kdb/src/main/java/kdb/presto / https://github.com/tuor713/trino-kdb
Kylin https://github.com/poiyyq/presto-kylin/tree/master/src/main/java/com/facebook/presto/plugin/kylin
Influx trinodb/trino#2397
SnappyData https://github.com/dawsongzhao/snappydata-presto-connector
HiveServer2 https://github.com/WeilerWebServices/Eventbrite/tree/master/presto/presto-hive-jdbc / https://github.com/leolorenzoluis/presto-csv-jdbc
Cloudwatch Logs https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-cloudwatch
Cloudwatch Metrics https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-cloudwatch-metrics
AWS Resource inventory CMDB https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-aws-cmdb
Obiba and REST https://github.com/obiba/presto-obiba
Manta https://github.com/joyent/presto-manta
Hazelcast https://github.com/ajermakovics/presto-hazelcast
TileDB https://github.com/TileDB-Inc/TileDB-Presto
Template https://github.com/jmrozanec/presto-template-connector
BioSamples https://github.com/EBIBioSamples/biosamples-presto-connector
FPGA https://github.com/supermt/presto-fpga-connector
Vitess https://github.com/vitessio/contrib/tree/master/presto-vitess-connector / https://github.com/yuokada/presto-vitess
Loki https://github.com/sdojjy/presto-loki/tree/master/src/main/java/io/prestosql/plugin/loki
LocalCSV https://github.com/dongqianwei/presto-localcsv
Spreadsheets https://github.com/fortitudetec/presto-plugins/tree/master/spreadsheet-storage-handler
AWS metadata? https://github.com/haitaoyao/presto-aws-plugin
Extract/Memory? https://github.com/yoloz/prestoSamples
Weiwodb/bytecode? https://github.com/photogamerun/weiwodb
Kubernetes https://github.com/xuxinkun/kubesql
TiDB/tikv https://github.com/zhihu/presto-connectors
Victoria Metrics prestodb/presto#13777
Pravega pravega/presto-connector#7
Twitter https://github.com/bitsondatadev/presto/tree/twitter-hacks / https://github.com/kokosing/trino-rest/tree/master/trino-rest-twitter/src/main/java/rocks/trino/rest/twitter
Apache DataSketches trinodb/trino#6643
Slack https://github.com/kokosing/trino-rest/tree/master/trino-rest-slack/src/main/java/rocks/trino/rest/slack / https://aws.amazon.com/blogs/big-data/create-a-custom-data-connector-to-slacks-member-analytics-api-in-amazon-quicksight-with-amazon-athena-federated-query/
Katta https://github.com/zhenqin/katta/tree/master/katta-presto/src/main/java/com/ivyft/katta/presto
Huawei Heti OpenLookEng (DataCenter/VDM) https://github.com/openlookeng/hetu-core
Yugabyte-db trinodb/trino#5708 / https://docs.yugabyte.com/latest/develop/ecosystem-integrations/presto/
Pixelsdb https://github.com/pixelsdb/pixels/tree/master/pixels-presto
Kairosdb https://github.com/xuhang1458/presto-kairosdb-connector
Apache IoTDB (Internet of Things) https://github.com/xuhang1458/presto-iotdb-connector
Git https://github.com/nineinchnick/trino-git
Rapid7 Armor https://github.com/rapid7/presto-armor-connector/tree/master/src/main/java/com/rapid7/presto/armor
Aerospike https://www.aerospike.com/docs/connect/access/presto/index.html
Hyena messaging daemon https://github.com/FCG-LLC/presto/tree/master/presto-hyena/src/main/java/co/llective/presto/hyena
EC2 instance https://github.com/haitaoyao/presto-aws-plugin/tree/master/src/main/java/presto/aws
AWS data (like EC2 instances or S3 buckets) https://github.com/nineinchnick/trino-cloud
Merged connectors (data sources):
Microsoft SQL Server
Oracle
Hive
PostgreSQL
MySQL
Amazon Redshift
MongoDB
Cassandra
BigQuery
Elasticsearch
Apache Kafka
Clickhouse
Druid
JMX
Kinesis
Phoenix
Google Sheets
Kudu
Redis
Thrift
Prometheus
Linkedin Pinot
Black Hole
Accumulo
Local File
Memory
MemSQL
System
TPCDS
TPCH
see https://trino.io/resources.html for up to date list
Supported file types in Hive Connector:
ORC
Parquet
Uber Hudi/hoodie (already in Hive connector)
Netflix Iceberg (already in Hive connector)
Avro
JSON (using org.apache.hive.hcatalog.data.JsonSerDe)
CSV (using org.apache.hadoop.hive.serde2.OpenCSVSerde)
TextFile
RCText (RCFile using ColumnarSerDe)
RCBinary (RCFile using LazyBinaryColumnarSerDe)
SequenceFile
XML (trinodb/trino#9219)
Supported filesystem storage for Hive Connector:
AWS S3 ("s3", "s3a")
Google Cloud Storage aka GCS ("gs")
Hadoop File System ("hdfs")
Windows Azure Storage Blob (WASB) ("wasb", "wasbs")
Azure Data Lake Storage (ADLS) ("adl")
Azure ADLS Gen2 - Azure Blob File System ("abfs", "abfss")
Aliyun Object Storage Service (OSS) - https://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html
Tencent Cloud Object Storage (COSN) - trinodb/trino#4978
IBM Cloud Object Storage (COS) - CODAIT/stocator#218 (comment) / https://docs.starburstdata.com/latest/connector/starburst-hive-ibm-cos.html
??? Oracle Cloud Infrastructure Object Storage (OCI) - oracle/oci-hdfs-connector#51
MinIO
Ceph
Dell EMC
Cloudian?
OpenIO?
SwiftStack / Stackpath / OpenStack?
Alibaba Cloud Object Storage Service?
DigitalOcean Spaces?
Huawei FusionStorage?
Baidu afs?
Qiniu Kodo?
Huawei Cloud OBS?
RackSpace Cloud files?
Vultr?
Pure Storage Flashblade?
Databricks file system?
viewfs?
Apache Ignite FS?
maprfs?
Clients
C#/.NET: https://github.com/koralium/EntityFrameworkCore.Presto
Perl: https://github.com/spiritloose/Net-Presto
PHP: https://github.com/360d-io-labs/PhpPrestoClient
Elixir: https://github.com/bbalser/ex_prestodb
Scala: https://github.com/nezihyigitbasi/presto-scala-client
C: https://github.com/easydatawarehousing/prestoclient/tree/master/C
Rust: https://github.com/nooberfsh/prusto
ODBC: Simba
Trino JDBC driver for Java/JVM
trino-go-client for Go
presto-client-node for Node.js
lento for Node.js
PyHive for Python
RPresto for R
trino-client-ruby for Ruby
Used in:
GUI: Hue, Sqlpad..etc
Google Cloud: Dataproc
Alibaba Cloud: Data Lake Analytics, E-MapReduce
Tencent Cloud: Elastic MapReduce
Huawei Cloud: MapReduce Service
Caching:
Data: Alluxio or Qubole Rubix
Directory/file listing
Metastore partitions
Performance tips:
Partitioning on VARCHAR columns (not high cardinality) that are included in WHERE of SELECTs
ANALYZE to gather statistics: https://prestosql.io/docs/current/optimizer/statistics.html
Compressed ORC/Parquet files with size between 64MB-1GB and the columns inside the files sorted so min/max ranges don't overlap for reduced network IO and better predicate pushdown (https://www.slideshare.net/databricks/the-parquet-format-and-performance-optimization-opportunities)
https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/
High CPU important for parallelism/concurrency
High Memory important for joins/group by/distinct
task.max-worker-threads
task.concurrency
Security:
LDAP/Kerberos/JWT/Cert/OAuth
TLS
Apache Ranger for column/table/schema/catalog level authorisation, column masking and row level filtering fine grained access
Notes:
Rarely achieve <400ms performance even for single table SELECTs/simple filters
Most useful feature is joining between data from any of the connectors (although if u want to create a view across them you need Hive Metastore).
Disclaimer: while Hive sources are suitable for data of large size (ie Petabytes) provided you have a lot of compute workers, JDBC sources are only performant on small tables. For example:
you have a 8 billion row txns table that is indexed/partitioned in Oracle.
Below query takes 38 seconds run in Oracle directly but 70 minutes when run in Presto -->
select custid, count(1) numtxns from oracle.txns
where txnmonth in (to_date('202002','YYYYMM'),to_date('201902','YYYYMM'))
group by custid
having count(1) > 4
Why Presto (hive connector) slower than pure MySQL (without presto) for small (ie <5GB) datasets?
remote storage ie s3 instead of local disk
limited caching - no data retained in memory for serving multiple clients running the same query
gathered stats like rowcount/# unique vals are not used to answer queries, just contribute to query plan decision
no indexes or primary keys, foreign keys
no histogram stats
too many small files
json rather than binary result format
query optimiser not as advanced, especially for OR conditions, functions or subqueries
partition pushdown might not work on timestamp/numeric columns
columnar, not row based. ie parquet is not suited to retrieving all columns for a few rows as it has to pivot the column reads into rows and scan a lot since rows are not stored contiguously
Missing in all JDBC connectors:
Complex filter pushdown trinodb/trino#7994 / trinodb/trino#402
ORDER BY pushdown trinodb/trino#8093
Aggregate pushdown trinodb/trino#6613
Join pushdown trinodb/trino#6620
DISTINCT pushdown trinodb/trino#4324
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent