I realize there is some refactoring going on so this might be out of date or you might already be aware of it.
The signature of the method HiveSession.getTables() looks like:
IHiveSession.getTables(request: TablesRequest): Promise<IOperation>
... after all the handling is done the response looks something like this:
[
{
TABLE_CAT: 'hive_metastore',
TABLE_SCHEM: 'sqltools_databricks_driver',
TABLE_NAME: 'parent',
TABLE_TYPE: '',
REMARKS: 'UNKNOWN',
TYPE_CAT: null,
TYPE_SCHEM: null,
TYPE_NAME: null,
SELF_REFERENCING_COL_NAME: null,
REF_GENERATION: null
},
{
TABLE_CAT: 'hive_metastore',
TABLE_SCHEM: 'sqltools_databricks_driver',
TABLE_NAME: 'parent_view',
TABLE_TYPE: '',
REMARKS: 'UNKNOWN',
TYPE_CAT: null,
TYPE_SCHEM: null,
TYPE_NAME: null,
SELF_REFERENCING_COL_NAME: null,
REF_GENERATION: null
}
]Notice that both of the returned objects have TABLE_TYPE: ''
I am calling it something like the following.
const tables = await session.getTables(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableTypes: ['TABLE'],
}
);
await handle(tables);
const views = await session.getTables(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableTypes: ['VIEW'],
}
);
await handle(views);I notice it makes no difference if I use a value in the tableTypes or if I leave that out completely, note that it is optional. The docs / code as inherited from the Hive project sadly don't explain the usage here- if it is supposed to filter or not. I assume that is its purpose.
export declare type TablesRequest = {
catalogName?: string;
schemaName?: string;
tableName?: string;
tableTypes?: Array<string>;
};I also note that the getTableTypes() method returns this, so I am confident that 'TABLE' or 'VIEW' are the correct values here:
[ { TABLE_TYPE: 'TABLE' }, { TABLE_TYPE: 'VIEW' } ]
side note
I was hoping to find some type definitions of these various responses, if they exist. I would appreciate being pointed
to them, otherwise I am recreating them just through testing them and looking at the responses. There is a fair bit of
indirection in the codebase and the method that eventually runs has an any return type (sadface)
OperationResult.getValue(): any
My main task here is to extract some of these fields and use in a different library's interface so I need to know what types to expect.
for example getColumns() returns some nulls and I wonder if these should be union types
type columnResponse = [
{
TABLE_CAT: string // 'hive_metastore',
TABLE_SCHEM: string // 'sqltools_databricks_driver',
TABLE_NAME: string // 'parent',
COLUMN_NAME: string // 'id',
DATA_TYPE: number // 4,
TYPE_NAME: string //'INT',
COLUMN_SIZE: number // 4,
BUFFER_LENGTH: any // null,
DECIMAL_DIGITS: number // 0,
NUM_PREC_RADIX: number // 10,
NULLABLE: number // 0,
REMARKS: string // '',
COLUMN_DEF: any // null,
SQL_DATA_TYPE: any // null,
SQL_DATETIME_SUB: any // null,
CHAR_OCTET_LENGTH: any // null,
ORDINAL_POSITION: 0,
IS_NULLABLE: string // 'YES',
SCOPE_CATALOG: any // null,
SCOPE_SCHEMA: any // null,
SCOPE_TABLE: any // null,
SOURCE_DATA_TYPE: any // null,
IS_AUTO_INCREMENT: string // 'NO'
}
]
Reproducing the issue
To quickly repro this I did something similar to the E2E test, but I've added a .env file and using the dotenv package to autoconfigure them.
SQLTOOLS_DATABRICKS_HOST='**********.cloud.databricks.com'
SQLTOOLS_DATABRICKS_PATH='/sql/1.0/endpoints/************'
SQLTOOLS_DATABRICKS_TOKEN='***********TOKENMCTOKENFACE****'
*test script, just running this like npx ts-node test_script.ts
import { DBSQLClient } from '@databricks/sql';
import IOperation from '@databricks/sql/dist/contracts/IOperation';
import dotenv from 'dotenv';
const utils = DBSQLClient.utils;
const client = new DBSQLClient();
dotenv.config();
const TARGET_DATABASE = 'hive_metastore';
const TARGET_SCHEMA = 'sqltools_databricks_driver';
const TARGET_TABLE = 'parent';
const TARGET_VIEW = 'parent_view';
type catalogsResponse = [
{
TABLE_SCHEM: string
}
];
type schemasResponse = [
{
TABLE_SCHEM: string
TABLE_CATALOG: string
}
];
type tableTypesResponse = [
{
TABLE_TYPE: string
}
];
type tablesResponse = [
{
TABLE_CAT: string // 'hive_metastore',
TABLE_SCHEM: string // 'sqltools_databricks_driver',
TABLE_NAME: string // 'parent',
TABLE_TYPE: string // '', // why is this blank?
REMARKS: string // 'UNKNOWN',
TYPE_CAT: string // null,
TYPE_SCHEM: string // null,
TYPE_NAME: string // null,
SELF_REFERENCING_COL_NAME: string // null,
REF_GENERATION: string // null
}
];
type columnResponse = [
{
TABLE_CAT: string // 'hive_metastore',
TABLE_SCHEM: string // 'sqltools_databricks_driver',
TABLE_NAME: string // 'parent',
COLUMN_NAME: string // 'id',
DATA_TYPE: number // 4,
TYPE_NAME: string //'INT',
COLUMN_SIZE: number // 4,
BUFFER_LENGTH: any // null,
DECIMAL_DIGITS: number // 0,
NUM_PREC_RADIX: number // 10,
NULLABLE: number // 0,
REMARKS: string // '',
COLUMN_DEF: any // null,
SQL_DATA_TYPE: any // null,
SQL_DATETIME_SUB: any // null,
CHAR_OCTET_LENGTH: any // null,
ORDINAL_POSITION: 0,
IS_NULLABLE: string // 'YES',
SCOPE_CATALOG: any // null,
SCOPE_SCHEMA: any // null,
SCOPE_TABLE: any // null,
SOURCE_DATA_TYPE: any // null,
IS_AUTO_INCREMENT: string // 'NO'
}
]
type response = (
catalogsResponse | schemasResponse | tableTypesResponse |
tablesResponse | columnResponse
);
async function handle(queryOperation: IOperation, logResult: boolean = true): Promise<response> {
await utils.waitUntilReady(queryOperation, false, () => {});
await utils.fetchAll(queryOperation);
await queryOperation.close();
const result = utils.getResult(queryOperation).getValue();
if (logResult) {
console.log(result);
}
return Promise.resolve(result);
};
client
.connect({
host: `${process.env.SQLTOOLS_DATABRICKS_HOST}`,
path: `${process.env.SQLTOOLS_DATABRICKS_PATH}`,
token: `${process.env.SQLTOOLS_DATABRICKS_TOKEN}`,
})
.then(async (client) => {
const session = await client.openSession();
const createSchema = await session.executeStatement(`create schema if not exists hive_metastore.sqltools_databricks_driver`, {runAsync: true});
await handle(createSchema, false);
const createTable = await session.executeStatement(`
create or replace table hive_metastore.sqltools_databricks_driver.parent (
id int not null,
name string not null,
desc string not null
)`
,{runAsync: true}
);
await handle(createTable, false);
const insertTable = await session.executeStatement(`
insert into hive_metastore.sqltools_databricks_driver.parent (id, name, desc)
values
(1, 'hey', 'yo'),
(1, 'whut', 'noway')`
,{runAsync: true}
);
await handle(insertTable, false);
const createView = await session.executeStatement(`
create or replace view hive_metastore.sqltools_databricks_driver.parent_view as
select * from hive_metastore.sqltools_databricks_driver.parent`
,{runAsync: true}
);
await handle(createView, false);
const databases = await session.getCatalogs();
await handle(databases);
const schemas = await session.getSchemas(
{ //schemaName: 'default', // don't provide this
catalogName: TARGET_DATABASE}
);
await handle(schemas);
const tabletypes = await session.getTableTypes();
await handle(tabletypes);
const tables = await session.getTables(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableTypes: ['TABLE'], // no MATERIALIZED_VIEW ...yet
}
);
await handle(tables);
const views = await session.getTables(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableTypes: ['VIEW'], // no MATERIALIZED_VIEW ...yet
}
);
await handle(views);
const table_columns = await session.getColumns(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableName: TARGET_TABLE,
//tableTypes: [TABLE, VIEW, MATERIALIZED_VIEW],
}
)
await handle(table_columns);
const view_columns = await session.getColumns(
{
catalogName: TARGET_DATABASE,
schemaName: TARGET_SCHEMA,
tableName: TARGET_VIEW,
//tableTypes: [TABLE, VIEW, MATERIALIZED_VIEW],
}
)
await handle(view_columns);
//Nonsense: compulsory functionName arg means this returns single function info
// not a list
//const functions = await session.getFunctions(
// {
// functionName: '',
// catalogName: 'hive_metastore', //samples
// schemaName: '',
// }
//)
//await handle(functions);
// requires unity catalog
// const primarykeys = await session.getPrimaryKeys(
// {
// schemaName: TARGET_SCHEMA,
// tableName: TARGET_TABLE,
// catalogName: TARGET_DATABASE, // - this is optional?? madness.
// }
// )
// await handle(primarykeys);
// requires unity catalog
// const foreignkeys = await session.getCrossReference(
// {
// parentCatalogName: '',
// parentSchemaName: '',
// parentTableName: '',
// foreignCatalogName: '',
// foreignSchemaName: '',
// foreignTableName: '',
// }
// )
// await handle(foreignkeys);
const cleanUp = await session.executeStatement(
`drop schema hive_metastore.sqltools_databricks_driver cascade`
,{runAsync: true}
);
await handle(cleanUp, false);
await session.close();
await client.close();
})
.catch((error) => {
console.log(error);
});
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent