FAILED_PRECONDITION: there was an error creating the session: the table has a storage format that is not supported about spark-bigquery-connector

[kmjung]

It looks like you are retrieving the anonymous table to which the query result was written, rather than setting the destination table when submitting the query for execution. All query results are written to tables in BigQuery, but the workaround is to specify an output table rather than letting the system create an anonymous table for you.

I don't know the BQQuerier object that you're using here, but at the BQ API level, this means using the jobs.insert API rather than jobs.query, and specifying the destination table in your JobConfigurationQuery.

[kmjung]

@sharadbhadouria the fix is being rolled out at the moment. I will update this bug when it's enabled globally.

[virender7]

Just to add the where clause works perfectly fine from BigQuery Query Editor UI. I tried this with another BigQuery table and the results were same.

[pmkc]

This is a KI with the read API, where the result of the query needs to be at least 10MB. We are working on fixing it on the backend when we do, it will work with no changes in the client.

I'll leave this open until it is fixed.

[virender7]

Thanks pmkc. Do you know (tentative date) by when it could be fixed?

[kmjung]

There is no tentative date for this work item at this point, although it's a high-priority item for us. I'll update this bug when we have something more to share here.

[sreenivasan-fathom]

Hi @kmjung Any updates on this, we want to query only a smaller subset (50 rows), but we are Unable to, because of this bug. Is there a work-around for this?

[kmjung]

As a short-term workaround, you can set a destination table on the original query and use the storage API to read from that table.

[sreenivasan-fathom]

@kmjung
We are currently setting a destination table

            bq = BQQuerier.instance()
            query_job = bq.get_bq_query_job(query=self._query)
            query_job.result() # execute query
            table = (f'{query_job.destination.dataset_id}.'
                     f'{query_job.destination.table_id}')

Can you please elaborate more on using storage API to read from the table? We are currently reading using Spark connector. Could you please give some references/ example of using Storage API to read from destination table?

        df = (spark
              .read
              .format('bigquery')
              .option('table', table)
      df = df.select(*self._columns).where(self._where)

[chile12]

The downside of this workaround is, of course, that you have to pay for storing these tables while the temp tables are without charge. So this issue is still of importance. To avoid creating tables for small result sets i added this function where the query is performed by Spark:

def selectFromBigQuery(query: String): DataFrame = {
    val logicalPlan = spark.sessionState.sqlParser.parsePlan(query)
    val allTables = logicalPlan.collectLeaves()
      .collect { case x: UnresolvedRelation => x }
      .map(_.tableIdentifier)
      .map(x => {
        val dataset = x.database.getOrElse(throw new IllegalArgumentException("A table id without a dataset id was encountered: " + x.table))
        TableId.of(parentProject, dataset, x.table)
      })
      .map { table =>
        val viewName = getSqlTableName(table).replace('.', '_') + "_view"
        readFromBigQueryPartition(table).createOrReplaceTempView(viewName) // we register the table as a temp view name
        table -> viewName
      }.toMap

    val transformed = logicalPlan.transformDown {
      case tab: UnresolvedRelation => // we replace the table name with the tem view name
        val tid = TableId.of(
          tab.tableIdentifier.database.getOrElse(throw new IllegalArgumentException("No dataset provided in addition to table name.")),
          tab.tableIdentifier.table)
        UnresolvedRelation(TableIdentifier(allTables.getOrElse(tid, throw new IllegalArgumentException), None))
    }
    val analyzed = spark.sessionState.analyzer.executeAndCheck(transformed)             // we have to resolve the plan first
    new org.apache.spark.sql.Dataset[Row](spark, analyzed, RowEncoder(analyzed.schema)) // now we can create a dataframe
  }

Which seems to work just fine.

[yan-hic]

@kmjung any update on the API fix ?
Not sure this is the right place to follow up on backend issues as many other libraries are impacted.

[kmjung]

We're targeting early Q2 2020 for a fix here.

[sharadbhadouria]

Hi @kmjung , is this still on your radar?

[emkornfield]

The fix has been rolled out globally now.