googlecloudplatform / bigquery-geo-viz

Visualize Google BigQuery geospatial data using Google Maps Platform APIs

BigQuery Geo Viz

Web tool and developer example for visualization of Google BigQuery geospatial data using Google Maps Platform APIs.

Getting started

• Tool: https://bigquerygeoviz.appspot.com/
• Documentation: https://cloud.google.com/bigquery/docs/gis-analyst-start

Development

Quickstart

# Start a dev server at http://localhost:4200/.
npm run dev

# Run unit tests with Karma.
npm test

Resources

• Google Maps JavaScript API documentation
• Google BigQuery REST API documentation
• Angular
• D3.js
• TypeScript

Terms & privacy

This tool is provided as a reference implementation of Google Maps and geospatial BigQuery API usage capabilities. It may be useful as a debugging and visualization resource. It is not an officially supported Google product and is provided without guarantees of maintenance.

• Google Terms of Service
• Google Cloud Platform Terms of Service
• Privacy

bigquery-geo-viz's Issues

FR: show color gradient scale

When data-dependent color fill is used, it would be
great to have an option to show color gradient scale,
displaying gradient plus min/max and maybe a few
intermediate values of the column driving coloring.

circleRadius data-driven not working

Following this tutorial in the BigQuery GIS documentation;
https://cloud.google.com/bigquery/docs/gis-tutorial-hurricane

In the "To format your map" section, step 9, the circleRadius data-driven formatting is not styling the points. Expect point radius to change based on the data as per the tutorial.

Disable tilt

When zoomed in sufficiently, such as with a tightly collocated result set, maps api may enable 45° tilt and hide the overlay. We should disable that: https://developers.google.com/maps/documentation/javascript/maptypes#Enabling45DegreeImagery

Query hangs if geom is null

The app just hangs as the query doesn't return. I have to reload the page and start over.
The app gets to prequery OK.

The issue occurs when the app detects a geometry column, but the geometry of a row evaluates to null.

Console printout:

TypeError: feature.geometry is null
map.component.ts:132:6
    updateFeatures map.component.ts:132
    forEach self-hosted:262
    updateFeatures map.component.ts:131
    set map.component.ts:67
    Angular 6
    View_MainComponent_0 MainComponent.html:16
    Angular 5
    query main.component.ts:242
    r2 cb=gapi.loaded_0:127
    xs cb=gapi.loaded_0:130
    Wq cb=gapi.loaded_0:130
    uea cb=gapi.loaded_0:130
    Ap cb=gapi.loaded_0:122
    Angular 9

An error handler here would be a nice enhancement.

The app will guess the geometry column. Usually there will only be one column that is a sensible candidate.

Workaround

By adding a filter to the query, along the lines of gem is not null, the problem is averted and the query renders fine.

Viz screen showing 0 results

Hi,

When I am trying to run the BQ GIS query as mentioned here https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/blogs/bqml/taxifare_bqml.ipynb I am getting 0 results in the UI. I believe that this issue is happening because we are converting the pickup and dropoff location to Text as shown in the below lines

ST_AsText(ST_SnapToGrid(pickup, params.RES)) AS pickup_gridpt,
    ST_AsText(ST_SnapToGrid(dropoff, params.RES)) AS dropoff_gridpt

If I remove the type cast the UI returns the correct row count but then it prohibits from applying the grouping function.

When I try to to apply the casting as below it agains returns 0 rows

SELECT
  ST_AsText(dropoff_gridpt),
  COUNT(error) AS numpts,
  -- mean absolute percent error
  AVG(100 * error) AS MAPE
FROM predictions
GROUP BY ST_AsText(dropoff_gridpt)
HAVING numpts > 100

BigQuery Geo Viz Permission Issues Surrounding Google Sheets

Hello, I have a BigQuery table which is basically just a connected Google Sheet. However, BigQuery Geoviz runs into permissions issues when trying to query this table. I am wondering, how to give permissions on the Google Sheet to BigQuery Geoviz?

I have attached a screenshot of the permission error message

Could this map be put into data studio as a chart?

I would love to see it.

Authorization does not work with incognito mode

When trying to authorize the application in an incognito browser it walks through the authorization flow, however, does not register the account. Additionally, when trying to authorize after going through the flow, it does not allow to attempt to authorize after clicking the button.

Erratic visualization behavior

Run the following query and then open in the geo viz app. Upon clicking to utilize street view, the behavior of the viz becomes erratic.

SELECT *
FROM bigquery-public-data.san_francisco_bikeshare.bikeshare_station_info as t1
WHERE
ST_WITHIN(ST_GEOGPOINT(lon, lat),
ST_MAKEPOLYGON(ST_GEOGFROMTEXT("""LINESTRING(
-122.41028583847944 37.797426773624345,
-122.39778035397875 37.79923820799601,
-122.39614117965742 37.79277567798291,
-122.40880525911727 37.790152515674635)"""))
)

Provide GMaps bounding box variable/macro to SQL script

It would be really cool if you could put $BBOX or similar in a query and have it replaced with a Geography representing the current bounding box of the google maps widget. This would allow easy usage with GEOGRAPHY clustering to limit results to just what you're looking at. If you want to look at different data, just move the map and hit Run again to refresh the results.

It seems like it would be simple to add, does this project accept contributions? It seems like not much has happened with this project recently so I wanted to check before spending time making the changes and sending a PR.

Authorize does not work

After pressing the "Authorize" button the google login pop-up does show, but afterwards nothing happens. "Authorize" button remains active and "Data" and "Style" label stay greyed-out. I did receive an email confirming the authorization. II tried it on two different systems.

Is the authorization (temporary?) broken or should I do something different?

Very much looking forward to a reply. Thanks in advance.

Can We add Map Styles option for better Visualization?

There is several Map styles from MAP API here https://docs.mapbox.com/api/maps/styles/#mapbox-styles
It would be cool if we can add it to the BigQuery Geo Viz.

Query results are not rendered (failed to execute shaderSource)

Query

SELECT
  ST_GeogPoint(longitude, latitude)  AS WKT,
  status,
  health,
  spc_common,
  user_type,
  problems,
  tree_dbh
FROM `bigquery-public-data.new_york_trees.tree_census_2015`
WHERE status = 'Alive'
LIMIT 5000000;

Console error

zone.js:199 Uncaught TypeError: Cannot read properties of undefined (reading 'setUniforms')
    at ScatterplotLayer.draw (scatterplot-layer.js:180)
    at layer.js:894
    at withParameters (with-parameters.js:17)
    at ScatterplotLayer.drawLayer (layer.js:893)
    at DrawLayersPass.drawLayerInViewport (layers-pass.js:164)
    at layers-pass.js:136
    at Array.forEach (<anonymous>)
    at DrawLayersPass.drawLayersInViewport (layers-pass.js:122)
    at layers-pass.js:64
    at Array.forEach (<anonymous>)

Browser

Chrome is up to date
Version 95.0.4638.54 (Official Build) (x86_64)

System

Model Name:	MacBook Pro
  Model Identifier:	MacBookPro16,1
  Processor Name:	6-Core Intel Core i7
  Processor Speed:	2.6 GHz
  Number of Processors:	1
  Total Number of Cores:	6
  L2 Cache (per Core):	256 KB
  L3 Cache:	12 MB
  Hyper-Threading Technology:	Enabled
  Memory:	16 GB
  System Firmware Version:	1554.140.20.0.0 (iBridge: 18.16.14759.0.1,0)

Graphics

AMD Radeon Pro 5300M:

  Chipset Model:	AMD Radeon Pro 5300M
  Type:	GPU
  Bus:	PCIe
  PCIe Lane Width:	x8
  VRAM (Total):	4 GB
  Vendor:	AMD (0x1002)
  Device ID:	0x7340
  Revision ID:	0x0043
  ROM Revision:	113-D3220E-190
  VBIOS Version:	113-D32207P1-019
  Option ROM Version:	113-D32207P1-019
  EFI Driver Version:	01.A1.190
  Automatic Graphics Switching:	Supported
  gMux Version:	5.0.0
  Metal Family:	Supported, Metal GPUFamily macOS 2
  Displays:
Colour LCD:
  Display Type:	Built-In Retina LCD
  Resolution:	3072x1920 Retina
  Framebuffer Depth:	30-Bit Colour (ARGB2101010)
  Main Display:	Yes
  Mirror:	Off
  Online:	Yes
  Automatically Adjust Brightness:	No
  Connection Type:	Internal
HP Z27:
  Resolution:	3840x2160 (2160p/4K UHD 1 - Ultra High Definition)
  UI Looks like:	1920 x 1080 @ 30.00Hz
  Framebuffer Depth:	30-Bit Colour (ARGB2101010)
  Display Serial Number:	CN494208MM  
  Mirror:	Off
  Online:	Yes
  Rotation:	Supported
  Automatically Adjust Brightness:	Yes
  Connection Type:	Thunderbolt/DisplayPort

Note: issue reproduces only with external monitor connected

Color picker should be shown in non data-driven mode.

Currently the color picker is only available in data-driven mode, and the non-data-driven input requires a color name or #ffffff hex value.

[FR] Checkbox to remove geo column from infowindow

When creating a query for visualization, the geometry column is always required - of course. As a result it also always shows up on the infowindow/bubble when clicking on a rendered feature. It also has a related issue of making the infowindow larger than necessary.
A simple toggle in the "style" or "data" section to hide this data when displayed would be a useful visual enhancement.

unable to use UDF

Thanks for great tool ! but I have been unable to use UDF in the past few days.

Syntax error: Unexpected keyword CREATE at [5:1]

main.component.ts:589 Maximum call stack size exceeded

Apologies in advance for this being vague. What I did was first ran a query from a table with a million rows. The visualization worked fine, I played with the style options as well, which worked fine. Then I wanted to try the same thing with ten million rows. I got the "Maximum call stack size exceeded" error, both in the UI and in the console. Then, I tried switching back to the first query, only selecting geography points, with a million rows, and it stopped at 163,857 rows and got the same error. The first query no longer worked. I tried switching back to the ten million one again, just selecting points, same error, same number: 163,857 rows. Refreshing the page and rerunning the query doesn't work.

When a user hasn't entered a project ID, make it more clear

Right now, the text "Project ID" looks like a header rather than an input box. Took me a minute or two and some minor frustration to figure out that's what was keeping the "Run" button from turning blue

Points are off the map

Running this example query and visualizing through bigquerygeoviz

SELECT ST_GEOGPOINT(lat, lon) as geo FROM properati-data-public.properties_ar.properties_sell_201802
WHERE lat_lon <> ''
limit 500

As you see results are outside the continent

Same point from resultset in Google Maps provides correct location

Styles are not working (e.g. circleRadius)

Hi,

I have been using bigquery-geo-viz for a while in several courses. Today, I was noticed that several features related to styles are not working as usual.

To be sure, I have followed the "hurricane track example" from https://cloud.google.com/bigquery/docs/gis-tutorial-hurricane?hl=es-419

Usually, when I reached step 2 in the bigquey-geo-viz (select geometry column field), the google map points were shown. Now, they remain the same.

Moreover, when I try to define styles (e.g. circleRadius) following the tutorial, nothing changes. Only some small red points are shown

Kind regards,
Daniel.

Consider wrapping longitudes over the 180th antimeridian

See https://github.com/uber/deck.gl/blob/master/docs/api-reference/layer.md#wraplongitude-boolean-optional, suggested by @pmast. Should this be an option in the UI, or on by default?

Possibility to embed the map generated by BigQuery Geo Viz

Hello BigQuery Geo Viz team,

I would like to start by saying that I am a big fan of the project, and it has been extremely helpful for my visualizations. Congratulations on the great work!

I have a question regarding the possibility of embedding the maps generated by BigQuery Geo Viz. Is it possible to embed the maps, and if so, how can it be done?

Thank you very much for your assistance so far.

Add option to disable planar conversion

Showing geodesic lines and polygons correctly requires a backend planar conversion, currently implemented by wrapping the user's SQL query. This has the side effect of breaking UDF queries (#3).

For some cases (e.g. point data) that conversion isn't necessary. The application can't tell automatically when that's the case, but we could perhaps offer an option to disable it, so the user's query will run verbatim.

Cannot type into query box or see anything

Hi

I am not able to type anything into the query box here. It also looks like its glitched out;

Visualization going Red on Zoom

Running into an issue where the screen is going red when zooming into the map. No styles have been applied. Has happened consistently with multiple data sets

BigQuery GeoViz query returns fewer results than standard BigQuery query

This issue may be related to #5

My dataset is an import of the US Census 2018 Coastline shapefile, which has 4258 rows. Each row has a non-null geojson field (all linestrings), from which I created a new geography field using the ST_GEOGFROMGEOJSON() function. My geography field is named g, and all of its values are of type LINESTRING.

In BigQuery GeoViz, when I run the query
SELECT g
FROM xxxxxxxxx.census_coastline2018;

I get 1496 results.

In standard BigQuery, when I run
SELECT COUNT(g)
FROM xxxxxxxxx.census_coastline2018;

I get back the 4258 as expected.

Looking at my logs, I see that the actual query that was passed to BigQuery from BigQuery GeoViz is

SELECT
ST_AsGeoJson(g) AS g
FROM (
SELECT
g
FROM
xxxxxxxxx.census_coastline2018 );

Finally, in standard BigQuery, when I run
SELECT
COUNT(ST_AsGeoJson(g))
FROM (
SELECT
g
FROM
xxxxxxxxx.census_coastline2018 );

I again get 4258.

In other words, the BigQuery GeoViz is omitting results that standard BigQuery returns.

Remove LocalStorage implementation

This can be simplified given #37.

add labels to points

Add ability to plot point labels with a data-based field.

View individual results from larger query

Sometimes we want to view the results of a query in Geo Viz one by one. You can use LIMIT and OFFSET to accomplish this but it is very slow because you have to requery BigQuery every time. It would be better if there was an option to keep the query the same (query all results) but visualize results on the map one-by-one so it would be easier to see trends/check results.

strokeColor by data value is not working

I am trying to style my LineStrings by a categorical value from one of my columns. But it looks like the feature is broken, the geometries are only being styled using the first color value.

STR

1. Query a LineString dataset
2. Go to STYLE and enable the data-driven option from the strokeColor section
3. Select a categorical column
4. Fill it with the proper unique values
5. Select any color combination

Expected result

LineStrings should be styled by categorical values.

Current Result

Some errors are shown in the console. LineStrings are only plotted using the first color value.

OS and device

Chrome 77
Ubuntu 18

Use deck.gl onLoad callback to detect when renderer is ready

As opposed to:

if (!this._deckLayer._deck.layerManager) return;

visgl/deck.gl#2042

Share link does not hold style settings

I tried to share some of my visualizations with my co-workers.
Unfortunately the styles are not being retained and they have to reconfigure.

Could there be a way to export those settings in a JSON or yaml file?
They could be reused online or offline.

No points rendered

I am experiencing troubles with geo viz not showing up any points. The query runs fine, the data is visible, the map zooms to the correct bounding box but no points are drawn. I have an M1 MacBook Air. I have tried Chrome and Safari, restarts and clearing cache have no effect. Works like a charm on my old MacBook Air.
Interestingly, the Estimated query size goes from showing a correct number to quickly switching to 0.0 bytes