GithubHelp home page GithubHelp logo

cartodb / toolkit Goto Github PK

View Code? Open in Web Editor NEW
9.0 7.0 3.0 3.69 MB

JS library to interact with CARTO APIs in a simple way

Home Page: https://toolkit-wheat.now.sh

License: BSD 3-Clause "New" or "Revised" License

TypeScript 90.77% JavaScript 2.57% HTML 6.66%

toolkit's Introduction

CARTO Toolkit - New Web-SDK!

This is a monorepo intended to contain all small-medium sized JS libraries that are used to create CARTO Apps. Currently contains:

  • An auth library, mainly for OAuth.
  • A SQL API library.
  • A Custom Visualizations library.
  • A Viz library, built on top of deck.gl
  • A Metapackage that exposes all

It uses Lerna to manage all the packages and inter-dependencies.

Quick steps to check that build works correctly

 nvm use
 npm install
 npm run build

Development

Linking Deck.gl

In order to work with your local version of Deck.gl in this project, you need to link the Deck.gl module or modules (take into account Deck.gl is a lerna project too, so it is a bunch of different projects) you need into this one.

  • In the root of deck.gl project
yarn build
  • Move to the module you want to link (in this example we want to link @deck.gl/layers)
cd modules/layers/
npm link
  • In the root of the toolkit project
npm link @deck.gl/layers

toolkit's People

Contributors

alasarr avatar jesusbotella avatar juandjara avatar manmorjim avatar neokore avatar oleurud avatar rjimenezda avatar victorvelarde avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

toolkit's Issues

Documentation

Use this issue to track how we want to tackle generating documentation for the packages.

Weird behavior for lines and points with high size

There is a weird behavior with points with high radius:
image

And the same with lines with high line width:
image

They seem to be affected by clipping extension. It is required for polygons in order to avoid artifacts and it was added by uber people for MVTLayers: visgl/deck.gl#4336

An suggestion would be removing it for lines and points layers.

[SQL] RequestManager should handle request authorization

Right now, RequestManager stores the API key, but implementations of it are responsible for adding the api_key to the request. In practice, the api_key is added to the url of the request that gets pushed to the queue, and will never change.

We should be smarter than this, and add it on RequestManager, which is the one actually calling fetch with the queue elements.

I think the SQL API supports Basic Auth, so it could be sent as a header, but we should be able to add it to the URL as well.

Why?

If you are using OAuth, and you have many requests queued that have to wait a long time, there's a chance that a request that has an old token will fire after the token has expired. This implies that requets have been waiting more than 5 minutes, but who knows.

Refactor Credentials use and other common utilities

We should consider a mechanism to share 'classes' & util methods between packages:

  • Credentials (username / apiKey) + several related properties (and some operations) are being used in several packages (eg. sql & maps).

  • encoding params in URLs and so on?

  • GET or POST depending on size?

[SQL] Improve return types of queries

Right now, queries return a Promise, which is pretty bad.

SQL API requests are not specially easy to type:

  • If something goes wrong, it's just an object with an error field.
  • If the return type is json, it's an object with fields, rows and some extra stuff.
  • If you specify a type, like CSV or SVG, it'll be just a string*

*If the response mime type is not application/json, it just returns response.text()

Toolkit UMD naming

The UMD module names are a bit of a mess right now. They are long, verbose, and not namespaced.

We should try to come up with a relatively short way of naming packages, maybe even namespacing?

IDEA: We could have different entry points for the UMDs, so they build a namespace. We shouldn't use carto. because I think VL and carto.js will not play nice.

Dynamic raidus scale for points size based on pixels

The radius size for points is based on distances (meters/degrees) rather than pixels so they are adapted depending on the zoom level:

Peek 2020-04-28 16-46

It is desirable to have radius size based on pixels since this is the most common form of representation for points. By this way, they are still visible in low zoom levels by keeping their size.

For lines there is a property called lineWidthUnits which accepts the value 'pixels'.

Unfortunately this options is not available for points so a workaround would be using dynamic radiusScale like is described in visgl/deck.gl#1259 (comment)

Something like setting radiusScale as pixel per meters multiplying factor that you can get from the viewport. This has to be updated everytime the zoom changes: https://github.com/visgl/deck.gl/blob/master/modules/core/src/viewports/viewport.js#L92

Add CI configuration

Prepare a simple CI integration.

CircleCI worked pretty well in other projects for us

Manage Typescript declaration files automagically

Right now, no .d.ts files are created automatically from source. As a result, some Type info between packages is not correctly detected.

eg

import { Credentials } from '@carto/toolkit-auth';

class Maps {
  private _credentials: any; // TODO Credentials class type is not detected
}

TODO: Review tsc compiling & rollup config, so Types are automatically included in the library (and / or externally published?)

Move to Typescript 3.7

Right now is in beta, so we should wait until it goes stable, but we could really use assert functions.

Node compatibility

Right now, I think 100% of the requests this library does use fetch, so this won't work on node at all, unless we use a library that shims it, or we switch to another library.

There might be other compatibility problems, so we should give it a try.

Some packages, like the Auth one, probably don't make a lot of sense in node, but it's worth checking it out.

[SQL] Improve error handling

Right now, rate limits are handled gracefully. However, we should also be wary of:

  • 503 errors, that might happen if the SQL API is not ready for whatever reason and nginx can't redirect the request to it. We should just retry after some time (maybe immediately)
  • Timeout errors, these might come back as a 429, but probably don't have the retry headers. To try this, PG_Sleep is probably the way.

Getting relevant data from Layer

An important part of what consists of our Web SDK is to let users access the data behind the visualization they are watching on their computers' screen, whether it is the features rendered on screen, the extent of the features of the layer or the number of rows of the whole dataset.

So for that matter, new functions have to be added to our Layer specification. Replicating the ones that we had in CARTO-VL, we thought of adding those ones:

Layer.getViewportFeatures()

This method will return an array containing GeoJSON objects for the features that are visible within the current viewport.

Proposed usage:

const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get current viewport features
(await?) layer.getViewportFeatures();

Depending on whether we do it with an SQL query or filtering features within a layer, we would need to add an await keyword. Both approaches have upsides and downsides.

  • Filtering the rendered feature data from our layer
    This approach is based on getting features from MVTLayer, filter them by the bounds of the current viewport and returning them in our function. It is completely synchronous, so no need to await for our results, but filtering happens on client. But given that our coordinates are local coordinates we would need to convert them into WGS84 coordinates before coordinate comparison

  • Requesting rows within viewport to SQL API
    This approach is much easier for us given that SQL API will perform the filtering and then we will return the rows' data in our function. The downside is that the request can take too much time depending on the data that is matched, and we will need to add an await keyword to our execution statement.

Layer.getExtent()

This method will return the bounds of our layer in the map, the coordinates of the rectangular area that encloses all layer features.

Proposed usage:

const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get layer extent
await layer.getExtent();

Return value:

[
 { longitude: -180, latitude:  90 },
 { longitude:  180, latitude:  90 },
 { longitude:  180, latitude: -90 },
 { longitude: -180, latitude: -90 },
]

To get layer's extent, we would need to request it to SQL API, so there's no discussion for this method's approach.

Layer.getNumberOfRows()

I think that was @VictorVelarde who added this method to Clubhouse's task, but I find it kind of strange to ask our layer for the number of rows. It is probably a better fit to have a .getDataset() or .getSource() method, and then request the number of rows to that object. What do you think?

Proposed usage:

const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get layer extent
await layer.getDataset().getNumberOfRows();
await layer.getSource().getNumberOfRows();

Platform package

Right now, the auth package has a small class that fetches the basic userinfo (API v4 /me) for you, since the URL is part of the OAuth response. I think that's fine, but it makes the App / OAuthApp split weird, since only OAuthApps have that information.

At some point we'll need to get that information for all Apps, not just OAuth apps, and more information of the platform (list datasets, for instance). I think we should move this to its own package.

Add linting to packages

We should have a linter setup for every package, under the command lint, so we can run it nicely on CI

Review visualization model from cartodb and add missing fields

As we're kinda building a visualization model ourselves, it would be good to mimick visualization model from cartodb so that we have as little properties mismatching as possible.

One of these can be a tags property that we definitely need to include.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.