deneb-viz / deneb Goto Github PK

It's been observed that if we try to supply a field containing a decimal point in it into a template, then these are not correctly inserted. This is due to how Vega-Lite handles special characters in the field encoding. Vega also has this consideration to make.

We will also need to ensure that autocomplete takes this into consideration. The simplest solution would be to advise that we surround field references with square brackets. Would also be better if we restrict our search and replace to field properties or datum references too.

The AppSource submission and review process favours NPM, so reserving this issue for any potential remediation (hopefully not though).

I'd like to make feature switches more controllable, so we'll move them out of where they are and set up a simple config API also.

The visual can be quite chatty and this may cause a lot of components to re-render when it's not absolutely necessary. We should review and itemise some low-hanging fruit that we can tighten-up in order to optimise perfomance.

In implementing #2, this solves the issue if the user is inside an instance of a JSON editor, but if they navigate to the Settings pane then the keys no longer work, as there's no editor on this pane.

It's highly-likey that we want a number of shortcut keys to work when the advanced editor is open, and not just when we're stuck in the JSON editor, so we should (if we can) separate out any editor-specific key bindings from UI-specific key bindings and get them all working as expected:

• Apply Changes
• Toggle Auto Apply
• Repair/Format JSON
• New Specification
• Get Help
• Select Pivot Item (Spec/Config/Settings)

In visuals API 3.6, the concept of no data role support was introduced, where visuals can be treated as functional, even if no data is supplied. This is desirable because we can enable the advanced editor function without having to make the user add data and this can improve onboarding.

It would be useful to provide at least some high-level information about the visual dataset in the visual editor, such as:

• Number of rows
• Measure/column metadata
• Limited preview of rows

When a template is selected in the Create New Specification dialog, and provider is changed, the selected template is reset (As expected) but the highlight in the list is incorrect. This is a fairly minor issue as the workflow is correct but creates an incorrect visual cue for the end-user.

Steps to Reproduce

1. Open the dialog
2. Select a Vega-Lite template, e.g. Colored Scatterplot
3. Change provider to Vega

Expected Outcome

The Vega templates are shown, with the first template selected ([empty]).

Actual Outcome

The [empty] template is selected, but the list item is showing Connected Scatterplot, e.g.:

We include some simple templates in the source but it would be much more useful if folks can build and import their own.

Current list of high-level requirements:

• Architect template API for opening/validating and 'exporting' templates
  • Determine v1 template structure/schema and identify gaps in code to confirm existing templates
  • Documentation on how to package a template manually suitable for template API
• Add Create Template from Specification dialog
  • Assign hotkey and stub out CommandService method
  • Template high-level metadata should include:
    • Provider (inferred from settings)
    • Provider schema URL (auto-provided for future-proofing)
    • Template GUID (auto-generated)
    • Template schema version
    • Name (mandatory; provided by creator)
    • Author Name (mandatory; provided by creator)
    • Description (mandatory; provided by creator; plain text only)
  • Placeholders should infer column/measure from dataset, as well as data types from metadata
  • Placeholder keys will be sequential numbers
  • Anything matching the displayName for a given column or measure will be replaced with the placeholder key
  • (optional) Description for creator to fill out for each placeholder
  • Any user input should be sanitised
  • Upon validation, template JSON will be provided for copy/paste > save as .json file (custom visuals/Power BI Desktop don't have native file system access; we might be able to allow user to open a link to browser w/JSON in the page for easier save, but this is for later.
• Add Open from Template functionality to dialog
  • .json files only
  • A template needs to be parsed and validated against schema prior to admission into the visual
  • Implementation of assistive text for placeholders into the dialog
  • Completion of data type restrictions for placeholder selection (only measure/grouping/both is implemented)

If we want a compliant visual for AppSource, we won't be able to look at storing/publishing templates at this time, as it involves HTTP and certified visuals should not include this functionality. We also can't save/export to a local file, so none of this is currently in-scope.

We could look at improving re-draws/re-renders when the visual is resized, notably:

• When the visual gets put into focus mode, the visuals SDK runs several update methods, when we might only expect one. This is possibly due to how Power BI resizes the iframe element for a custom visual (note: it might be worth asking MS if this can be done in the developer visual host also, as it has benefits to all custom visuals this way).

• When a user is resizing the developer visual, the update method is triggered. We already do some things to avoid re-processing data (as it doesn't get reloaded), but we can probably do more to selectively debounce or throttle the amount of times we re-render a visual specification whilst in the process of resizing.

With changes in #21 implemented, we can potentially create finer-grained selectors for column/measure combinations. This means that we could also potentially attempt to create selectors for layers containing transforms away from the grain of the data view.

We should add-in functionality to the handler (in the absence of the in-built __key__ field in the datum) to attempt to create a selector for partial fields, if they can be identified by name/value in the data view. This should then resolve report page tooltips accordingly.

This will not catch all cases, but will enhance the number of scenarios we can apply report page tooltips to. This should also set us in good stead for figuring out #6 and #5.

Currently the preview window in the editor will stetch the visual viewport to fit the focus mode viewport, les the editor pane. This is nice, but can mean that the version you're editing might look differently when scaled back down after returning to the report. We should allow creators the option of preserving the visual viewport dimensions when opening the editor, and we can allow scrolling as necessary if it doesn't fit.

Could you send a pull request to add Deneb to http://vega.github.io/vega-lite/ecosystem.html?

For template imports, we'll validate the JSON against our schema, but for the purposes of simplicity, we leave spec as an object that can have any structure and we assume that it's correct based on the validation done for 'export'. however if a developer hand-cranks their own, the spec could conceivably be invalid.

On import, once the initial schema has been validated, we should then validate against the provider (i.e. Vega or Vega-Lite) to confirm the whole template is good.

Ability to easily add pattern fill to marks via html/svg defs

It will be handy from a UX perspective, if we can also prompt or warn the user about this so that they can take remedial action.

I've probably taken a somewhat casual approach to ensuring that the editor UI has been labelled-up correctly, so we need to review the UI and fix up any elements or controls that aren't picking-up correctly.

Vega/Vega-Lite use d3 formatting, which is not immediately intuitive for Power BI users. There's the ability to add custom formatters to a view, so let's see if we can get this working for as many cases as possible where a user wishes to apply custom formatting.

Any way to remove the alerts on the empty template so as not to alarm the user upon first opening?

During prototyping, we identified issues with implementing the context menu generically, as it depends on Power BI's selection API and some compliance from the developer of a specification. At a high-level, we should achieve the following as part of this work:

• Prototype suitable approaches and support for specifications
• Migrate out of legacy selectionHandlerService into selection API
• Abstract as much as possible into signal handlers
• Scope and document commitment from developer to implement, as well as gotchas and requirements.
• Ensure regular context menu shows, as this is a certification requirement; this is whether the menu picks up selection IDs/datapoints or not

We currently need to manually change our locale using Power BI settings or use external tooling to do this, which makes the dev/debug cycle hard when testing locale-based stuff like formatting. IDeally, we should add selectable locale to the developer options (if enabled) with a shorter pre-defined list that we can use instead of the environment settings.

With default tooltips, if any formatting is applied to the column or measure in the data model, we don't currently resolve it. We need to try and manage this in the TooltipHandler where possible.

Has a dependency on #13, as we need powerbi-visuals-utils-formattingutils in the build to make things work.

It's envisaged that we'll have up to 4 different versions of the visual:

• The AppSource version, which will not allow loading of external data and images and will be certified
• The standalone version, which will allow as much as Power BI lets custom visuals do within the CORS restrictions it imposes
• Alpha channel build (upon pull request; may not represent intended release build)
• Beta channel build (upon tagging; intended to represent ideal release candidate builds)

Each one of these should have its own guid and other metadata, and ideally a unique icon so if a creator has them side-loaded, then it's easy to see what's going on in a report.

As the SDK doesn't have a process to allow us to specify a pbiviz.json file, we'll need to temporarily patch the one in the root folder with appropriate properties, package the visual and then revert. If this works, we can consider making a pull request to the SDK for a property that can allow people to specify an external pbiviz.json file for the package process.

The AppSource build will keep the 4-dot versioning system, but this will now keep the fourth digit at zero rather than using 'build' number (e.g. 1.1.0.0), as this is unsustainable if/when we get multiple collaborators. This will be manually managed when changes are ready to make available properly. The other three package methods will move to a major.minor.patch.yyyyMMdd#commit version number, which will be generated by our script.

Ideally, we should extend the CI processes upon the following events as follows:

• Alpha: on pull request to master, run existing CI workflow, and then build the alpha version. Deploy as a pre-release with the alpha label or similar. Bonus points if we can take the PR text and add it to the body of the release
• Beta: TBC, but likely upon tagging the master branch accordingly to trigger the build and pre-release with an appropriate label or similar
• Standalone: if we can do this when a release is published, and attach the built .pbiviz files, that would be ideal. If not, then we can look at a similar process to Beta

We should improve the landing page to better illustrate how to get started with the visual, i.e.:

• Add data
• Choose existing template or import one
• Save or export

We should also consider what to display if the report is being viewed rather than being edited, as this may not completely make sense, depending on context.

Currently, if we swap out, remove or re-name columns or measures (either in the data model or in the visual), this breaks the spec, as we refer to them by display name. In these cases, we can do a replace all in the editor or manually fix them up, but it's a bit onerous.

What would help a lot, is if we can persist our column and measure metadata to the visual properties, and compare these with the current metadata on any changes (either on constructor or update).

If any changes exist, we should prompt the creator to re-assign placeholder values and replace them in the spec using a similar technique to the templating process, if possible.

In 0.1.0.80, we added the ability for the editor to flip sides, based on user preference. Internally we still call the left (resizable) pane editorPane*, so I'd just like to sort this out so it's a bit more sensible for code spelunkers or anyone looking to get involved.

Will certainly make it easier to 'peek' at the visual without interrupting workflow too much.

Also: doing this without saving changes causes them to be lost (presumably due to an unmount), so we should apply changes also.

During prototyping, we identified issues with implementing selection/cross-highlighting generically, as it depends on Power BI's selection API and some compliance from the developer of a specification. At a high-level, we should achieve the following as part of this work:

• Prototype suitable approaches and support for:
  • single view
  • layer view
  • concat view
  • facet view
  • repeat view
• Migrate out of legacy selectionHandlerService into selection API
• Abstract as much as possible into signal handlers
• Scope and document commitment from developer to implement, as well as gotchas and requirements.

Sometimes the canvas renderer leaves a tooltip visible on mouseout, if this happens quickly enough to not be detectable. In the example below the cursor is back in the editor after testing the tooltip in the preview area:

This can also happen when the visual is in view mode.

Workaround is to move the mouse back into the visual and then back out, or trigger an interaction in another visual. But would be nice to figure out.

As v8 is now available, we should upgrade to the standard version, particularly as we're using some beta functionality.

With situations like where we want to add layers with enpty data (for reference lines etc.), the visual refuses to render. This also happens if we mutate the data object in the spec, so we can't prototype with static or overriden data etc.

Currently 5.20.1 and 5.1.0 respectively.

We need to add appropriate shortcut keys to allow selection of:

• Specification tab
• Config tab
• Editor tab

When importing a template, if we've selected a file for import, this is stored ready to stream in. If we were to, say, change the underlying file and then attempt to re-import it, this doesn't happen as the name is the same. We should ensure that the current file is cleared when the import button is clicked, so that we can replace it more seamlessly.

Early on (waaaaay before the recent builds and things got refactored), we had a simple dirty flag and status for the end-user if they weren't applying their changes automatically and we should add this back in to prevent accidental loss of work.

We should also prompt the user if moving from advanced editor, back to the report if the specification/config is dirty and confirm if they want to save/discard changes.

The current TooltipService is instantiated as a class and mirrors the vega-tooltip handler, but can be broken-down and re-written to work in a more functional way. This will also make it easier to extend, particularly for #37 and #8.

We're currently using the table data view mapping so that we can handle data fairly generically. However we lose flexibility here when resolving selection IDs for interactivity purposes, as these represent the entire dataset row. This can cause us to lose context if we transform data within the visual or mutate it.

It's should theoretically be possible to produce a 'table-like' mapping using categorical. The working idea here is that we can produce a similar-looking dataset for the specification, and then we can generate more granular selection IDs on the fly from partial fields or measures if they end up attached to a mark.

This could be handy for things like facet plots or legends, which might represent a higher level of grain or grouping, and with the table mapping, we'd need to add all affected rows to the selection manager to ensure we get everything. Current prototyping shows that this generates a lot of work server-side, and between visuals on the same page. This way, we could match a single dataset row by category/value for higher-level groupings and this would suffice.

All existing functionality, including report page tooltip (or base selection IDs) and data limit handling will need to be regression tested.

As we introduced the ability to switch sides for the editor in 0.1.0.80, the handling of the correct orientation of the chevron is done as a ternary expression in both EditorHeadingExpanded.tsx and EditorPaneCollapsed.tsx. These upset the linter and should ideally have been fixed before finalisation of 0.1, so we'll fix up ASAP.

Once we tab into the JSON Editor, we can't tab out, so we need a hotkey to allow us to 'escape' back to the pivot.

The Create New Specification dialog is pretty cramped on small screens, e.g.:

This is partially due to custom visuals only being able to use the visual viewport, but we should re-assess the layout, particularly once templates are out there, as we can potentially remove a lot of redundant info from this page.

It may make more sense to make this part of the application's main state, rather than popping up a dialog, which might be an anti-pattern in this case - this would allow us to use the entire canvas and keep the user focused on the current operation.

We should ideally do this for import/create and export workflows, and also standardise the layouts for placeholders to make maintenance easier, e.g. DetailsLists rather than specific logic for each.

As Vega uses the d3 formatters, we've implemented the locales in d3 that cross-over with the Power BI locales for the beta, but there are a number of gaps commented in src/config/locales.ts. We need to complete these as much as we can to ensure that we can support all Power BI locales.

In some situations when we're producing a valid Vega-Lite spec, but then there's an underlying error with the render - such as when a param is declared at the wrong level and introduces duplicate signals in the Vega portion of the render - results in a blank visual with no indication as to what went wrong.

Reproducing these in the Vega editor confirms that there are indeed underlying issues that need addressing, so this is not clearly communicated from our side. We'll need to identify these situations and try to at least signal that something is up for the creator.

Currently, we pass in the values of the viewport to react-vega for height and width, and these are ignored if stated in the specification. This means that single and layered plots will want to stretch to fit, and we should respect the creator's decisions here if they have specific values they want to use. If these are included in the spec, we should resolve them into the component when it's rendered.

For this issue, acceptance criteria would be:

• Creator can specify the following and they'll work with the container as per the doc VL | V
• For Vega-Lite, using "container" will work as intended for either width, height or both.
• For Vega, in-built height/width signals will also work as intended
• [CONSIDER] If a spec is missing height or width at the top-level, impacts of patching signals for the end-user. It may be better just to update the templates with correct values rather than doing unknown stuff.

Probably a good companion for #15

When a template is imported, we list the dataset placeholders, but we could also enrich this dialog with metadata from the imported template.