elastic / sense

A JSON aware developer's interface to Elasticsearch. Comes with handy machinery such as syntax highlighting, autocomplete, formatting and code folding.

HEADS UP: This repo is deprecated. Sense is now included as Console in Kibana 5.0. File issues over at elastic/kibana

Sense

A JSON aware developer's interface to Elasticsearch. Comes with handy machinery such as syntax highlighting, API suggestions, formatting and code folding.

Installation

Sense is a Kibana app. To get up and running you will first need to download Kibana and install as instructed here. Once Kibana is installed, you can simply install Sense running the following command from your Kibana folder:

$ ./bin/kibana plugin --install elastic/sense

Now start your Kibana server by running:

$ ./bin/kibana

Sense should be available on http://localhost:5601/app/sense (assuming Kibana defaults).

For more information and advanced setting please see the documentation.

Screenshots

Handy API suggestions

Sense offers handy contextual suggestion to the Elasticsearch API.

Format validation

Sometimes it is hard to find that little missing comma. Sense automatically highlights and explains invalid input.

Scope collapsing

Working on a big JSON query can be distracting. Using the mouse or via a handy keyboard shortcut (Ctrl/Cmd+Option+0) , Sense allows you to focus on a sub section and fold others away.

Auto formatting

Type your commands however you want and let Sense format them for you.

Submit multiple requests at once

When testing or trying things out you frequently need to repeat the same sequence of commands. Just write them all in Sense, select and submit multiple requests to Elasticsearch.

Copy and Paste cURL commands

Once done, you can copy one or more requests as cURL commands (and of course paste them back)

Results in:

# Delete all data in the `website` index
curl -XDELETE "http://localhost:9200/website"

# Create a document with ID 123
curl -XPUT "http://localhost:9200/website/blog/123" -d'
{
  "title": "My first blog entry",
  "text":  "Just trying this out...",
  "date":  "2014/01/01"
}'

Documentation

Visit elastic.co for the full documentation.

sense's Issues

Generating a diff between REST API Spec and Sense shows they have differences, here they are.

Here are the differences between the 2.0 REST API Spec and the 2.0 Sense API model:

First, API endpoints that match the URL patterns but do not have HTTP Method found:

 WARN!  apiSpec [indices.exists_alias HEAD] does not match any sense patterns+methods
         but did match pattern of [_post_alias,_delete_alias,_get_alias]
  WARN!  apiSpec [indices.exists_type HEAD] does not match any sense patterns+methods
         but did match pattern of [index_doc_no_id,_get_index]
  WARN!  apiSpec [indices.exists HEAD] does not match any sense patterns+methods
         but did match pattern of [__create_index__,__delete_indices__,_get_index]
  WARN!  apiSpec [exists HEAD] does not match any sense patterns+methods
         but did match pattern of [_get_doc,_delete_doc,index_doc]
  WARN!  apiSpec [get_script GET] does not match any sense patterns+methods
         but did match pattern of [_put_script]
  WARN!  apiSpec [delete_script DELETE] does not match any sense patterns+methods
         but did match pattern of [_put_script]
  WARN!  apiSpec [snapshot.delete_repository DELETE] does not match any sense patterns+methods
         but did match pattern of [put_repository]
  WARN!  apiSpec [indices.exists_template HEAD] does not match any sense patterns+methods
         but did match pattern of [_delete_template,_get_template,_put_template]
  WARN!  apiSpec [snapshot.get_repository GET] does not match any sense patterns+methods
         but did match pattern of [put_repository]

Sense specs that do not have patterns+HTTP Method that matches API spec:

  WARN!  unused sense specs: [_cluster/nodes/stats,create_doc,_delete_mapping,_put_mapping,_put_percolator,all_snapshots]

For those that match the URL pattern and HTTP Method but when comparing the URL query parameters of the REST API Spec against the union of matching Sense endpoints query parameters:

  [indices.get_upgrade] api matches sense [_upgrade_status]
    WARN:  [indices.get_upgrade] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [human]
           only in SENSE []
  [cat.thread_pool] api matches sense [_cat/thread_pool]
    WARN:  [cat.thread_pool] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h, full_id]
           only in SENSE [bytes]
  [count_percolate] api matches sense [_percolate_count]
    WARN:  [count_percolate] query parameters do not match
           in both [routing, preference, ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [percolate_index, percolate_type, version, version_type]
           only in SENSE [percolate_format]
  [get] api matches sense [_get_doc]
    WARN:  [get] query parameters do not match
           in both [parent, routing, _source, _source_exclude, _source_include, version]
           only in API [fields, preference, realtime, refresh, version_type]
           only in SENSE []
  [cat.indices] api matches sense [_cat/indices]
    WARN:  [cat.indices] query parameters do not match
           in both [bytes, h, help, pri, v]
           only in API [local, master_timeout]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [nodes.stats] api matches sense [_nodes/stats]
    WARN:  [nodes.stats] query parameters do not match
           in both []
           only in API [completion_fields, fielddata_fields, fields, groups, human, level, types, timeout]
           only in SENSE []
  [cluster.state] api matches sense [_cluster/state]
    WARN:  [cluster.state] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local, master_timeout, flat_settings]
           only in SENSE []
  [snapshot.restore] api matches sense [restore_snapshot]
    WARN:  [snapshot.restore] query parameters do not match
           in both [wait_for_completion]
           only in API [master_timeout]
           only in SENSE []
  [count] api matches sense [_count]
    WARN:  [count] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards, min_score, preference, routing]
           only in API [q, analyzer, analyze_wildcard, default_operator, df, lenient, lowercase_expanded_terms]
           only in SENSE [terminate_after]
  [cluster.reroute] api matches sense [_cluster/reroute]
    WARN:  [cluster.reroute] query parameters do not match
           in both [dry_run, explain]
           only in API [metric, master_timeout, timeout]
           only in SENSE []
  [cat.aliases] api matches sense [_cat/aliases]
    WARN:  [cat.aliases] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [cat.health] api matches sense [_cat/health]
    WARN:  [cat.health] query parameters do not match
           in both [help, ts, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [indices.put_mapping] api matches sense [_put_type_mapping]
    WARN:  [indices.put_mapping] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [timeout, master_timeout, update_all_types]
           only in SENSE []
  [indices.get_mapping] api matches sense [_get_mapping]
    WARN:  [indices.get_mapping] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local]
           only in SENSE []
  [indices.delete] api matches sense [__delete_indices__]
    WARN:  [indices.delete] query parameters do not match
           in both []
           only in API [timeout, master_timeout]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [delete_template] api matches sense [_get_delete_search_template]
    WARN:  [delete_template] query parameters do not match
           in both []
           only in API [version, version_type]
           only in SENSE []
  [indices.clear_cache] api matches sense [_cache/clear]
    WARN:  [indices.clear_cache] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [field_data, fielddata, fields, query, index, recycler, request]
           only in SENSE []
  [cat.segments] api matches sense [_cat/segments]
    WARN:  [cat.segments] query parameters do not match
           in both [help, v]
           only in API [h]
           only in SENSE [bytes]
  [cluster.pending_tasks] api matches sense [_cluster/pending_tasks]
    WARN:  [cluster.pending_tasks] query parameters do not match
           in both []
           only in API [local, master_timeout]
           only in SENSE []
  [cat.nodes] api matches sense [_cat/nodes]
    WARN:  [cat.nodes] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [index] api matches sense [index_doc,index_doc_no_id]
  [indices.recovery] api matches sense [_recovery]
    WARN:  [indices.recovery] query parameters do not match
           in both [detailed, active_only, human]
           only in API []
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [nodes.hot_threads] api matches sense [_nodes/hot_threads]
    WARN:  [nodes.hot_threads] query parameters do not match
           in both []
           only in API [interval, snapshots, threads, ignore_idle_threads, type, timeout]
           only in SENSE []
  [search] api matches sense [_search]
    WARN:  [search] query parameters do not match
           in both [analyzer, analyze_wildcard, default_operator, df, explain, fields, from, ignore_unavailable, allow_no_indices, expand_wildcards, lowercase_expanded_terms, preference, q, routing, scroll, search_type, size, sort, _source, _source_exclude, _source_include, terminate_after, timeout, track_scores, request_cache]
           only in API [fielddata_fields, lenient, stats, suggest_field, suggest_mode, suggest_size, suggest_text, version]
           only in SENSE [scroll_id]
  [indices.delete_template] api matches sense [_delete_template]
    WARN:  [indices.delete_template] query parameters do not match
           in both []
           only in API [timeout, master_timeout]
           only in SENSE []
  [indices.flush] api matches sense [_flush]
  [indices.delete_alias] api matches sense [_delete_alias]
    WARN:  [indices.delete_alias] query parameters do not match
           in both []
           only in API [timeout, master_timeout]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [indices.refresh] api matches sense [_refresh]
    WARN:  [indices.refresh] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [force, operation_threading]
           only in SENSE []
  [snapshot.create] api matches sense [put_snapshot]
    WARN:  [snapshot.create] query parameters do not match
           in both [wait_for_completion]
           only in API [master_timeout]
           only in SENSE []
  [cat.plugins] api matches sense [_cat/plugins]
    WARN:  [cat.plugins] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [cluster.put_settings] api matches sense [put_cluster/settings]
    WARN:  [cluster.put_settings] query parameters do not match
           in both []
           only in API [flat_settings, master_timeout, timeout]
           only in SENSE []
  [indices.get_field_mapping] api matches sense [_get_field_mapping]
    WARN:  [indices.get_field_mapping] query parameters do not match
           in both [include_defaults, ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local]
           only in SENSE []
  [indices.shard_stores] api matches sense [_shard_stores]
    WARN:  [indices.shard_stores] query parameters do not match
           in both [status, ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [operation_threading]
           only in SENSE []
  [indices.put_template] api matches sense [_put_template]
    WARN:  [indices.put_template] query parameters do not match
           in both []
           only in API [order, create, timeout, master_timeout, flat_settings]
           only in SENSE []
  [indices.update_aliases] api matches sense [_post_aliases]
    WARN:  [indices.update_aliases] query parameters do not match
           in both []
           only in API [timeout, master_timeout]
           only in SENSE []
  [snapshot.create_repository] api matches sense [put_repository]
    WARN:  [snapshot.create_repository] query parameters do not match
           in both []
           only in API [master_timeout, timeout, verify]
           only in SENSE []
  [cluster.get_settings] api matches sense [get_cluster/settings]
    WARN:  [cluster.get_settings] query parameters do not match
           in both []
           only in API [flat_settings, master_timeout, timeout]
           only in SENSE []
  [indices.create] api matches sense [__create_index__]
    WARN:  [indices.create] query parameters do not match
           in both []
           only in API [timeout, master_timeout, update_all_types]
           only in SENSE []
  [cat.nodeattrs] api matches sense [_cat/nodeattrs]
    WARN:  [cat.nodeattrs] query parameters do not match
           in both [h, help, v]
           only in API [local, master_timeout]
           only in SENSE []
  [cluster.health] api matches sense [_cluster/health]
    WARN:  [cluster.health] query parameters do not match
           in both [level]
           only in API [local, master_timeout, timeout, wait_for_active_shards, wait_for_nodes, wait_for_relocating_shards, wait_for_status]
           only in SENSE []
  [cat.master] api matches sense [_cat/master]
    WARN:  [cat.master] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [percolate] api matches sense [_percolate,_percolate_id]
  [field_stats] api matches sense [_field_stats]
  [cat.allocation] api matches sense [_cat/allocation]
    WARN:  [cat.allocation] query parameters do not match
           in both [bytes, help, v]
           only in API [local, master_timeout, h]
           only in SENSE []
  [indices.put_warmer] api matches sense [_put_warmer]
    WARN:  [indices.put_warmer] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [master_timeout, request_cache]
           only in SENSE []
  [put_script] api matches sense [_put_script]
    WARN:  [put_script] query parameters do not match
           in both []
           only in API [op_type, version, version_type]
           only in SENSE []
  [indices.put_alias] api matches sense [_post_alias]
    WARN:  [indices.put_alias] query parameters do not match
           in both []
           only in API [timeout, master_timeout]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [termvectors] api matches sense [_termvectors,_termvectors_id]
    WARN:  [termvectors] query parameters do not match
           in both [term_statistics, field_statistics, dfs, fields, offsets, positions, payloads, preference, routing, parent, version, version_type]
           only in API [realtime]
           only in SENSE []
  [indices.put_settings] api matches sense [_put_settings]
    WARN:  [indices.put_settings] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [master_timeout, flat_settings]
           only in SENSE []
  [delete] api matches sense [_delete_doc]
    WARN:  [delete] query parameters do not match
           in both [parent, routing, version, version_type]
           only in API [consistency, refresh, timeout]
           only in SENSE []
  [get_template] api matches sense [_get_delete_search_template]
    WARN:  [get_template] query parameters do not match
           in both []
           only in API [version, version_type]
           only in SENSE []
  [update] api matches sense [_update]
    WARN:  [update] query parameters do not match
           in both [consistency, fields, parent, refresh, retry_on_conflict, routing, timeout, timestamp, version, version_type]
           only in API [lang, script, script_id, scripted_upsert, ttl, detect_noop]
           only in SENSE []
  [indices.upgrade] api matches sense [_upgrade]
    WARN:  [indices.upgrade] query parameters do not match
           in both [allow_no_indices, expand_wildcards, ignore_unavailable, wait_for_completion]
           only in API [only_ancient_segments]
           only in SENSE []
  [indices.stats] api matches sense [_stats]
    WARN:  [indices.stats] query parameters do not match
           in both [completion_fields, fielddata_fields, fields, level, types]
           only in API [groups, human]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [indices.get_warmer] api matches sense [_get_warmer]
    WARN:  [indices.get_warmer] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local]
           only in SENSE []
  [indices.delete_warmer] api matches sense [_delete_warmer]
    WARN:  [indices.delete_warmer] query parameters do not match
           in both []
           only in API [master_timeout, name]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [indices.analyze] api matches sense [_analyze]
    WARN:  [indices.analyze] query parameters do not match
           in both [analyzer, char_filters, field, filters, text, tokenizer]
           only in API [index, prefer_local, format]
           only in SENSE [ignore_unavailable, allow_no_indices, expand_wildcards]
  [indices.segments] api matches sense [_segments]
    WARN:  [indices.segments] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [human, operation_threading]
           only in SENSE []
  [indices.flush_synced] api matches sense [_flush_synced]
  [search_shards] api matches sense [_search_shards]
  [search_template] api matches sense [_search_template]
    WARN:  [search_template] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [preference, routing, scroll, search_type]
           only in SENSE []
  [snapshot.status] api matches sense [_snapshot_status]
    WARN:  [snapshot.status] query parameters do not match
           in both []
           only in API [master_timeout]
           only in SENSE []
  [indices.get_alias] api matches sense [_get_alias]
    WARN:  [indices.get_alias] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local]
           only in SENSE []
  [cat.recovery] api matches sense [_cat/recovery]
    WARN:  [cat.recovery] query parameters do not match
           in both [bytes, help, v]
           only in API [master_timeout, h]
           only in SENSE []
  [indices.get_template] api matches sense [_get_template]
    WARN:  [indices.get_template] query parameters do not match
           in both []
           only in API [flat_settings, master_timeout, local]
           only in SENSE []
  [snapshot.get] api matches sense [single_snapshot]
    WARN:  [snapshot.get] query parameters do not match
           in both []
           only in API [master_timeout]
           only in SENSE []
  [put_template] api matches sense [_put_search_template]
    WARN:  [put_template] query parameters do not match
           in both []
           only in API [op_type, version, version_type]
           only in SENSE []
  [get_source] api matches sense [_get_doc_source]
    WARN:  [get_source] query parameters do not match
           in both [parent, routing, _source_exclude, _source_include, version]
           only in API [preference, realtime, refresh, _source, version_type]
           only in SENSE []
  [indices.get] api matches sense [_get_index]
    WARN:  [indices.get] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards]
           only in API [local, flat_settings, human]
           only in SENSE []
  [cat.count] api matches sense [_cat/count]
    WARN:  [cat.count] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [render_search_template] api matches sense [_render_search_template,_render_search_template_with_id]
  [indices.get_aliases] api matches sense [_get_aliases]
    WARN:  [indices.get_aliases] query parameters do not match
           in both []
           only in API [timeout, local]
           only in SENSE []
  [cat.pending_tasks] api matches sense [_cat/pending_tasks]
    WARN:  [cat.pending_tasks] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [cat.shards] api matches sense [_cat/shards]
    WARN:  [cat.shards] query parameters do not match
           in both [help, v]
           only in API [local, master_timeout, h]
           only in SENSE [bytes]
  [indices.get_settings] api matches sense [_get_index_settings,_get_settings]
    WARN:  [indices.get_settings] query parameters do not match
           in both [ignore_unavailable, allow_no_indices, expand_wildcards, flat_settings]
           only in API [local, human]
           only in SENSE []
  [indices.validate_query] api matches sense [_validate_query]
    WARN:  [indices.validate_query] query parameters do not match
           in both [explain, ignore_unavailable, allow_no_indices, expand_wildcards, rewrite]
           only in API [operation_threading, q, analyzer, analyze_wildcard, default_operator, df, lenient, lowercase_expanded_terms]
           only in SENSE []
  [nodes.info] api matches sense [_nodes/info]
    WARN:  [nodes.info] query parameters do not match
           in both []
           only in API [flat_settings, human, timeout]
           only in SENSE []
  [snapshot.delete] api matches sense [single_snapshot]
    WARN:  [snapshot.delete] query parameters do not match
           in both []
           only in API [master_timeout]
           only in SENSE []

Autosuggest for Watcher API

Would be great to have autosuggest for the Watcher API. Ideally the auto-suggestions would only be presented if the cluster has the Watcher plugin installed.

_mget with "get" rather than "GET" sends as querystring

This gets interpreted to send the request as a querystring for some reason:

get /employees/_mget
{
  "docs" : [
    {
      "_type" : "sales",
      "_id" : 1
    }
  ]
}

versus

GET /employees/_mget
{
  "docs" : [
    {
      "_type" : "sales",
      "_id" : 1
    }
  ]
}

Always get the welcome box when you go to sense

It would be good to have a cookie (or something) to not show the welcome message if the user has opened sense before

Support Swagger

I love Sense! It would be really awesome to be able to point this to any Swagger API and have it read the API description and automatically provide intellisense for the API.

Make font size box display correctly

The font size box is not currently tall enough and is truncated in the settings.

Write docs

Sense logo goes away

Add License autocomplete to KB

Multi request fails to stop on failure...

Add release notes

Most of the docs are done in #62 , but there is still a release not list to compile

Sense breaks on HEAD verb

# add a document
PUT /website/blog/123
{
  "title": "hello there"
}

# get to make sure it's in... it is!
GET /website/blog/123

# break Sense :)
HEAD /website/blog/123

Spits out

Error: Uncaught TypeError: Cannot read property 'promise' of undefined (http://localhost:5601/bundles/sense.bundle.js:23015)
at window.onerror (http://localhost:5601/bundles/commons.bundle.js:57730:25)

Hover on copy in curl results in blue highlight

as opposed to the grey one we need...

Sense initialises with empty server field

Change default proxyFilter

Currently it's only documented: https://www.elastic.co/guide/en/sense/current/installing.html#securing_sense

A localhost default strikes the right balance between out of the box experience and security.

Update Sense to work on latest commit for Kibana 4.2

Figure out deplyoment/build

Remove unneeded dependencies (lodash, bootstrap, jquery-ui)

Sense have some legacy dependencies that can probably go away and replaced by the standard kibana libs.

Re-intrduce dark theme

Sense result pane only shows 14 lines on IE 11

Using Kibana 4.2.0-snapshot on Windows 10.
I installed sense, restarted Kibana, opened sense in IE 11, ran the default query, the results came back but only shows lines 1 - 14. I can scroll down on the right pane and see more results but the lower 3/4 of the pane stays blank (see screenshot).

But once I resize the IE just slightly, the right pane repainted and showed all lines correctly. Running the query again shows all result lines correctly. So in my case it was just a problem on the first opening of the sense page.

I closed IE and reopened it, went to sense, ran the query, it worked fine. I'm not sure if I could reproduce it at this point. It could be an issue with my UHD video driver on my ASUS laptop.

Seems to work fine on Microsoft Edge browser.

Here's the output of me installing sense;

C:\kibana-4.2.0-snapshot-windows\kibana-4.2.0-snapshot-windows\bin>kibana.bat plugin -i elastic/sense
Installing sense
Attempting to extract from https://download.elastic.co/elastic/sense/sense-latest.tar.gz
Downloading 318236 bytes....................
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

C:\kibana-4.2.0-snapshot-windows\kibana-4.2.0-snapshot-windows\bin>

View in Sense is no longer working

The View in Sense buttons in the docs no longer work. There were issues with the website itself (see https://github.com/elastic/website-www.elastic.co/issues/744) but these have been resolved. Now I think the issue is with Sense/Kibana.

When trying to load this URL: http://localhost:5601/app/sense?load_from=http://www.elastic.co/guide/en/elasticsearch/reference/current/snippets/copy-to/1.json

I get this in the console:

unable to load ace/lib/es5-shim
sense?load_from=http://www.elastic.co/guide/en/elasticsearch/reference/current/snippets/copy-to/1.j…:1 XMLHttpRequest cannot load http://www.elastic.co/guide/en/elasticsearch/reference/current/snippets/copy-to/1.json. 
Request header field kbn-xsrf-token is not allowed by Access-Control-Allow-Headers in preflight response.

So first the JSON snippet doesn't load. But then there is a second issue caused (I think) by the ?load_from... query string. If I try to run a command in Sense, I get:

Uncaught TypeError: Cannot read property 'indexOf' of undefined (http://localhost:5601/bundles/sense.bundle.js:28371)
Version: 4.3.0-snapshot
Build: 9367
Error: Uncaught TypeError: Cannot read property 'indexOf' of undefined (http://localhost:5601/bundles/sense.bundle.js:28371)
at window.onerror (http://localhost:5601/bundles/commons.bundle.js:64531:25)

Unicode characters seem to encode incorrectly?

From: https://discuss.elastic.co/t/smart-chinese-analysis-returns-unicodes-instead-of-chinese-tokens

Unicode is far from my expertise, so I may be very wrong about this. It seems that Sense is urlencoding unicode characters, which is preventing them from being properly decoded at Elasticsearch?

For example, if we setup a smartcn analyzer and analyze some chinese characters:

PUT /test_chinese
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "default": {
            "type": "smartcn"
          }
        }
      }
    }
  }
}

GET /test_chinese/_analyze?text='我说世界好!'

The tokens are incorrect:

{
  "tokens": [
    {
      "token": "\u0011",
      "start_offset": 1,
      "end_offset": 2,
      "type": "word",
      "position": 1
    },
    {
      "token": "�",
      "start_offset": 2,
      "end_offset": 3,
      "type": "word",
      "position": 2
    },
    {
      "token": "\u0016",
      "start_offset": 3,
      "end_offset": 4,
      "type": "word",
      "position": 3
    },
    {
      "token": "l",
      "start_offset": 4,
      "end_offset": 5,
      "type": "word",
      "position": 4
    }
  ]
}

If we look at what get's sent over the wire:

http://localhost:5601/api/sense/proxy?uri=http%3A%2F%2Flocalhost%3A9200%2Ftest_chinese%2F_analyze%3Ftext%3D%27%E6%88%91%E8%AF%B4%E4%B8%96%E7%95%8C%E5%A5%BD!%27&_=1450180681591


Decoded:  http://localhost:5601/api/sense/proxy?uri=http://localhost:9200/test_chinese/_analyze?text='我说世界好!'&_=1450180681591

So that part looks ok (unlike previous versions of Sense), So I suspect the proxy portion is what's incorrectly encoding. I pulled out a packet sniffer and this is what the proxy is sending to ES:

GET /test_chinese/_analyze?text=%27���L}!%27 HTTP/1.1
connection: keep-alive
x-forwarded-proto: http
accept: text/plain, */*; q=0.01
referer: http://localhost:5601/app/sense
kbn-xsrf-token: 959b10246601e4bc85e7f57d254ea23c31800cd60b36ee50627d0b6ef84f52f7
accept-encoding: gzip, deflate, sdch
x-forwarded-for: 127.0.0.1
accept-language: en-US,en;q=0.8
x-forwarded-port: 59597
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36
Host: localhost:9200

Full Packet dump:

0000   02 00 00 00 45 00 02 53 b2 0a 40 00 40 06 00 00  ....E..S..@.@...
0010   7f 00 00 01 7f 00 00 01 e8 ce 23 f0 b9 86 e9 fd  ..........#.....
0020   ce 74 fb d2 80 18 31 d7 00 48 00 00 01 01 08 0a  .t....1..H......
0030   1b bd 77 d1 1b bd 77 d1 47 45 54 20 2f 74 65 73  ..w...w.GET /tes
0040   74 5f 63 68 69 6e 65 73 65 2f 5f 61 6e 61 6c 79  t_chinese/_analy
0050   7a 65 3f 74 65 78 74 3d 25 32 37 11 f4 16 4c 7d  ze?text=%27...L}
0060   21 25 32 37 20 48 54 54 50 2f 31 2e 31 0d 0a 63  !%27 HTTP/1.1..c
0070   6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 70 2d  onnection: keep-
0080   61 6c 69 76 65 0d 0a 78 2d 66 6f 72 77 61 72 64  alive..x-forward
0090   65 64 2d 70 72 6f 74 6f 3a 20 68 74 74 70 0d 0a  ed-proto: http..
00a0   61 63 63 65 70 74 3a 20 74 65 78 74 2f 70 6c 61  accept: text/pla
00b0   69 6e 2c 20 2a 2f 2a 3b 20 71 3d 30 2e 30 31 0d  in, */*; q=0.01.
00c0   0a 72 65 66 65 72 65 72 3a 20 68 74 74 70 3a 2f  .referer: http:/
00d0   2f 6c 6f 63 61 6c 68 6f 73 74 3a 35 36 30 31 2f  /localhost:5601/
00e0   61 70 70 2f 73 65 6e 73 65 0d 0a 6b 62 6e 2d 78  app/sense..kbn-x
00f0   73 72 66 2d 74 6f 6b 65 6e 3a 20 39 35 39 62 31  srf-token: 959b1
0100   30 32 34 36 36 30 31 65 34 62 63 38 35 65 37 66  0246601e4bc85e7f
0110   35 37 64 32 35 34 65 61 32 33 63 33 31 38 30 30  57d254ea23c31800
0120   63 64 36 30 62 33 36 65 65 35 30 36 32 37 64 30  cd60b36ee50627d0
0130   62 36 65 66 38 34 66 35 32 66 37 0d 0a 61 63 63  b6ef84f52f7..acc
0140   65 70 74 2d 65 6e 63 6f 64 69 6e 67 3a 20 67 7a  ept-encoding: gz
0150   69 70 2c 20 64 65 66 6c 61 74 65 2c 20 73 64 63  ip, deflate, sdc
0160   68 0d 0a 78 2d 66 6f 72 77 61 72 64 65 64 2d 66  h..x-forwarded-f
0170   6f 72 3a 20 31 32 37 2e 30 2e 30 2e 31 0d 0a 61  or: 127.0.0.1..a
0180   63 63 65 70 74 2d 6c 61 6e 67 75 61 67 65 3a 20  ccept-language: 
0190   65 6e 2d 55 53 2c 65 6e 3b 71 3d 30 2e 38 0d 0a  en-US,en;q=0.8..
01a0   78 2d 66 6f 72 77 61 72 64 65 64 2d 70 6f 72 74  x-forwarded-port
01b0   3a 20 35 39 35 39 37 0d 0a 75 73 65 72 2d 61 67  : 59597..user-ag
01c0   65 6e 74 3a 20 4d 6f 7a 69 6c 6c 61 2f 35 2e 30  ent: Mozilla/5.0
01d0   20 28 4d 61 63 69 6e 74 6f 73 68 3b 20 49 6e 74   (Macintosh; Int
01e0   65 6c 20 4d 61 63 20 4f 53 20 58 20 31 30 5f 31  el Mac OS X 10_1
01f0   31 5f 31 29 20 41 70 70 6c 65 57 65 62 4b 69 74  1_1) AppleWebKit
0200   2f 35 33 37 2e 33 36 20 28 4b 48 54 4d 4c 2c 20  /537.36 (KHTML, 
0210   6c 69 6b 65 20 47 65 63 6b 6f 29 20 43 68 72 6f  like Gecko) Chro
0220   6d 65 2f 34 37 2e 30 2e 32 35 32 36 2e 37 33 20  me/47.0.2526.73 
0230   53 61 66 61 72 69 2f 35 33 37 2e 33 36 0d 0a 48  Safari/537.36..H
0240   6f 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 3a 39  ost: localhost:9
0250   32 30 30 0d 0a 0d 0a                             200....

For comparison, if you run the command via curl, you get the proper tokens back:

$ curl -XGET -v "http://127.0.0.1:9200/test_chinese/_analyze?text='我说世界好!'&pretty"

* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
> GET /test_chinese/_analyze?text='我说世界好!'&pretty HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 1749
< 
{
  "tokens" : [ {
    "token" : "￦",
    "start_offset" : 1,
    "end_offset" : 2,
    "type" : "word",
    "position" : 1
  }, {
    "token" : "ﾈ",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "word",
    "position" : 2
  }, {
    "token" : "ﾑ",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "word",
    "position" : 3
  }, {
    "token" : "￨",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "word",
    "position" : 4
  }, {
    "token" : "ﾯ",
    "start_offset" : 5,
    "end_offset" : 6,
    "type" : "word",
    "position" : 5
  }, {
    "token" : "ﾴ",
    "start_offset" : 6,
    "end_offset" : 7,
    "type" : "word",
    "position" : 6
  }, {
    "token" : "￤",
    "start_offset" : 7,
    "end_offset" : 8,
    "type" : "word",
    "position" : 7
  }, {
    "token" : "ﾸ",
    "start_offset" : 8,
    "end_offset" : 9,
    "type" : "word",
    "position" : 8
  }, {
    "token" : "ﾖ",
    "start_offset" : 9,
    "end_offset" : 10,
    "type" : "word",
    "position" : 9
  }, {
    "token" : "￧",
    "start_offset" : 10,
    "end_offset" : 11,
    "type" : "word",
    "position" : 10
  }, {
    "token" : "ﾕ",
    "start_offset" : 11,
    "end_offset" : 12,
    "type" : "word",
    "position" : 11
  }, {
    "token" : "ﾌ",
    "start_offset" : 12,
    "end_offset" : 13,
    "type" : "word",
    "position" : 12
  }, {
    "token" : "￥",
    "start_offset" : 13,
    "end_offset" : 14,
    "type" : "word",
    "position" : 13
  }, {
    "token" : "ﾥ",
    "start_offset" : 14,
    "end_offset" : 15,
    "type" : "word",
    "position" : 14
  }, {
    "token" : "ﾽ",
    "start_offset" : 15,
    "end_offset" : 16,
    "type" : "word",
    "position" : 15
  } ]
}

Sense fails with SSL (using own RootCA)

Using 4.2 kibana snapshot with config

abonuccelli@w530 /opt/elk/PRODSEC/kibana/kibana-4.2.0-snapshot-linux-x64 $ egrep '^[^#]' config/kibana.yml 
server.host: "0.0.0.0"
elasticsearch.url: "https://localhost:9200"
elasticsearch.username: kibana_server_user
elasticsearch.password: r1ng3r
elasticsearch.ssl.ca: /opt/elk/PRODSEC/FS/secure/cacert.pem
elasticsearch.ssl.verify: true
logging.verbose: true

RootCa cert

abonuccelli@w530 /opt/elk/PRODSEC/kibana/kibana-4.2.0-snapshot-linux-x64 $ openssl x509 -in /opt/elk/PRODSEC/FS/secure/cacert.pem -inform pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 18411508252604821347 (0xff82d13517aeef63)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN="Tony-Secure-CA"
        Validity
            Not Before: Sep 29 11:11:46 2015 GMT
            Not After : Sep 26 11:11:46 2025 GMT
        Subject: CN="Tony-Secure-CA"
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c9:10:af:e5:f4:30:5f:78:b8:7c:ec:a8:3c:ac:
                    bb:4f:c8:d8:ef:83:1d:44:15:7a:80:a6:c4:e9:b2:
                    59:bb:0f:3e:26:d8:32:8a:48:ce:f7:60:28:a4:b2:
                    e2:55:23:e8:36:1c:a1:ac:6f:33:9f:8a:34:c2:af:
                    f1:cb:d1:6d:a7:f9:3d:05:68:f3:68:cc:d1:ca:db:
                    af:3d:5f:a5:73:d8:03:5f:39:65:b0:cc:4d:54:43:
                    af:1e:66:23:6f:9b:29:ba:2b:4d:a9:b8:85:73:f9:
                    b1:8c:ea:62:d9:8a:18:15:b4:7d:f5:71:48:c5:60:
                    5e:ac:b6:6c:a7:e7:3f:93:82:22:5b:21:52:f5:3c:
                    e6:c5:d4:ff:55:f7:2a:19:3c:d4:c2:b8:04:3d:4d:
                    e7:1c:64:c7:a4:3d:28:1d:2f:82:d0:62:4b:4a:02:
                    71:80:26:f9:20:de:5e:7c:9b:b0:f4:e9:f8:21:e9:
                    44:7a:62:bb:33:cb:0e:50:a8:70:23:34:10:7d:ac:
                    ca:0a:0f:7e:86:53:ec:97:99:ad:c8:08:ed:ca:d0:
                    68:03:8b:a3:bc:f6:de:da:4a:8a:de:34:70:fa:6c:
                    a1:eb:50:37:b1:1f:5b:bc:80:a0:60:21:94:1e:9a:
                    7d:f8:a2:f4:67:c2:5c:ee:d8:fb:15:84:1b:0c:3b:
                    06:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:TRUE
            X509v3 Subject Key Identifier: 
                CE:F3:A7:6E:CC:1A:F2:1E:D5:41:9C:00:48:DC:2B:4C:6C:98:8C:33
            X509v3 Authority Key Identifier: 
                keyid:CE:F3:A7:6E:CC:1A:F2:1E:D5:41:9C:00:48:DC:2B:4C:6C:98:8C:33
                DirName:/CN="Tony-Secure-CA"
                serial:FF:82:D1:35:17:AE:EF:63

    Signature Algorithm: sha256WithRSAEncryption
         8f:bd:46:55:dd:c4:4a:9f:d3:a7:47:78:7d:40:50:59:99:d3:
         08:cf:0c:e9:dc:14:89:6e:5b:34:de:63:d8:84:9c:78:95:a5:
         a7:3c:5d:93:90:ea:f6:72:ed:a3:1e:59:7a:d9:7b:36:a2:2a:
         d4:1b:bc:a1:d6:4f:be:44:5a:cd:50:7a:ae:77:30:69:80:54:
         61:09:c7:5e:d2:08:71:ed:d1:6a:42:87:ae:b4:fb:35:24:d9:
         97:dd:a8:44:2a:79:3a:d4:b8:e1:8f:39:25:8a:7e:8b:bd:a9:
         88:24:2e:0d:88:4a:3d:ef:dd:9f:e6:7c:f2:aa:17:9a:c9:c1:
         d3:09:1b:50:8f:f3:3f:fc:fe:a6:c1:5c:56:4f:62:f3:00:7e:
         bc:48:37:4c:ac:a3:8f:30:d7:9e:bb:91:c2:52:69:f0:8f:da:
         27:78:b2:b2:34:60:7c:1f:54:d3:0e:35:0f:a5:fa:73:d1:f4:
         65:9a:ed:a0:bf:fa:65:b1:43:9f:96:e8:91:dc:14:35:11:46:
         7e:02:e1:03:66:7b:06:82:c9:56:52:8e:04:7f:e1:45:48:73:
         05:1d:c5:81:34:d0:96:ed:e7:a3:e9:76:10:e1:f5:f0:c2:d5:
         0e:aa:0a:db:66:82:79:b9:58:ce:3e:c0:3d:42:e0:8f:14:7d:
         85:b2:46:ee

generated as per https://www.elastic.co/guide/en/shield/current/certificate-authority.html

Re-introduce in-flight request indication

When people submit heavy requests to ES it may take time for it to respond. We used to have a crude indication of the fact that we had an in flight request but we lost it in the re-design. Adding an issue to remember to introduce an alternative.

Do not use global autocomplete rules if the are ones locally defined

At the moment every time a key is specified which is hooked up to global rules, the globals rules will be applied:

_search
{
   filter {
     ... global rules used here
   }
}

However it should be possible to disable this and supply local rules instead. See #27 for an example

make autocomplete __one_of work with multiple types

We have a few cases where a field can legally be different types. The most prominent example of this is the Script API where a script can be specified as a string (example in kb rules format):

"script": ""

or as an object:

"script": {
    __template: {
        inline: ""
    },
    inline: "",
    file: "",
    id: "",
    lang: "",
    params: {}
}

We want script to be a global rule and support either of these suggestions but the following does not work:

api.addGlobalAutocompleteRules('script', { __one_of: [ "", {
    __template: {
        inline: ""
   },
   inline: "",
   file: "",
   id: "",
   lang: "",
   params: {}
} ] });

At runtime when asking for autocomplete for a script field this logs a warning as follows as does not suggest anything:

error while getting completion terms TypeError: component.getTerms is not a function
    at http://sense.dev/app/autocomplete/body_completer.js:290:47
    at Function.forEach (http://sense.dev/vendor/lodash.js:3639:15)
    at ScopeResolver.cls.getTerms (http://sense.dev/app/autocomplete/body_completer.js:289:9)
    at http://sense.dev/app/autocomplete/engine.js:331:30
    at Function.forEach (http://sense.dev/vendor/lodash.js:3639:15)
    at http://sense.dev/app/autocomplete/engine.js:330:13
    at Function.forEach (http://sense.dev/vendor/lodash.js:3639:15)
    at Object.exports.populateContext (http://sense.dev/app/autocomplete/engine.js:328:11)
    at addBodyAutoCompleteSetToContext (http://sense.dev/app/autocomplete.js:618:29)
    at getAutoCompleteContext (http://sense.dev/app/autocomplete.js:246:13)

I think the problem might be because json_rule_walker.js expects that all entries in __one_of are the same type in the getRulesType() method

Document lockdown

lowercase get method in mget causes failed to derive from xcontent

The following call doesn't work from sense because of the lowercase get, although with other apis that (GET vs get) doesn't seem to be a problem:

get _mget
{
  "docs" : [
    {
      "_index":"my_index",
      "_type":"my_type",
      "_id":"1"
    }
  ]
}

causes a "failed to derive from xcontent error returned from elasticsearch, I guess because of the incorrect request body that gets sent.

Following call works perfectly though:

GET _mget
{
  "docs" : [
    {
      "_index":"my_index",
      "_type":"my_type",
      "_id":"1"
    }
  ]
}

Server url is only updated on focus

Sometimes the current server is updated from JavaScript (mostly upon initial load). This is not always immediately synced back to the UI but rather requires a focus change.

To reproduce use the following url:
http://localhost:5601/app/sense?load_from=http://www.elastic.co/guide/en/elasticsearch/guide/current/snippets/030_Data/10_Create_doc_123.json

Dependency on kibana is undesirable

Sense now requires elasticsearch and kibana to be installed. This is sub optimal as kibana uses up unnecessary memory in tightly constrained environments. For example when deploying a docker container containing elasticsearch, it is now necessary to run a seperate docker container to run kibana and link the two together.

Ideally it would be possible to install sense as an elasticsearch plugin.

sense offline install

how to manually (offline) install sense.
what do i need to download and what are the command/s to install ?

auto completion for array of values in url parameters

Currently we cannot define autocompletion for url parameters if the parameter can take a list of values such as h=host,ip,port .

Once this is fixed we also need to fix kb for _cat/nodeattrs , see
#16

Add phone home data and banner

once settled down...

Sense now has weird ES dependencies based upon Kibana

You have to have a local ES instance running in order to access Sense at all, which seems like a weird requirement given its current focus as a (much, much smarter) "cURL replacement".

1. Start Kibana (but don't start ES)
2. Navigate to the Sense app

You get the Kibana cluster status page only being red. This makes it impossible to, for example, use Sense locally against only a remote cluster (without also having a local instance running)

GET request ends up indexing a document

If you have a search api call but you omit the _search endpoint in the url, you end up indexing a document:

GET test/my_type
{
  "query" : {
    "match_all": {}
  }
}

Same doesn't happen via command line, where you get a "feature not found" as the request goes to the get index api and the type is not recognized as a feature. Good as it throws an error rather than writing. We should never write with GET!

If the server field is empty you get no auto complete

No really sure if this is a bug as the server field shouldn't ever really be blank anyway. This is only evident at the moment because of #7

Oops when typing and then doing shift + enter

The following error is being returned:

Steps to reproduce

1. Open Kibana
2. Open Sense
3. Type anything with a space (example: sadasd dsadass), and press Shift and Enter

Beta1 labeling

As of today, the latest Sense release is 2.0-beta1. The "beta" tagging is causing some confusion in the field for the end user expects the version 2.0 to match up with ES 2.0. Pretty sure this is expected today, so just creating this ticket to discuss if we can sync up Sense and ES versioning for future releases.

Remove license headers and add license.md

As follow up to our build discussion, a license.md file is sufficient. Opening this issue to make sure we don't forget to do this once all the dust settles around the refactorign

Left over KB todos from ES 2.0

Add reverse nested aggregation #5485
Double check and bring node / indices stats (metrics) up to do date
Add "external_gt", "external_gte", and "force" version types #4213, #2946
Add analyzer end point
Add children aggregation #6936
Add an option to early terminate document collection when searching/counting #6885
Add inner hits to nested and parent/child queries #8153
Add an unlike parameter #8674
Support terms filtering #9561 IGOR

Update to latest design

See attachments. One thing that's not designed is an activity indication. Let's try to come up with something and o.w. ask jurgen.

Bottom of code window hidden

On Chrome and Safari (at least) the last few lines of the left hand code window are not visible. I have to add blank lines to the end in order to be able to scroll down sufficiently.

eg, if you put the following into the code window, i only see up to line 47 (results may vary based on screen size)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Make sure sense works on other browsers too

I wasn't aware of the fact that right now Kibana only works on chrome, but when Kibana works on other browsers too, we should verify if Sense does too.

Sense doesn't save previous session state

Unlike the previous version of sense, the state in the request view is not saved when you reload

http response not captured

when I execute comand and there is error, but sense can't show the error info.
version 2.1

Option to proxy a specific cluster, and that cluster only

To deploy Sense on Found, we need to be able to have the proxy only proxy requests to a pre-configured cluster, i.e. the cluster that Kibana is pointing at. Currently it can connect to anything, which makes it a risk to deploy.

In that mode, it would make sense to hide the server form, and just have it connect to the cluster it is attached to.

Black Theme

Looks like the Black Theme has been removed from SENSE. This was used by many developers as well as for Demos.

Autosuggest for Shield API

Would be great to have autosuggest for the Shield API. Ideally the auto-suggestions would only be presented if the cluster has the Shield plugin installed.

Display warning if > 10K words per line are used

Currently Ace stops parsing after 10K words on a line, which means things can get funky if a user has a massive query.

We should display a warning when we hit this limit so it's obvious to users.

Switch out zeroclipbard for clipboard.js

Every browser except safari supports the api's necessary to support the "copy as curl" functionality without requiring flash or zeroclipboard.

clipboard.js wraps these apis and provides a fallback for safari which focusses an input box in the background and instructs the user to press command+c in order to complete the copy.