quickwit-oss / quickwit-datasource Goto Github PK
View Code? Open in Web Editor NEWQuickwit data source for Grafana
License: GNU Affero General Public License v3.0
quickwit-datasource's People
Contributors
Stargazers
Watchers
Forkers
hoangphuocbk ppaulweber jpds idrissneumann etolbakov ddelemeny superfly wjordan silentsokolovquickwit-datasource's Issues
Trigger a search on key 'Shift-Enter' in lucene query editor
Now that no query is run on query changes, we need to execute the query by clicking on the run button. From my point of view, we can't release with such a behavior. Did I miss something?
It's possible to have custom key binding with CodeMirror so we can fix that.
const keyMapExtension = keymap.of([
{
key: 'Shift-Enter',
run: () => {
// Do something
return true
}
}
])
Document with nested arrays are not correclty displayed in the explorer
Update grafana deps, skim deprecations
Add sort direction
Sometimes, there's a need to change the sorting direction (timestamp only). Unfortunately, the defalut options available in the Grafana interface may not allow for changing the sorting direction.
Could we consider adding a special option for that?
Default query should be a conjunction.
right now by default, we consider the query timed out as timed OR out, it should be interpreted as timed AND out.
(the elasticsearch API has a query parameter for that)
Remove the timestamp and timestamp format fields from datasource configuration
Those fields can be found using this endpoint on the datasource save event:
Keeping query context while changing query types
Example to reproduce the lost of query context:
- Enter a metrics query (term or date histogram)
- Execute it
- Change logs query
- Go back to metrics query: you lost your previous query.
It would be nice to keep the metrics query context.
Exclude query not working
When using using the "Filter out value in query" button in Grafana Explore view, it produces a query which doesn't work.
Moreover if you exclude two fields, the query fails due to the syntax, I believe:
Console warn "Duplicate field names in DataFrame: message" seen on a Grafana instance with 0.4.0-beta.2 version:
I did not reproduce it locally. I tested a Grafana 10.3.3 with Quickwit 0.4.0-beta.2 installed.
Correlate logs to traces
The elasticsearch plugin allows to define how to link a log to a trace. See the datalinks config part.
We should be able to have the same feature and provide this dataLinks configuration.
Here is a config example coming from the doc:
dataLinks:
- datasourceUid: my_jaeger_uid # Target UID needs to be known
field: traceID
url: '$${__value.raw}' # Careful about the double "$$" because of env var expansionTo test the setup, we need 2 datasources: one quickwit datasource for the logs and one jaeger datasource for traces.
Improve UX when defining an index with its timestamp field and output format
The UI should validate the output format and we should autofill the input by calling retrieving the output format on quickwit API.
Use errorsource from grafana-plugin-sdk
Throw an error when quickwit is responding a status_code >= 400
Today with the #23 PR, we got this piece of code to improve the debugging:
if res.StatusCode >= 400 {
jsonResponseBody, _ := json.Marshal(res.Body)
jsonQueryParam, _ := json.Marshal(queryParams)
jsonRequestBody, _ := json.Marshal(r.Requests)
c.logger.Error("Error on multisearch: statusCode = " + strconv.Itoa(res.StatusCode) + ", responseBody = " + string(jsonResponseBody) + ", queryParam = " + string(jsonQueryParam) + ", requestBody = " + string(jsonRequestBody))
}Improve this by throwing directly an error, you'll probably have to fix some unit tests.
Add dynamic fields in the list of available field in aggregation query editor
When forging an aggregation query, we need to choose the fields to aggregate on. Currently, the list of proposed fields is limited to fields available in the index doc mapping. But, in the case of a dynamic mapping, this means that most of the fields can be missed. Instead of using the doc mapping to get the list of available fields, we should use the field capabilities endpoint of Quickwit when it will be added.
Endpoint to use: _elastic/{index-id}/_field_caps
Docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html
Failed to load log volume for this query
Hello
Im not sure if its grafana / quickwit plugin,
we are getting this error around 50% of the times:
Failed to load log volume for this query
Internal error: `Aborting aggregation because memory limit was exceeded. Limit: 500.00 MB, Current: 1.36 PB`.
Grafana version: 10.2.2
Quickwit datasource plugin version: 0.2.4
image for reference
:
Handling several indexes per datasource
Currently, a quickwit datasource requires to define an index.
We want a user to be able to query different indexes (using the index pattern) or query multiple indexes with a single datasource.
Here is the proposed spec.
Datasource parameters
- timestamp field(mandatory): This field is used across all views and ensures that all indexes used are on the same timestamp field. We could make it optional but we currently don't support querying multiple indexes with different timestamp fields.
- timestamp output format (mandatory): we don't want the user to set this value, but I think we don't really have the choice for now. This format is used by the Grafana plugin to show correctly the datetime. We could guess the format at the backend plugin level though.
- default index(optional): either an index ID or an index pattern or empty.
Query builder
We should add an index field filled with the default value in the query builder.
Validation
If an index or index pattern is defined, we can validate the timestamp field and timestamp field output format.
Add tests on template variable interpolation
Query variable sometimes shows 'None'
Context:
It seems that variables are sometimes lost when refreshing (F5) a dashboard with predefined query variables.
This is my settings in grafana:
By checking the browser inspector, it appears to be related to a call with _elastic/_msearch:
{"size":0,"query":{"bool":{"filter":[{"range":{"":{"gte":"1706814729878","lte":"1706836329878"}}}]}},"aggs":{"1":{"terms":{"field":"attributes.environment","size":500,"order":{"_key":"asc"}}}}}
As we can see, the timestamp_nanos field is empty, causing a bad request when attempting to retrieve different possible values for this field.
Environment:
Grafana version: v10.1.2, v10.3.1
Improve context feature
We should be able to replicate what is available for Loki: https://grafana.com/blog/2023/09/12/new-in-grafana-10-better-log-context-for-better-log-analysis/
Reduce the number of bars in the log volume panel in explore view
As reported by a user, here is a screenshot where we can see that we have a bit too many bars
Valid query considered invalid by the editor
default fields inexistent in trace
By default, when searching traces in grafana, the default field is not set.
We have some relevant field we should be able to search into.
span_name for instance, but also event_names, which is a tokenized name that is not even visible from the user without looking into our source code.
Remove the "script" input in the plugin UI
Reported by @yangshike
"missing" field feature for aggregations
In the aggregation API, there is a "missing" field available in order to configure a default value for documents where the aggregated field does not exist. It would be nice if we could have the same feature when we use aggregations in Grafana.
For example for each term, we could configure a default value in case the field is missing.
Customize display of a log line in the explore view
Currently, it's possible to define a field to display a log line in the explore view.
It's not very flexible, and it would be nice to be able to customize it a bit more. We need to define some spec (and probably look at the last evolutions in Grafana explore view)
Remove `TimeOutputFormat` from the datasource
Queries that retrieve documents are all sorted by the timestamp field. At request time, we enforce the output format of the sorted value in nanosecond with b.Sort(es.SortOrderDesc, defaultTimeField, "epoch_nanos_int").
The ES response contains a sort field in each hit. This sort field is an array of values with a size > 0, and we know that the first element of the array is the timestamp value in nanosecond.
So we don't need to know the TimeOutputFormat, we have already the info in the hit, we just need to use it.
Add support for ad hoc filters
Fix the breaking change with the last version of grafana lib
On the github action of all opening pull requests:
But if you upgrade the dependancy to 10.2.2, you'll see a huge amount of breaking changes to fix:
assets by status 814 KiB [cached] 10 assets
orphan modules 516 KiB [orphan] 258 modules
runtime modules 1.13 KiB 5 modules
built modules 354 KiB [built]
modules by path external "@grafana/ 126 bytes
external "@grafana/data" 42 bytes [built] [code generated]
external "@grafana/runtime" 42 bytes [built] [code generated]
external "@grafana/ui" 42 bytes [built] [code generated]
./module.ts + 90 modules 353 KiB [built] [code generated]
external "lodash" 42 bytes [built] [code generated]
external "rxjs" 42 bytes [built] [code generated]
external "react" 42 bytes [built] [code generated]
external "@emotion/css" 42 bytes [built] [code generated]
external "redux" 42 bytes [built] [code generated]
ERROR in ./src/components/QueryEditor/index.tsx:21:56
TS2344: Type 'QuickwitDataSource' does not satisfy the constraint 'DataSourceApi<Elasticsearch, QuickwitOptions, {}>'.
Types of property 'importQueries' are incompatible.
Type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-...' is not assignable to type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery...'.
19 | import { QueryTypeSelector } from './QueryTypeSelector';
20 |
> 21 | export type ElasticQueryEditorProps = QueryEditorProps<ElasticDatasource, ElasticsearchQuery, QuickwitOptions>;
| ^^^^^^^^^^^^^^^^^
22 |
23 | export const QueryEditor = ({ query, onChange, onRunQuery, datasource, range, app }: ElasticQueryEditorProps) => {
24 | return (
ERROR in ./src/components/QueryEditor/index.tsx:53:29
TS2345: Argument of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { root: string; queryItem: string; }' is not assignable to parameter of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { root: string; queryItem: string; }'.
Types of parameters 'theme' and 'theme' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2'.
51 |
52 | export const ElasticSearchQueryField = ({ value, onChange }: { value?: string; onChange: (v: string) => void }) => {
> 53 | const styles = useStyles2(getStyles);
| ^^^^^^^^^
54 |
55 | return (
56 | <div className={styles.queryItem}>
ERROR in ./src/components/QueryEditor/index.tsx:73:29
TS2345: Argument of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { root: string; queryItem: string; }' is not assignable to parameter of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { root: string; queryItem: string; }'.
71 | const dispatch = useDispatch();
72 | const nextId = useNextId();
> 73 | const styles = useStyles2(getStyles);
| ^^^^^^^^^
74 |
75 | const showBucketAggregationsEditor = value.metrics?.every(
76 | (metric) => metricAggregationConfig[metric.type].impliedQueryType === 'metrics'
ERROR in ./src/components/QueryEditor/MetricAggregationsEditor/MetricEditor.tsx:55:28
TS2345: Argument of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2' is not assignable to parameter of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2'.
53 |
54 | export const MetricEditor = ({ value }: Props) => {
> 55 | const styles = getStyles(useTheme2(), !!value.hide);
| ^^^^^^^^^^^
56 | const query = useQuery();
57 | const dispatch = useDispatch();
58 | const getFields = useFields(value.type);
ERROR in ./src/components/QueryEditor/QueryEditorRow.tsx:22:29
TS2345: Argument of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { iconWrapper: string; icon: string; }' is not assignable to parameter of type '(theme: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2) => { iconWrapper: string; icon: string; }'.
Types of parameters 'theme' and 'theme' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2'.
The types of 'colors.primary' are incompatible between these types.
Property 'borderTransparent' is missing in type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ThemeRichColor' but required in type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ThemeRichColor'.
20 | hidden = false,
21 | }: PropsWithChildren<Props>) => {
> 22 | const styles = useStyles2(getStyles);
| ^^^^^^^^^
23 |
24 | return (
25 | <InlineFieldRow>
ERROR in ./src/components/QueryEditor/SettingsEditorContainer.tsx:41:28
TS2345: Argument of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").GrafanaTheme2' is not assignable to parameter of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").GrafanaTheme2'.
39 |
40 | const theme = useTheme2();
> 41 | const styles = getStyles(theme, hidden);
| ^^^^^
42 |
43 | return (
44 | <InlineSegmentGroup>
ERROR in ./src/datasource.ts:174:3
TS2416: Property 'getDataProvider' in type 'QuickwitDataSource' is not assignable to the same property in base type 'DataSourceWithSupplementaryQueriesSupport<Elasticsearch>'.
Type '(type: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").SupplementaryQueryType, request: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/src/dataquery.gen").Elasticsearch>) => import("/...' is not assignable to type '(type: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").SupplementaryQueryType, request: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/src/dataquery.gen").Elasticsearch>) => import("/...'. Two different types with this name exist, but they are unrelated.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryResponse> | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryResponse> | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryResponse>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryResponse>'.
Types of property 'source' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<any> | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<any> | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<any>'.
Types of property 'operator' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any> | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any> | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any>'.
Types of property 'call' are incompatible.
Type '(subscriber: import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>, source: any) => import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/types").TeardownLogic' is not assignable to type '(subscriber: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>, source: any) => import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/types").TeardownLogic'.
Types of parameters 'subscriber' and 'subscriber' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>'.
Property 'isStopped' is protected but type 'Subscriber<T>' is not a class derived from 'Subscriber<T>'.
172 | * supplementary query type and original request.
173 | */
> 174 | getDataProvider(
| ^^^^^^^^^^^^^^^
175 | type: SupplementaryQueryType,
176 | request: DataQueryRequest<ElasticsearchQuery>
177 | ): Observable<DataQueryResponse> | undefined {
ERROR in ./src/datasource.ts:266:7
TS2345: Argument of type 'this' is not assignable to parameter of type 'DataSourceApi<Elasticsearch, QuickwitOptions, {}>'.
Type 'QuickwitDataSource' is not assignable to type 'DataSourceApi<Elasticsearch, QuickwitOptions, {}>'.
Types of property 'importQueries' are incompatible.
Type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-...' is not assignable to type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery...'.
Type '(queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-d...' is not assignable to type '(queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery,...'.
Types of parameters 'originDataSource' and 'originDataSource' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceJsonData, {}>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafan...'.
Types of property 'importQueries' are incompatible.
Type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery...' is not assignable to type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-...'.
Type '(queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery,...' is not assignable to type '(queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-d...'.
Types of parameters 'originDataSource' and 'originDataSource' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafan...' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceJsonData, {}>'.
Types of property 'query' are incompatible.
Type '(request: DataQueryRequest<DataQuery>) => Observable<DataQueryResponse> | Promise<DataQueryResponse>' is not assignable to type '(request: DataQueryRequest<DataQuery>) => Promise<DataQueryResponse> | Observable<DataQueryResponse>'.
Types of parameters 'request' and 'request' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery>'.
The types of 'scopedVars.__dataContext' are incompatible between these types.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataContextScopedVar | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataContextScopedVar | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataContextScopedVar' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataContextScopedVar'.
The types of 'value.frame.fields' are incompatible between these types.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").Field<any, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").Vector<any>>[]' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").Field<any, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").Vector<any>>[]'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").Field<any, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").Vector<any>>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").Field<any, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").Vector<any>>'.
Types of property 'type' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").FieldType' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").FieldType'.
Property 'nestedFrames' is missing in type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").FieldType'.
264 |
265 | return queryLogsVolume(
> 266 | this,
| ^^^^
267 | { ...logsVolumeRequest, targets },
268 | {
269 | range: request.range,
ERROR in ./src/datasource.ts:488:18
TS2345: Argument of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/src/dataquery.gen").Elasticsearch>' is not assignable to parameter of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQueryRequest<import("/Users/ineumann/quickwit-datasource/src/dataquery.gen").Elasticsearch>'.
Types of property 'scopedVars' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ScopedVars' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ScopedVars'.
486 |
487 | return lastValueFrom(
> 488 | this.query(contextRequest).pipe(
| ^^^^^^^^^^^^^^
489 | catchError((err) => {
490 | const error: DataQueryError = {
491 | message: 'Error during context query. Please check JS console logs.',
ERROR in ./src/datasource.ts:539:50
TS2345: Argument of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ScopedVars | undefined' is not assignable to parameter of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ScopedVars | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ScopedVars' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ScopedVars'.
537 |
538 | interpolateLuceneQuery(queryString: string, scopedVars?: ScopedVars) {
> 539 | return this.templateSrv.replace(queryString, scopedVars, formatQuery);
| ^^^^^^^^^^
540 | }
541 |
542 | interpolateVariablesInQueries(queries: ElasticsearchQuery[], scopedVars: ScopedVars | {}): ElasticsearchQuery[] {
ERROR in ./src/datasource.ts:547:3
TS2416: Property 'applyTemplateVariables' in type 'QuickwitDataSource' is not assignable to the same property in base type 'DataSourceWithBackend<Elasticsearch, QuickwitOptions>'.
Type '(query: Elasticsearch, scopedVars: ScopedVars) => Elasticsearch' is not assignable to type '(query: Elasticsearch, scopedVars: ScopedVars) => Record<string, any>'.
Types of parameters 'scopedVars' and 'scopedVars' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ScopedVars' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ScopedVars'.
Types of property '__dataContext' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataContextScopedVar | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataContextScopedVar | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataContextScopedVar' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataContextScopedVar'.
Types of property 'value' are incompatible.
Property 'data' is missing in type '{ frame: DataFrame; field: Field<any, Vector<any>>; rowIndex?: number | undefined; calculatedValue?: DisplayValue | undefined; }' but required in type '{ data: DataFrame[]; frame: DataFrame; field: Field<any, Vector<any>>; rowIndex?: number | undefined; frameIndex?: number | undefined; calculatedValue?: DisplayValue | undefined; }'.
545 |
546 | // Used when running queries through backend
> 547 | applyTemplateVariables(query: ElasticsearchQuery, scopedVars: ScopedVars): ElasticsearchQuery {
| ^^^^^^^^^^^^^^^^^^^^^^
548 | // We need a separate interpolation format for lucene queries, therefore we first interpolate any
549 | // lucene query string and then everything else
550 | const interpolateBucketAgg = (bucketAgg: BucketAggregation): BucketAggregation => {
ERROR in ./src/datasource.ts:574:91
TS2345: Argument of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").ScopedVars' is not assignable to parameter of type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").ScopedVars'.
572 | };
573 |
> 574 | const finalQuery = JSON.parse(this.templateSrv.replace(JSON.stringify(expandedQuery), scopedVars));
| ^^^^^^^^^^
575 | return finalQuery;
576 | }
577 | }
ERROR in ./src/datasource.ts:625:80
TS2345: Argument of type 'Promise<DataQueryResponse> | Observable<DataQueryResponse>' is not assignable to parameter of type 'ObservableInput<any>'.
Type 'Observable<DataQueryResponse>' is not assignable to type 'ObservableInput<any>'.
Type 'Observable<DataQueryResponse>' is not assignable to type 'Observable<any>'.
Types of property 'source' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<any> | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<any> | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Observable").Observable<any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Observable").Observable<any>'.
Types of property 'operator' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any> | undefined' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any> | undefined'.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Operator").Operator<any, any>'.
Types of property 'call' are incompatible.
Type '(subscriber: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>, source: any) => import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/types").TeardownLogic' is not assignable to type '(subscriber: import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>, source: any) => import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/types").TeardownLogic'.
Types of parameters 'subscriber' and 'subscriber' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/node_modules/rxjs/dist/types/internal/Subscriber").Subscriber<any>'.
Property 'isStopped' is protected but type 'Subscriber<T>' is not a class derived from 'Subscriber<T>'.
623 |
624 | const queryResponse = datasource.query(logsVolumeRequest);
> 625 | const queryObservable = isObservable(queryResponse) ? queryResponse : from(queryResponse);
| ^^^^^^^^^^^^^
626 |
627 | const subscription = queryObservable.subscribe({
628 | complete: () => {
ERROR in ./src/LanguageProvider.ts:9:3
TS2416: Property 'datasource' in type 'ElasticsearchLanguageProvider' is not assignable to the same property in base type 'LanguageProvider'.
Type 'QuickwitDataSource' is not assignable to type 'DataSourceApi<any, any, {}>'.
Types of property 'importQueries' are incompatible.
Type '((queries: DataQuery[], originDataSource: DataSourceApi<DataQuery, DataSourceJsonData, {}>) => Promise<Elasticsearch[]>) | undefined' is not assignable to type '((queries: DataQuery[], originDataSource: DataSourceApi<DataQuery, DataSourceJsonData, {}>) => Promise<any[]>) | undefined'.
Type '(queries: DataQuery[], originDataSource: DataSourceApi<DataQuery, DataSourceJsonData, {}>) => Promise<Elasticsearch[]>' is not assignable to type '(queries: DataQuery[], originDataSource: DataSourceApi<DataQuery, DataSourceJsonData, {}>) => Promise<any[]>'.
Types of parameters 'originDataSource' and 'originDataSource' are incompatible.
Type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceJsonData, {}>' is not assignable to type 'import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery, import("/Users/ineumann/quickwit-datasource/node_modules/@grafan...'.
7 | declare request: (url: string, params?: any) => Promise<any>;
8 | declare start: () => Promise<any[]>;
> 9 | datasource: ElasticDatasource;
| ^^^^^^^^^^
10 |
11 | constructor(datasource: ElasticDatasource, initialValues?: any) {
12 | super();
ERROR in ./src/module.ts:8:44
TS2344: Type 'QuickwitDataSource' does not satisfy the constraint 'DataSourceApi<Elasticsearch, QuickwitOptions, {}>'.
Types of property 'importQueries' are incompatible.
Type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/runtime/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-...' is not assignable to type '((queries: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery[], originDataSource: import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataSourceApi<import("/Users/ineumann/quickwit-datasource/node_modules/@grafana/data/dist/index").DataQuery...'.
6 | import { QuickwitOptions } from 'quickwit';
7 |
> 8 | export const plugin = new DataSourcePlugin<QuickwitDataSource, ElasticsearchQuery, QuickwitOptions>(QuickwitDataSource)
| ^^^^^^^^^^^^^^^^^^
9 | .setConfigEditor(ConfigEditor)
10 | .setQueryEditor(QueryEditor);
We're opening this issue for fixing all those breaking changes
Improve screenshot for the dashboard view.
Add autocompletion in the search query editor
To add it, we need to wait for the field capabilities endpoint in the Quickwit ES compatible API.
Context: a Quickwit index has generally many dynamic fields, this means that the doc mapping does not contain fields that can be present in documents. Instead of using the doc mapping to be aware of existing fields, we should use the field capabilities endpoint to retrieve existing fields.
Add max and min timestamp on the field_caps endpoint.
We recently added two new query parameters in the ES field_caps endpoint, ref.
We should use them to restrict the field cap call to the Grafana time window.
Unable to properly install the 0.3.0 with grafana helm chart - it gets downloaded, installed, run and then overridden by the 0.2.4 from official plugin catalog
Hi,
Following the documentation to download and run external grafana plugins, I was able to see it was properly downloaded at helm deployment time as shown in the logs:
2024-01-14T17:15:07.582415949Z ✔ Downloaded and extracted quickwit-quickwit-datasource v0.3.0 zip successfully to /var/lib/grafana/plugins/quickwit-quickwit-datasource
2024-01-14T17:15:07.582497033Z Please restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.
The plugin seems to be properly initialized:
2024-01-14T17:15:09.377098358Z logger=local.finder t=2024-01-14T17:15:09.376027472Z level=debug msg="Loading plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/plugin.json
2024-01-14T17:15:09.460300900Z logger=plugins.signature t=2024-01-14T17:15:09.460216857Z level=debug msg="Plugin signature valid" id=quickwit-quickwit-datasource
2024-01-14T17:15:09.460331025Z logger=plugin.signature.validator t=2024-01-14T17:15:09.460316358Z level=debug msg="Plugin has valid signature" id=quickwit-quickwit-datasource
2024-01-14T17:15:09.463630727Z logger=plugins.registration t=2024-01-14T17:15:09.463570434Z level=info msg="Plugin registered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:09.463705436Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.46367231Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 args=[/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64]
2024-01-14T17:15:09.464339526Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.463979272Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 pid=33
2024-01-14T17:15:09.464370859Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.464021939Z level=debug msg="waiting for RPC address" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64
2024-01-14T17:15:09.471223265Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471165681Z level=debug msg=Profiler enabled=false
2024-01-14T17:15:09.471238973Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471221973Z level=debug msg=Tracing enabled=false propagation=
2024-01-14T17:15:09.471261057Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471241223Z level=debug msg="Serving plugin" plugins="[stream diagnostics resource data]"
2024-01-14T17:15:09.471573977Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471518185Z level=debug msg="using plugin" version=2
2024-01-14T17:15:09.471694520Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.471620519Z level=debug msg="plugin address" address=/tmp/plugin3134522783 network=unix
2024-01-14T17:15:09.472412611Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:09.472347777Z level=debug msg="Successfully started backend plugin process"
2024-01-14T17:15:09.472528362Z logger=plugin.loader source=external t=2024-01-14T17:15:09.47249507Z level=debug msg="Plugin source loaded" plugins=quickwit-quickwit-datasource duration=96.626891ms
2024-01-14T17:15:09.472544279Z logger=plugin.store t=2024-01-14T17:15:09.47251682Z level=info msg="Plugins loaded" count=56 duration=127.512258ms
Then for some reasons it gets stopped and the version 0.2.4 gets downloaded and installed instead
2024-01-14T17:15:16.996485963Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.996325795Z level=debug msg="Stopping plugin process"
2024-01-14T17:15:16.997524058Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.997064386Z level=debug msg="Plugin server exited"
2024-01-14T17:15:16.997639309Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.997593433Z level=info msg="plugin process exited" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 id=33
2024-01-14T17:15:16.997646309Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:16.9976196Z level=debug msg="plugin exited"
2024-01-14T17:15:16.997647851Z logger=plugins.deregister t=2024-01-14T17:15:16.997623767Z level=debug msg="Plugin unregistered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:17.005295806Z logger=plugins.external.registration t=2024-01-14T17:15:17.005210972Z level=debug msg="Skipping HasExternalService call. The feature is behind a feature toggle and needs to be enabled."
2024-01-14T17:15:17.005348932Z logger=plugin.repository t=2024-01-14T17:15:17.005277264Z level=debug msg="Installing plugin from https://grafana.net/api/plugins/quickwit-quickwit-datasource/versions/0.2.4/download"
2024-01-14T17:15:17.473611708Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:17.473515582Z level=debug msg="Plugin decommissioned"
2024-01-14T17:15:22.683321927Z logger=installer.fs t=2024-01-14T17:15:22.683203551Z level=info msg="Downloaded and extracted quickwit-quickwit-datasource v0.2.4 zip successfully to /var/lib/grafana/plugins/quickwit-quickwit-datasource"
2024-01-14T17:15:22.683378386Z logger=local.finder t=2024-01-14T17:15:22.683345344Z level=debug msg="Loading plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/plugin.json
2024-01-14T17:15:22.745257541Z logger=plugins.registration t=2024-01-14T17:15:22.745192832Z level=info msg="Plugin registered" pluginId=quickwit-quickwit-datasource
2024-01-14T17:15:22.745341250Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.745293625Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 args=[/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64]
2024-01-14T17:15:22.746285135Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.74624576Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64 pid=46
2024-01-14T17:15:22.746304427Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.746279843Z level=debug msg="waiting for RPC address" plugin=/var/lib/grafana/plugins/quickwit-quickwit-datasource/gpx_quickwit_linux_arm64
2024-01-14T17:15:22.753273709Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.75317925Z level=debug msg=Profiler enabled=false
2024-01-14T17:15:22.753461378Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753392502Z level=debug msg=Tracing enabled=false propagation=
2024-01-14T17:15:22.753466253Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753433836Z level=debug msg="Serving plugin" plugins="[stream diagnostics resource data]"
2024-01-14T17:15:22.753762422Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753711214Z level=debug msg="using plugin" version=2
2024-01-14T17:15:22.753804881Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.753788089Z level=debug msg="plugin address" address=/tmp/plugin3988986528 network=unix
2024-01-14T17:15:22.754194135Z logger=plugin.quickwit-quickwit-datasource t=2024-01-14T17:15:22.75416401Z level=debug msg="Successfully started backend plugin process"
2024-01-14T17:15:22.754250344Z logger=plugin.loader source=external t=2024-01-14T17:15:22.754233344Z level=debug msg="Plugin source loaded" plugins=quickwit-quickwit-datasource duration=71.002793ms
After that I have the Datasource creation automation as shown here
2024-01-14T17:15:22.802052308Z logger=datasources t=2024-01-14T17:15:22.802005141Z level=debug msg="Received command to add data source" url=http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger
2024-01-14T17:15:22.802058683Z logger=datasource t=2024-01-14T17:15:22.80201985Z level=debug msg="Applying default URL parsing for this data source type" type=jaeger url=http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger
2024-01-14T17:15:22.847466455Z logger=datasources t=2024-01-14T17:15:22.847396913Z level=debug msg="Received command to add data source" url=http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/
2024-01-14T17:15:22.847472955Z logger=datasource t=2024-01-14T17:15:22.847409996Z level=debug msg="Applying default URL parsing for this data source type" type=quickwit-quickwit-datasource url=http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/
And here are the values.yml meaningful customization
grafana:
grafana.ini:
plugins:
allow_loading_unsigned_plugins: quickwit-quickwit-datasource
plugins:
- "https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.3.0/quickwit-quickwit-datasource-0.3.0.zip;quickwit-quickwit-datasource"
env:
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: quickwit-quickwit-datasource
GF_LOG_LEVEL: debug
GF_DEFAULT_APP_MODE: development
And the datasources:
- Jaeger:
{
"name": "Jaeger",
"type": "jaeger",
"typeName": "Jaeger",
"typeLogoUrl": "public/app/plugins/datasource/jaeger/img/jaeger_logo.svg",
"access": "proxy",
"url": "http://quickwit-searcher.quickwit.svc.cluster.local:7280/api/v1/otel-traces-v0_7/jaeger",
"user": "",
"database": "",
"basicAuth": false,
"isDefault": false,
"jsonData": { "timeout": -2 },
"readOnly": false
}
- Quickwit
{
"name": "Quickwit",
"type": "quickwit-quickwit-datasource",
"typeName": "Quickwit",
"typeLogoUrl": "public/plugins/quickwit-quickwit-datasource/img/logo.svg",
"access": "proxy",
"url": "http://quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/",
"user": "",
"database": "",
"basicAuth": false,
"isDefault": false,
"jsonData": {
"index": "otel-logs-v0_6",
"logLevelField": "level",
"logMessageField": "body.message",
"timeField": "timestamp_nanos",
"timeOutputFormat": "unix_timestamp_nanos"
},
"readOnly": false
}
Any idea of what I might be doing wrong here ?
Thank you !
Basic Auth for the datasource
It could be nice :-)
Display Quickwit 4XX response body in the grafana UI
Currently, if Quickwit returns a 4XX, the plugin backend does not handle properly the response and the user has no feedback on the error.
We should display the error in the Grafana UI.
Duplicates in autocompleted fields
In my configuration, I have many fields which are displayed twice (or more times) in the autocompleted fields list for aggregations when I do count aggregations by terms.
In the following screenshot, the following fields are displayed twice in the list: "freediskstorage", "sigma_infos.def_url" and "auditscore"
Make sure older logs can be explored
Limits
AFAIK, there's no notion of result limits in grafana, the pagination only relies on timestamps and the amount of results expected for a given timerange is unbounded. There's an edge case to this mode where a lot of logs have the same timestamp. If the datasource doesn't limit the results, the plugin may explode (could be an attack vector) . If the datasource puts a limit, logs may be lost between pages or the pagination may fail to compute bounds for the next page.
IMO, Grafana should implement a more robust pagination scheme to solve this case (we should probably open an issue on that), but we could work around it by implementing unbounded DataQueries as a sequence of bounded ones (the result is an observable). Not ideal, but it would avoid losing logs and allow throttling strategies to mitigate the risk of DoS.
Sorting
Grafana allows the user to sort logs results ascending or descending and adapts querying actions accordingly. This SHOULD be enough to cover both use cases of chronological and backwards exploration.
However, the limit on results interferes as we don't specify if the results should be fetched from the beginning or the end of the specified range.
Pagination
Up to Grafana 10.3.x, the pagination component tends to break and fail to recognize which page it's in. Current main branch fixes that bug and implements infinite scrolling as an experimental feature.
Msearch API
Each query in a multi-search request defines its own limits, pagination must take multi-requests into account.
See also:
- Known edge case when a lot of logs have the same timestamp
grafana/grafana#64041 - Infinite scrolling feature (feature toggle :
logsInfiniteScrollingin 10.4)
grafana/grafana#76348 - Pagination bug fixed in 10.4
grafana/grafana#80167
Support annotations
Display the number of matched document in the explore view
Currently, the UI does not display the number of matched documents.
Backend log levels are too loud ?
According to grafana's best practices for plugin development, backend log levels should be Debug instead of Info
Embed dashboards for logs and traces index
Depends on #2
Support search_after to use if for getting documents after and before a given document
The search_after feature is now available in Quickwit.
We can now use it to implement the log context feature of the explore view.
Code pointers:
Unknown field `end_timestamp`
On the Explore page
v0.4.0-beta.1
/api/datasources/uid/<>/resources/_elastic/<>/_field_caps?start_timestamp=1708489939&end_timestamp=1708511540
{
"indices": [
"<>"
],
"fields": {
DATA
},
}v0.4.0-beta.4
/api/datasources/uid/<>/resources/_elastic/<>/_field_caps?start_timestamp=1708489939&end_timestamp=1708511540
{
"message": "unknown field `end_timestamp`, expected one of `allow_no_indices`, `expand_wildcards`, `fields`, `ignore_unavailable`"
}quickwit: 0.7.1
Keeps refreshing output while typing query
It's pretty annoying behaviour. I suppose it's coming from the datasource and not Grafana - correct me please if I'm wrong.
Expectation would be to not reload anything automatically. There are often situtation where you have a log entry open and checking some fields and their values and only then want to construct the whole query.
If it keeps running in the meantime the "preview" of particular log is gone.
EDIT:
Plugin version: v0.4.0 / v0.3.2
quickwit: 0.7.1
Add screenshots in plugin.json
Improve log context field filtering
cf : #45 (comment)
Fields shown in the log context sidebar may not be relevant filters.
There may not be a sensible default exclusion list.
How can this be improved ?
Support Grafana tracing view(s)
To support the Jaeger grafana plugin without the jaeger-query layer, we need to implement the following REST endpoints:
GET api/servicesGET api/services/{service}/operationsGET api/traces?service={service}&start={start_in_ns}&end={end_in_ns}&lookback=customGET api/traces/{trace-id-base-64}
The handlers of those endpoints should use the jaeger gRPC service.
Check the "no limit" size parameter for term aggregation
I'm unsure of the meaning of "no limit"
No graph in the explore view on a metric query
Support for logging the Query
It could be helpful for debugging and troubleshooting to log the underlying Query sent from Grafana to Quickwit.
Maybe look for a global Grafana setting such as debug and log the Query in the web console based on the value.
Server-side logging might be fine for this. It all depends on how easily a user can grab the logged query and send it for troubleshooting assistance.
Add highlight searched words
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.