opensearch-project / opensearch-go Goto Github PK

If people want to connect to an Amazon OpenSearch Service cluster, they have to figure out how to use this client alongside some third-party signing library and/or the AWS SDK, which is non-trivial due to how we (by design) conceal the underlying HTTP requests that the client makes. We should offer IAM signing as an option, just like the OpenSearch CLI does.

Is your feature request related to a problem?
If you want to use this client with an Amazon OpenSearch Service cluster that has IAM authentication rather than basic authentication, good luck.

What solution would you like?

When initializing the client, an AuthType option. If basic (or null), accept username and password. If IAM, accept options for access key, secret key, session token, region, and service.

What alternatives have you considered?

An additional Go IAM signing library for use on top of the client. But given that it would only work with the client, it seems better and easier to just add it to the client.

Do you have any additional context?

opensearch-project/OpenSearch#1400

CVE-2021-38561 - High Severity Vulnerability

Vulnerable Library - github.com/golang/text-v0.3.6

[mirror] Go text processing support

Dependency Hierarchy:

• github.com/aws/aws-sdk-go-v1.42.27 (Root Library)
  • github.com/golang/net/go.mod-4163338589ed626ba11276b63facaef1f55349d4
    • ❌ github.com/golang/text-v0.3.6 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

Due to improper index calculation, an incorrectly formatted language tag can cause Parse
to panic, due to an out of bounds read. If Parse is used to process untrusted user inputs,
this may be used as a vector for a denial of service attack.

Publish Date: 2021-08-12

URL: CVE-2021-38561

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://osv.dev/vulnerability/GO-2021-0113

Release Date: 2021-08-12

Fix Resolution: v0.3.7

In this milestone, we focus on setting up main branch that can work with existing 7.10.2 version of ODFE

• Delete all older branches < 7.10
• Delete all released tags
• Delete main branch(this branch potentially tracks of all 8.x features as well)
• create main branch from 7.x branch
• Remove product checker commit if present (this is required to make clients work with oss cluster)
• Copy template files related to OpenSearch guidelines.
• set up CI and ensure tests pass for Elasticsearch OSS Docker image. This means all the test cases related to oss features from 7.10.2 executes successfully.

Add badge for

1. unit test
2. integration test
3. build
4. codecov
5. PR
6. chat

** What kind of business use case are you trying to solve? What are your requirements?

Need golang server to have more control over our destiny and hate the JVM ( sorry )

What is the problem? What is preventing you from meeting the requirements?

We are golang coders in the team and so using a golang server is much easier.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

THIS: https://github.com/prabhatsharma/zinc

Its taking off. Does not have clustering yet
but there is another clustered one here https://github.com/mosuka/phalanx, but it needs the ziny APi on top of it.

What are your assumptions or prerequisites?

Dont think i am assuming much. but i probably dont know what i dont know.

What are remaining open questions?

i would like to know if this interests the community.
I can put time into it, but it would be nice to know if others like this idea and is they also want to get involved.

CVE-2015-9251 - Medium Severity Vulnerability

Vulnerable Library - golang.org/x/tools-v0.1.12

Library home page: https://proxy.golang.org/golang.org/x/tools/@v/v0.1.12.zip

Dependency Hierarchy:

• ❌ golang.org/x/tools-v0.1.12 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

jQuery before 3.0.0 is vulnerable to Cross-site Scripting (XSS) attacks when a cross-domain Ajax request is performed without the dataType option, causing text/javascript responses to be executed.

Publish Date: 2018-01-18

URL: CVE-2015-9251

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2015-9251

Release Date: 2018-01-18

Fix Resolution: jQuery - 3.0.0

What is the bug?
the api.update.go using wrong HTTP request path.

How can one reproduce the bug?
Steps to reproduce the behavior: create UpdateRequest and execute it.

What is the expected behavior?
Swap line 101 & 103 of "opensearchapi/api.update.go".

What is your host/environment?

• OS: macos
• Version 12
• Plugins

Do you have any screenshots?
None.

Do you have any additional context?
None

CVE-2022-27191 - High Severity Vulnerability

Vulnerable Library - golang.org/x/crypto-v0.0.0-20210921155107-089bfa567519

Library home page: https://proxy.golang.org/golang.org/x/crypto/@v/v0.0.0-20210921155107-089bfa567519.zip

Dependency Hierarchy:

• ❌ golang.org/x/crypto-v0.0.0-20210921155107-089bfa567519 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

The golang.org/x/crypto/ssh package before 0.0.0-20220314234659-1baeb1ce4c0b for Go allows an attacker to crash a server in certain circumstances involving AddHostKey.

Publish Date: 2022-03-18

URL: CVE-2022-27191

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2022-27191

Release Date: 2022-03-18

Fix Resolution: golang-golang-x-crypto-dev - 1:0.0~git20220315.3147a52-1;golang-go.crypto-dev - 1:0.0~git20220315.3147a52-1

In this milestone, we focus on cleaning up existing code and make it ready for OpenSearch development

• Remove X-Pack and non oss features/tests
• Remove docs
• Remove commits not related to >7.10 from main branch
• Update CI to run with OpenDistro docker image with out security plugin

Hi, We have a problem with opensearch-go client but we can't find it so we want to instrument the client in order to improve the observability. Is there an easy way to achieve that goal? Except forking the project and adding instrumentation to the forked version and using it.

We should create examples and move to documentation website

Is your feature request related to a problem?
Currently there's no support for the ISM APIs in opensearchapi. This means to use that feature you need to manually extend the API yourself

What solution would you like?
It would be nice if this was included.

What alternatives have you considered?
Currently we have written our own implementation.

CVE-2021-43565 - High Severity Vulnerability

Vulnerable Library - golang.org/x/crypto-v0.0.0-20210921155107-089bfa567519

Library home page: https://proxy.golang.org/golang.org/x/crypto/@v/v0.0.0-20210921155107-089bfa567519.zip

Dependency Hierarchy:

• ❌ golang.org/x/crypto-v0.0.0-20210921155107-089bfa567519 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

The x/crypto/ssh package before 0.0.0-20211202192323-5770296d904e of golang.org/x/crypto allows an attacker to panic an SSH server.

Publish Date: 2022-09-06

URL: CVE-2021-43565

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2021-43565

Release Date: 2021-11-10

Fix Resolution: golang-golang-x-crypto-dev - 1:0.0~git20211202.5770296-1;golang-go.crypto-dev - 1:0.0~git20211202.5770296-1

• Metadata handler
• API compatibility

copy template files like RELEASING, LICENSE, MAINTAINER from .github project.

Hi, go mod requires tags to start with a "v" prefix. Otherwise, it's invalid.
I've noticed this when trying to update a go.mod to use the latest release.

Could you please retag it to be v1.0.0?

Context:
golang/go#30146 (comment)
golang/go#32945

Add installation instructions, developer guide and usage.

Coming from opensearch-project/project-meta#17

A Developer Certificate of Origin is required on commits in the OpenSearch-Project.

See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.

• DCO Check Workflow
• CONTRIBUTING.md DCO Section

Hello,

As I can see, currently there is no support for the users/roles API in opensearchapi. Could you please include this API to the client?

Second minor release

• Create 1.1 branch
• Generate release notes
• Create release tag

Hi all, i am new here and asking for some help .I query my documents by code below (golang), but it would search from all the index, how should i change the queryString to make that onle searching with setted index

queryString := `{
		"size": 1,
		"query": {
			"multi_match": {
			"query": "{name}",
			"fields": ["{fields}"]
			}
	    }
	}`
queryString = strings.Replace(queryString, "{name}", fmt.Sprintf("%v", "person1_login"), -1)
queryString = strings.Replace(queryString, "{"+"fields"+"}", fmt.Sprintf("%v", "user_login"), -1)
content := strings.NewReader(queryString)

search := opensearchapi.SearchRequest{
Body: content,
}

Any advices would be appreciated, thanks.

Add support to run integration tests across various versions of OpenSearch and publish a compatibility matrix for the client version vs OpenSearch version.

PR Reference: opensearch-project/opensearch-java#122
Compatibility: https://github.com/opensearch-project/opensearch-java/blob/main/COMPATIBILITY.md

Coming from meta issue opensearch-project/opensearch-clients#14:

• Add auto-backport PR creation Github action to make sure all the PRs are backported for a particular release.
• Add auto-delete for merged auto-backport PRs to clean up the branches created during auto-backport.
• Add documentation for the same.

Is your feature request related to a problem?
Currently the bulk index request structs do not support the version or routing options.

What solution would you like?
Let's add these fields to the structs and update the writer to build the json correctly.

What alternatives have you considered?
I'm not sure there are any, with how the current implementation of the bulk indexer works we're unable to set these fields unless they are available on the structs.

Do you have any additional context?
https://opensearch.org/docs/opensearch/rest-api/document-apis/index-document/ the version, routing, and version_type fields are the ones requested

Remove/update files inside .ci folder that are not related to opendistro/opensearch

To support backward compatibility, if addresses are not specified in config during new client api, we will look for addresses either from ELASTICSEARCH_URL or OPENSEARCH_URL environment variable. if we find both, then will exit with error message that both can't be set .

CVE-2022-28948 - Medium Severity Vulnerability

Vulnerable Library - github.com/go-yaml/yaml-v2.4.0

YAML support for the Go language.

Dependency Hierarchy:

• ❌ github.com/go-yaml/yaml-v2.4.0 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

An issue in the Unmarshal function in Go-Yaml v3 causes the program to crash when attempting to deserialize invalid input.

Publish Date: 2022-05-19

URL: CVE-2022-28948

CVSS 3 Score Details (5.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-fm53-mpmp-7qw2

Release Date: 2022-05-19

Fix Resolution: v3.0.0

Since we will be supporting only 7.10 onwards, update main to 7.x

Release the client version 2.0.0. Before releasing, the following items need to be done:

• Add OpenSearch 2.0 to the integration test matrix.
• Bump the client version to 2.0.0.
• Cut a 2.0 branch for the client.

What kind of business use case are you trying to solve? What are your requirements?

Hi. First, thank you for releasing this client.

I had been using the original one for the last couple of weeks and I noticed that it uses some patterns (like) that aren't so commonly found on Go clients (generated or not).

I imagine that having a leaner API using a more traditional (as in Go idiomatic) approach would be a good thing, at least for casual API consumers that are primarily Go developers – although I'm not so sure about people with plenty of experience using OpenSearch.

What is the problem? What is preventing you from meeting the requirements?
Let me try to give you a concrete example:

I imported the package to my code for the first time, and then I started looking around to see how I could do what I needed.
I quickly discovered that I needed to instantiate a client by calling opensearch.NewClient.

Then, the next thing I discovered was the Search function there, except that I immediately got confused as to why it was defined in the confusing API struct in an oddly named api._.go file.

and I started looking for usage examples just after I noticed the following

type Search func(o ...func(*SearchRequest)) (*Response, error)

and then I discovered there are functions defined on top of this SearchRequest type

which are really decorators(-ish?) on top of the much more idiomatic SearchRequest param:

I see several problems with decorator functions like these.

I also noticed that gopls wasn't helping me much discover the API for the library, and I attribute part of the problem to how you're expected to build the code kind indirectly using decorators – making browsing code and documentation slower than it may be.

• There are other usage complications too, but I feel I need more time using it to share a more thoughtful opinion.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

• Primarily, remove these functional/decorator functions and make some adjustments to the API to make it feel more like a traditional Go API.
• Also, improve the high-level documentation (probably after making the API leaner).

In short, my suggestion is to do a clean-up to make the API more idiomatic to the Go ecosystem before releasing a v1 version.
What do you think?

What are your assumptions or prerequisites?
I'm assuming the current API can improve developer experience getting rid of some, apparently unimportant, odd patterns (to the Go programming language), which makes the API itself bigger than it needs to be, and harder to master.

Best regards,
Henrique.

CVE-2020-11022 - Medium Severity Vulnerability

Vulnerable Library - golang.org/x/tools-v0.1.12

Library home page: https://proxy.golang.org/golang.org/x/tools/@v/v0.1.12.zip

Dependency Hierarchy:

• ❌ golang.org/x/tools-v0.1.12 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

In jQuery versions greater than or equal to 1.2 and before 3.5.0, passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others) may execute untrusted code. This problem is patched in jQuery 3.5.0.

Publish Date: 2020-04-29

URL: CVE-2020-11022

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-11022

Release Date: 2020-04-29

Fix Resolution: jQuery - 3.5.0

What is the bug?
The project does not follow the Go versioning requirements of a /v2 being added to the package in the go mod file when the first major version bump occurs. Because of this the client is not usable as adding it to the go.mod file as a dependency results in a failed build due to the version not existing, or being an invalid version because the package doesn't contain a /v2 at the end. Even adding the +incompatible does not seem to fix it.

How can one reproduce the bug?
Steps to reproduce the behavior:
Create a project and add this project as a dependency
run go mod tidy

What is the expected behavior?
Go dependency management will fail to find the project and fail the build

What is your host/environment?

• OS: Darwin
• Version Latest
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
no but can get some if needed adding exact wording of error message.
server response: not found: github.com/opensearch-project/[email protected]+incompatible: invalid version: module contains a go.mod file, so module path must match major version ("github.com/opensearch-project/opensearch-go/v2")

Do you have any additional context?
no

CVE-2012-6708 - Medium Severity Vulnerability

Vulnerable Library - golang.org/x/tools-v0.1.12

Library home page: https://proxy.golang.org/golang.org/x/tools/@v/v0.1.12.zip

Dependency Hierarchy:

• ❌ golang.org/x/tools-v0.1.12 (Vulnerable Library)

Found in HEAD commit: 3f3ab782590b20cf59f880bfdbd70556a225006e

Found in base branch: main

Vulnerability Details

jQuery before 1.9.0 is vulnerable to Cross-site Scripting (XSS) attacks. The jQuery(strInput) function does not differentiate selectors from HTML in a reliable fashion. In vulnerable versions, jQuery determined whether the input was HTML by looking for the '<' character anywhere in the string, giving attackers more flexibility when attempting to construct a malicious payload. In fixed versions, jQuery only deems the input to be HTML if it explicitly starts with the '<' character, limiting exploitability only to attackers who can control the beginning of a string, which is far less common.

Publish Date: 2018-01-18

URL: CVE-2012-6708

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2012-6708

Release Date: 2018-01-18

Fix Resolution: jQuery - v1.9.0

Is your feature request related to a problem?
AWS has released v2 of their GoLang SDK. OpenSearch Go already has support for AWS SDK v1, but no support for this newer SDK. Users of SDK v2 thus have to add SDK v1 as a dependency if they want to use OpenSearch Go.

What solution would you like?
A signer only using v2 SDK imports to be added.

What alternatives have you considered?
No alternatives.

Do you have any additional context?
No additional context.

Update module name to github.com/opensearch-project/opensearch-go

Remove unwanted make commands.

Get started with incorporating the breaking changes for 2.0 from OpenSearch 2.0.

To do so:

1. Branch off a 1.x from main if not already done so that development for 2.0 can start on main.
2. Incorporate breaking changes as mentioned in opensearch-project/opensearch-clients#12.
3. Add integ tests support to run against unreleased OpenSearch.

Example PR from java-client: opensearch-project/opensearch-java#132

Handle application/vnd.elasticserach+json;compatible-with=7 in backward compatible way

Add support for signed requests via AWS sigv4.

https://github.com/opensearch-project/opensearch-go/security/dependabot/_benchmarks/benchmarks/go.mod/github.com%2Ftidwall%2Fgjson/open
https://github.com/opensearch-project/opensearch-go/security/dependabot/_examples/instrumentation/go.mod/go.elastic.co%2Fapm/open

What is the bug?
It seems that with a BulkIndexer with 2 workers, I am getting unexpected latency on BulkIndexer.Add(). It seems that somehow the workers are not consuming the queue within any reasonable sort of timeframe, I'm seeing delays of over 20s!

For example, in the last hour I've 53 cases of >1s latency on just Add() out of a total of 174 calls.

How can one reproduce the bug?
With 2 workers running, adding items from different goroutines and a relatively busy search cluster.

What is the expected behavior?
Sub-millisecond latencies, basically the time it takes to shove something into a channel.

What is your host/environment?

• OS: Ubuntu 20.02
• Version: 1.1.0 (but nothing has changed to the bulkindexer since the fork from ES)

Do you have any screenshots?

Add support for inclusive naming by deprecating the master nomenclature.

See meta issue opensearch-project/opensearch-clients#16 for details.

update package/file from es -> opensearch

What is the bug?
Example at UserGuide (https://github.com/opensearch-project/opensearch-go/blob/main/USER_GUIDE.md) is incomplete. It does not create an index with specified mapping.

How can one reproduce the bug?
Execute example at https://github.com/opensearch-project/opensearch-go/blob/main/USER_GUIDE.md

What is the expected behavior?
Use this api to create mapping before inserting documents.

What is your host/environment?
N/A

Do you have any screenshots?
N/A

Do you have any additional context?
N/A