Comments (9)
Hey, thanks for opening the issue. The use of client.Perform()
is not recommended as it skips all the error handling.
The correct way would be following:
func (logger *MyLogger) SendLogEvent(event *MyLogEvent) error {
// Marshal the struct
s, err := json.Marshal(event)
if err != nil {
return WrapError("failed to marshal log event", err)
}
// Do the request
_, err := logger.client.Index(
context.Background(),
opensearchapi.IndexReq{
Index: logger.logsIndexName,
Body: bytes.NewReader(s),
},
)
if err != nil {
return err
}
// Success
return nil
}
from opensearch-go.
OK, this morning I'm not getting the errors. We had terrible connectivity yesterday and I think that was causing it. (I think I had changed the settings, see my next comment)
I found where the error was being raised:
/usr/local/go/src/net/http/transfer.go
line 391
ncopy, err = t.doBodyCopy(w, io.LimitReader(body, t.ContentLength))
if err != nil {
return err
}
var nextra int64
nextra, err = t.doBodyCopy(io.Discard, body)
ncopy += nextra
}
if err != nil {
return err
}
}
...
if !t.ResponseToHEAD && t.ContentLength != -1 && t.ContentLength != ncopy {
return fmt.Errorf("http: ContentLength=%d with Body length %d",
t.ContentLength, ncopy)
}
Seems like not all of the request body was being written yet there were no I/O errors.
Thanks for the info on how to use the API. I have updated my code.
from opensearch-go.
@merlinz01 Glad this was helpful. Should we close this?
from opensearch-go.
Now I'm getting some of these errors again even though the connectivity has much improved. So far they only show up when I set CompressRequestBody
to true. I wonder if there is a bug in the gzipping of the requests.
from opensearch-go.
OK, I'm getting it. So when it retries a failed attempt, the second time the request body is an empty bytes.Buffer
reader. It seems like it tries to re-read the body on the second attempt, but as it was fully read the first time, the result is zero bytes. It passes this zero-byte buffer to the writer, which writes zero bytes and succeeds, and wonders why only zero bytes were written when the indicated Content-Length is XXX bytes.
opensearch-go/opensearchtransport/opensearchtransport.go
Lines 239 to 268 in ea3e57a
On line 248, an io.nopCloser
for the exact same (possibly already consumed) bytes.Buffer
is returned every time req.GetBody
is called.
from opensearch-go.
Here's how I fixed it:
req.GetBody = func() (io.ReadCloser, error) {
// We have to return a new reader each time so that retries don't read from an already-consumed body.
// This does not do any copying of the data.
return io.NopCloser(bytes.NewReader(buf.Bytes())), nil
}
The code for non-gzipped bodies (line 261) does a similar thing it by implicitly copying the buf
by value before its internal state changes. I would say it is less clear what is happening so maybe we should update it to be similar to this fix.
I can open a PR if you want.
from opensearch-go.
I can open a PR if you want.
Most definitely, with tests please.
I think ideally the reader would be rewound only if retry is needed.
from opensearch-go.
We could implicitly copy the buffer like is done for non-gzipped bodies but IMHO it's better to be explicit. The second method below only makes two small struct allocations so it's not really a performance question.
req.GetBody = func() (io.ReadCloser, error) {
b := *buf
return io.NopCloser(&b), nil
}
vs.
req.GetBody = func() (io.ReadCloser, error) {
reader := bytes.NewReader(buf.Bytes())
return io.NopCloser(reader), nil
}
from opensearch-go.
See #543.
(#542 can be ignored; I committed to the wrong branch.)
from opensearch-go.
Related Issues (20)
- [BUG] Add missing global io usage field for nodes stats
- [FEATURE] Use pooled gzip writer for request body gzip compression
- [BUG] 429 error does not follow standard error format HOT 5
- [BUG] not reporting when failed index HOT 5
- [FEATURE] ML client for pipeline setup and neural search HOT 2
- [BUG] Request context cancellation is ignored when `retryBackoff` is configured
- [FEATURE] Exploring similar functionality in Go as Python classes {Object, Nested, Text, Keyword}. HOT 3
- Cleanup unused branches HOT 1
- [BUG]Unable to install because of checksum error HOT 3
- ISM Plugin `Conditions` struct is invalid
- [BUG] MappingFieldResp struct is not parsed correctly. HOT 2
- [BUG] Tests are flaky HOT 1
- Add MatchedQueries field to opensearchapi.SearchHit HOT 3
- [BUG] Index name is added two times to a query when using SearchRequest HOT 1
- [BUG] opensearchapi.Client.Document.Delete bugged error handling (v4) HOT 13
- [FEATURE] Make all response structs their own type
- [FEATURE] Support `search_pipeline` query parameter for search API requests HOT 1
- [BUG] ISM Allocation action does not match Opensearch server expectation HOT 1
- [FEATURE] Functionality to create aggregation queries when moving from olivere to opensearch-go
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opensearch-go.