Comments (9)
In case of OpenCensus implementation, we have a safety check in place to handle end of span call multiple times. You might see "Calling end() on an ended Span." log when calling end more than once.
from java-spanner.
Thanks @mayurkale22. That's what I suspected. I'm trying to reproduce this with the ResumableStreamIteratorTest
but I can end the span as many times as I want and it doesn't seem to be an issue.
So it's not so much that the span is ended many times. It's more that the span has gone out of scope by the time end() is called. I haven't yet figured out how to recreate this situation.
Adding @dinooliva to see if he has any thoughts.
from java-spanner.
Also adding @olavloite to see if he has any ideas.
from java-spanner.
So it's not so much that the span is ended many times. It's more that the span has gone out of scope by the time end() is called.
AFAIK a default Span will be used (which does nothing) if no Span is associated with the current Context/Scope. I would not worry about out of scope scenario.
Looking at the implementation (com.google.perftools.opencensus.trace.SpanImpl
), reference (refCount
) is increased only when the Span is created and decreased when the span is ended. The refCount
is dropping below means there are more number of ended span than started, which also indicates spanner instrumentation is calling end()
on an ended Span.
One of the suspect is this, there is a call TraceUtil.endSpanWithFailure(span, e)
which calls span.end()
and a second end()
is getting called in the finally block. For this case, I would not end the span in endSpanWithFailure
and let the finally block take care of it.
This is my initial assessment. WDYT?
from java-spanner.
Yup I agree with you @mayurkale22. I thought that was the issue and was going to replace some of the TraceUtil.endSpanWithFailure()
calls with span.setStatus()
calls then allow the finally
and/or close
to end the span once.
But when I tried out this idea with the ResumableStreamIteratorTest
by calling end()
a few times, I don't get this refcount warning. Maybe I'm not running the tests in a similar environment to those people that reported the issue.
from java-spanner.
See https://github.com/mayurkale22/java-spanner/pull/1 for a small test framework for checking whether we are or are not 'overkilling' spans.
from java-spanner.
Test framework looks great, I have integrated with my PR with some checks on all the started span and their ended status. Two things I observed:
-
CloudSpanner.ReadWriteTransaction span doesn't end after
transactionRunner
call. See: https://github.com/googleapis/java-spanner/pull/75/files#diff-711d9312cd4ab6691f226f63f857fd54R197. I am not sure what is expected behavior, should we close the span in finally block? -
CloudSpannerOperation.ExecuteStreamingQuery span getting ended twice in case of
singleUse().executeQuery
due to this, I think stream is not closed properly in case ofGrpcResultSet
. @olavloite thoughts?
from java-spanner.
Test framework looks great, I have integrated with my PR with some checks on all the started span and their ended status. Two things I observed:
- CloudSpanner.ReadWriteTransaction span doesn't end after
transactionRunner
call. See: https://github.com/googleapis/java-spanner/pull/75/files#diff-711d9312cd4ab6691f226f63f857fd54R197. I am not sure what is expected behavior, should we close the span in finally block?- CloudSpannerOperation.ExecuteStreamingQuery span getting ended twice in case of
singleUse().executeQuery
due to this, I think stream is not closed properly in case ofGrpcResultSet
. @olavloite thoughts?
Ad 1: I would say we should end the span in the finally block of this method:
Ad 2: The problem is not that stream
is not closed properly, it is because it is closed twice. The problem is that the fact that the stream is closed is not recorded, so it is closed twice and the span is ended twice.
The reason that the span in 2 is closed twice, is that single use read contexts will automatically close the result set once all rows have been consumed. This happens here. The result set is then closed a second time when the try-with-resources block ends.
Closing a result set multiple times should not have any side effects. I think the solution in this case should be to set stream = null;
here. That way, the span and the stream are not ended/closed more than once.
from java-spanner.
Thanks for the suggestion, I made the change 10d5ffc. I think things look good now.
from java-spanner.
Related Issues (20)
- Library Compatibility with Protobuf-Java 4.26.x+
- spanner.DatabaseClientImplTest: testSessionPoolExhaustedError_containsStackTraces failed HOT 3
- ITEmulatorConcurrentTransactionsTest is flaky with multiplexed sessions HOT 1
- ITReadOnlyTxnTest is flaky with multiplexed sessions
- OpenTelemetry traces show a missing SpanID HOT 1
- OpenTelemetrySpanTest.transactionRunner is flaky with multiplexed sessions
- Fix RESOURCE_EXHAUSTED admin quota limit exceed failures in test pipelines.
- Flaky failure in com.google.cloud.spanner.it.ITDmlReturningTest.dmlReturningWithExecuteQuery[Dialect = POSTGRESQL]
- Flaky test failure in AsyncTransactionManagerTest.asyncTransactionManagerWaitsUntilAsyncUpdateHasFinished with MultiplexedSessions=true
- Flaky failure in ITDmlReturningTest.dmlReturningWithExecuteQuery[Dialect = GOOGLE_STANDARD_SQL]
- spanner.it.slow.ITBackupTest: test02_RetryNonIdempotentRpcsReturningLongRunningOperations failed HOT 1
- spanner.it.slow.ITBackupTest: test03_Delete failed HOT 1
- spanner.SessionPoolOptionsTest: testToBuilder[min sessions = 500, max sessions = 600] failed HOT 1
- spanner.BatchCreateSessionsTest: testSpannerReturnsAllAvailableSessionsAndThenNoSessions failed HOT 1
- spanner.connection.SavepointMockServerTest: testKeepAlive[dialect = GOOGLE_STANDARD_SQL, useVirtualThreads = true] failed HOT 2
- spanner.connection.SavepointMockServerTest: testKeepAlive[dialect = GOOGLE_STANDARD_SQL, useVirtualThreads = false] failed HOT 2
- spanner.connection.SavepointMockServerTest: testKeepAlive[dialect = POSTGRESQL, useVirtualThreads = false] failed HOT 2
- SavepointMockServerTest is flaky
- spanner.SessionPoolOptionsTest: testToBuilder[min sessions = 600, max sessions = 500] failed HOT 1
- BatchCreateSessionsTest is flaky
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from java-spanner.