permutive-engineering / fs2-google-pubsub Goto Github PK

In case of internal Google PubSub library error (i.e. io.grpc.StatusRuntimeException: NOT_FOUND) stream does not fail and it's impossible to track/handle/log the error.

I propose (at least) to add custom listeners to com.permutive.pubsub.consumer.grpc.PubsubGoogleConsumerConfig and apply them to the subscriber.

Hello!
Readme says :

Doesn't work with the official PubSub emulator (is in feature backlog)

Link to that backlog is no longer working. I've looked though google cloud docs and they show here usage of java client with the emulator.

String hostport = System.getenv("PUBSUB_EMULATOR_HOST");
ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext().build();

Does it mean that this library could be adjusted for emulator support?

There is a cancelation leak in FutureInterop, very similar to this one

We should investigate the usage of LinkedBlockingQueue - I believe that that queue is used in SPSC mode in which case the locking behaviour is a massive overhead. There are a few options floating about in the ecosystem (potentially one about to land in cats effect IIRC) that should be much more performant when combined with a dispatcher to run the enqueue actions

ERROR - [PubSub] Unknown PubSub error occurred. Body is: PubSubErrorResponse(PubSubErrorMessage(Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.,INVALID_ARGUMENT,400))

From the docs:

In the case of exactly-once delivery, the modifyAckDeadline or acknowledgment request to Pub/Sub fails when the acknowledgment id is already expired. In such cases, the service considers the expired acknowledgment id as invalid, since a newer delivery might already be in-flight. This is by design for exactly-once delivery. You then see acknowledgment and ModifyAckDeadline requests return an INVALID_ARGUMENT response. When exactly-once delivery is disabled, these requests return OK in cases of expired acknowledgment ids.
To ensure that acknowledgment and ModifyAckDeadline requests have valid acknowledgment ids, consider setting the value for minDurationPerAckExtension to a high number.

Exactly once client support is added in version 1.116.0 which exposes the setting minDurationPerAckExtension which should be added to the PubsubGoogleConsumerConfig in this repo

I have raised a fix here #465 for the grcp client but not sure what to do for http client...

I noticed one thing. When I'm trying to connect to not existing subscription (or not yet created) stream is not failing. It hangs and it's not possible to capture the error in any way. On debugger I can see that this part of the code is invoked:

queue.put(Left(InternalPubSubError(failure)))

but somehow takeNextElement method is not returning anything.

https://github.com/permutive/fs2-google-pubsub/blob/d70fc53535e4eeaf3da2b83e9c08d664c9ce4b52/fs2-google-pubsub-http/src/main/scala/com/permutive/pubsub/http/oauth/GoogleOAuth.scala#L34

Allegedly the above URL is deprecated and we should use https://oauth2.googleapis.com/token. I can't find anything exact and all Google documentation regarding this endpoint have disappearing from the Cloud documentation.

I found grahamearley/FirestoreGoogleAppsScript#43 which solves authentication problems (which we, as of writing this, don't have) by simply changing the URL.

Greetings!

I am having trouble connecting to the Google PubSub emulator (latest version). According to the README, the emulator should work when using HTTP, however, I am consistently getting timeouts. It just seems to hang when attempting to connect.

I have the emulator started in a docker container on port 9050.

Here is a code sample that will encounter the error:

  private val httpClient = AsyncHttpClient.resource[IO](
    AsyncHttpClient.defaultConfig
  )

  private implicit def unsafeLogger: Logger[IO] =
    Slf4jLogger.getLoggerFromName[IO]("com.permutive.fs2-google-pubsub")

  private def publish(request: MyRequestType): IO[MessageId] = {
    val mkProducer = HttpPubsubProducer.resource[IO, MyRequestType](
      projectId = Model.ProjectId(config.projectId),
      topic = Model.Topic(config.topic),
      googleServiceAccountPath = "",
      config = PubsubHttpProducerConfig(
        host = "localhost",
        port = 9050,
        isEmulator = true
      ),
      _
    )
    httpClient.flatMap(mkProducer)
    .use { producer =>
      producer.produce(request)
    }
  }

This code consistently times out on the .use { producer => line: the issue is happening when attempting to acquire the resource. Here is a stacktrace:

[info] - test *** FAILED *** (47 seconds, 248 milliseconds)
[info]   java.util.concurrent.TimeoutException: Request timeout to localhost/127.0.0.1:9050 after 45000 ms
[info]   at org.asynchttpclient.netty.timeout.TimeoutTimerTask.expire(TimeoutTimerTask.java:43)
[info]   at org.asynchttpclient.netty.timeout.RequestTimeoutTimerTask.run(RequestTimeoutTimerTask.java:50)
[info]   at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
[info]   at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
[info]   at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
[info]   at java.lang.Thread.run(Thread.java:748)
[info]   at async @ org.http4s.client.asynchttpclient.AsyncHttpClient$.$anonfun$allocate$3(AsyncHttpClient.scala:52)
[info]   at apply @ org.http4s.client.asynchttpclient.AsyncHttpClient$.resource(AsyncHttpClient.scala:66)
[info]   at use @ com.mlbam.adtech.googleAdIngest.AdSubscriberIT.publish(AdSubscriberIT.scala:57)
[info]   at flatMap @ com.permutive.pubsub.consumer.http.internal.HttpPubsubReader.$anonfun$read$4(HttpPubsubReader.scala:66)
[info]   at flatMap @ com.permutive.pubsub.consumer.http.internal.HttpPubsubReader.$anonfun$read$3(HttpPubsubReader.scala:61)
[info]   at flatMap @ com.permutive.pubsub.consumer.http.internal.HttpPubsubReader.$anonfun$read$2(HttpPubsubReader.scala:60)
[info]   at flatMap @ com.permutive.pubsub.consumer.http.internal.HttpPubsubReader.<init>(HttpPubsubReader.scala:52)
[info]   at unsafeRunSync @ com.mlbam.adtech.commontest.SharedIOTest.testIO(IOFixtureFreeSpec.scala:44)

And this is the log output from the emulator:

pubsub_1  | [pubsub] Dec 16, 2020 6:37:29 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
pubsub_1  | [pubsub] INFO: Detected non-HTTP/2 connection.
pubsub_1  | [pubsub] Dec 16, 2020 6:37:31 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
pubsub_1  | [pubsub] INFO: Detected non-HTTP/2 connection.
pubsub_1  | [pubsub] Dec 16, 2020 6:37:46 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
pubsub_1  | [pubsub] INFO: Detected non-HTTP/2 connection.

I have tried multiple HTTP client backends, but they all run into the same issue. Since I don't get any useful output from the emulator itself I have gotten stuck debugging the issue.

Do you have any advice here, or have seen this error before? Thank you for your time!

This would be really nice but currently we depend on Java libs for JWT and logging. Are there ScalaJS compatible alternatives?

https://github.com/permutive-engineering/functional-google-clients/tree/main/google-auth

We currently can't build against JDK 17 because of an illegal reflective access used to update environment variables for tests