GithubHelp home page GithubHelp logo

forcedotcom / pub-sub-api Goto Github PK

View Code? Open in Web Editor NEW
127.0 15.0 111.0 216 KB

Sample project to get started with the Pub/Sub API

License: Creative Commons Zero v1.0 Universal

Python 15.95% Go 16.64% Shell 0.25% Java 67.16%
grpc pubsub salesforce

pub-sub-api's Introduction

Getting Started with the Pub/Sub API

About Pub/Sub API

Welcome to Pub/Sub API! Pub/Sub API provides a single interface for publishing and subscribing to platform events, including real-time event monitoring events, and change data capture events. Based on gRPC and HTTP/2, Pub/Sub API enables efficient delivery of binary event messages in the Apache Avro format.

This repo contains the critical proto file that you will need to use the API.

gRPC

gRPC officially supports 11 languages, but there is unofficial community support in more. To encode and decode events, an Avro library for your language of choice will be needed. See below for which officially supported languages have well-supported Avro libraries:

Supported gRPC Language Avro Libraries
C# AvroConvert
Apache Avro C# (docs are not great)
C++ Apache Avro C++
Dart avro-dart (last updated 2012)
Go goavro
Java Apache Avro Java
Kotlin avro4k
Node avro-js
Objective C ObjectiveAvro (but read this)
PHP avro-php
Python Apache Avro Python
Ruby AvroTurf

Documentation, Blog Post and Videos

Code Samples from Salesforce

Salesforce provides these samples for demonstration purposes. They aren't meant to be used in production code. Before you use these samples in production, make sure you perform thorough functional and performance testing.

Code Samples from the Developer Community

These examples are developed by the community. They aren't supported by Salesforce. Use at your own discretion.

If you have a code sample for Pub/Sub API that you would like to add a link to in this section, submit a PR with the modified readme page. We don't guarantee that we can link to all samples. Priority will be given to samples implemented in a programming language that is not represented in this repository's samples.

pub-sub-api's People

Contributors

echenran avatar herinckc avatar jason-alaya-sf avatar judy-lin avatar knhage avatar pozil avatar saumitramohan avatar sidd0610 avatar svc-scm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pub-sub-api's Issues

Unable to Receive Events from Salesforce Pub/Sub API using Dart and gRPC

I'm trying to subscribe and listen to events from the Salesforce Pub/Sub API using Dart and gRPC. I've been able to successfully implement this in Python following the quick start guide, but I'm experiencing difficulties in Dart. Despite following a similar pattern, I'm not receiving any events in Dart while the Python script receives them in real time.

Here is the Dart code:

import 'package:grpc/grpc.dart';
    import 'package:pubsub_testing/src/generated/salesforceProtoFile.pbgrpc.dart' as pb_grpc;
    import "package:pubsub_testing/src/generated/salesforceProtoFile.pb.dart" as pb2;
    
    // Session ID, instance URL, and tenant ID are provided.
    
    final authMetadata = CallOptions(metadata: {
      'accesstoken': sessionId,
      'instanceurl': instanceUrl,
      'tenantid': tenantId,
    });

    final channel = ClientChannel(
      'api.pubsub.salesforce.com',
      port: 7443,
    );
    
    final stub = pb_grpc.PubSubClient(channel);
    
    Stream<pb2.FetchRequest> fetchReqStream(String topic) async* {
      while (true) {
        yield pb2.FetchRequest(
          topicName: topic,
          replayPreset: pb2.ReplayPreset.LATEST,
          numRequested: 100,
        );
      }
    }
    
    Future<void> subscribe(String mySubTopic) async {
      print('Subscribing to $mySubTopic');
      try {
        final substream =
            stub.subscribe(fetchReqStream(mySubTopic), options: authMetadata);
        print("substream: $substream");
    
        await for (var event in substream) {
          print("Got an event!\n");
          if (event.events.isNotEmpty) {
            print("Number of events received: ${event.events.length}");
            var payloadbytes = event.events[0].event.payload;
            var schemaid = event.events[0].event.schemaId;
    
            var schema = await stub.getSchema(pb2.SchemaRequest(schemaId: schemaid),
                options: authMetadata);
    
            print("Got an event!\n");
          } else {
            print("[${DateTime.now()}] The subscription is active.");
          }
        }
      } catch (e) {
        print('An error occurred during subscription: $e');
      }
    }

The code runs without throwing an error, and it seems to successfully subscribe because the terminal does not close, indicating that it's waiting for events. However, no events are being printed to the console, even though I know they are being published because my Python script is receiving them.

I've checked the permissions, and the session ID I'm using has the necessary permissions to subscribe to the topic.

Does anyone have experience with this type of situation or could suggest possible solutions? I would appreciate any insights or assistance.

The following is the Python implementation that works(Of course I removed sensible info). Also, I commented out some lines such that I made the code as minimal as possible so it would be easier to replicate in Dart. This file works and receives events in real time:

from __future__ import print_function
# import grpc
import requests
import threading
import io
import pubsub_api_pb2 as pb2
import pubsub_api_pb2_grpc as pb2_grpc
# import avro.schema
# import avro.io
import time
import certifi
import json

semaphore = threading.Semaphore(1)

latest_replay_id = None

# with open(certifi.where(), 'rb') as f:
#     creds = grpc.ssl_channel_credentials(f.read())

with grpc.secure_channel('api.pubsub.salesforce.com:7443', grpc.ssl_channel_credentials(None)) as channel:



    # Store Auth 
    sessionid = ''
    instanceurl = ''
    tenantid = ''
    authmetadata = (('accesstoken', sessionid),
    ('instanceurl', instanceurl),
    ('tenantid', tenantid))
    
    # Generate Stub
    stub = pb2_grpc.PubSubStub(channel)

    # Subscribe to event channel
    def fetchReqStream(topic):
        while True:
            semaphore.acquire()
            yield pb2.FetchRequest(
                topic_name = topic,
                replay_preset = pb2.ReplayPreset.LATEST,
                num_requested = 1)
    
    # Decode event message payload
    def decode(schema, payload):
        schema = avro.schema.parse(schema)
        buf = io.BytesIO(payload)
        decoder = avro.io.BinaryDecoder(buf)
        reader = avro.io.DatumReader(schema)
        ret = reader.read(decoder)
        return ret

    # Make the subscribe call
    mysubtopic = "/event/RS_L__ConversationEvent__e"
    print('Subscribing to ' + mysubtopic)
    substream = stub.Subscribe(fetchReqStream(mysubtopic),
            metadata=authmetadata   )
    for event in substream:
        if event.events:
            semaphore.release()
            print("Number of events received: ", len(event.events))
            payloadbytes = event.events[0].event.payload
            schemaid = event.events[0].event.schema_id
            schema = stub.GetSchema(
                    pb2.SchemaRequest(schema_id=schemaid),
                    metadata=authmetadata).schema_json
            decoded = decode(schema, payloadbytes)
            print("Got an event!", json.dumps(decoded), "\n")
            # print(f"payloadbytes: {payloadbytes}")
            # print(f"schemaid: {schemaid}")
        else:
            print("[", time.strftime('%b %d, %Y %l:%M%p %Z'),
            "] The subscription is active.")
        # latest_replay_id = event.latest_replay_id

Retry feature

Hi Siddartha,
Please let me know what the logic is for the retry feature in the OnError method (in the com.optum.pubsub.Subscribe class).
In case if there is any failure and I want to try to consume a failed batch of the messages from the target topic...I just need some guidance.

Thanks,
Meruzha

Support of MQTT Last Will and testament

Via MQTT it is possible to provide a last will: https://www.hivemq.com/blog/mqtt-essentials-part-9-last-will-and-testament/:

"Last Will and Testament (LWT) is a powerful feature in MQTT that allows clients to specify a message that will be automatically published by the broker on their behalf, if or when an unexpected disconnection occurs. It provides a reliable means of communication and ensures that clients can gracefully handle disconnections without leaving topics in an inconsistent state. This feature is particularly valuable when clients must notify others of their unavailability or convey important information upon an unexpected disconnection."

The PubSub SDK should support the possibility to specify such a “last will” for each pubished dataset which should then be provided as “last will” via MQTT.

C# example

Would it be possible to create an example for C# as well? Unfortunately, the Java code does not help me.
I would be particularly interested in how I can subscribe to a platform event in C#.

Missing production endpoint

The endpoint is currently set to api.pilot.pubsub.salesforce.com:7443 - at least in the Go example. What's the GA production endpoint? I see api.pubsub.salesforce.com:7443 works with my own code but it would be good to have the right endpoint in the repo.

Publishing from outside platform - Inexact CreatedById causes event not to fire triggers or flows

Hi,

When non-salesforce platform publishers publish Platform Events through the PubSub API, they are required to meet the Schema of the platform event.
This also includes two fields that are generally not provided by salesforce platform publishers (apex, flows, etc) :

  • CreateDate : appears to be ignored, and the Event bus handles the timestamping.
  • CreateById : MUST match the id of user authenticated to the pubsub api

Failing to do that :

  1. Message is accepted by the pubsub API and returns a replay ID
  2. No automation (Platform event Flow or Platform event trigger) are triggered. Nor is any log provided to help trouble shoot

This is a follow up to issue pozil/pub-sub-api-node-client#17
This was also discussed with support in case 500Hx00000K444jIAB (#45348057)

Steps to reproduce

  1. Create a new platform event, no custom field required
  2. Add a simple Platform Event Trigger (that only throws an exception) or Platform event Flow (that only causes) on this platform event
  3. Enable finest debug log on the intergration user
  4. Implement any client outside salesforce platform using any language (i went for Node using @pozil 's https://github.com/pozil/pub-sub-api-node-client/)
  5. Publish a platform event with CreatedById=WHATEVER as the intergration user

Observed Outcome

The pubsub api provides the client with a replay id
NO Executition of the trigger nor flow (no log generated for the interview nor Code Unit)

Expected outcome (Strict)

The pubsub api rejects the message with error Invalid user Id : Must match the Id of Authenticated user or something along the way

Expected outcome (devFriendly)

The pubsub accepts the message, but overrides the value provided in CreatedById to match user posting the platform event through pubsub api
The flow and trigger execute without issue.

[Java] The Replay ID validation failed when specify a replayId

I have input a custom replay Id as the following in configuration file:

Replay ID in ByteString

REPLAY_ID: [50, 53, 50, 52, 51, 50, 52, 56]

When starting subscription, it throws an exception:

=== GRPC Exception ===
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The Replay ID validation failed. Ensure that the Replay ID is valid. rpcId: 73ffdb43-42aa-438f-a81a-809e94e7ddd0
at io.grpc.Status.asRuntimeException(Status.java:534)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)

Trying another way to define replay ID but it doesn't work:

ByteString.copyFrom("25243248".getBytes(StandardCharsets.UTF_8))

subscribeLongLived

The file that was added to show the retry logic and long lived subscription not appearing on main branch, was it deleted? I am having issues keeping the stream alive, just implemented some changes from your class to keep retrying but it still fails. Are you able to keep the stream/subscription alive for a long time without any network error or retry failures?

Receiving a large commitNumber in ChangeEventHeader causes an error

This issue was initially raised here in the JS client repo but I believe that it affects all clients as it seems to be related to the schema definition:

A large commitNumber value under ChangeEventHeader can exceed the allocated Avro type buffer space and cause an error. For example 1698100138468892700 does not fit in the expected Long type and this causes issues with Avro when deserializing.

As I stated in the original issue, I don't think that this can be fixed on the client side. Can we ensure that the Avro schema supports all possible values for this field?

PubSub Java Publisher error

I am getting a null pointer exception when publishing a message to Salesforce event

2023-09-07 11:42:04,227 [main] java.lang.Class - Using grpcHost api.pubsub.salesforce.com and grpcPort 7443
2023-09-07 11:42:27,543 [main] java.lang.Class - Error During PublishStream
2023-09-07 11:42:27,552 [main] java.lang.Class -  === Exception ===
java.lang.NullPointerException: null
	at org.apache.avro.generic.GenericRecordBuilder.set(GenericRecordBuilder.java:124)
	at org.apache.avro.generic.GenericRecordBuilder.set(GenericRecordBuilder.java:113)
	at utility.CommonContext.createCarMaintenanceRecord(CommonContext.java:198)
	at genericpubsub.PublishStream.generateProducerEvent(PublishStream.java:123)
	at genericpubsub.PublishStream.generatePublishRequest(PublishStream.java:143)
	at genericpubsub.PublishStream.publishStream(PublishStream.java:68)
	at genericpubsub.PublishStream.main(PublishStream.java:203)

Anyone faced this issue before? any guidance here is helpful. I am following steps from Salesforce documentation.

Go code doesn't have any bitmap processing examples

Was throw for quite a loop when I received these bizarre bitmap-encoded strings in the ChangeEventHeader because Googling ChangeEventHeader changedFields led me to misleading examples of changedFields in other contexts (where they are plaintext field names).

It would be very helpful if there were examples of handling the bitmap-encoded changedFields in Go.

Does PubSub API support multiple record crud operations for custom channels?

I am subscribing to a custom channel with channel members of AccountChangeEvent and another custom object. I receive the events just fine when I update one of each record, but if I do any CRUD operations on more than one type of object (say I update 2 account records) then I never receive the event.

I experimented with the custom channel and cometd in java and it seemed to receive all the events.

I replicated this issue with the pubsub api java and javascript implementations provided by salesforce and I am unable to receive events from custom channels when updating two or more accounts.

Are there issues with these implementations or are they are done correctly and the pubsub api just does not support multiple record operations for custom channels?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.