Management Operations about azure-service-bus-dotnet

[mythz]

@ChristianWolf42 Please take the time to listen to library users feedback. Not being able to create Tables in ORM's or MQ's in client libraries is a major inconvenience that no other popular MQ or RDBMS library has, this should be sending alarm bells when designing libraries with this limitation - you're inventing inconveniencing no-one else has to put up with that given a choice will be choosing a different technology that doesn't impose artificial restrictions with what can be automated in code.

So creating tables in SQL or queues in service bus should be handled independently from sending receiving messages.

Yes creating RDBMS tables or MQ's is independent from inserting rows or sending MQ messages, this does not mean that the ability to create Tables or MQ's should not be possible in code! It means it should ideally be done on Startup where we can create tables/MQ's that don't exist allowing Apps to be able to create any Tables/MQ's they need instead of breaking and requiring manual intervention.

So in many cases up-taking new management libraries or actions should not be causing too much disruption over time and can be done when appropriate.

IMO this just indicates limited knowledge on how the technology is used in the real world. Make no mistake this is a user-hostile limitation that no-one else has to live with coming from a different technology. Not having the ability to automate this in code just means we're forced to workaround this limitation - this does not mean "should not be causing too much disruption", it's a major inconvenience that's imposed on every developer/user of Azure Service Bus.

As a library author we have to document why some features are not possible in Azure Service Bus, something we have to shield support issues for because the technology is incapable of supporting basic use-cases everyone else supports.

[SeanFeldman]

I would like to re-open this issue.
Here is the reason why it should be addressed and discussed.

The old client provided management operations out of the box. For good or for bad, it was providing the ability to perform CRUD operations and report message counts. With the new client, customers will be facing a few challenges. First, there will be sheer conversion of the ASB code. Then, there will be the shift to the AAD based authentication and ARM based approach. And then, there will be wheel reinvention since every single customer will have to implement what NamespaceManager was already doing. For that implementation, there's an Azure ResourceManagement project, that is auto-generating API either based on AutoRest branch or Fluent branch.

Here's an example of a typical confusion and amount of work each customer will have to go through to get the NamespaceManager equivalent: finding number of messages. Or an example posted here that was answered by an MVP (and not the repo maintainers) that leads to a documentation page that is far from a good example. The separate management library is rough edge and requires plumbing code to be written and tested by each customer for now obvious reason. Not to mention that there clearly a gap between ASB and Management teams:

• ServiceBus management client v1.0 feedback
• Trying to get answers for the missing API
• Reporting non-implemented properties

While there's no doubt that eventually management operations and run-time operations should not be mixed and the right step in that direction is to split the library into two, there's got to be a better story for users.

Update 2017-09-27
An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

[ChristianWolf42]

I wanted to provide an update on what we concluded today:
As @clemensv mentioned above we will do multiple things to opimize and improve the situation:

1. We will support role based access control and managed service identity. Through this it will be possible to use the same token for: Access to Azure Resource Manager, the Queue / Topic client, metrics
2. We will support a light weight management library for the .Net Standard client. It will have the following capabilities:

• Get - limited to exists check and give metadata back
• GetRuntimeInformation (Gets all counts and last state, Approximate count, accurate within 10s)
• GetQueueNames, GetTopicNames (List entity names)
• Create entity
• Delete entity
• Update (Need details on what metadata to update, can be done during implementation)
• FindOrCreate (Upsert - queue doesn't exists create it)
• UpdateOrCreate (Upsert)

Additionally to 1. and 2. we will keep enhancing and optimizing the metrics we make available and emit and work with the monitoring team on event grid uptake as mentioned in above post.

In regards to timeline we will aim for releasing these optimizations incrementally. We start asap but it will take us some time till we have everything baked and released. Till everything is fully in place it will likely take us till summer 2018. We will try to release critical pieces first and update on progress here and on our blog: https://blogs.msdn.microsoft.com/servicebus/

[jtaubensee]

Hi @sealightPT - So far we have decided not to include management operations in this library. As a part of Azure Resource Manager, we now have multiple ways to dynamically provision/manage entities.

The functionality you are looking for is now included in the Microsoft.Azure.Management.ServiceBus library:

• GitHub repo - https://github.com/Azure/azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus
• NuGet package - https://www.nuget.org/packages/Microsoft.Azure.Management.ServiceBus/
• Sample - https://github.com/Azure-Samples/service-bus-dotnet-management

Using Resource manager is the standard way to manage Azure Resources, and we have chosen as a team to adhere to that standard. We also think that the separation is a good idea, and we do not want to encourage patterns where management operations are invoked at every run-time operation.

For example checking if a queue exists before each send, will cripple your performance. As much as this sounds obvious, we do see it regularly.

[Mossharbor]

I created a nuget package that has the CRUD functionality (create/delete/exists get descriptions) from the old assembly but now targeted at .net core 2.0. This should provide a temporary unblocker for those stuck in their migration process.

Install-Package Mossharbor.AzureWorkArounds.ServiceBus
https://github.com/Mossharbor/AzureWorkArounds.ServiceBus

[SeanFeldman]

🎉 😍 🥂
On behalf of the entire ASB .NET customer community - THANK YOU!!!

[SeanFeldman]

now that everything is in Azure Monitor I have no need to have management capabilities in the SDK since all infrastructure is now available in the general Azure ARM SDK

That's the key - you don't have the need 🙂
I have customers that want to run with a single connection string rather than having several pieces that would be required to authenticate. There are also scenarios where application has to mutate topology. For these scenarios the simplicity of connection string outweights AAD option.

There's also another one, don't piss off your customers 😃

[clemensv]

Reopened.

We'll stick our heads together in the next few days and figure out a good way forward that addresses the concerns brought up here.

[SeanFeldman]

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

The good news - you can with Microsoft.Azure.Management.ServiceBus.Fluent.
The bad news - it's not a 3 liner anymore.

var azureCredentials = SdkContext.AzureCredentialsFactory.FromFile(path); // or different way
var serviceBusManager = ServiceBusManager.Authenticate(azureCredentials, subscriptionId);

// namespace manager
var @namespace = serviceBusManager.Namespaces.GetByResourceGroup(resourceGroupName, namespaceName);

// check if queue exists
var queue = @namespace.Queues.List().FirstOrDefault(x => x.Name == queueName);
if (queue == null)
{
   // create a queue
   var createdQueue = @namespace.Queues.Define(queueName)
                    .WithSizeInMB(1024)
                    .WithMessageMovedToDeadLetterQueueOnMaxDeliveryCount(10)
                    .WithMessageLockDurationInSeconds(30)
                    .Create();
}

[ChristianWolf42]

Right now we are aiming for this being available sometime around June 2018.

[erick-thompson]

@tomkerkhove There are a number of reasons. First, the key requirement is that the service needs to create topics (and sometimes subscriptions) during the course of normal operation.

I use MSI for accessing Key Vault, and so having to add all the Service Bus access policies makes the deployment system more complex. All other resources used (SQL Azure connection strings, Azure storage keys, Event Hub connection string, etc.) are stored in Key Vault, which can then use MSI, certificates, or even client id/secret. Service Bus is the only Azure service that doesn't allow for this common pattern to be followed. So additional complexity is required to now have a totally separate authentication mechanism. And it makes locking down the MSI account harder.

Aside from the complexity and oddness as compared to other resources, using the ARM APIs is very slow. Getting an access token can sometimes take a second or two (and sometimes up to 5), which forces any operation that need to use the token to be something driven by a queue or other long running operation mechanism. Additional complexity, whereas the SAS approach is fast enough to be handled as part of a service call.

Finally, it feels wrong. I don't use Azure ARM/REST to create a blob container, so why do I need to use it to create a topic? Creating a storage account, sure. Creating a Service Bus namespace, sure. But a topic? It doesn't fit with the model used in other Azure services.

[mythz]

@ChristianWolf42 The end solution should be straight forward, provide the ideal code API for developers to be able to create/delete queues/topics in a single library. This is what developers want and expect from any MQ, RDBMS or any other data store library. We need to be able to detect existing queues/topics and create them when they don't exist, that's the basic feature we need.

Any compromises or obstacles that's needed in addition to calling a simple API on this library, imposes that additional undue complexity onto every developer that has to work with it, which should be contra to a library's goals of being useful and making it easy to develop with. Having to reference an additional library to perform basic tasks is workable but definitely not ideal. I honestly don't understand the desire to make it more complicated that what's expected, where did the technology inspiration come from to require a separate library or have queue's created manually?

[ChristianWolf42]

Hi All, we had a meeting on this early this week and started writing up how we want to address this. We will need a little more time but will come back to this thread soon.

[SeanFeldman]

I can see this question repeating itself in the future. Perhaps README.MD should contain a link/reference to the new management library/samples @jtaubensee ?

[sbosell]

I can understand the need for connecting via the resource manager to create service bus accounts but not to create queues inside of a service bus. It is a pretty common operation for most systems which I think most if not all of the test code that has been published does. Can you please add back the capability to manage queues without having to use the Azure management libraries?

[SeanFeldman]

An option to bring to the table - a new Microsoft backed OSS to provide NamespaceManager like functionality.

[mythz]

@ChristianWolf42 We'd like to be able to create queues/topics with the client library using the normal Azure Service Bus connection string which AFAIK isn't possible. The use-case is being able to easily detect missing queues and create them on Startup, e.g. 1 API call to detect existing MQ's, 1 API call to create them. Referencing an additional library is workable but I don't see any benefit of needing to, it certainly doesn't make the library any easier to use.

[janpieterz]

This sounds very odd, is the plan to move all Azure management operations to separate libraries backed by Azure AD? Like, are we going to need Azure AD credentials and resource manager/subscription knowledge to create tables in a hosted SQL database for example? Create tables, containers in storage? 😕

It sounds way better to use non-management and management connection strings (like we've been doing since forever with read/write/create permissions in SQL) than to jump through all these extra hoops to get the right stuff and need knowledge of subscription and resource groups into applications and tests to run and be deployed to new/changed underlying infra.

Please provide the NamespaceManager like capabilities. Either through something open sourced or simply included in the default library, with no need for extra configuration items (just connection strings) and no need for knowledge for apps where they're going to run. This approach will considerably increase configuration and subscription management with, for us at least, no return.

[rajrao]

We used to create a subscription with a short time to live when a website came online. That way it could get messages published to a topic. An example message was a ping like message, that would cause all websites to respond with a message. With the old library, each website, would check to see if a subscription existed for it and if not, it would create it (The short time to live, would then auto delete the subscription if that server hosting the site went away). With the .net standard library, I cant do it. And the ARM libraries dont work, as the information required by them are tightly controlled at our company.

Would love to get the NamespaceManager functionality in the .net standard library.

[shahbarkha]

It would be terrific if we can address this issue where one does not have to use the AAD to just create the entities in the Service Bus. Connection string should be enough for the user to deal with creating entities like it used to be in the old library. I have submitted PR for Foundatio community that has new implementation of azure service bus and it looks really ugly having to deal with management library in the service bus implementation.
@SeanFeldman I would really love it if you could review my PR and give any comments whenever you have a chance :)
FoundatioFx/Foundatio.AzureServiceBus#1

[SergeBerwert]

I struggle with the same problem right now. I think it should be possible to manage namespace in this library. In the cloud we wane have software that is easy to scale on demand that means a software must be capable to create its own subscriptions or even its one topic. I don’t want rollout security on arm level to achieve that. I think it’s not smart from security perspective.
The problem is all I want is a library exactly the same it was bevor but for .net standard. Why can you not make the old library capability to use in .net standard like they did with the storage library.
The decision you made creates only problems and costs with no additional benefit for us. And you slow down us on fast migration to .netstandard.

[niemyjski]

Any timeline for when the connection string management will come back? We have a abstraction for the new client and we will not be supporting the new client or releasing the update until it's released. It's unacceptable to have a requirement on AD inside of a third party library and a bad practice to include AD related auth in deployed applications and services.

[ChristianWolf42]

@tomkerkhove Yes, we will have a very lightweight / approximate count based on an emitted / cached result. We try to have it accurate within 10s. As @SeanFeldman explains in his blog https://weblogs.asp.net/sfeldman/reading-azure-service-bus-metrics the current implementation is pretty expensive. With the new implementation it will be much cheaper but also only an approximate count.

[niranjan-sharma]

Any thought, When it(Creating Topic or Queue ) will be available ?

[niemyjski]

If you need anyone to help test the management changes let me know. I have a branch waiting for these changes ready to go :)

[alex-bogomaz]

Hello, this would be really helpful for us too if Microsoft will provide functionality similar to:

var ns = NamespaceManager.CreateFromConnectionString(connectionString);
ns.QueueExists(queueName);

in NetStandard library,
so we could create queues, topics inside service bus, having only connection string and do not use AAD auth for that.

[clemensv]

The connection string based management that is used by the .NET Framework client will continue to exist inside the service. We have a doc update rolling through the publishing system right now that'll show up here (check the publishing date) and that clarifies further how that protocol works. Our Java client actually has a substantial (but not complete) implementation in its repo for testing purposes. (I'll get back to this below)

Here's an update for everyone following this issue on where we are with the discussion and bit more on the motivation.

First, NamespaceManager as-is needed to go because it covers too much ground. Event Hubs and Service Bus and Relay are clearly distinct capabilities now and therefore it will have to be broken up.

The use-case that @SeanFeldman pointed about above about obtaining the number of messages was actually a key factor in holding off on a straight port of NamespaceManager as it stands, even in scoped down form and just for Service Bus. We have far too many customers running GetQueue in tight loops to collect the queue length observations. On a partitioned entity that is super expensive and dings everyone's performance on the SB Standard SKU, because it needs to collect and roll up the info across the cluster.

The key reason for why developers collect the counts at runtime appears to be driving policies for scaling resources and whether they should attempt to receive from the queue. This blog post by Tom Kerkhove shows how to scale up/down using the info we push into the monitoring pipeline. In addition, we will add an Event Grid trigger shortly that will fire when a queue has messages while there is no active consumer, meaning you can spin up a consumer reactively and not by polling the counts. And we're discussing for how to raise such events based on custom monitoring rules similar to what Tom's blog post shows. When we have those new options available, I don't see much in terms of urgent use-cases that would justify serving up the counts to the client on the data plane surface area.

Another related behavior is that we have seen customers stick application configuration into the "UserMetadata" property and for their apps to pull that before every interaction with the entity. That wasn't the intent.

All operations that enumerate entity collections are, consequently, even more expensive, because they all return entity descriptions that hold runtime stats.

The puzzle we are trying to solve is how to get everyone to a place where they can realize their key scenarios and we can yet make sure that we prevent behavior that harms performance and reliability, especially also on the lower level SKUs that run fully multitenant without resource reservation.

Current status of the discussion is that the new client surface area for the "direct" management operations through the data plane will require a SAS token with "Manage" permission like before, but we're going to constrain it such that it is focused on the core runtime requirement to create/update/delete individual entities (queues, topics, subscriptions). The subscription client already allows managing rules. The new client will not provide access to user-supplied metadata, it will not have enumeration (paging) for entities, and it will not yield metrics. The latter requires some server work, because suppressing the metrics requires a protocol change. We also want to ditch XML, but that may be a further iteration.

That new client surface area will likely be shipped as Java/.NET packages separate from the core client (similar to "Event Processor Host" for Event Hubs) because we don't want to lead all new users onto this path automatically and too obviously. The default management model will still be through Azure Resource Management, and for many mainstream customers, the confusing problem of having different security contexts for control and data plane will largely go away within the coming months as we roll out the next iteration of AAD/RBAC integration (which we already have for the control plane now).

We also want to make the resulting set of gestures available over pure AMQP. Management operations over AMQP 1.0 aren't standardized, yet, but is something that other AMQP 1.0 brokers already do loosely based on a spec draft that has been sitting in the OASIS AMQP TC for a while and [Service Bus also uses gestures leaning on that draft for updating rules and some other actions}(https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-request-response). We've picked up some of the basics (like request/response) in the OASIS AMQP TC recently and I expect that we'll get to a point that we'll end up with a standardized set of gestures.

For "right here and right now", there is a fully supported REST API endpoint that you can continue to use, and doing so will completely unblock any blocked scenarios you might have. I hope the doc update will be helpful. It's a few lines with WebClient and some XML handling. It'd probably be easy for me to share code for this, but then it's "the official Microsoft thing" that gets picked up, with the effect that we shot ourselves in the foot on the guidance goals above.

We're now splitting some scenarios across Monitoring, Event Grid, Resource Manager, and the runtime API. We only had the latter when we started with all of this, and with the new SDKs we want to come out with a clean and sustainable story that guides developers to the right practices.

Adding that new surface area is not super hard, but it is work and it will require some server-side support, so it's something that competes with all other visible and invisible priorities for a time slot to get done. Therefore no firm ETA at this point, but we rarely write specs for things that we aren't shipping within the coming 3-6 months.

PS: Relevant tweet

[ConX-Ryan]

Waiting with baited breath for this old functionality in the new .NET Standard packages

[xplicit]

@SeanFeldman I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

Endpoint=sb://domain.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<key is here>

Can you point me to the code which does that job? Seems that old windows version of Azure Service Bus is closed source (or maybe I could not find its source) so I was not able to look how NamespaceManager implemented and reuse that code to keep compatibility between .NET and .NET Core library.

[mythz]

Have to agree, requiring a separate library or use a UI to manually create queue's is absurd for an MQ library. I'm surprised anyone puts up with this limitation, It makes everything so much more difficult and you're likely losing Customers at the door when they hit this first hurdle.

Checking/Creating RDBMS tables also shouldn't be done at runtime either, but App's should still be able to create them on Startup if they don't exist. It would be unacceptable if an RDBMS handicapped ORM's from not being able to create them within the library, likewise for MQ Brokers. I don't know any other data store or MQ with this limitation.

[paulstancer]

I'm in the process of migrating a our Azure Service Bus libraries over to .NET Standard 2, and I have to agree with everything raised here.

Access to the Service Bus management through the connection string is pretty important. We allow auto-creation of queues and topics on creation when the SAS key allows it. We also use this to monitor queue length, which can prioritize queue polling, and also programmatically trigger WebApp scale-out events when the queue length grows.

To not have access to this functionality without the need to get an Azure management token is going to cause significant issues. At the moment, I might just bypass this library altogether and just go direct to the API and generate my own SAS token to do the management that I need. I'm not sure that is the desired behavior that you are looking for.

Also, these management libraries are an abomination. Don't be surprised when no one uses these Swagger generated monstrosities.

https://github.com/Azure/azure-sdk-for-net/blob/3b013cd9b9aac9407f3d9bea221922d037c1596a/src/SDKs/ServiceBus/Management.ServiceBus/generated.cmd

Thanks,

Paul.

[mscrivo]

@SeanFeldman and @nemakam We've been having a ton of problems with SB lately, lots of throttling, random exceptions. We've written a ton of code to handle all the failure conditions and we're finally able to tread water at least. Anyhow, all that aside, we've done what you guys have asked and switched from using the GET API call to get queue counts to using the Metrics API instead for counts and the numbers we're see are way off. For example, for one of our queues, GET is reporting 11m active messages, whereas the metrics API is reporting 800k. Any idea what's going on there?

Again, we use these numbers to automatically scale up/down our various worker tasks and need these counts to be somewhat accurate. We used to poll the GET on each queue every 3-5 mins, and now we're querying the metrics every 5.

[rogerdawson]

@mscrivo , I hear you on the service bus issues, we have our own retry logic on top of the service bus retry logic to try and cope with the SB issues but they still get through. The quality of SB standard has nose dived over the past year. We've been pushed by MS to SB premium in some cases to reduce the exceptions and throttling but that's not got great performance/scalability and the costs are astronomical in comparison to standard. It's easily our biggest maintenance headache and support generator on Azure. Sorry that this is slightly off-topic.

[mscrivo]

@rogerdawson thanks for that. It's reassuring that we're not the only ones.

[clemensv]

@tomkerkhove I specifically point to your post since we'll definitely keep doing that.

@niemyjski The messaging systems you will have dealt with are not running a queue in a way that there are some 16 distinct partitions with designated primary owners managing the partition indices are spread across a 64-node (distinct VMs) cluster for reliability and load balancing, and from which the partial counts must be rolled up. The common messaging systems are also not massively multitenant. The problem with caching is that customers are keenly interested in accuracy, especially when it's about a queue that has 0 messages vs. one that has any messages.

What is your scenario for the metrics, i.e. what are you doing with that information? As I explained above, we have/plan several paths for making the counts or count-based triggers available, so the question is why those would not be sufficient.

[SeanFeldman]

There's going to be some sort of metrics helper and that's part of the spec/design proposal that @ChristianWolf42 currently works on.

👍

[rogerdawson]

We also would like this features. I find it very surprising this wasn't implemented in the first place. For what it's worth, our use case is:

Create a topic/queue/subscription and with various different parameters (express bus, server side batching, size etc.)

I was in the process of converting our code to .net standard but I think we'll have to put it on hold for now. I hope the same mistake hasn't been made with blob and table management as we dynamically create those in code too.

[jtaubensee]

Good point @SeanFeldman - I will add that.

[jtaubensee]

Closing based on #66

[barkhashah]

@SeanFeldman Can you please point us to the github link that points to the latest 1.0.0 management library released on 30th june. The above links are old and reference to pre-release versions.

thanks
barkha shah

[barkhashah]

@SeanFeldman https://github.com/Azure/azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus
This link is pointing to old source code. Management library 1.0.0 is way different than this. When we use nuget package manager, we do get the latest release however, I need to see the source code.

[SeanFeldman]

You're looking at a specific branch. That team might be releasing from different branches. Raise an issue in appropriate repo to get an answer since I'm not a Microsoft-ie 🙂

…

On Aug 4, 2017 14:31, "barkhashah" ***@***.***> wrote: @SeanFeldman <https://github.com/seanfeldman> https://github.com/Azure/ azure-sdk-for-net/tree/AutoRest/src/ResourceManagement/ServiceBus This link is pointing to old source code. Management library 1.0.0 is way different than this. When we use nuget package manager, we do get the latest release however, I need to see the source code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#65 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABP7tkUAco-kA63a8jPQVHry7PvSFZMrks5sU3-3gaJpZM4LeHvm> .

[xplicit]

Asked this question along time ago Azure-Samples/service-bus-dotnet-management#9 but didn't get an answer. In new management API I did not find how to check queue existing by using connection string.

What is the equivalent for this code snippet from WindowsAzure.ServiceBus in new management API?

var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);

namespaceManager.QueueExists(queueName);
namespaceManager.CreateQueue(new QueueDescription(queueName));

[SeanFeldman]

/cc @Azure/azure-service-bus-write

[SeanFeldman]

@xplicit ironically, precisely the reason I'm asking to re-open this issue and re-evaluate what's been done so far.

I still don't understand how can I get resourceGroupName, subsriptionId and other settings when I have only this connection string?

You can't. The management library doesn't connect using connection string. Instead, you need to go through Azure Active Directory. See Authentication with Azure Management Libraries doco that explains what is needed and how to get it using Azure CLI (locally or using the portal).

[niemyjski]

Yeah, it is pretty absurd but I understand why it was done initially. However, making it so hard to do anything (like requiring AAD based authentication) is pretty unacceptable. Having to use AAD just to consume a service bus in a deployed app.... This is a security and deployment nightmare. Why can't I just use my existing connection string?

[SeanFeldman]

tagging @ChristianWolf42

[ChristianWolf42]

So first we will not retire NamespaceManager anytime soon. On the other hand we are investing in aligning with Azures Management story (ARM).
In most cases Management code should be separate from interactive code. So creating tables in SQL or queues in service bus should be handled independently from sending receiving messages. So in many cases up-taking new management libraries or actions should not be causing too much disruption over time and can be done when appropriate.

[ChristianWolf42]

@mythz and @SergeBerwert I do understand your frustration and want to add a few points:
Correct we push the usage of ARM across Azure. That is implying some inconveniences like using different credentials e.g. and there are for sure many others.
At the same time also correct our old library is not on .net standard, hence not platform independent.
Additionally in any case we want at least two libraries and never have one and the same library for management and or client.

Now let us talk a bit more what would help you exactly. I make an attempt of describing something and are happy to discuss with our engineering org what can be done and also roughly when it can be done. I can't give you any promise at this point but do want to help:
@SergeBerwert if I understand you correctly if we would just have a seperate management library which works similar to the old namespacemanager but is in .net standard this would be ok for you?
Would it be a solution for you if we would expose all management operations as http apis?
Please let me know.
@mythz please let me know if what I propose above would resolve your concern or if you would like to see something else / something additional.
If we cannot resolve it in this thread I am also happy to facilitate a call.

[ChristianWolf42]

@mythz Please let me ask a few clarifying questions here: Do you want an upsert like pattern, e.g. FindOrCreate?
I am not sure I understand, you do not need to create queues manually. I still would want to understand why you would be against having a separate library or let us keep speaking on the scenario level for now. Let us also put aside the code migration effort:
If you start a new project and have one or two libraries, or even a set of rest calls and a library. Would that still meet your requirements?
Now let us add the migration component: Let us say anything which you currently have implemented will keep working (As said above, NamespaceManager is not going to go away anytime soon). You only need to worry if you write net new code.
Please let me know what I am missing so I can represent that accordingly over here.

[SergeBerwert]

@ChristianWolf42 I could totally live with a separate assembly for namespace management something like Microsoft.Azure.ServiceBus.NamespaceManager . But no AD credentials needed. I think separation of concerns is in general the right way to do. In my opinion, it's maybe a bit too much because I think a lot of developers wane create subscriptions and topics just bevor send and receiving the message. But I don’t want to do it with ARM. First of all, because of security, I don’t want have more credentials in my app then absolutely needed. And second, as much as I like ARM to create infrastructure it definitely has his issues. ARM sometimes is not reliable its sometimes slow and often cannot relay tell you what the problem is when something goes wrong. When my containers are rearranged during the night because of infrastructure issues or load or whatever I would not like if the container could not work because it cannot connect to the service bus. So fare service bus is the backbone of our applications and it works very reliably.

[clemensv]

Thank you, @paulstancer. We're on it. We also know that the libraries based on the Resource Manager HTTP API surface area shape are not particularly pretty.

[paulstancer]

Great. Thanks. Any estimation on how long this might take?

[paulstancer]

Appreciate that you're now looking at this. There also seems to be a disconnect regarding some of the Azure Service Bus documentation. This post is telling everyone to migrate away from ACS to SAS; not the other way around: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-migrate-acs-sas

[ChristianWolf42]

@paulstancer I am not quite sure I understand. We definitely want everybody to migrate to SAS and away from ACS.
We deprecated ACS, hende people should migrate to SAS:
https://azure.microsoft.com/en-us/blog/acs-access-control-service-namespace-creation-restriction/
Let us discuss this in a seperate thread please if we should discuss it in greater detail as I would not necessarily like to fork this topic.

[niemyjski]

We have queue metric behaviors and those can be retrieved every few seconds, so that they can be reported to any service like statsd / etc I think if the main driving force behind this is that getting stats is so expensive then an overhaul needs to be done on the backend... Getting queue lengths and number of working items should be super cheap operation.... I know in the other messaging systems that I've dealt with it's pretty dang cheap.. But yes, it should not be done in a tight loop. Maybe you should rate limit this (Or cache the information a little longer)?

[tomkerkhove]

@clemensv I presume Service Bus will still send Message Count to Azure Monitor as a metric for dashboarding purposes or is that something that is not going to happen anymore?

[SeanFeldman]

What is your scenario for the metrics, i.e. what are you doing with that information?

@clemensv an example would be tools such as ServiceBus Explorer or ServiceBus360. Customers use it to troubleshoot issues and rely on message counts to understand what's happening. Normally, it would not be a tight loop and constant polling, but a specific request per entity.

[tomkerkhove]

@clemensv Sounds perfect to me, only remaining roadblock could be for tooling like @SeanFeldman mentioned.

[clemensv]

@SeanFeldman Tools are already much better off using the Resource Manager as a starting point right now, because that provides complete access to all user subscriptions and resources and not just to one namespace. Access to the data plane from there can either be done by picking up or making a rule for that purpose or, soon, by just using the same identity for send/receive. If Service Bus Explorer were started today, that'd clearly be the preferred direction. Having to paste the connection string into the tool is rather clunky at this point. I would much prefer for it to pop a login dialog and then have access to all my namespaces. For logs and metrics we feed into the dedicated Azure services from which tooling (like the Azure portal) can pick that up.

[SeanFeldman]

@clemensv I'm in agreement on what the course of a greenfield project should be.

Access to the data plane from there can either be done by picking up or making a rule for that purpose or, soon, by just using the same identity for send/receive.

That's not public yet 🙂

For logs and metrics we feed into the dedicated Azure services from which tooling (like the Azure portal) can pick that up.

For that I'd love to see a package that allows to read metrics w/o going through custom built REST API wrapper.

When a functionality is going away w/o a clear understanding on the replacement option(s), it generates worry. Unjustified worry, if you know what's coming. All in all, sounds like a good plan, but needs to be elaborated a bit more.

[clemensv]

The mentioned REST API clarification is now public.

A key enhancement in that section are that the Overview now clearly explains what goes into the Atom entry content (xxxDescription elements) and how subscriptions and rules stack underneath topics, even though I didn't go as far as to separately document an XSD for those elements. I believe the element structure is easy to infer from the Framework reference doc. I also touched all the operation docs to clarify that the Authorization header accepts SAS tokens.

@SeanFeldman Something becomes public when we make it public, so I probably just did. There's going to be some sort of metrics helper and that's part of the spec/design proposal that @ChristianWolf42 currently works on.

[niemyjski]

@clemensv In foundatio https://github.com/FoundatioFx there are several usages where we get stats

1. We have a Queue Job that can run until empty. We are looking into ways where we can cache this info better or exit when a dequeue returns null (no items) instead of relying on stats.
2. We have Metric Queue Behaviors that you can attach to any queue abstraction and get the count, working count and dead letter. You can then report those metrics to any stats implementations (We report our apps to statsd so we have nice dashboards...). This happens on a user defined schedule...
3. Allow users to get the queue stats via the api.

[tomkerkhove]

It think most of them are already mentioned:

• 1 They will provide notifications to let you know when your queue is no longer empty. Once there are no messages you will basically have to wait for that event to come in or periodically check.
• 2 & 3 will have to be done by querying the Azure Monitor metric for Service Bus

I think it's good to mention that the current count is actually an estimate as well so I would make bets on it.

[danielmarbach]

Great news! Thanks for all your support

[tomkerkhove]

@ChristianWolf42 So getting the count of messages on a queue will stick around then or what count are your refering to with GetRuntimeInformation

Thanks for the update!

[tomkerkhove]

@ChristianWolf42 I get that it is expensive so I'm amused that there will even be a light version - Great!

[sfalik]

Will we also be able to create rules/keys on a newly created topic? My use case is:

1. create a topic
2. create a subscription
3. create a 'listen' rule on the topic
4. generate a SAS token for the subscription/topic

In my case, I have a SAS token for the entire namespace with 'manage' claim. I then want to subdivide the namespace and give out SAS Tokens at a granular level. I can of course do this with the full .net api, but this is not available from the REST API or this .net standard library.

[niemyjski]

@rogerdawson You could use our Foundatio Abstraction for now :) https://github.com/FoundatioFx/Foundatio.AzureServiceBus and then when management is enabled will merge the pr for .net core support.

[prapicault]

Has anyone tried to mix and match the new library (to get send and receive messages) and the old one (to create subscription programmatically)?

[mscrivo]

Is there any update on when the lightweight queue counts management abilities will be coming to this library?

We were stopped dead in our tracks trying to convert our app to .NET core by this issue. Our worker auto-scaling algorithms depend on getting specific queue counts to scale up/down particular workers. AFAIK, this is not something that can be done via the portal or with the current metrics, so it would be really nice to have that functionality back, even if it's not super accurate.

[tomkerkhove]

@mscrivo It depends what you want, if you want to have message count it is available as a metric and can be used to autoscale on. Example in this post.

[mscrivo]

@tomkerkhove That doesn't seem to be what we need. That seems to only allow scaling for the entire cloud service based on a very limited set of service bus metrics (of which message counts is not one of them).

In our system, we have sets of worker role cloud services that each run a whole bunch of different workers. Each worker has a 1-1 mapping with a queue/topic/subscription and our autoscaling mechanism is able to scale an individual worker running within that worker role based on the queue count specific to it.

[ChristianWolf42]

@mscrivo we have it in our current engineering plan to be delivered in the next 6 month. We are trying to get it out sooner (like in 3 month time) but that may not be easily possible.
Seperately from the management library we also work on adding the queue count and queue size via metrics but are having a few internal dependencies here which hold us up a little.

[tomkerkhove]

@mscrivo I have the same scenario and works with the built-in metrics out of the box.

You can see all roles for your cloud service under Autoscale:

You can configure multiple scaling rules for each individual role, for one or more queues depending on your scenario.

This looks like this:

Configuration looks like this:

[MikaelGRA]

I am currently trying to migrate away from WindowsAzure.ServiceBus to Microsoft.Azure.ServiceBus, so I can support netstandard, and that brings me to my question:

Is it even possible to create Subscriptions with the Microsoft.Azure.ServiceBus package? All I can see is a few operation to change existing filtering rules on an already existing subscription.

Say I have an application that creates subscriptions with customized SqlFilters on an existing Topic so they only listen to a subset of all messages (I know, crazy use-case, right?). Is it possible to do this with the new this API?

[nemakam]

@MikaelGRA
As of today, it is not supported in this repo. The right way to do it is mentioned in #65 (comment)

[SeanFeldman]

@MikaelGRA @nemakam more relevant comment was made later in the thred, here #65 (comment)

Are there any preview bits that can be shared regarding the mgmt plane operations for a .NET Core app - the performance of the Microsoft.Azure.Management.ServiceBus leaves a lot to be desired wrt perf and the REST API is short #365

We are blocked moving forward leveraging .NET Core and Service Bus mgmt operations.

[erick-thompson]

I would appreciate any update on this as well. I'm currently using the AD based approach, and it is providing to be difficult due to performance and management of the service identity. The functionality I need is to be able to create Topics and Subscriptions using the SAS authentication approach.

[michaljaros84]

Honestly the design decisions around the .net core lib for service bus are confusing. Especially that I've just written a Python client that can use SAS to manipulate namespace. Why wouldn't this functionality by provided for the .net core?

[erick-thompson]

I kind of understand, as I think there is a desire to move management/infrastructure functions to ARM/AzureRM/REST, and the creation of topics/queues/etc straddles the line between infrastructure and application. Nonetheless, I am waiting with bated breath for this functionality.

[tomkerkhove]

Honestly, now that everything is in Azure Monitor I have no need to have management capabilities in the SDK since all infrastructure is now available in the general Azure ARM SDK. Is there a specific reason why this does not fit you needs?

[0xced]

Right now we are aiming for this being available sometime around June 2018.

@ChristianWolf42 Are you still on track to deliver something next month?

I created a nuget package that has the CRUD functionality

@Mossharbor Thanks for this workaround! For me, it’s still just missing GetQueues() and GetTopics(), do you know if it would be possible to implement these two methods with the API you are using?

[0xced]

Well, it turns out that implementing GetQueues and GetTopics was fairly easy. A pull request is waiting for you: Mossharbor/AzureWorkArounds.ServiceBus#6 😉

[nemakam]

@0xced
Yes. Aiming at June.

[SeanFeldman]

Any idea what's going on there?

Metrics do not report accurate numbers at the moment AFAIK.
@ShubhaVijayasarathy might help and shed some light on the recent developments.

[awr]

Huh. Then why the recommendation to use it over regular counts?

Does it work on some skus and not others? Seems like the numbers on Premium are more accurate than on Standard. Any timelines on when this will work correctly? Otherwise we'll have to switch back to the old approach once again...

[nemakam]

@mscrivo
800k to 11m. That's way off. This might need some investigation to be done. Could you raise a support ticket online (through Azure Portal) and our Support executives can find out about the discrepancy you are seeing.

[mscrivo]

@nemakam already done.

[SeanFeldman]

Removed Backlog milestone as it's a WIP.

[SeanFeldman]

@nemakam should this be 3.1.0 or 4.0.0?

[nemakam]

There is no breaking change. 3.1.0 seems valid according to SemVer.