plagueho / cosmosdb Goto Github PK

See for example: https://github.com/PlagueHO/PowerShellGet

When I try to create an included path which contains a spatial index the request fails because the precision property is always returned from New-CosmosDbCollectionIncludedPathIndex whether I specify it or not. The error returned is "The 'Spatial' index type should not have a precision specified."

There is no simple cmdlet available for creating a new CosmosDB Account in the AzureRM cmdlets. There is a command in the Azure CLI.

It would be useful to have a simple function that could use an ARM Template to easily create a Cosmos DB Account.

When I try to provision an index with the indexing mode set to none, the New-CosmosDbCollectionIndexingPolicy function returns an error as this value does not exist in the ValidateSet.

• [5.1.16299.431] Version of PowerShell you're using
• [Console] PowerShell host you're using
• [Win10] Operating system you're running
• [2.0.12.418 ] Version of CosmosDB you're using

How does one get the error details when CosmosDB returns an error? When using New-CosmosDbDocument, for example, I occasionally receive this:

Invoke-RestMethod : The remote server returned an error: (500) Internal Server Error.
At C:\Program Files\WindowsPowerShell\Modules\CosmosDB\2.0.12.418\lib\utils.ps1:512 char:23
+         $restResult = Invoke-RestMethod @invokeRestMethodParameters
+                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

I can't see any way to get more information. Sometimes, the error is 400 Invalid Request, which is often missing or malformed data, but I have to figure out what's missing by trial and error.

https://docs.microsoft.com/en-us/rest/api/documentdb/user-defined-functions

Add parameter that sets x-ms-documentdb-expiry-seconds header.

See
https://docs.microsoft.com/en-us/rest/api/documentdb/get-a-permission

https://docs.microsoft.com/en-us/rest/api/documentdb/attachments

Currently all the build, test and release code is contained in the AppVeyor.yml file. This should be moved out into a PSake file.

Azure Cosmos DB now supports sharing reserved throughput across multiple collections in a database:
https://docs.microsoft.com/en-us/azure/cosmos-db/set-throughput#provision-throughput-for-a-set-of-containers-or-at-the-database-level

This is not yet documented in the REST APIs:
https://docs.microsoft.com/en-us/rest/api/cosmos-db/create-a-database

When this is documented/available support should be added.

This is available through the PUT method of the REST API for Collections:
https://docs.microsoft.com/en-us/rest/api/cosmos-db/replace-a-collection

Function New-CosmosDbAuthorizationToken incorrectly converts the ResourceLink (ResourceId variable) to lowercase here.

The MS documentation states

The ResourceLink portion of the string is the identity property of the resource that the request is directed at. ResourceLink must maintain its case for the id of the resource. Example, for a collection it will look like: "dbs/MyDatabase/colls/MyCollection"

• [5.1] Version of PowerShell you're using
• [Console host and VS Code] PowerShell host you're using (eg. Console Host, ISE, Visual Studio)
• [Win10 x64] Operating system you're running
• [2.0.7.342] Version of CosmosDB you're using (use Get-Module -Name CosmosDB)

Some functions may take a large amount of time to complete. So enabling these cmdlets to run async would be a useful scenario.

Before submitting your issue for the CosmosDB project, please take a moment to provide the following details:

• Version of PowerShell you're using
• PowerShell host you're using (eg. Console Host, ISE, Visual Studio)
• Operating system you're running
• Version of CosmosDB you're using (use Get-Module -Name CosmosDB)

If you have a general question or suggestion, feel free to post on the CosmosDB Gitter Chat at https://gitter.im/PlagueHO/CosmosDB. This is also a great place to just say Hi, ask any questions you might have or get help.

Thanks for contributing your feedback and support! You can optionally submit a Pull Request against this project, if you have a fix you'd like to share.

A custom CosmosDB exception object should be created and returned when the CosmosDB server returns an error. This object should contain all the additional information that is returned by the CosmosDB server.

Convert "Connection" Functions and Parameters to Context. This is to align better with how the AzureRM modules are named (e.g. New-AzureRmApiManagementContext).

Aliases should be used for the old function names and parameter names to ensure the change isn't breaking.

Currently PlatyPS Documentation Generation is performed manually. This should be performed automatically during release. The Docs folder should only be generated at release time.

I have issues when the database name is capital

If I run that script above it will correctly query the collection
If I make a 2nd database with Example it will 401 error

The creation Error is below :

Code for that is :

PSVersion 5.1.16299.251 , I get same error on both 2.08.350 and 2.0.7.288

https://docs.microsoft.com/en-us/rest/api/documentdb/offers

Hello Team ,

I have successfully created the new user for the COsmos DB .Could you please let me know how to use the use to connect to the Cosmos DB using 3T or any way to connect via this user

Thanks
Atul Gupta

I tried 'localhost', 'localhost:8081' and other patterns. As per Tweet chat with Dan, this is not yet supported.
Dan: "This doesn’t work because internally the URI to the CosmosDB is built by combining the $account with ‘.documents.azure.com’. However, there should be a simple work around you could use (until I add built in support): After creating the $connection variable, set the BaseUri property to ‘http://localhost:8081’. E.g. $connection.BaseUri = [uri] ‘http://localhost:8081’. I’m assuming here the emulator uses HTTP rather than HTTPS. What I’ll do is add a way to set the base URI through a parameter when creating the connection object."
Thanks, great tool (a must IMHO).

PartitionKey is handled when -Id is passed

@ https://github.com/PlagueHO/CosmosDB/blob/dev/src/lib/documents.ps1#L156-L159

    if (-not [String]::IsNullOrEmpty($Id))
    {
        # A document Id has been specified
        if ($PSBoundParameters.ContainsKey('PartitionKey'))

But is not handled in the else block. @ https://github.com/PlagueHO/CosmosDB/blob/dev/src/lib/documents.ps1#L173-L259

Error Produced:

PS /Users/jason.chester/_dev/LDM/scripts/generic> Measure-CosmosDbPartitionDocuments -PartitionKey "bmi"
Invoke-CosmosDbRequest : A parameter cannot be found that matches parameter name 'PartitionKey'.
At /usr/local/share/powershell/Modules/CosmosDB/2.0.10.388/lib/documents.ps1:242 char:42
+         $result = Invoke-CosmosDbRequest @PSBoundParameters `
+                                          ~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidArgument: (:) [Invoke-CosmosDbRequest], ParameterBindingException
+ FullyQualifiedErrorId : NamedParameterNotFound,Invoke-CosmosDbRequest

Attatched file contains edits to documents.ps1 which resolved my specific issue.
documents.ps1.txt

• Version of PowerShell you're using

Name                           Value
PSVersion                      6.0.0
PSEdition                      Core
GitCommitId                    v6.0.0
OS                             Darwin 17.6.0 Darwin Kernel Version 17.6.0: Fri Apr 13 19:57:44 PDT 2018; root:xnu-4570.60.17.0.1~3/RELEASE_X86_64
Platform                       Unix

• PowerShell host you're using (eg. Console Host, ISE, Visual Studio)
  pwsh in terminal & VS Code Host

• Operating system you're running
  macOS High Sierra
  version 10.13.5 Beta (17F45c)

• Version of CosmosDB you're using (use Get-Module -Name CosmosDB)
  CosmosDB 2.0.10.388

This would to allow navigating the CosmosDB graph like a drive in PowerShell.

See https://github.com/PowerShell/SHiPS

The Set-CosmosDbOffer cmdlet is currently failing with the following output:

Invoke-RestMethod : {"code":"Unauthorized","message":"The input authorization   
token can't serve the request. Please check that the expected payload is built 
as per the protocol, and check the key being used. Server used the following 
payload to sign: 'put\noffers\n{removed}\nmon, 23 apr 2018 15:47:13 
gmt\n\n'\r\nActivityId: {removed}, 
Microsoft.Azure.Documents.Common/1.21.0.0"}
At /usr/local/microsoft/powershell/6.0.0-beta.7/Modules/CosmosDB/2.0.9.364/lib/
utils.ps1:402 char:23
+         $restResult = Invoke-RestMethod @invokeRestMethodParameters
+                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Method: PUT, Re...ication/jso 
   n
}:HttpRequestMessage) [Invoke-RestMethod], HttpResponseException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe 
   ll.Commands.InvokeRestMethodCommand

The fix seems to be relatively simple and straightforward. There just needs to be a .ToLowerInvariant() added to the end of Line 327 in the /src/lib/utils.ps1 file.

My local environment details:

$PSVersionTable | Format-List               


Name  : PSVersion
Value : 6.0.0-beta

Name  : PSEdition
Value : Core

Name  : GitCommitId
Value : v6.0.0-beta.7

Name  : OS
Value : Darwin 17.5.0 Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST 
        2018; root:xnu-4570.51.1~1/RELEASE_X86_64

Name  : Platform
Value : Unix

Name  : PSCompatibleVersions
Value : {1.0, 2.0, 3.0, 4.0...}

Name  : PSRemotingProtocolVersion
Value : 2.3

Name  : SerializationVersion
Value : 1.1.0.1

Name  : WSManStackVersion
Value : 3.0

Get-Module -Name CosmosDB | Format-List     


Name              : CosmosDB
Path              : /usr/local/microsoft/powershell/6.0.0-beta.7/Modules/Cosmos
                    DB/2.0.9.364/CosmosDB.psm1
Description       : This module provides cmdlets for working with Azure Cosmos 
                    DB databases, collections, documents, attachments, offers, 
                    users, permissions, triggers, stored procedures and user 
                    defined functions.
ModuleType        : Script
Version           : 2.0.9.364
NestedModules     : {}
ExportedFunctions : {Get-CosmosDbAttachment, 
                    Get-CosmosDbAttachmentResourcePath, 
                    Get-CosmosDbCollection, 
                    Get-CosmosDbCollectionResourcePath...}
ExportedCmdlets   : 
ExportedVariables : 
ExportedAliases   : New-CosmosDbConnection

https://docs.microsoft.com/en-us/rest/api/documentdb/collections#indexing-policy

https://docs.microsoft.com/en-us/rest/api/documentdb/create-a-collection

https://docs.microsoft.com/en-us/rest/api/documentdb/documents

Use the Open Code of Conduct

• Version of PowerShell you're using: 5.1.14393.1770
• PowerShell host you're using: ISE
• Operating system you're running: Windows 10
• Version of CosmosDB: 1.0.10.108

Issuing the following command;

$params = @(@{name = "@in_newrelic";value="false";})
$documents = Get-CosmosDbDocument -Database devops -Connection $Connection -CollectionId 'inventory' -Query "SELECT * FROM inventory WHERE (inventory.in_newrelic = @in_newrelic)" -QueryParameters $params

returns the following response;

Invoke-WebRequest : The remote server returned an error: (400) Bad Request.
At C:\Program Files\WindowsPowerShell\Modules\CosmosDB\1.0.10.108\lib\utils.ps1:477 char:23
+ ... estResult = Invoke-WebRequest -UseBasicParsing @invokeRestMethodParam ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
ConvertFrom-Json : Cannot bind argument to parameter 'InputObject' because it is null.
At C:\Program Files\WindowsPowerShell\Modules\CosmosDB\1.0.10.108\lib\documents.ps1:294 char:54
+         $tempObject = (ConvertFrom-JSON -InputObject $result.Content)
+                                                      ~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [ConvertFrom-Json], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.ConvertFromJsonCommand

When I run the following,

$Query = "select * from inventory where (in_newrelic = @in_newrelic)"
$QueryParameters = @(@{name = "@in_newrelic";value="false";}) # @(@{name = "@in_newrelic";value = "false"})

**$bodyObject = @{ query = $Query }
if (-not [String]::IsNullOrEmpty($QueryParameters))
{
    $bodyObject += @{ parameters = $QueryParameters }
}
ConvertTo-Json -InputObject $bodyObject**

the output JSON object appears to match the desired format according to the CosmosDB RestAPI documentation.

{
    "query":  "select * from inventory where (in_newrelic = @in_newrelic)",
    "parameters":  [
                       {
                           "value":  "false",
                           "name":  "@in_newrelic"
                       }
                   ]
}

The bold portion of the script was taken from the source code. I wanted to ensure that the resulting body passed to the RestAPI was in the desired format, which it is. I also checked and the source code seems to be setting the "x-ms-documentdb-isquery" header, so I cannot tell what else could be wrong here.
I can return documents from the table using the following command;
$documents = Get-CosmosDbDocument -Database devops -Connection $Connection -CollectionId 'inventory' -MaxItemCount 5

Version of PowerShell: 5.1.1
Running in: VSTS, ISE

New-CosmosDbDocument fails with a 400 error when the collection is partitioned. This looks to be down to the x-ms-documentdb-partitionkey header missing.

Example code:

    $document = @"
{
    `"id`": `"$([Guid]::NewGuid().ToString())`",
    `"content`": `"Some string`",
    `"more`": `"Some other string`"
}
"@

    $cosmosDbConnection = New-CosmosDbConnection -Emulator -Database 'Database'
    
    New-CosmosDbDatabase -Connection $cosmosDbConnection -Id 'Database'

    New-CosmosDbCollection -Connection $cosmosDbConnection -Id 'Collection' -OfferThroughput 2500 -PartitionKey "id"

    New-CosmosDbDocument -Connection $cosmosDbConnection -CollectionId 'Collection' -DocumentBody $document

If you remove the -PartitionKey "id" from the above it works fine. I have tested a local change with PartitionKey as a parameter to New-CosmosDbDocument, doing something like:

	if ($PSBoundParameters.ContainsKey('PartitionKey'))
    {
        $headers += @{
            'x-ms-documentdb-partitionkey' = '["' + $PartitionKey + '"]'
        }
        $null = $PSBoundParameters.Remove('PartitionKey')
    }

And that all seems OK, I haven't put in a PR as I'm unsure what approach you'd prefer :)

Following the contributions.md guidance, I'm writing an issue prior to submitting an enhancement.

When you use non-partitioned collections, Microsoft imposes a hard size limit of 10 GB. Microsoft allows you to set up alerting based on size, but it is the aggregate account size, not the collection size, rendering it moot.

The CosmosDB PowerShell module contains a lot of the foundation plumbing required to get this information. I've written a small enhancement to return size information about a collection. https://docs.microsoft.com/en-us/azure/cosmos-db/monitor-accounts indicates that this info can be retrieved by doing a GET on the collection, and the response contains usage information for the collection in the x-ms-resource-usage header.

The design introduces a new function in lib\collections.ps1 named Get-CosmosDBCollectionSize which returns information for a given collection, and leaves the existing Get-CosmosDBCollection function untouched. The params for Get-CosmosDBCollectionSize are identical to Get-CosmosDBCollection.

The response contains a hashtable containing the values of the following properties:
triggers, collectionSize, documentsCount, functions, storedProcedures, documentsSize, documentSize. Unfortunately, these do not appear to be documented; some discussion about the potential values are here: https://stackoverflow.com/questions/35274760/understanding-the-x-ms-resource-usage-in-documentdb-response-header.

With this enhancement, I've been able to set up basic alerting based on collection size. If this is of interest, I'm happy to submit a pull request.

While testing out a performing complicated aggregations in cosmosdb using stored procs (https://github.com/lmaccherone/documentdb-lumenize) I ran smack into some limitations of the stored proc Implementation.

In order to address the limitations Invoke-CosmosDbStoredProcedure should

• Support for specifying -PartitionKey
  -- this is required when executing a stored procedure in a partitioned collection
• Change StoredProcedureParameters from string[] to object[]
  -- Should be able to handle most cases with ConvertTo-Json
• Add -EnableScriptDebugging parameter when executing a stored procedure
  -- This appears to be missing from rest api documentation
  -- Relevant SO https://stackoverflow.com/questions/47855094/how-to-debug-azure-cosmos-db-stored-procedures
  -- Consider adding this to the cosmos request automatically when Powershell -Debug is used.

Planning to put together a PR for these if time allows.

This may be more of a cosmosDB concepts issue rather than a CosmosDB Powershell issue, but thought I'd ask - feel free to close if out of scope.

I have a context for a given database, and have obtained a list of offers

$cosmosDbContext = New-CosmosDbContext -Account $account -Database $dd -ResourceGroup $rg -MasterKeyType 'SecondaryMasterKey'; $offers = Get-CosmosDbOffer -Context $cosmosDbContext; $colls = Get-CosmosDbCollection -Context $cosmosDbContext;

Then I have code that iterates through the offers, and eventually tries to match up the offer to a collection as follows:

foreach ($offer in $offers) { $collName = ($colls | where-object {$_._rid -eq $rid} ).Id }

However, it looks like those two objects really don't match up as noted by the different resource IDs below:

Offers:
Id: h-5E OfferVersion: V2 OfferType: invalid Etag: 00007801-0000-0000-0000-5acbb2610000 ResourceId: h-5E Timestamp: 4/9/2018 2:35:13 PM Uri: offers/h-5E/

Collections
id : Foo indexingPolicy : @{indexingMode=consistent; automatic=True; includedPaths=System.Object[]; excludedPaths=System.Object[]} _conflicts : conflicts/ _docs : docs/ _etag : "00000700-0000-0000-0000-5acd17070000" _rid : Q4QdAMy0QwE= _sprocs : sprocs/ _triggers : triggers/ _ts : 1523390215 _udfs : udfs/

Any thoughts on how i could match these two up?

Received the following response when trying to create this kind of index:

"The specified index type value 'Spacial' is invalid. Please choose among 'Hash', 'Range', and 'Spatial' index types."

The link to the Style Guidelines (STYLEGUIDELINES.md) is broken.

The CodeCov artifact is failing:
https://ci.appveyor.com/project/PlagueHO/cosmosdb#L1076

Error 429 is returned by Cosmos DB when a collection has exceeded it's assigned RU's. The typical way of managing this is to back off a short amount of time and repeat the request. This should be configurable within the context object or via some global parameter.