azure / azure-sdk-for-node Goto Github PK

To align with other services.

Would it be alright if we enabled the Wiki so that the community can add content?

MSDN is a great place for official tutorials and blogs -- but I'm thinking that the community surrounding Windows Azure NodeJS could add their tutorials/demos/examples to this wiki...

Just a thought!

Cheers,
Jaime

Wasn't able to lock down a solid repro for this, but it looks like you can get in a state with the queue service where base64 encoding does not work correctly. Here is a stack trace:

if (encStr.charCodeAt(i - 2) == 61) {
^
TypeError: Object # has no method 'charCodeAt'
at Object.decode64 (C:\node-deploy\areader\WorkerRole1\node_modules\azure\li
b\util\base64.js:37:14)
at Function.parse (C:\node-deploy\areader\WorkerRole1\node_modules\azure\lib
\services\queue\models\queuemessageresult.js:70:47)
at C:\node-deploy\areader\WorkerRole1\node_modules\azure\lib\services\queue
queueservice.js:526:55
at Array.forEach (native)
at C:\node-deploy\areader\WorkerRole1\node_modules\azure\lib\services\queue
queueservice.js:525:18
at IncomingMessage. (C:\node-deploy\areader\WorkerRole1\node_modu
les\azure\lib\services\serviceclient.js:230:13)
at IncomingMessage.emit (events.js:88:20)
at HTTPParser.onMessageComplete (http.js:137:23)
at Socket.ondata (http.js:1124:24)
at TCP.onread (net.js:348:27)

They take too much space.

When you query an Azure table in version 0.5.2, the returned RowKeys are sometimes truncated. This isn't noticeable if the RowKeys are short, but it's a problem if one uses a GUID. I tested this against the storage emulator.

It's possible that other properties are truncated, but I've only noticed it with RowKeys.

Hey all,

@mikeal wrote an interesting post about dependencies on modules within Node projects: http://www.mikealrogers.com/posts/nodemodules-in-git.html

Do we want to have node_modules included in the repository to move forward with developers practising having local modules?

I'm +1 for node_modules being in the root if that's the pattern that the Node community is starting to adopt.

Cheers,
Jaime

When round-tripping certain values to table storage and back the value is mangled. In some cases new line characters appear to be stripped on the way back. When querying the row from .NET I can see the original line breaks. So it appears there is no corruption, but rather just mangling somewhere in the node sdk. It doesn't happen for all values.

This popped up when storing a JavaScript function body in an azure table. Somehow this value (and any other similar snippet) is affected.

IN:

for ( var i in [1,2,3] ) { 
 console.log( i );
}

OUT:

for ( var i in [1,2,3] ) { console.log( i );
}

When using unicode in table field values, an "Invalid Input" error is being thrown.

Currently we provide IsInBatch(), which returns whether or not you are in a batching context. However, if you try to call commit batch and haven't added any operations, we throw an exception.

The scenario I'm trying to implement is one where I am selectively updating my table entries (as I iterate through determining if any of them need updating - if they do, I add them to the batch). In the case that there are none that need updates, I will end up trying to commit an empty batch.

Currently, if you call

azure.ISO8061Date.format(new Date());

We will return something like this:

2011-12-17T21:31:46.0000773Z

If you then try to convert this back into a date, the milliseconds will be set to zero. The format that works is the following:

2011-12-17T21:31:46.773Z

The following program doesn't work when deployed to staging or production. node.exe appears to crash due to unhandled exception leaving only this behind in server.js.logs:

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: getaddrinfo ENOENT
    at errnoException (dns.js:31:11)
    at Object.onanswer [as oncomplete] (dns.js:140:16)

server.js

var http = require('http');
var port = process.env.port || 1337;

var KEY1 = "xxx";
var KEY2 = "zzz";

var util = require('util');
var azure = require('azure');
var tables = azure.createTableService(KEY1, KEY2);

http.createServer(function (req, res) {
    tables.createTableIfNotExists("Table", function (error) {
        res.writeHead(200, { 'Content-Type': 'text/plain' });
        if (error) {
            res.end('ERROR: ' + util.inspect(error,true));
        } else {
            res.end('EXISTS!');
        }
    });
}).listen(port);

Web.cloud.config (default, except enabled logging)

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <modules runAllManagedModulesForAllRequests="false" />

    <!-- NOTE: By default, debugging and logging are unsecure and should not be enabled for production applications in the cloud.-->
    <iisnode 
      debuggingEnabled="false"
      loggingEnabled="true"
      devErrorsEnabled="false"
    />

    <!-- indicates that the server.js file is a node.js application 
    to be handled by the iisnode module -->
    <handlers>
      <add name="iisnode" path="server.js" verb="*" modules="iisnode" />
    </handlers>
    <rewrite>
      <rules>
        <clear />
        <rule name="app" enabled="true" patternSyntax="ECMAScript" stopProcessing="true">
            <match url="server\.js.+" negate="true" />
            <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
            <action type="Rewrite" url="server.js" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>

Projection is not supported by table emulator and any attempt to use projection from SDK leads to an exception. Unfortunately, this prevents me from writing the code I want to write. I am left to use Array.map() to get what I want, but this isn't the code I would have liked to use. And adding branching all over the place isn't too appealing. Projection is obviously a better alternative. The situation is made more awkward by node SDK injecting some properties into entities beyond my control (id, etag, updated).

Why not emulate projection in the SDK code by stripping properties client-side as specified by select()? I think that would present developers with a more coherent toolset.

Almost all applications on the web require some level of cryptographic security even if just for hashing passwords and other sensitive data. There should be a uniform cross platform API for accessing the native and most appropriate (openssl or windows Cryptographic service) library to allow developers to leverage the same code regardless of deployment target. For example, I use the node-BCrypt library (https://github.com/ncb000gt/node.bcrypt.js) for hashing passwords, it obviously doesn't compile on the azure platform because openssl libraries aren't present. I am not even sure if the WCS provides for BCrypt, and if it doesn't it would definitely be something to get plumbed in for a variety of reasons, not the least of which is the reduced acceptance of SHA/MD5 hashes as "secure enough". In terms of API, Nick Campbell (ncb000gt) has a pretty good start at the JS level one that fits my needs and is abstract enough that it should work for any implementation of BCrypt library, I would start there and modify as appropriate and needed.

A common pattern is for one to copy our samples and just deploy them. Currently you have to modify code as the Azure module is relatively referenced in the require.

A fix would be to add logic that tries to resolve relative, if not found it does a simple require.

Currently we process the responses based on the batch operations. However, after commiting the batch, another batch can be started and the operations replaced.

Currently blog uploader sample says "Upload MSI". Nothing about the app is specific to MSIs. It should say "File"

npm install azure

doesn't work on ubuntu

jdavis@LittleDog:~/web$ node tablestore.js

node.js:201
       throw e; // process.nextTick error, or 'error' event on first tick
             ^
Error: Cannot find module './services/table/tableservice'
   at Function._resolveFilename (module.js:334:11)
   at Function._load (module.js:279:25)
   at Module.require (module.js:357:17)
   at require (module.js:368:17)
   at Object. (/home/jdavis/node_modules/azure/lib/azure.js:22:20)
   at Module._compile (module.js:432:26)
   at Object..js (module.js:450:10)
   at Module.load (module.js:351:31)
   at Function._load (module.js:310:12)
   at Module.require (module.js:357:17)
jdavis@LittleDog:~/web$

Is there are recommended SQL Server Driver for SQL Azure? See a few out there, none of them look maintained.

This will cause cache to be refreshing more than needed.

#86

Dev estimate: 0
Test Estimate: 3

The repro code runs a call to getMessages from a queue service in a tight loop. Eventually you will get to an error where the socket has hung up which looks like this:

events.js:48
throw arguments[1]; // Unhandled 'error' event
^
Error: socket hang up
at createHangUpError (http.js:1104:15)
at Socket.onend (http.js:1167:27)
at TCP.onread (net.js:369:26)

At which point your role needs to recover from the crash. Is this some kind of throttling from Azure? Can we possibly handle this better?

Repro code:

var azure = require('azure')

var quickQueueService = azure.createQueueService();

//MAIN CODE FLOW
var eventLoop = function () {
quickQueueService.getMessages('foo', function(err, message) {
if(!err) {
console.log(new Date() + ' Success');
} else {
console.log(new Date() + 'Failure');
}
});
}
try {
setInterval(eventLoop, 5);
} catch (e) {
console.log('Error');
}

I tried to run the library on Heroku without success since it only supports node v0.4.7 and the azure npm package engine is set to >= 0.6.4

I forked this repo and published an npm package 'azure-0.4.7' to perform a test on Heroku and it seems to be working fine.

is there any reason to keep the v0.6.4 as the minimun required version? can you change it to 0.4.7?

thanks
.jpg

When a user specifies cerdentials in the TableService constructor, but does not specify whether or not path style URIs should be used, the constructor should make the choice based on the credential provided, not based on the EMULATED variable.

Should still use the EMULATED variable when no credentials are provided

Currently we provide IsInBatch(), which returns whether or not you are in a batching context. However, if you try to call commit batch and haven't added any operations, we throw an exception.

The scenario I'm trying to implement is one where I am selectively updating my table entries (as I iterate through determining if any of them need updating - if they do, I add them to the batch). In the case that there are none that need updates, I will end up trying to commit an empty batch.

The README.md file is HTML, not Markdown. We should do this right and 'convert' it to Markdown.

An important side benefit is that we will then be able to use GFM (GitHub Flavored Markdown)'s excellent source code syntax highlighting - today our Node JavaScript doesn't have this and we'll get it for free with backticking.

When running the tests, I get this error:

Actual Results

✖ testInsertOrMerge

TypeError: Cannot call method 'queryTables' of undefined
    at Object.<anonymous> (C:\Projects\Github\azure-sdk-for-node\test\services\t
able\tableservice-tests.js:57:18)
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\lib\core.js:2
57:35
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\deps\async.js
:453:21
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\deps\async.js
:158:13
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\deps\async.js
:118:13
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\deps\async.js
:134:9
    at C:\Projects\Github\azure-sdk-for-node\node_modules\nodeunit\deps\async.js
:157:9
    at Object.mapSeries (C:\Projects\Github\azure-sdk-for-node\node_modules\node
unit\deps\async.js:147:23)
    at Object.series (C:\Projects\Github\azure-sdk-for-node\node_modules\nodeuni
t\deps\async.js:451:19)
    at Object.<anonymous> (C:\Projects\Github\azure-sdk-for-node\node_modules\no
deunit\lib\core.js:261:22)

✔ testNotEmulatedExplicitCredentials
✔ testEmulatedExplicitCredentials
✔ testEmulatedWithoutParameters
✔ testNotEmulatedWithoutParameters

FAILURES: 83/149 assertions failed (456ms)

I'm currently using the latest on master.

Expected Result

• tests should pass...
• queryTables should be a method that exists in the tableService object

Steps to reproduce

• git clone git://github.com/WindowsAzure/azure-sdk-for-node.git
• cd azure-sdk-for-node
• npm install .
• node .\test\runtests.js

Check environment variables or config.json for values

This causes the test to fail when runnign NPM test

I'm using TableStorage for storing server access logs in my Azure CoffeeScript tutorial. I also have a worker process that calculates access counts per URL.

To make this idempotent, I'm using a modified URL as the RowKey, and tried to use insertOrReplaceEntity. That always failed so I had to rewrite my code as:

countRequests url, (total) ->
  key = "url-#{url.replace('/', '-')}"
  totalEntry =
    total: total
    RowKey: key
    PartitionKey: 'totals1'
  table.insertEntity 'requestTotal', totalEntry, (error) ->
    return unless error
    table.updateEntity 'requestTotal', totalEntry, (error) ->
      console.log "Failed to update request count for #{url} to #{total}", error if error

A table storage insert with a true boolean value, such as:

{ one: 'uno',
  two: { '@': { type: 'Edm.Double' }, '#': 2 },
  three: { '@': { type: 'Edm.Boolean' }, '#': true },
  four: 
   { '@': { type: 'Edm.DateTime' },
     '#': '2010-12-23T23:12:11.234Z' },
  PartitionKey: 'foo',
  RowKey: 'bar' }

works correctly. However, the same insert with a false value:

{ one: 'uno',
  two: { '@': { type: 'Edm.Double' }, '#': 2 },
  three: { '@': { type: 'Edm.Boolean' }, '#': false },
  four: 
   { '@': { type: 'Edm.DateTime' },
     '#': '2010-12-23T23:12:11.234Z' },
  PartitionKey: 'foo',
  RowKey: 'bar' }

generates an error ("One of the request inputs is not valid"). When I dump the XML of this failing request, it shows an extra set of <#> tags around the false value:

<m:properties>
  <d:one>uno</d:one>
  <d:two m:type="Edm.Double">2</d:two>
  <d:three m:type="Edm.Boolean"><#>false</#></d:three>
  <d:four m:type="Edm.DateTime">2010-12-23T23:12:11.234Z</d:four>
  <d:PartitionKey>foo</d:PartitionKey>
  <d:RowKey>bar</d:RowKey>
</m:properties>

A work around is to enclose the false value in quotes:

 { one: 'uno',
  two: { '@': { type: 'Edm.Double' }, '#': 2 },
  three: { '@': { type: 'Edm.Boolean' }, '#': 'false' },
  four: 
   { '@': { type: 'Edm.DateTime' },
     '#': '2010-12-23T23:12:11.234Z' },
  PartitionKey: 'foo',
  RowKey: 'bar' }

which causes the resulting XML to be generated correctly.

Current implementation basically emulates the GetEntity functionality. It would be highly beneficial to allow whereKeys to be passed with a partitionKey and a null rowKey, so that queries can be further optimized to return all records in a partition.

NextRowKey value can be null for queryentities queries.

When I create a QueueService with explicit credentials and try to invoke listQueues:

var azure = require('azure');

var queueService = azure.createQueueService(testCredentials.account,testCredentials.key);

queueService.listQueues(function(err,queues) {
  if (err) {
    console.dir(err);
  } else {
    console.dir(queues);
  }
});

I get an authentication error with the following information:

Server used following string to sign: \'GET x-ms-date:Sun, 22 Jan 2012 19:38:21 GMT\nx-ms-version:2011-08-18\n/account/\ncomp:list\'.'

And this is the request being sent to Azure:

{ method: 'GET',
  path: 'http://account.queue.core.windows.net:80/?comp=list',
  host: 'account.queue.core.windows.net',
  port: 80,
  headers: 
   { 'Content-Type': '',
     'x-ms-version': '2011-08-18',
     'x-ms-date': 'Sun, 22 Jan 2012 19:38:21 GMT',
     Accept: 'application/atom+xml,application/xml',
     'Accept-Charset': 'UTF-8',
     Host: 'account.queue.core.windows.net:80',
     Authorization: 'SharedKey account:...=' 
   }
}

(where account is the name of my Azure storage account)

Should document AZURE_SERVICEBUS_NAMESPACE, AZURE_SERVICEBUS_ACCESS_KEY

Hi, there

I'm planning to use a virtual machine to host the azure emulator, while developing the app on a Mac.

Is it possible to make a connection to azure storage emulator? Any examples available?

Thanks.

The windows azure tables query does not support substring searches i.e. substring('nikhil', fullname) eq true or some similar format so one can perform a query equivalent to the SQL "like". I am basing the above off the oData spec. but it can take any form as long as the facility to perform substring searches is available.
There are often scenarios, especially around AJAX autocomplete where such a facility is required.
The minimum requirement for this query should be partitionKey. substring should be supported on each attribute in addition to rowKey.
Ideal implementation would be an extension of the query object and response would be an array.

The SharedKeyLite for blob storage may be missing the "comp" part of the querystrin:
http://msdn.microsoft.com/en-us/library/windowsazure/dd179428.aspx

test/azuretest.js - teardown restores storage client environment variables, but does not restore servicebus environment variables.

Today when using ServiceClient.performRequestInputStream (from cloudblobclient) it is impossible to relay the actual response headers into an incoming request stream.

To circumvent this we added the following line in beginning of the http.request callback :
writeStream.emit('start', response);

This allows us to copy the response headers into the incoming request response stream. Otherwise we are not able to relay the headers and status code.

It would be nice if you would support this directly with a dedicated method of the form:
ServiceClient.proxyRequest(requestParameters, rawData, incomingResponse, callback)
or by adding a dedicated prewrite event as the once we have added which allows the users to add this functionality themselves.

I am writing a function that encodes a URL. The resulting partition key is:

aHR0cDovL2ZlZWRzLmZlZWRidXJuZXIuY29tL2ppbXdhbmdzYmxvZw==

However, when I use our TableQuery object to create a query containing this string like so:

var tableQuery = azure.TableQuery.select()
.from(feedDataTable)
.where('PartitionKey eq '' + base64.encode64(feedUrl) + ''');

The resulting query that gets pushed to the server is

PartitionKey eq 'aHR0cDovL2ZlZWRzLmZlZWRidXJuZXIuY29tL2ppbXdhbmdzYmxvZweq'

Which will return no results as the trailing == have been replaced by eq as per the _replaceOperators function in TableQuery. We should consider updating this to be a little smarter about when it does the replacement.

The server.js.logs folder and its contents is clearly visible inside the package (which is just some nested zip files).

On top of that iisnode.exe running in Azure exposes the folder without any regard to the setting in the Web.cloud.config.

Put these two together and you will leak a snapshot of your emulator log when publishing which includes console.log output. Development logs can reveal all sorts of secrets.

For example in blobservice.js, there are 15 lines where errors are thrown. It's code that is being called with a nice callback structure and thus should the callback been invoked with the err as it's first parameter. For example in doing fs.stat in the same file this happens. So at the moment the module is mixing up errors and callbacks with errors as parameter. That's very unusable and can bring down a consuming service if not expected.

jsDocs are missing on the static methods in the azure.js file.

L680 return process.env[ServiceClient.EnvironmentVariables.EMULATED] ? true : false;

Should check the type. and not return true if EMULATED is simply set. Example, it could be set to "false"