james-jw / xq-promise Goto Github PK

You convinced me in indicating that the fork-join code is indeed not dependent on the promise pattern, so I’m currently wondering if we should define an Async Module (alternative names are welcome), which includes your promise:fork-join function, and possibly some more functions based on the ideas that have been developed by @apb2006 and discussed in BaseXdb/basex#1211. Do you think that would make sense? We could e.g. define this module as W3C EXPath Module:

(: Namespace: async = 'http://expath.org/spec/async' :)
async:fork-join($functions as function(*))) as item()*,
async:fork-join($functions as function(*)), $options as map(*)) as item()*

Keys allowed for $options:

• threads (?): number of threads to allow in the pool (default, maybe implementation-defined: number of CPU cores available on the current system)
• thread-size (?): number of functions to be evaluated by each thread (default: 1)

If an XML transformation is within one of the callbacks, (transform clause) query results may be inconsistent depending on the query.

Currently when only accepts functions or deferred objects but regular non function values should be supported as well and be passed along unchanged to the when's callbacks.

For example:

let $doc := doc('')
let $fork := p:fork($do-something)
let $promise := p:when($doc, $fork) => p:then($do-something-else)
return 
   $promise()

jQuery and Q (Promises/A+) have a few differences in functionality and terminology, as covered here: Coming from jQuery: Q Promises

Q has a static when() function that has different semantics: Q.when

During a failure, if the fail callback mitigates the issue, the following success callback chain will be initiated twice.

Am I doing something wrong? Or should this indeed work?

let $worker := function($fname, $lname) { 'Hello, ' || $fname || ' ' || $lname }
let $promise := promise:defer($worker, ('James', 'Wright'), map {
  'then': function($p) { 'then: ' || $p },
  'fail': function ($result as item()*) {
            trace($result, 'Request failed!') => prof:void()
          }
})
return $promise(())

Result:

Evaluating:
Request failed!James
Request failed!Wright

Expected:

then: Hello, James Wright

Same problem with a query found in README:

let $req := <http:request method="GET"/>
let $uri := 'http://www.google.com'
let $worker := http:send-request(?, ?)
let $extractListItems := function ($res as map(*)) { $res?list?* }
let $error := function ($result as item()*) {
     trace($result, 'Request failed!') => prof:void()
}
let $retrieve := promise:defer($worker, ($req, $uri), map { 
           'then': parse-json(?), 
           'fail': $error 
}) 
let $extract := $retrieve(map { 'then': $extractListItems  })
return
   $extract(())

[edited from 1st version]
Hi James,
This is not working the way I am looking for. Am I doing it wrong?

import module namespace promise = 'org.jw.basex.async.xq-promise';

let $query := 'prof:sleep(10000),admin:write-log("test1")'
let $promise := promise:fork(xquery:eval(?), $query)
return ( 2,$promise())

This does not return until the $query has completed. I want to kick off the query and forget about it.

Hi James,
I am trying this with BaseX 8.4 beta 165f5fd

import module namespace promise = 'org.jw.basex.async.xq-promise';
let $work := http:send-request(<http:request method="GET" />, ?)
let $extract-doc := function ($res) { $res[2] }
let $extract-links := function ($res) { $res//a[@href => matches('^http')] }
let $promises :=
  for $uri in ((1 to 5) !  ('http://www.google.com', 'http://www.yahoo.com', 'http://www.amazon.com', 'http://cnn.com', 'http://www.msnbc.com'))
  let $defer := promise:defer($work, $uri, map {
       'then': ($extract-doc),
       'done': trace(?, 'Results found: ')})
  return 
     promise:attach($defer, map {'then': $extract-links })
return 
promise:fork-join($promises,1)

I get the error:

[XPTY0004] XqPromise:forkJoin(item()+, xs:integer): org.basex.query.QueryContext.<init>(Lorg/basex/query/QueryContext;Z)V.

Should this work?

Looking at the java component of this impl I was wondering what the barriers were to a pure xquery solution ?

ps- enjoyed your talk at XML Prague

If an array is used for passing on arguments, no flattening would take place. See https://www.w3.org/TR/xpath-functions-31/#func-apply as example for an existing XQuery function…

Hi @james-jw,

Very cool stuff. I finally found some time to look at this properly in detail and it is a very interesting piece of work.

Having read your paper from XML Prague and tried to understand Promise in the ECMA spec, and Deferred in the jQuery documentation, I am a little confused. I am wondering if the terminology you use in xq-promise is meant to be directly comparable to the ECMA or jQuery? For example do you consider a "Promise" in xq-promise to have the same semantics as a "Promise" in ECMA?

From what I can understand the major difference is the execution model. It is clear to me that in xq-promise a Promise is never executed until it is passed to fork-join or fork. In both EMCA and jQuery it seems that the function passed to either Promise or Deferred is immediately evaluated with the arguments necessary it to complete the promise, although that completion may happen later if the function itself calls other asyncronous functions (e.g. setTimeout).

Also it seems that in xq-promise a promise is completed by the return value of the function passed to promise:deferred being evaluated. Whereas in ECMA and jQuery, the inner function has to call the resolve or failed functions that it is passed as arguments. This is perhaps a subtle difference, but perhaps an important one?

I guess I am confused by how you use the terms executed and execution. If I understand correctly with ECMA and jQuery a Promise may have been "completed" before you compose with it via then to it, but that is okay, the result is still the same.

Just wondering what your thoughts are on this?

When a fail callback is called. It should be provided a single argument which is map that contains all the error information.

I propose:

map {
  'err': {
     'description': Same as standard error bindings
     ...
   },
  'args': array [ ... ]
}

This way, the fail callback has the information needed to either report a useful error, including the bad arguments or continue the work somehow.

Long time coming (Sorry for delay)

I didn’t check this out so far by myself… But maybe it will get easier with your new XQuery-based solution to add support for real updates?

Hi James, I checked out your text in the XML Prague Conference Proceedings (Page 139 ff.), and I stumbled upon your interesting thoughts on how the FLWOR expressions could be extended to allow for an asynchronous execution. I wrote down some more examples, and I tried to formalize it:

Query 1:

for async $a in 1 to 2
return $a

Query 1 (rewritten):

(: return :) p:fork-join(
  for $a in 1 to 2
  return function() { (: return :) $a }
)

Query 2:

for       $a in 1 to 2
for async $b in 3 to 4
return $a * $b

Query 2 (rewritten):

for $a in 1 to 2
return p:fork-join(
  for $b in 3 to 4
  return function() { (: return :) $a * $b }
)

Query 3:

for async $a in 1 to 2
for       $b in 3 to 4
return $a * $b

Query 3 (rewritten):

(: return :) p:fork-join(
  for $a in 1 to 2
  return function() {
    for $b in 3 to 4
    return $a * $b
  }
)

Query 4:

for async $a in 1 to 2
for async $b in 3 to 4
return $a * $b

Query 4 (rewritten):

(: return :) p:fork-join(
  for $a in 1 to 2
  return function() {
    p:fork-join(
      for $b in 3 to 4
      return function() { (: return :) $a * $b }
    )
  }
)

Query 5:

for       $a in 1 to 2
for async $b in 3 to 4
order by $b descending
return ($a * $b)

Query 5 (rewritten):

…does not work out, as the rewritten version would have a nested FLWOR expression, which will lead to different results. Similar problems arise with count, group by and window clause. A pragmatic solution would be to limit the usage of async to cases in which all following clauses are for, let and return.

As a result, we might end up with the following rewriting rules:

• All for clauses in a FLWOR expression are checked for the async keyword; the most inner clause will be checked first.
• If a for clause has the async keyword, it will be replaced with a return clause, and the expression of this clause will be a p:fork-join function call.
• The argument of p:fork-join will be a new FLWOR expression, which consists of:
  • the original, replaced for clause (excluding the async keyword), and
  • a return clause with a function item as expression, which contains the remaining clauses of the original FLWOR expression as function body.
• If any of the FLWOR expressions (the original one, or the newly created ones) only consists of a return clause, it must be simplified and replaced with its expression (e.g.: return p:fork-join(...) → p:fork-join(...); return function() { $a } → function() { $a }).

As I’ve surely missed something, anyone’s input is welcome!