peburrows / diplomat Goto Github PK

Is there a better way to find something by a name/id? This below works, but seems like the named parameter way should work too.

def find(email) do
    # Works fine, but I want to use the key
    # Diplomat.Query.new("select * from `User` where email_address = @email", %{email: email} )

    "select * from User where __key__ = KEY(User, \"#{email}\")"
    |> Diplomat.Query.new
    |> Diplomat.Query.execute
    |> hd
  end

When I use the names params:

 def find2(email) do
    Diplomat.Query.new("select * from User where __key__ = KEY(User, @email)", %{email: email})
    |> Diplomat.Query.execute
  end

I get:

iex(1)> u = User.find2("[email protected]")
{:error,
 %Diplomat.Proto.Status{
   code: 3,
   details: [],
   message: "Encountered \"@email\" at line 1, column 46.\nWas expecting one of:\n    <INTEGER> ...\n    <SINGLE_QUOTE_STRING> ...\n    <DOUBLE_QUOTE_STRING> ...\n    "
 }}
iex(2)>

Semi-new to Google Data Store so I could be missing something simple.

Thanks!

The documentation for Datastore states the following about transactions:

When two or more transactions simultaneously attempt to modify entities in one or more common entity groups, only the first transaction to commit its changes can succeed; all the others will fail on commit. Because of this design, using entity groups limits the number of concurrent writes you can do on any entity in the groups. When a transaction starts, Cloud Datastore uses optimistic concurrency control by checking the last update time for the entity groups used in the transaction. Upon commiting a transaction for the entity groups, Cloud Datastore again checks the last update time for the entity groups used in the transaction. If it has changed since our initial check, an error is returned.

Essentially, two or more transactions may not concurrently modify any given entity (or any set of entities in the same entity group). If more than one transaction attempts to make concurrent modifications, only the first to commit will succeed and any others will receive an error. I have verified this behavior using a Ruby script and the google-cloud-datastore Ruby gem.

When using Elixir and Diplomat, however, it seems as if concurrent modifications by multiple transactions do not exhibit this behavior. Instead, all transactions succeed on commit and data written by one transaction will be overwritten by data from the next transaction. It should be the case that only the first transaction should successfully commit, and subsequent transactions should fail with an error.

This seemingly incorrect behavior can be verified by having multiple processes updating the same entity within a transaction. Each update in each process would:

1. Start a new transaction with Diplomat.Transaction.begin/0.
2. Read the current value of the entity from Datastore using a simple key lookup.
3. Sleep for a configurable period of time. (See below for explanation.)
4. Update some attribute of the entity.
5. Save the updated entity with Diplomat.Transaction.update/2 and commit the change with Diplomat.Transaction.commit/1.

The reason for the sleep in step 3 is so that if we start multiple processes with sufficiently different sleep times, this ensures that there will be an update collision, since the slower process will start a transaction and the faster process will concurrently start its own transaction and commit it before the slower process can complete.

If transactions exhibited the documented behavior, then this scheme should result in an error when the slower process tries to commit a transaction for which the faster process has already successfully committed changes to the same entity. However, when testing this scenario, I found that all transactions always succeed and that the slower process simply overwrites the data written by the faster process. I haven't yet discovered why this happens with Elixir and Diplomat, but does not happen with Ruby, but I suspect it might be due to the fact that that Diplomat is still using the v1beta3 protobuf schema, whereas Ruby is using v1.

Below is an Elixir script file that implements the scenario outlined above, for reference.

alias Diplomat.{Key, Entity, Transaction}

defmodule DatastoreWriter do
  require Logger

  def start_link(uuid, delay) do
    state = %{
      uuid: uuid,
      delay: delay,
      key: Key.new("Rule", uuid, Key.new("Service", "transactional",  Key.new("Company", "transactional"))),
      rule: %{"id" => uuid, "version" => 0}
    }
    Agent.start_link(fn -> state end)
  end

  def insert_initial(agent) do
    state = get_state(agent)
    result = Transaction.begin fn(tx) ->
      Transaction.insert(tx, Entity.new(state.rule, state.key))
    end
    # Logger.info("insert_initial -> #{inspect result}")
  end

  def write_versions(agent, versions) when is_list(versions) do
    Task.async fn ->
      for version <- versions, do: write_version(agent, version)
    end
  end

  def write_version(agent, version) do
    state = get_state(agent)
    tx = Transaction.begin()
    new_rule =
      state.key
      |> lookup_key()
      |> Entity.properties()
      |> Map.update!("version", fn(_) -> version end)
    Process.sleep(state.delay)
    result = try do
      tx
      |> Transaction.update(Entity.new(new_rule, state.key))
      |> Transaction.commit()
    rescue
      e ->
        Logger.error("!!!!! Transaction error: #{inspect e}")
        Transaction.rollback(tx)
        {:error, e}
    end
    # Logger.info("update_version(#{version}) -> #{inspect result}")
    result
  end

  defp get_state(agent) do
    Agent.get(agent, &(&1))
  end

  def lookup(agent) do
    lookup_key(get_state(agent).key)
  end

  def lookup_key(key) do
    [entity] = Key.get(key)
    entity
  end
end

uuid = UUID.uuid1()
{:ok, slow_writer} = DatastoreWriter.start_link(uuid, 500)
{:ok, fast_writer} = DatastoreWriter.start_link(uuid, 100)
DatastoreWriter.insert_initial(fast_writer)

slow_writer_task = DatastoreWriter.write_versions(slow_writer, Enum.map(1..10, &("slow.#{&1}")))
fast_writer_task = DatastoreWriter.write_versions(fast_writer, Enum.map(1..10, &("fast.#{&1}")))

slow_writer_results = Task.await(slow_writer_task, 60_000)
fast_writer_results = Task.await(fast_writer_task, 60_000)

IO.puts("***** fast_writer_results:")
IO.inspect(fast_writer_results)
IO.puts("***** slow_writer_results:")
IO.inspect(slow_writer_results)

IO.puts("Final value written:")
IO.inspect(DatastoreWriter.lookup(fast_writer))

I created a key with a namespace, and then used that key to create an entity, but the entity was insert into the [default] namespace in Datastore.

Is there support for configuring namespaces? It would be nice to easily specify different namespaces for dev/stage/prod.

https://cloud.google.com/datastore/docs/concepts/multitenancy

I'm guessing that I'm totally missing something.

One should be able to use Diplomat as an Ecto adapter for Datastore. Lots of work to do here, especially around query generation and all that, but once we get to v1.0, I want to look at tackling this.

Are you open to transferring this project over to my noizu-labs org. I rely on this for a few projects and have some upstream changes to merge in.

Hello 👋 ,

It seems that the RPC API exposes cursors for both EntityResult and QueryResultBatch but we seem to be discarding them here.

Ofc, that alone wouldn't be enough to add pagination support. Diplomat.Query would also need to support that functionality.

Are there any plans to support pagination when fetching a batch of Entities?

Thank you!

Attempting to push something like:

%{"my.key" => "value"}

Will result in the following error:

{:error, 
  %Diplomat.Proto.Status{code: 3, 
    details: [], 
    message: "property.name contains a path delimiter, \
              and the entity contains one or more indexed entity value."}}

This is expected, but there ought to be a way to have Diplomat transparently account for this and prevent the error.

According to Hex, Diplomat.Entity.insert/1's spec is:

insert([t()] | t()) :: {:ok, Diplomat.Key.t()} | Diplomat.Client.error()

However, it seems that it returns [Diplomat.Key.t()] when success.
I think it should return {:ok, Diplomat.Key.t()}.

I recently started to get a mix deps.compile error without changing any major package version.

==> diplomat
Compiling 10 files (.ex)

== Compilation error in file lib/diplomat.ex ==
** (Protobuf.Parser.ParserError) [missing_occurrence, missing_occurrence, missing_occurrence, missing_occurrence]
    lib/exprotobuf/parser.ex:101: Protobuf.Parser.parse!/3
    (elixir) lib/enum.ex:2986: Enum.flat_map_list/2
    lib/exprotobuf/parser.ex:8: Protobuf.Parser.parse_files!/2
    lib/exprotobuf.ex:194: Protobuf.parse/2
    expanding macro: Protobuf.__using__/1
    lib/diplomat.ex:10: Diplomat.Proto (module)
    (elixir) expanding macro: Kernel.use/2
    lib/diplomat.ex:10: Diplomat.Proto (module)

could not compile dependency :diplomat, "mix compile" failed. You can recompile this dependency with "mix deps.compile diplomat", update it with "mix deps.update diplomat" or clean it with "mix deps.clean diplomat"

Version details:
Elixir : 1.8
Erlang: Erlang/OTP 20.1
Diplomat: 0.12.1

Any help will be appreciated.
Thanks

Hello, is Diplomat mature enough for daily use (or is production ready)?

I'm building a Google Cloud APIs collection https://github.com/sashaafm/gcloudex. My next intention is to implement the Google Datastore API. Can this library be used for that purpose as dependency?

Also if you're interested in contributing to the project you're welcome :)

Errors come back from the API and are not parsed in any way, so all the Diplomat user gets is something like this:

{:error,
 <<8, 3, 18, 52, 99, 97, 110, 110, 111, 116, 32, 119, 114, 105, 116, 101, 32,
   109, 111, 114, 101, 32, 116, 104, 97, 110, 32, 53, 48, 48, 32, 101, 110, 116,
   105, 116, 105, 101, 115, 32, 105, 110, 32, 97, 32, 115, 105, 110, 103, ...>>}

Would be very nice if this was something more useful 😄

Hi,

My system dependencies are:
Erlang/OTP 19 [erts-8.3] [64-bit] [smp:4:4] [async-threads:10]
Elixir 1.4.4

Part of my project dependencies:
diplomat ~> 0.7
goth ~> 0.4

I unsuccessfully tried to update 'hackney' dependency.

Stacktrace

 ** (UndefinedFunctionError) function :unicode_util.lowercase/1 is undefined (module :unicode_util is not available)
        :unicode_util.lowercase('www.googleapis.com')
        (idna) c:/projects/SpeechRecognitionServer/deps/idna/src/idna.erl:57: :idna.lowercase_list/1
        (idna) c:/projects/SpeechRecognitionServer/deps/idna/src/idna.erl:10: :idna.to_ascii/1
        (hackney) c:/projects/SpeechRecognitionServer/deps/hackney/src/hackney_url.erl:96: :hackney_url.normalize/2
        (hackney) c:/projects/SpeechRecognitionServer/deps/hackney/src/hackney.erl:291: :hackney.request/5
        (httpoison) lib/httpoison/base.ex:432: HTTPoison.Base.request/9
        (goth) lib/goth/client.ex:53: Goth.Client.get_access_token/2
        (goth) lib/goth/token.ex:94: Goth.Token.retrieve_and_store!/1
        lib/diplomat/client.ex:138: Diplomat.Client.auth_header/0
        lib/diplomat/client.ex:109: Diplomat.Client.call/2
        lib/diplomat/client.ex:43: Diplomat.Client.commit/1
        lib/diplomat/entity.ex:138: Diplomat.Entity.insert/1
        (web) web/controllers/page_controller.ex:12: Web.PageController.myTest/2

As the title says.

1. is this already supported by diplomat and I am just to dumb to find it?
2. otherwise would it be possible to implement it without a big re-write of the whole library? if yes then I can provide a PR

Once River is ready, we'll use that as the basis for our gRPC support.

After deploying a hot update with distillery connection begins to fail with an :undefined response.
Hot update using distillery version 1.5.2

GoldenRatio.Appengine.UserEntity.entity!("r6EhunDmTsbYuJqEcN7bNq0Yedg2")
** (MatchError) no match of right hand side value: :undefined
(hackney) /mnt/data/code/lax-alerts/src/ingressor/src/deps/hackney/src/hackney_connect.erl:69: :hackney_connect.
create_connection/5
(hackney) /mnt/data/code/lax-alerts/src/ingressor/src/deps/hackney/src/hackney_connect.erl:37: :hackney_connect.
connect/5
(hackney) /mnt/data/code/lax-alerts/src/ingressor/src/deps/hackney/src/hackney.erl:315: :hackney.request/5
(httpoison) lib/httpoison/base.ex:439: HTTPoison.Base.request/9
(goth) lib/goth/client.ex:49: Goth.Client.get_access_token/2
(goth) lib/goth/token.ex:94: Goth.Token.retrieve_and_store!/1
(diplomat) lib/diplomat/client.ex:141: Diplomat.Client.auth_header/0
(diplomat) lib/diplomat/client.ex:109: Diplomat.Client.call/2