Rate-limiting about shopify

[Ninigi]

Afaik not even the ruby gem (maintained by shopify) has this feature, would be really cool to have this!

I guess it would be a bigger feature to implement, if you want to take a first shot at it, I would be happy to help 👍

[humancopy]

I've actually implemented this in a little script I did for transferring data from a different system, following the Ruby example from https://help.shopify.com/api/getting-started/api-call-limit, but I'm wondering how would you go about doing this in the package?

I kept the last call time with an Agent, passing it around the module.

Something like this:

defmodule LimitedShopify do
  @cycle 0.5

  def run do
    {:ok, agent} = Agent.start_link fn -> DateTime.to_unix(DateTime.utc_now(), :microsecond) end
    agent |> get_product("handle_for_product")
  end

  defp get_product(agent, handle) do
    {:ok, product_response} = agent |> shopify_call(fn -> Shopify.session |> Shopify.Product.all(%{handle: handle}) end)
  end

  # Code inherited from https://help.shopify.com/api/getting-started/api-call-limit
  defp shopify_call(agent, function) do
    start_time = Agent.get(agent, fn time -> time end)
    stop_time  = DateTime.to_unix(DateTime.utc_now(), :microsecond)
    IO.puts "Last batch processing started at #{start_time |> DateTime.from_unix!(:microsecond) |> DateTime.to_string()}"
    IO.puts "The time is now #{stop_time |> DateTime.from_unix!(:microsecond) |> DateTime.to_string()}"
    processing_duration = (stop_time - start_time)/1000000
    IO.puts "The processing lasted #{Float.ceil(processing_duration, 2)} seconds."
    wait_time = Float.ceil(@cycle - processing_duration, 2)
    IO.puts "We have to wait #{wait_time} - #{@cycle - processing_duration} - #{Float.ceil(wait_time, 4)} seconds then we will resume."
    if wait_time > 0, do: :timer.sleep(round(Float.ceil(wait_time, 4)*1000))
    Agent.update(agent, fn _ -> DateTime.to_unix(DateTime.utc_now(), :microsecond) end)

    function.()
  end
end

It's not the prettiest solution but it works really good... 😃

I'd be happy helping out implementing this feature in the package, let me know how you thought to tackle this one ...

[Ninigi]

@humancopy I haven't really put any thought into this, since my approach always was to run it in a supervised process and just let it crash if the request limit was reached... Following the good ol' "let it crash" philosophy, that might even be a solution we can think about for the package.

For now, I think completing the resources should be priority, but I will come back to this once that milestone is reached.

[nsweeting]

At the moment, I would categorize this as perhaps outside the scope of this project. But we can perhaps circle back to it at another time.

Typically I have used the "constant pressure" approach - pattern matching on the results, and calling the function again if required. A simple outline:

def get_products(session) do
  session |> Shopify.Product.all() |> handle_result(session)
end

def handle_result({:error, %{code: 429}}, session) do
  :timer.sleep(500)
  get_products(session)
end

def handle_result(result, _session) do
  result
end

Closing for now.

[humancopy]

I was also considering this approach, but the main two reasons I decided to go for rate limiting in the code were wasting bandwidth and the possibility of Shopify changing their policy in the future and "punishing" for multiple rate limit violations.

I've improved a bit the code to use the name attribute with the agent so no need to pass it along. Also using milliseconds instead of microseconds so the math is much simpler now and it's in :timer.sleep format. 😄

I think we could find a way to integrate this in the package somehow as an optional interface, just not sure yet exactly how to go about it. I'll make a pull request once I have an idea 😉

defmodule LimitedShopify do
  @cycle 500

  def run do
    Agent.start_link(fn -> DateTime.to_unix(DateTime.utc_now(), :millisecond) end, name: __MODULE__)
    get_product("handle_for_product")
  end

  defp get_product(handle) do
    {:ok, product_response} = shopify_call(fn -> Shopify.session |> Shopify.Product.all(%{handle: handle}) end)
  end

  # Code inherited from https://help.shopify.com/api/getting-started/api-call-limit
  defp shopify_call(function) do
    start_time = Agent.get(__MODULE__, fn time -> time end)
    stop_time  = DateTime.to_unix(DateTime.utc_now(), :millisecond)
    Logger.debug "Last batch processing started at #{start_time |> DateTime.from_unix!(:millisecond) |> DateTime.to_string()}"
    Logger.debug "The time is now #{stop_time |> DateTime.from_unix!(:millisecond) |> DateTime.to_string()}"
    processing_duration = stop_time - start_time
    Logger.debug "The processing lasted #{processing_duration} milliseconds."
    wait_time = @cycle - processing_duration
    Logger.debug "We have to wait #{wait_time} - #{@cycle - processing_duration} - #{wait_time} milliseconds then we will resume."
    if wait_time > 0, do: :timer.sleep(wait_time)
    Agent.update(__MODULE__, fn _ -> DateTime.to_unix(DateTime.utc_now(), :millisecond) end)

    function.()
  end
end