Drab websocket frame size / performance about drab

[grych]

Thanks for reporting.
Actually, I can't see the case when you need to live modify @conn. It is added automatically to the assigns, and I will remove it from Drab's assigns.

[grych]

True, session and stuff is in the @conn.
Assigns are not encrypted in this version. I was planning to do it in some future release, on the way to stable version, but I think I will do it a little bit earlier :)

[grych]

Reopening the issue to remember to improve the performance of transmitting assigns.

Drab does not have to transmit every assign all the time. If we assume that assigns can be changed only with poke, and there is a cache in Drab's genserver, the assigns must be transmitted only on connect. And only changed assigns must be transmitted back.

[OvermindDL1]

This is a good idea, it will shrink it. Anyway, before, we need to identify which fields except the endpoint could be used in the helper functions.

I'd have it be configurable like you have assigns be now, but for the entire conn. Perhaps just a simple matcher on a map with true values or so, perhaps the default being:

conn_pass_through_or_whatever_name: %{
  private: %{
    phoenix_endpoint: true,
  },
}

And maybe some others, whatever is useful and small.

The function to do it could be something like:

  def deep_filter_map(%{__struct__: _} = struct, map_filter) do
    deep_filter_map(Map.from_struct(struct), map_filter)
  end
  def deep_filter_map(original, map_filter) do
    original
    |> Enum.flat_map(fn {key, value} = set ->
      case map_filter[key] do
        true -> [set]
        %{} = map_filter when is_map(value) ->
          value = deep_filter_map(value, map_filter)
          if map_size(value) === 0, do: [], else: [{key, value}]
        _ -> []
      end
    end)
    |> Enum.into(%{})
  end


  def deep_merge_map(base, to_merge) do
    to_merge
    |> Enum.reduce(base, fn
      ({key, %{} = value}, base) ->
        sub = base[key] || %{}
        sub = if is_map(sub), do: deep_merge_map(sub, value), else: sub
        Map.put(base, key, sub)
     ({key, value}, base) -> 
       Map.put(base, key, value)
    end)
  end

With those you could easily do a filter to filter out parts of an entire map, then on the other end you can merge it back into a default/fake conn that could be passed around. :-)

[OvermindDL1]

Actually, I can't see the case when you need to live modify @conn. It is added automatically to the assigns, and I will remove it from Drab's assigns

As well as that could be a massive security issue if they ever managed to decrypt it (is it sent encrypted?) as it contains the server's private keys and user state data and much more. ^.^;

[grych]

Is the assumption that no other js code can modify the assigns?

Yes. Now stronger than ever, as assigns are encrypted on the server side.

Anyway, it will be done in a future. Performance is not a priority in the early version, and there are more important issue to solve.

Thanks!

[GBH]

About @conn getting removed. I see that in 0.5.5 I can't render links in drab templates now (they kinda need that @conn). How do I get around that?

To clarify, I'm not messing @conn inside the Commander, it's just the template needs it to generate links.

[OvermindDL1]

@conn must be back. I am going release a fix today.

Hold up! Path helpers do not need conn!

They only accept conn because they can get the endpoint off of it. For note, these are entirely interchangeable:

some_path(@conn, :index)
some_path(MyServerWeb.Endpoint, :index)

It needs the endpoint so it can look up the endpoint url information. You do not need the @conn.

[OvermindDL1]

True, but I don't want to force users to change their Phoenix habits. The goal is to reuse existing Phoenix template as much as we can. Removing @conn broke this rule. We will deal with the performance in the other way.

However you can make a fake @conn structure and fill in specific things for things to work, such as the endpoint module, and perhaps also the conn.assigns that the user chose to keep too?

[grych]

v0.6.0 (coming 2017-10-31) will introduce new features to reduce the frame size:

• expressions with conn as an only one assign are not drabbed, so in the most cases conn will be not treated as a Drab assign and transmitted back to the server
• nodrab option allows to turn off Drab.Live features on any expression to save the data
• Drab assigns are transmitted only on connect, and stored in the Drab genserver; no need to resend it on each event anymore

[grych]

@OvermindDL1
Finally I've used your idea for shrinking the conn. And I took your deep_ functions, they worked out of the box, without any change. Thank you very much!

[grych]

Fix included in the premature release v0.5.5

[tjheeta]

Wow, thanks for the quick release.

Just tested and the request is now around half the size at 150k.

__amperes: {gi2dimbsga4dg: ["gmzdiojvgy2tm", "gyztqmbqhe2do", "ha3tknzvgi2tk"]}
__assigns:
gi2dimbsga4dg: {
  objects: "150k of data"
  text: "QTEyOEdDTQ.Ua88q6Lu-SxI5EVdz3xZv4iXTyfEUHR9AUQ41mxSJ2b3BtTzj2D"
}

[grych]

You must have quite a hudge assign called @objects, yes?

[OvermindDL1]

Just tested and the request is now around half the size at 150k.

That is still huge! What all is in it?! o.O

[tjheeta]

It is essentially rebuilding a 100k document. Then there's some overhead from erlang terms.

My initial instinct is not to send any of the assign data back to the server except for what has been changed. It does look like it is possible at drab/lib/drab/live.ex:287 to assign only changes, but this looks like it also affects peek.

If we cache the assigns in genserver on the websocket, we need to make sure the full assigns are re-transmitted after a network failure.

Is the assumption that no other js code can modify the assigns?

[grych]

@GBH - true. I completely forgot about it, I need to add some partial with link or img to tests.

@conn must be back. I am going release a fix today.

Thanks all!

[grych]

True, but I don't want to force users to change their Phoenix habits. The goal is to reuse existing Phoenix template as much as we can. Removing @conn broke this rule. We will deal with the performance in the other way.

[grych]

This is a good idea, it will shrink it. Anyway, before, we need to identify which fields except the endpoint could be used in the helper functions.

[OvermindDL1]

And I took your deep_ functions, they worked out of the box, without any change.

Hah, cool, it was entirely untested. ^.^;

It 'could' be more efficient, but not enough to really matter though unless the maps got huge. :-)