Spark write mode "Ignore" support about spline-spark-agent

[cerveada]

The SaveMode.Ignore works the similar way as CREATE TABLE IF NOT EXISTS in sql. The data are stored only when the target destination is empty (the file doesn't exist for example), otherwise the write is just ignored.

The problem is to get the information which of those two happend. The best way to do it seems to be write metric numFiles. This seems to work, but not for every source. For example Excel source doesn't produce any metrics at all. To deal with this there should be warning when the information is not there and the write will be ignored by spline as well.

[wajda]

It seems to me that we are dealing with false positive vs false negative dilemma. If we treat absence of change metrics are ignored write we might loose some lineage information. Otherwise we might record lineages that did not actually happen.
What if we make this strategy configurable in the agent, so the user can decide which (mis)behavior is more tolerable for his needs?
E.g.

spline.ignored_write_detection_strategy=BY_METRICS_ABSENCE | BY_METRICS_ZERO

[cerveada]

When metric is absent we still always want to ignore by metric zero. What about:
spline.ignored_write_metrics_missing_action=IGNORE_WRITE | CAPTURE_WRITE

[wajda]

let's do it even cleverer, using a future-proof approach. Let's use a strategy pattern for the ignore write detection logic with the built-in default implementation. And the property we are are discussing will then be a property of the default implementation. Hence the hierarchical configuration (spline -> strategy -> impl -> property):

spline.iwd_strategy.default.on_missing_metrics=IGNORE_LINEAGE | CAPTURE_LINEAGE

If somebody wants to override it with say com.foo.MyIWDStrategy then the config would look like

class MyIWDStrategy extends IWDStragery {
  override val name = "myImpl"
  def propFoo = ???
  def propBar = ???
}

spline.iwd_strategy.class = com.foo.MyIWDStrategy
spline.iwd_strategy.myImpl.propFoo = ...
spline.iwd_strategy.myImpl.propBar = ...