Need the ability to cancel a current job.

Should stream any stdout to the web interface so you can have an idea on progress and how things are running.

Right now when submitting jobs it waits with an open connection until it is done. This should change to submit-then-poll so that longer running jobs do not time out.

Add support to submit jobs via the command line. This is ideal for running jobs on a server on a regular basis or if you prefer your editor locally.

There should be a warning when a large amount of data may be collected to the Exalt SQL app. Maybe warn on anything over 10k rows?

Actions that transfer data to the client:

• .each
• .rows
• .cols
• .show

Need an initial test suite to verify core functionality works

• job submission
• check job status

Create a docker image for easy development and deployment.

Currently you can submit a job multiple times, this shouldn't happen.

Make exalt sql installable by a gem. This way you can easily start the web ui or run via command line by requiring the gem.

A running job shouldn't die because the browser closed or refreshed.

To see if this project can actually be viable, I'd like to implement a production workload I have running in spark.

Goal

Given a list of customer items, determine how many items each customer contributes to an overall unique list of items by customer item count descending.

For example, given 100 unique items, customer A have contributes 20 items to the list, customer B contributes 10 items to the list etc.

Dataset

Table: customer_items

Code

customer_contributor_counts = []

customers = query("select customer from customer_items group by customer")
unique_items = query("select item from customer_items group by item")
unique_customer_items = query("select customer, item from customer_items group by customer, item")

processed_customers = []

customers.each do |c|
  if customer_contributor_counts.length == 0
    diff_count = query("select count(*) from #{unique_customer_items.table} where customer = #{c["customer"]}").rows[0][0]
  else
     diff_count = query("""
       SELECT
         COUNT(*) AS cnt
       FROM (
         SELECT item FROM #{unique_customer_items.table} WHERE customer = #{c["customer"]}
         EXCEPT
         SELECT item FROM #{unique_customer_items.table} WHERE customer IN (#{processed_customers.join(",")})
        )
     """).rows[0][0]
  end
  customer_contributor_counts << {customer: c["customer"], item_contribution_count: diff_count}
  processed_customers << c["customer"]
end