Feature Request: Add ability to delete content from a channel about hub

[moulliet]

So this would allow for the deletion of an item, only after it has expired from the Spoke cache, ttl.

[aashishpatel]

yes, that will be perfect.

[allumbra]

@aashishpatel What is the use case you are solving for?

[aashishpatel]

@allumbra
Basically, our application will store indexes for the data separately from the where the data is actually stored(postgres and s3, respectively).

1. The hub API will be used as a REST interface for s3
2. because our IT team hosts and administers these hub instances, they are also already managing the encryption keys, access control etc. So we/they won't have anything more do.

The data and indexes for this data will initially be stored in the postgres RDS for quick retrieval. After the data is considered old, we will archive it to the hub permanently(trips old than 6 months, leaving the index behind). Now we can use postgres for searching and the hub for archived data retrieval. This should limit the amount/cost of data stored in the RDS.

[moulliet]

@aashishpatel what interface do you need to this feature?
Delete unique items?
Delete all items prior to a time?

[aashishpatel]

So far we only have a case for delete unique items.

[chriskessel]

Doesn't this cause the next/previous references to be unreliable now? Previously, if you had an item URI you could rely on it always existing (up to the TTL). With delete, can't someone use delete to blow a hole in clients that are following next/previous links?

I thought a big part of the hub value was the immutability of written data. I can understand wanting a front end to S3, but Aashish's use case sounds more like a CRUD name/value interface.

[breedx2]

I agree with @chriskessel here. I don't think that DELETE belongs in the hub like this, as it fundamentally changes the nature of the time-oriented data stream. I think it's a mistake to use the hub as a REST API to S3 (um, s3 already has a REST interface!)

I would vote to have this feature removed...or at least this should be a nonstandard channel option that us disabled by default.

[chriskessel]

I certainly understand the motivation to use Hub V2 since it hides/fixes the S3 write unreliability (black holes) with the spoke cache.

Perhaps keep channels which maintain immutability and add, uh, "buckets" that are basically S3 pass throughs? Use the same hub/spoke underlying technology to support 2 different storage needs?

[crozierm]

Plus one.

And provide group callbacks on bucket mutations.

On Wed, Mar 4, 2015 at 7:52 AM, Chris Kessel [email protected]
wrote:

I certainly understand the motivation to use Hub V2 since it hides/fixes
the S3 write unreliability (black holes) with the spoke cache.

Perhaps keep channels which maintain immutability and add, uh, "buckets"
that are basically S3 pass throughs? Use the same hub/spoke underlying
technology to support 2 different storage needs?

—
Reply to this email directly or view it on GitHub
#104 (comment).

[email protected]
Voice: 503.445.4233
Fax: [phone]

[aashishpatel]

Jason, we're not advocating using the Hub as a REST API for S3 here ONLY. With this interface as a company we have radically removed barriers around access control and encryption and guarantee durability. The only thing we are doing differently is using the hub as permanent storage for trip information. This will reduce IT's burden, and our own in having to implement yet another S3 client at conducive and all that goes along with access control and encrypting S3 buckets. We've seen how slow we are at doing this sort of stuff so this was just our attempt at utilizing existing products and infrastructure.

The way I see it, the Hub is ACTUALLY conducive's own version of a CRUD S3 REST client that stored information in time series and provides easy access control, encryption and guarantee durability of this information. It was simply missing a delete feature.

Kessel is right in that it would change the behavior of group callbacks, but Crozier has a great complementing idea with group callbacks for bucket mutations.

[moulliet]

I don't think next/previous will be unreliable, they remain relative links.

I think there is a difference between items changing and the list changing. You still can not change the content of an item. The list already does mutate with the TTL.

We can easily make this a non-standard option at the channel level.

I'm struggling to understand a use case for group callback on deleted items. Would we send the url of the deleted item that can no longer be retrieved? What could a client do with that?

[crozierm]

I was responding to Kessel's suggestion for a separate K/V store, not
asking for callbacks on the deleted items. I don't think there should be a
delete option in the datahub.

On Wed, Mar 4, 2015 at 9:24 AM, moulliet [email protected] wrote:

I don't think next/previous will be unreliable, they remain relative links.

I think there is a difference between items changing and the list
changing. You still can not change the content of an item. The list already
does mutate with the TTL.

We can easily make this a non-standard option at the channel level.

I'm struggling to understand a use case for group callback on deleted
items. Would we send the url of the deleted item that can no longer be
retrieved? What could a client do with that?

—
Reply to this email directly or view it on GitHub
#104 (comment).

[email protected]
Voice: 503.445.4233
Fax: [phone]

[crozierm]

Kessel is right in that it would change the behavior of group callbacks,
but Crozier has a great complementing idea with group callbacks for bucket
mutations.

To be clear, I was suggesting a callback feature for the K/V store that
Kessel suggested.

[email protected]
Voice: 503.445.4233
Fax: [phone]

[chriskessel]

By unreliable, I meant code now has to deal with items missing within a
sequence. With TTL, if you're walking backwards and hit an item that's gone
then you're done. You're guaranteed nothing else exists.

With delete being added, there can be holes. So if I'm walking backwards
and get a 404, it could be because that was deleted between my request and
when I followed the "previous" pointer.

Now on a 404 I'm not necessarily done, it could be a hole. So now I have to
refetch that item and look to see if it has a new previous link. Assuming
that item can be refetched and it wasn't also blown away as part of a
multi-item hole. And now my code has to do something like remember the last
X attempts and keep falling back to find item that hasn't been deleted so I
can get a valid "next" reference again.

There are similar issues if I use the new "previous 20" interface as well
in that I now have to handle the possibility of holes in the sequence as I
go walk them.

On Wed, Mar 4, 2015 at 9:30 AM, crozierm [email protected] wrote:

Kessel is right in that it would change the behavior of group callbacks,
but Crozier has a great complementing idea with group callbacks for
bucket
mutations.

To be clear, I was suggesting a callback feature for the K/V store that
Kessel suggested.

[email protected]
Voice: 503.445.4233
Fax: [phone]

—
Reply to this email directly or view it on GitHub
#104 (comment).

[moulliet]

@chriskessel - is this a problem if the channel you are using does not support deletes?

[chriskessel]

If delete was disabled/enabled per channel, that'd probably be sufficient.

The place I could see this being problematic is if someone like data analytics was walking lots of different channels to crunch numbers for various analytic purposes. They'd have to write a client that was able to handle deleted items regardless since they might be walking channels with and without deletion ability.

Personally, I'd rather leave channels as is and have a new domain concept for channels with mutable streams (aka the "bucket"). Something like a "http://hub-v2/bucket" hierarchy.

[moulliet]

@aashishpatel's use case went away, and with the concerns raised, we will remove this feature

[aashishpatel]

Thanks for everyone's input.
@benFlightStats @moulliet I am amazed how fast you guys got this feature done :)