Question: Differentiating btween lamba and spark architectures about tinytelematics

[chollinger93]

Hey,

thanks :)

This is a good point, but I am not a huge fans of persistent branches (as opposed to release branches) for functionality. However, the README.md could definitely use some work.

The long term plan is to add a 3rd version that re-uses part of the Spark code, but uses less Hadoop focused technologies.

[jmandel1027]

For sure, yeah release tags would totally do the trick and provide an easy way to access the correct state for each version.

Ooh interesting how so? As like a more optimized way to mount the files from s3 and perform the queries over spark? Thats super interesting. Athena is very powerful but sometimes not well suited for things in a real time fashion.

One thing that might be interesting to explore is a lambda to activate a Fargate task, this would give you more flexibility to execute the spark jobs without worrying about hitting a timeout although i think they're like 15 mins now so might be moot. Pairing a lambda step function + Fargate tasks might be an ergonomic way to handle the query and return the results.

[chollinger93]

Created separate branches