Comments (3)
Suggested API:
class Cache[T](path : String) {
def load : T
}
object Cache {
def apply[T](value : T) = new Cache[T](serializeAndPlaceInDistCache(value))
}
Usage:
val cachedData = Cache(loadLocalData)
lazy val data = cachedData.load
from scalding.
Was thinking of picking this up, but it turns our we have it already:
https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/filecache/DistributedCacheFile.scala added in 7fd4b15
A nice to have here would be to expose the add__ToClasspath
methods, but in general the existing implementation by @slyphon seems to resolve this task.
from scalding.
yes, closed by #450
@ktoso the classpath stuff would be great to have.
from scalding.
Related Issues (20)
- Use cats PairingHeap for PriorityQueueMonoid
- Optimizing KyroCoder in beam backend HOT 7
- changes to continuous integration HOT 2
- Scalding on Beam discussion about Joins HOT 1
- migrate to github CI HOT 10
- scalding-jbdc current use / discussion on support/deprecation HOT 4
- flake on optimization test HOT 1
- Set up automatic publishing HOT 16
- scalding-hadoop3 backend OR spark backend upgrade to 3.X HOT 12
- Re-enable code coverage HOT 3
- copy dagon into this tree HOT 1
- Beam backend is missing some pipes
- [Proposal] Support more sinks/sources in scalding-spark HOT 1
- Support CounterPipe in spark-backend HOT 8
- [bug/nit] code format enforcement lost
- make codecov not suck
- push new tags HOT 4
- incompatibility with recent java8 runtime environments due to hadoop
- duplicate tags in pom files HOT 1
- my account is closed HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scalding.