Light

intel-bigdata / ssm Goto Github PK

View Code? Open in Web Editor NEW

132.0 132.0 66.0 10.21 MB

Smart Storage Management for Big Data, a comprehensive hot/cold data optimized solution

License: Apache License 2.0

Shell 1.61% Java 82.38% HTML 3.19% CSS 1.23% JavaScript 6.79% ANTLR 0.10% Scala 0.50% Batchfile 0.19% XSLT 0.13% Python 1.44% Roff 0.96% Thrift 0.07% CMake 0.28% C 1.13%

ssm's People

Contributors

Stargazers

Watchers

ssm's Issues

[SSM] Implement SSM rule parser

Sub-tasks for this issue:

Parse and translate rule conditions
Parse and translate rule trigger
Parse and translate rule commands

Add inotify event fetch logic

Add EventBatch serializer

[SSM] Adjust code/doc organization

[SSM-SM] Poll NN for name space and maintain these info for SSM

[SSM] Refact code for developement base

[SSM] Define interface for Rule and StatesManager

[SSM] NN-end access count metric to sequence file

[SSM] NN checkpoint plugin mechanism

[SSM-AE] ActionPool and duplicate/loop check

Enhance Action Pool

Action Pool (queue)
Duplicate action check
Loop action check

[SSM] Implement interface to insert and retrieve info from table 'commands'

Implement getCommandsTableItem

add getCommandsTableItem() and change getCommands() to private

Implement command translation and generation in RM

[SSM-RESTful] Define RESTful APIs

[NN] Extend HDFS Cache to support caching partial file

Currently, only file level cache supported in HDFS.

[SSM] Load SSM database checkpoint for recovery

[SSM-WEB] Implement RESTful API server

Create RESTful API service in SSM

Periodically pull namespace into DB

[NN] Disable metrics collection if dfs.ssm.enabled is set to false

[SSM-AE] Action basic

SSM action basic module and UT

Move to SSD
Move to cache
Move to archive
UT

[SSM-WEB] Submit rule to SSM

[SSM-RULE] Implement SSM rule paser

[SSM] Refactor access table aggregation into non-blocking implemention

Apply inotify event into DB

[SSM] Define interface for Rule and RuleManager

Implement Rule executor to process time based rules

[SSM] Integrate ANTLR into SSM project

[SSM-WEB] Show rule related info/results

Implement rule executor task

Parse time based trigger

[SSM] Read access count sequence file and generate tables

Add InotifyEventFetcher test

[SSM-AE] Execute actions and maintain state/result

[SSM-WEB] Implement web console through RESTful API

[SSM-WEB] Manage rules in SSM

[SSM] SSM exits when another instance is functioning

Keep SSM single instance in the system.

[SSM-AE] Implement actions supported

Thread safe enhancement for RuleManager

[NN] Persist metric data unfetched in time into DB

[SSM] Persist critical data and checkpoint to HDFS

[SSM-RULE] Convert result of rule-paser to internal expression

[NN] Collect file access count metric from NN

This includes the following sub-tasks:

Implement the metric collection.
Provide rpc call for SSM to poll.

[SSM-AE] Make 'mover' tool parallelable

[SSM] SSM service bootstrap

[SSM] Integration test

[SSM] implement FilesAccessInfo using access history list

[SSM] Implement RPC server

Create and run RPC server in SSM end
Protobuffer based RPC infrastructure
Add RPC calls
Implement RPC client

[SSM] Support HDFS HA

[SSM] Poll storages info from NN

[SSM-RM] Check rule conditions and generate actions accrodingly

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs