muphblu / mapreduce

MapReduce

Methods

• Files:
  • Read
  • Write
  • Delete
  • SizeQuery
• Directory
  • List
  • Create
  • Delete
• ObjectType(folder, file, neither)

Structure:

StorageServer

started on ports 8000 - (8000+n) where n number of servers

• byte[] read(id) // read chunk from server
• write(chunk_id, content) //where list contains tuples(chunkId, content)
• size()
• status delete(list[]) // list of chunkIds
• size: ??

NamingServer

started on 8000 port

• listTuples read(path) // returns ordered list of tuples where key is serverId and value is chunkId
• listTuples write(path, countChunks) // returns ordered list with tuples where key is serverId and value is chunksIds
• status delete(path) // Name server tells storages to delete particular chunks
• size: ??
• list[] list(path) // list of files/directories
• status mkdir(path)
• status rmdir(path)
• type get_type(path)

Questions

1. Slava: Can we use plain structure for storage servers (we can use)
2. Slava: can we store file size on the namingServer?
3. Slava: Can we unite rmdir and delete?

mapreduce's Issues

Allocate features to different classes(Node, NameServer, Client)

Create features list

Adapt client to work from slave(storage_server)

add method do_the_job

Cover storage_server by tests

Implement I/O lib

Implement map_reducer

Adapt naming server to work with job tracker

Adapt naming server to notify Job tracker about up and down slaves

Implement job tracker

job_tracker

test

Create skeleton(code)

Create debug/release configurations