gwadvnet2015 / gw-kv-store Goto Github PK

We plan to finish integration of the api into our code. Once the demarhsall client function is working we will integrate into our code and test it on a memcached server. This will complete the client.

Define remaining Issues - each team must clear out old issues and create new ones with specific plans to complete before next class. Your issue should not just say "finish the project", it should say specifically which parts you will have done before class--make it something reasonable that you can actually finish this week. Due Tuesday at noon.

• Client team
• Server Infrastructure Team
• Server Logic Team
• Parsing Team

When your team has created the issue(s), check off your tick box here--this will alert us that we can begin looking through and commenting on your plans. I've created Labels so we can tag the issues by team.

Comment on Issues - team members should indicate in comments which issues they will solve. Members from other teams should comment on other Issues to indicate which pieces are most important to them or list functions which they cannot make progress without. Due Wednesday at noon.

• Client team -- comment on Parsing team issues
• Server Infrastructure Team -- comment on Server Logic team issues
• Server Logic Team -- comment on Parsing team and Server Infrastructure team issues
• Parsing Team -- respond to comments from other teams
• Phil -- comment on all teams
• Tim -- comment on all teams

Write Some Code! - each team should work on their code. Team leaders should continually bug their team members to ensure progress is being made. Ideally you should do small commits and make pull requests for each small feature you add--if your pull request includes hundreds of lines of code changed it will probably be rejected and you will have to redo your work! Pull requests are due Saturday at noon.

• Client team
• Server Infrastructure Team
• Server Logic Team
• Parsing Team

Pull Request Reviews - Phil and Eric get the lovely job of looking through the pull requests submitted by each team. They will undoubtedly send you feedback about what should be fixed. Continue to update your code in response to their suggestions/rants. Due Monday at noon.

• Phil reviews
• Eric reviews
• Client team respond to reviews
• Server Infrastructure Team respond to reviews
• Server Logic Team respond to reviews
• Parsing Team respond to reviews

Think of this project as your final exam. Your final grade will be based in part on how the overall kv-store comes together and on the individual pieces you contributed. That is why it is important that you make it clear who is working on what!

@pradeep-k @hantasm @kstats @Atryus @phil-lopreiato -- you are authorized to check off boxes in this issue as your teams finish things. If you need feedback or have questions, post comments a well (I don't think we get any notification for checked boxes).

We use thread pool. There is a task list. The procedure is the following:

1. Create the defined threads.
2. When a new connection comes, create a task and put to the tail of a task list.
3. Each thread will get a task from the task list and process it.
4. Use mutex and conditional variable to protect and synchronize the shared task list.

Need other groups to provide the parse_message and process_operation functions for us.

void parse_message(char *message, int len, struct operation *operation);
void process_operation(struct operation *operation, char *message, int *len);

Be sure this works and import it so we can use it in the client libraries
gwAdvNet2015/adv-net-samples#12

@akorzan has to complete this gwkv_demarshal_server
@huyang1988 has to fix the bug in gwkv_marshal_server. I believe gwkv_demarshal_client is done. If not flag it asap.
@pradeep-k has finished gwkv_marshal_client.
@Grace-Liu has finished writing the testcase. If you haven't created the pull request, please do this asap.

Since, other teams are depending on these APIs, we need to make this up first. I am targeting it to be finished by Wednesday 11:59 PM.
@akorzan and @huyang1988 : Let us know if you guys need any help in finishing this by that time.

Do it

Enabled automated continuous integration unit testing through http://travis-ci.org

See an example

What I'd like is to have these files in the client directory:

• client-api.c - this contains the functions which could be used by several different clients
  • This should have reasonable names for all external functions
  • Put the author name header at the very top of the file
  • @kstats @jdk514 @sacul29 - please make any necessary fixes for this part and update #73
• client-cmd.c - this is a command line-based client that reads arguments from the command line about whether to do a set or get and uses the library to make the request
  • @kstats @jdk514 @sacul29 - please make any necessary fixes for this part and update #73
• client-perf.c - this is a performance tester. It should issue N requests (a command line parameter) and should measure the time taken by each request and store it in a histogram. Remember that each request will require creating a new socket connection. At the end of the program it should print out the histogram.
  • @SMonaghan and @Chryel - please work on this once the client-api.c is merged with final function names.
• client-tests.c - this is a test program that can be used to test whether the server is working properly. It might for example might set key1 and then set key2 and then try to get them back to verify the values are correct. Then it might try to get key3 (which should not exist) and handle that appropriately. If the keys don't match or if the program segfaults that means the test fail. It may not be possible to run this as part of an automated Travis test because it will require that a server is actually running. That's OK with me.
  • I'll assign this to some of the others who didn't do a GENI project once we know what other tasks remain.

I don't think we need other automated tests for the client API since all of the interesting logic is related to sockets, and there is no easy way to test that with Travis.

Extend the simple server #9 to support multiple threads. In this case, all threads should be created at program start up and they should continuously poll to see if there is a new client socket with a request that must be handled.

all files should begin with:

/************************************************
 *                     GW KV
 *  https://github.com/gwAdvNet2015/gw-kv-store
 *
 * Copyright 2015 Author 1, Author 2, 
 *   Author 3, Author 4
 *
 * This program is licensed under the MIT license.
 *
 * filename.c - parses incoming messages and
 *   and prints them to the screen.
 *************************************************/

Replace the authors with anyone who worked on the file and put a short description of what the file does.

Extend the server #9 so that it has a single thread that calls select to multiplex multiple client sockets.

Write the overall server architecture that binds a server socket and accepts client connections one at a time. For each request it must:

• Parse the request #2
• add or retrieve the data from the KV store #8
• send a reply back to the client

Handler should do the following

• process command received from socket
  • call demarshal functions to parse and get operation struct
• use switch statement to determine GET or SET
  • call gwkv_set/get from wrapper
• then do the sending once all data is acquired and set up

How should we structure the server? How do we store data? How do we support concurrency? How should we define our application-level protocol?

We have successfully been able to read the first line of the server response by reading byte by byte. We now need to process the data block of the server response in order to finish. @jdk514 @kstats

Extend the simple server #9 to support multiple threads (one created per client)

This version of the client load generator should be similar to #4, but instead of using multiple threads it should use the select call to have a single thread monitor several different sockets. This can then be extended to support multiple threads, each using select.

Will attempt to complete the timer utility's last timer_end function with histogram features.

@huyang1988 @Grace-Liu @akorzan
We need to finish the Marshalling/Demarshalling stuff.
Here is the summary that we did:
1st and 4th API: Done by Yang.
2nd API done by Pradeep
3rd API is being developed by Anthony
We need to finish this as per plan.

Division of the work:
Grace: Please make the code compilable. You can see there is a Makefile and main.c inside lib directory speicifically designed for marshalling/demarshalling. Write a small test function in main.c so that we are able to call any of the APIs for testing. You don't have to write in-depth testing framework except making it run. Then, write one testcases for each of 4 APIs at least. You can write more.

Yang, pradeep and Anthony will then test their apis writing more testing code in main.c or a new file mimicing your written code as per their wish.

I will be merging the repository to main repository by end of day today so that you can fork the main repository.

Write a simple client that issues N get or set requests. Test it using a standard memcached server.

Write a client workload generator that can be used for evaluating overall performance. This needs to be high performance enough to stress the server. It should create N threads and have each repeatedly open a socket to the server and send a request.

It should record statistics for each thread - the response times of each request and the number of failed requests. Use the histogram and timer libraries #6 and #7.

We will work to finish converting our simple working string based client to a fully functioning API client. This involves fixing how we parse the incoming message and using demarshal for read response.

Write wrapper functions for the hashtable in the samples repo.

You can see the interface in this file

Development is happening here

@SMonaghan is working on implementing a MD5 hash function
@Chryel is working on the node comparison function

Finish this and import it to this project so we can use it in our client:
gwAdvNet2015/adv-net-samples#8

It breaks.

in server/test you should be able to run make and have the compiled file run without error

Make a wrapper around the hash table library so that it can be easily used to support get, set, and inc operations.

https://github.com/gwAdvNet2015/adv-net-samples/tree/master/datastructs/hashtable

Write functions to parse a request and call the required function.

When a key is not found, the server should only return END.