meow-watermelon / eoss Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 249 KB

Eric's Object Storage System

Python 98.21% Shell 1.79%

object-storage storage

eoss's People

Contributors

Watchers

eoss's Issues

add service architecture graph

add data replication feature

this task is to add object data replication feature for EOSS. the initial idea is to use the single-leader replication method. i will attach a design document about the implementation.

as an object storage service, it must ensure the data is stored on the disk before sending an acknowledgment to the client. this issue is to track the effort to improve the data store resilience by using os.fsync().

ref.: https://docs.python.org/3/library/os.html#os.fsync

typo on WSGIRequestHandler.protocol_version

# enable HTTP/1.1
WSGIRequestHandler.protocol_version = "HTTP/1.0"

concurrent write is not working

sample commands to upload the same file:

terminal 1

$ curl -X PUT -T testfile.100M http://localhost:4080/eoss/v1/object/testfile100m -H "X-EOSS-Object-Version: ver15.0" -v
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 127.0.0.1:4080...
* Connected to localhost (127.0.0.1) port 4080 (#0)
> PUT /eoss/v1/object/testfile100m HTTP/1.1
> Host: localhost:4080
> User-Agent: curl/7.85.0
> Accept: */*
> X-EOSS-Object-Version: ver15.0
> Content-Length: 104857600
> Expect: 100-continue
> 
* Done waiting for 100-continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 527 UNKNOWN
< Content-Type: text/html; charset=utf-8
< Content-Length: 20
< X-EOSS-Request-ID: d207772c-906e-482f-944f-519099e95040
< 
* Connection #0 to host localhost left intact
EOSS Rollback Failed

terminal 2

$ curl -X PUT -T testfile.100M http://localhost:4080/eoss/v1/object/testfile100m -H "X-EOSS-Object-Version: ver15.0" -v
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 127.0.0.1:4080...
* Connected to localhost (127.0.0.1) port 4080 (#0)
> PUT /eoss/v1/object/testfile100m HTTP/1.1
> Host: localhost:4080
> User-Agent: curl/7.85.0
> Accept: */*
> X-EOSS-Object-Version: ver15.0
> Content-Length: 104857600
> Expect: 100-continue
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 521 UNKNOWN
< Content-Type: text/html; charset=utf-8
< Content-Length: 21
< X-EOSS-Request-ID: 68b9b316-6a07-435e-afac-6ff5950ccc3f
* HTTP error before end of send, stop sending
< 
* Closing connection 0
MDS Execution Failure

eoss.log:

2023-05-09 21:29:23,341 - uwsgi_file_eoss - INFO - object_filename: testfile100m object_version: ver15.0 object_name: dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:23,341 - uwsgi_file_eoss - INFO - object_filename: testfile100m object_version: ver15.0 object_name: dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - metadata database initialized
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - metadata database initialized
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - object existence flag for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w is False
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - object existence flag for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w is False
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - HTTP request method PUT detected for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:23,342 - uwsgi_file_eoss - INFO - HTTP request method PUT detected for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:23,351 - uwsgi_file_eoss - INFO - initial data for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w set done
2023-05-09 21:29:23,351 - uwsgi_file_eoss - INFO - object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w state initialized
2023-05-09 21:29:23,362 - uwsgi_file_eoss - ERROR - failed to set initial object data for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:24,481 - uwsgi_file_eoss - INFO - object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w data is saved in temp file
2023-05-09 21:29:29,512 - uwsgi_file_eoss - ERROR - failed to set object state 2 for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w: database is locked

object_client.log:

2023-05-09 21:29:23,341 - eoss.object_client - INFO - object filename: testfile100m; object name: dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w; object version: ver15.0
2023-05-09 21:29:23,341 - eoss.object_client - INFO - object filename: testfile100m; object name: dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w; object version: ver15.0
2023-05-09 21:29:23,342 - eoss.object_client - INFO - metadata database initialized
2023-05-09 21:29:23,342 - eoss.object_client - INFO - metadata database initialized
2023-05-09 21:29:23,351 - eoss.object_client - INFO - object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w initialized done in MDS database
2023-05-09 21:29:23,351 - eoss.object_client - INFO - set state on object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w: 1
2023-05-09 21:29:23,362 - eoss.object_client - ERROR - failed to set initial object data for object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w
2023-05-09 21:29:24,481 - eoss.object_client - INFO - set state on object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w: 2
2023-05-09 21:29:29,512 - eoss.object_client - ERROR - failed to set state on object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w: database is locked
2023-05-09 21:29:34,556 - eoss.object_client - WARNING - [ROLLBACK] failed to delete object record dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w in MDS: database is locked
2023-05-09 21:29:34,556 - eoss.object_client - WARNING - [ROLLBACK] rollback procedure on object dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w failed

mds_client.log:

2023-05-09 21:29:23,341 - eoss.mds_client - INFO - initialized metadata database file /home/ericlee/EOSS/mds/mds.sql
2023-05-09 21:29:23,341 - eoss.mds_client - INFO - initialized metadata database file /home/ericlee/EOSS/mds/mds.sql
2023-05-09 21:29:23,342 - eoss.mds_client - INFO - SQL executable: SELECT state FROM metadata WHERE id = ? parameters: ('dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w',)
2023-05-09 21:29:23,342 - eoss.mds_client - INFO - SQL executable: SELECT state FROM metadata WHERE id = ? parameters: ('dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w',)
2023-05-09 21:29:23,342 - eoss.mds_client - INFO - SQL executable: INSERT INTO metadata VALUES (?, ?, ?, ?, ?, ?) parameters: ('dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w', 'testfile100m', 'ver15.0', None, None, 1)
2023-05-09 21:29:23,342 - eoss.mds_client - INFO - SQL executable: INSERT INTO metadata VALUES (?, ?, ?, ?, ?, ?) parameters: ('dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w', 'testfile100m', 'ver15.0', None, None, 1)
2023-05-09 21:29:23,351 - eoss.mds_client - INFO - SQL executable: UPDATE metadata SET state = ? WHERE id = ? parameters: (1, 'dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w')
2023-05-09 21:29:23,361 - eoss.mds_client - ERROR - failed to execute INSERT INTO metadata VALUES (?, ?, ?, ?, ?, ?) - error: UNIQUE constraint failed: metadata.id
2023-05-09 21:29:24,481 - eoss.mds_client - INFO - SQL executable: UPDATE metadata SET state = ? WHERE id = ? parameters: (2, 'dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w')
2023-05-09 21:29:29,511 - eoss.mds_client - ERROR - failed to execute UPDATE metadata SET state = ? WHERE id = ? - error: database is locked
2023-05-09 21:29:29,529 - eoss.mds_client - INFO - SQL executable: DELETE FROM metadata WHERE id = ? parameters: ('dGVzdGZpbGUxMDBtOnNub29weTp2ZXIxNS4w',)
2023-05-09 21:29:34,555 - eoss.mds_client - ERROR - failed to execute DELETE FROM metadata WHERE id = ? - error: database is locked

access.log:

2023-05-09 21:29:23,363 68b9b316-6a07-435e-afac-6ff5950ccc3f 22 127.0.0.1 PUT /eoss/v1/object/testfile100m 521 curl/7.85.0
2023-05-09 21:29:34,556 d207772c-906e-482f-944f-519099e95040 11215 127.0.0.1 PUT /eoss/v1/object/testfile100m 527 curl/7.85.0

support override writing in PUT method

1st PUT:

$ curl -X PUT -T testfile.100M http://localhost:4080/eoss/v1/object/testfile1000m -H "X-EOSS-Object-Version: ver10.0" -v
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 127.0.0.1:4080...
* Connected to localhost (127.0.0.1) port 4080 (#0)
> PUT /eoss/v1/object/testfile1000m HTTP/1.1
> Host: localhost:4080
> User-Agent: curl/7.85.0
> Accept: */*
> X-EOSS-Object-Version: ver10.0
> Content-Length: 104857600
> Expect: 100-continue
> 
* Done waiting for 100-continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 CREATED
< Content-Type: text/html; charset=utf-8
< Content-Length: 15
< X-EOSS-Request-ID: 507deccc-5971-4f60-82cb-4005db0681dd
< 
* Connection #0 to host localhost left intact
Object Uploaded

2nd PUT:

$ curl -X PUT -T testfile.100M http://localhost:4080/eoss/v1/object/testfile1000m -H "X-EOSS-Object-Version: ver10.0" -v
Note: Unnecessary use of -X or --request, PUT is already inferred.
*   Trying 127.0.0.1:4080...
* Connected to localhost (127.0.0.1) port 4080 (#0)
> PUT /eoss/v1/object/testfile1000m HTTP/1.1
> Host: localhost:4080
> User-Agent: curl/7.85.0
> Accept: */*
> X-EOSS-Object-Version: ver10.0
> Content-Length: 104857600
> Expect: 100-continue
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 442 UNKNOWN
< Content-Type: text/html; charset=utf-8
< Content-Length: 21
< X-EOSS-Request-ID: 1b83cf96-fc41-4b1b-a3db-64bae88e1424
* HTTP error before end of send, stop sending
< 
* Closing connection 0
Object Exists Already

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble