laurent-simon / storage-at-desk Goto Github PK

The unique identifier for a machine is now based on it's hostname.  The
assumption is that within a domain there will be unique hostnames.  But if
we want cross/domain interaction or the ability to change hostnames then we
should come up with a better solution.

Idea:

Have the admin web interface generate unique ID's and the user must go to
the website to generate a UID and it will be saved in the config folder

An external executable is required for random access file read and writes
on windows.

The problem is in FileDisk.java

An explanation of the possible reason that nativefile is needed is here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734

It would be desirable to have file access all written in Java without
having to rely on an external binary that may need to be re-compiled on
different machine configurations.

What steps will reproduce the problem?
1. Run the StorageServer on windows machine
2. Connect with iSCSI
3. Transfer a large file

What is the expected output? What do you see instead?
It should work.
It hangs.

The Problem:
The combination of Windows XP SP3 and Sun Java JVM 1.6 has file locking 
issues with log4j.

http://forums.sun.com/thread.jspa?threadID=5326294

Please use labels and text to provide additional information.

The heartbeat operation does not update the IP address of the storage node.
 This means that if the IP address of a storage node changes it is
essentially lost forever unless the IP address is manually updated in the
database.

What steps will reproduce the problem?
1. Run the storage machine on ubuntu
2. turn the iscsi server on and off many times

What is the expected output? What do you see instead?
You should be able to startup and shutdown the iscsi server indefinitely, 
however, each time the iscsi server contacts the storage machine the 
storage machine opens the files again (never closing them)

This leads to a situation where the same file is opened multiple times, to 
the point of failure

Please use labels and text to provide additional information.

What steps will reproduce the problem?
1. Request a new volume
2. set Volume Chunk Size to 10x size of what's being requested
3. try to assign mappings

What is the expected output? What do you see instead?
This should find 1 volume Chunk and assign it to the new volume.
Instead it thinks that 0 chunks are required to satisfy the request.

Output:

443565 18:07:15,149 [RMI TCP Connection(7)-192.168.5.32] INFO 
volumecontroller.VolumeController  - Registering a volume
443566 18:07:15,150 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - No volume with name iqn.edu.virginia.cs.storagedesk:disk2
443566 18:07:15,150 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - To Insert Volume iqn.edu.virginia.cs.storagedesk:disk2
443569 18:07:15,153 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - Volume exists in the DB (id 5)
443573 18:07:15,157 [RMI TCP Connection(7)-192.168.5.32] DEBUG
database.Volume  - Mapping 1 copies 0 chunks
443574 18:07:15,158 [RMI TCP Connection(7)-192.168.5.32] INFO 
volumecontroller.VolumeController  - Getting a mapping for a volume
443575 18:07:15,159 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - Mapping doesn't exist
443575 18:07:15,159 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Machine  - Get all available machines whose chunk size is 1024000000
443583 18:07:15,167 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - 0 machines available
443584 18:07:15,168 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - Machine has 0 chunks vs. volume [0] asks for 0 chunks
443584 18:07:15,168 [RMI TCP Connection(7)-192.168.5.32] INFO 
database.Volume  - Assign mappings to 0 machines
443584 18:07:15,168 [RMI TCP Connection(7)-192.168.5.32] DEBUG
database.Volume  - Mapping 1 copies 0 chunks

Currently the Wrapper class relies on external binaries (.so and .dll's) to
function properly.

We have yet to get the correct paths to the .so and .dll's to work however
there may be a better solution than figuring Wrapper out and that's just to
drop it all together.

Wrapper's purpose is to provide an interface to the OS to treat our
application as a service.  I believe that there is a better way to do this
at the installer level.  This will make our app more lightweight and not
dependent on external binaries that need to be compiled for different
platforms.

The volume name should be unique since it is used to lookup the mappings of
a volume.  ex, query volume iqn.cs.virginia.edu:disk1 > volume id = 8 >
lookup mappings of volume id 8

If there are multiple volume's with the same name then it will not work.

The process of creating storage volumes for users will be handled by the
administrator

Using the globalSAN iSCSI initiator the following error appears:

37856 16:32:52,919 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
LOGIN SUCCESSFUL
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
ISCSI op code 0x3 complete
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
Ready to read from initiator
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
ISCSI op code 0x1
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
SCSI command
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.ScsiRequest  -
Decap scsi command request
37857 16:32:52,920 [pool-1-thread-2] DEBUG storageserver.ScsiRequest  -
decap finishes
37858 16:32:52,921 [pool-1-thread-2] DEBUG storageserver.TargetThread  -
SCSI Command (CmdSN 2, op 0x12)
37858 16:32:52,921 [pool-1-thread-2] DEBUG storageserver.TargetThread  - No
AHS to read
37858 16:32:52,921 [pool-1-thread-2] DEBUG common.Disk  - SCSI op 0x12 (LUN 0)
37858 16:32:52,921 [pool-1-thread-2] DEBUG common.Disk  - INQUIRY
Exception in thread "pool-1-thread-2" Executer shutdown: false
java.lang.ArrayIndexOutOfBoundsException: 7
    at edu.virginia.cs.storagedesk.common.Disk.executeCommand(Disk.java:105)
    at
edu.virginia.cs.storagedesk.storageserver.TargetThread.executeSCSICommand(Target
Thread.java:700)
    at
edu.virginia.cs.storagedesk.storageserver.TargetThread.execute(TargetThread.java
:196)
    at
edu.virginia.cs.storagedesk.storageserver.TargetThread.run(TargetThread.java:124
)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:8
86)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

What steps will reproduce the problem?
1. Start up services
2. Connect with iSCSI initiator 
3. Try to format drive, requests fail
4. Try to reconnect

What is the expected output? What do you see instead?

The connection is accepted but the new TargetThread is never executed by
the ExecutorService threadPool

Perhaps there is a problem with threadPool or blocking withing
TargetThread's run() method.

What steps will reproduce the problem?
1. Run the Volume controller with user name that isn't root
2. Watch the errors form

What is the expected output? What do you see instead?
This is a connection error on the volume contoller

Please use labels and text to provide additional information.

When a storage volume is created in the administrator's web interface, the
client associated with the volume will need to poll the database
periodically to see if it should accept the volume.

While the system will work with multiple Volume Chunk Sizes, we should have
it standardized across all parts of the system.

Storage Node:
Volume Chunk Size = size of the data file
Num Chunks = number of data files

Creating A New Volume:
Checks for all machine with a free volume of specified chunk size.  This
could be different from what everyone else has put for their volume chunk
sizes and so while there may indeed be enough free space, the user creating
a new volume may not be able to find it.

Proposed Solution:
Keep it in the xml config file and don't provide an interface to change it
OR
Hardcode into config.java and don't ever overwrite in Util.java

Currently all config is done through the Config class and overwritten by
the [email protected] configuration file.

The configuration variables should be split up by application.

1) Only the volume controller should be able to see database configuration
variables
2) File locations are only needed by StorageMachine
3) StorageServer is the only one that needs Target Name

There may be others but the big one is the database configuration, the
StorageMachine and StorageServer should not have database config information.

Make a determination of when a storage-node disappears for an extended
period of time and its data needs to be replicated to a different storage-node.

Can easily make determination by the heartbeat-timestamp

Problem is with journaling of writes and copying data chunks

What steps will reproduce the problem?
1. Startup services
2. Connect to iSCSI
3. Format Drive

What is the expected output? What do you see instead?
The format should just work. Instead it hangs at 100% and the following is
seen over and over again

349925 16:26:21,850 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
ISCSI op code 0x1 complete
349925 16:26:21,850 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
Ready to read from initiator
349923 16:26:21,848 [Thread-1  ] DEBUG storageserver.VirtualDisk  - To get
the journal queue size 1
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.Journal  - To get the
queue
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.Journal  - get returns
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Writer
is running with size 1
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Read,
from the journal, entry of Version 252
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Writing
updates to disks
349925 16:26:21,850 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Read
from aux file 43739136, 32768
349926 16:26:21,851 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Read
from data file, bytes size 32768
349926 16:26:21,851 [Thread-1  ] DEBUG storageserver.VirtualDisk  - write
task begins
349926 16:26:21,851 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Try
replica 0
349926 16:26:21,851 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Write
for replica 0 virtual chunk 0 from machine (id
a49d8384-1a04-38ae-ac9e-5e6b5ce03a91) at 43739136
349926 16:26:21,851 [Thread-1  ] DEBUG storageserver.Version  - read from
file: 251
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
ISCSI op code 0x1
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
SCSI command
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.ScsiRequest  -
Decap scsi command request
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.ScsiRequest  -
decap finishes
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
SCSI Command (CmdSN 677, op 0x28)
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.TargetThread  -
No AHS to read
349928 16:26:21,853 [pool-1-thread-1] DEBUG common.Disk  - SCSI op 0x28 (LUN 0)
349928 16:26:21,853 [pool-1-thread-1] DEBUG common.Disk  - READ 10
349928 16:26:21,853 [pool-1-thread-1] DEBUG common.Disk  - Read 10 (lba
42714, len 32 blocks)
349928 16:26:21,853 [pool-1-thread-1] DEBUG storageserver.VirtualDisk  -
Read 32768 bytes at 43739136
349929 16:26:21,854 [pool-1-thread-1] DEBUG storageserver.VirtualDisk  -
Try replica 0
349929 16:26:21,854 [pool-1-thread-1] DEBUG storageserver.Version  - read
from file: 251
349929 16:26:21,854 [pool-1-thread-1] DEBUG storageserver.Version  - read
from file: 252
349929 16:26:21,854 [pool-1-thread-1] DEBUG storageserver.VirtualDisk  -
waiting for writes
349993 16:26:21,918 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Write
32768 bytes from machine (id a49d8384-1a04-38ae-ac9e-5e6b5ce03a91) at 43739136
349993 16:26:21,918 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Write ends
349994 16:26:21,919 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Removed
Version 252
349994 16:26:21,919 [Thread-1  ] DEBUG storageserver.Journal  - Remove 252
from the journal queue
349994 16:26:21,919 [Thread-1  ] DEBUG storageserver.VirtualDisk  - Remove,
from the journal, entry of Version 252
349994 16:26:21,919 [Thread-1  ] DEBUG storageserver.VirtualDisk  - The
journal (size 0)

What steps will reproduce the problem?
1. Start StorageServer
2. Start StorageMachine
3. Connect with ISCSI and make read/writes

What is the expected output? What do you see instead?

You should be able to do read/writes because the storage machine is indeed
up. However you get the error:

Replica # has been marked down

This is because when a replica is marked as down it is always marked as
down.  VirtualDisk never checks to see if the replica is back up yet so if
a replica is down for a brief period of time it will appear to be down
until StorageServer is rebooted.

The database holds the path to the datafiles for each machine.  However the
StorageMachine never reads or even updates the path field after the initial
setup.

This means that a user can move the files around their machine and as long
as they have the correct path in the config xml it will work HOWEVER the
information in the database will be incorrect.

Suggest possibly removing the path field from the machine table schema.