Use generic java.nio.files interfaces in FileSystemStorage about bt

[ogregoire]

@MorrisLaw I wrote this ticket, but didn't start anything yet. So from what I can see, it's indeed up for grabs.

While java.nio indeed defines FileSystem and friends, the real win is by using FileChannel and MappedByteBuffer. Especially given how BitTorrent works.

Additionally, the download itself can use java.nio as well to take the full power of multithreading.

[MorrisLaw]

@ogregoire @atomashpolskiy Ok guys, I'm going to attempt to tackle this one.

[atomashpolskiy]

@ogregoire ,

I guess there must be some misunderstanding here...

There might be a misunderstanding, because

The thing is that currently you always go through a byte[], which is far from efficient.

I'm not sure where this comes from; there's a FileChannel, that is passed a ByteBuffer. No byte arrays are involved at this level.

The idea is to get rid of this temporary buffer by using the buffers we get from the MappedByteBuffer.

Regarding memory, it's a mappedbytebuffer, so it's memory that's directly mapped to the drive. No system calls. I don't see why it should be a user-preference: the user will want it to be written to disk, according to the best way to handle the disk. This is what is done by default.

I completely understand your idea. I'm just trying to argue, that:

1. We don't always want to create a MappedByteBuffer. Specifically we don't want to map anything to memory when we're streaming (what's the point of having an intermediate in-memory buffer, if we want data to be flushed to disk immediately and call force() after each write()? Seems like wasting RAM to me, directly working with FileChannel is clearly a superior option in this case)

2. When we do want to have an intermediate buffer in the form of MappedByteBuffer (e.g. when we're downloading a 100GB collection of files and don't care, if the data is on disk on in memory, until the download is finished), we also want to control and limit the total amount of RAM being used. Simply mapping each individual file to memory isn't going to work, which means that creating a MappedByteBuffer inside the storage unit (which is equivalent to a file in Bt terms) does not make much sense. It should be done in some upper layer, that manages all read/write operations.

I might be communicating not clearly enough though, excuse me if so, because English is not my first language :)

---

@MorrisLaw ,

Is this still the case:
What this issue is really about is switching to using generic java.nio.files.Path interface instead of java.io.File

Absolutely. Whatever we do with regards to intermediate buffering will be built upon the initial refactoring of FileSystemStorageUnit to get rid of java.io.File (though there will still be a constructor FileSystemStorage(File) for backwards compatibility reasons).

[atomashpolskiy]

Wow, this whole topic is a hell of a lot of words. I personally am going to go and read something about mapped memory, because it might be that I just misunderstand the concept and how it works. Cheers

[ogregoire]

You're saying we're providing byte[] at various places in the code.
I like your attitude! No joking :)

Sorry, that's a deformation from French in which "we" can be both "us" and the undefined pronoun (for French-speaking guys, I'm speaking about "on", not "nous").

Regarding the rest, I discovered your repo through your HN post last week, I was impressed and wanted to contribute. I haven't had the time to understand all the design yet. I know how the BitTorrent protocol works in the big lines (b-encoding, distribution of chunks, etc). But I most certainly haven't delved in it like you did to develop this project. However I have some experience with nio, though it's not natural for me, I still prefer good ol' Input/OutputStream.

So indeed I might be naive on some topics, but I hope to help in the best of my knowledge.

[atomashpolskiy]

@ogregoire , man, come on, I'm glad that you're contributing, and I'm very thankful. I immensely enjoy our conversation and under no circumstances would like to seem "smarter" or whatever. In fact I'm sure as hell that you are much more knowledgeable than me, and I hope you will continue to share your knowledge. I just wanted this conversation to pave a road from A (where we are) to B (where we'd like to be), so I thought it would be beneficial to start a more in-depth discussion. And I totally imply that I might be wrong in what I'm saying, so you're welcome to correct and provide your own arguments! Peace!

[MorrisLaw]

Is this enhancement being worked on, or is it up for grabs?

[magJ]

Another advantage of using java.nio.file is that you can use an alternative FileSystem implementation like the in-memory filesystem jimfs

[atomashpolskiy]

@MorrisLaw , Working with files is completely isolated inside Storage, so it should not be hard to write an alternative version, making it a good candidate for contributing

[atomashpolskiy]

Also see #9 and the last comment about jimfs, might be worth looking into

[atomashpolskiy]

Additionally, the download itself can use java.nio as well to take the full power of multithreading.

This is a very good point. While network, message encoders/decoders and storage already use NIO channels and buffers, between those the data is transferred as a binary array (encapsulated in bt.protocol.Piece). It should very easy to switch to using ByteBuffer instead, so that the data is piped between storage and sockets without creating additional byte[] objects.

This can be done in parallel with #8 (buffering several subsequent blocks before flushing them to disk)

[MorrisLaw]

@ogregoire @atomashpolskiy Just updating everyone on the progress. I'm running into issues with the buildClient(int port) class in examples and tests are failing. Here are some of the changes so far. Any input on the branch I'm working on is also welcome.

https://github.com/MorrisLaw/bt/tree/performance/file-storage

[atomashpolskiy]

@MorrisLaw , I've looked through your changes. I appreciate your effort, but there is a couple of issues with your code. I understand that you're a beginner, so it's fine. Let me give you some tips:

1. You can't cast an object reference to some arbitrary type, just to the same type or a supertype (plus a few more specific cases), otherwise you'll get a ClassCastException in runtime. E.g.

// bt/cli/CliClient.java:119
Files.getFileStore((Path) options); // => java.lang.ClassCastException: bt.cli.Options cannot be cast to java.nio.file.Path

You might want to skim through the corresponding JLS Chapter 5. Conversions and Promotions, specifically section 5.1.6. Narrowing Reference Conversion.

Keep in mind that in most cases casting is bad and probably means that you're doing something wrong.

2. Some nullity checks in FileSystemStorageUnit don't make sense, e.g.

// bt/data/file/FileSystemStorageUnit.java:53
if (Files.createFile(file) == null) { // => always false
  ...
}

If you look inside these methods, you'll see that they always return the first argument (Path to file) or throw an exception.

3. Another issue with casting:

// bt/data/file/FileSystemStorageUnit.java:185
public long size() {
   ...
   return (long)Files.size(file); // => casting is redundant, Files.size(Path) already returns `long`
   ...
}

4. As a general rule you shouldn't (and per this issue don't need to) change public API. Some client code may depend on certain methods/constructors having specific signatures, and changing the signature means breaking the client code, e.g.

    /*
     * @since 1.0
     */
   // bt/data/file/FileSystemStorage.java:62
    public FileSystemStorage(File rootDirectory) { // => can't just change the parameter type to java.nio.file.Path
        ...
    }

This one is especially bad:

// bt/data/StorageUnit.java:42
ByteBuffer readBlock(long offset, int length); // => wrapping a byte array into buffer does not make much sense

Of course the API is not limited to methods only; it includes class and package names, thrown exceptions, behavior that is promised in the javadoc, even side effects. So in most cases it's preferable to only add new API without changing/removing the old one (if old API is obsolete, it should be marked as deprecated and noticed in upgrade notes, so that the client code can migrate safely before it's finally removed).

5. dependency-reduced-pom.xml is generated by maven-shade-plugin and should not be checked into git repository. It's my fault that I haven't added the corresponding rule to .gitignore, I've updated it in add949a.

To sum up: I suggest that you discard your changes and start over, rebasing onto the latest commit in master. Keeping the above comments in mind, you need to do just the following:

• add a new constructor FileSystemStorage(Path). Old constructor FileSystemStorage(File) should delegate object creation to the new constructor by calling this(rootDirectory.toPath())
• update FileSystemStorageUnit internals to use java.nio.files interfaces
• verify that all tests pass

[MorrisLaw]

Thank you for the thorough response !

…

On Aug 5, 2017 8:50 AM, "Andrei Tomashpolskiy" ***@***.***> wrote: @MorrisLaw <https://github.com/morrislaw> , I've looked through your changes. I appreciate your effort, but there is a couple of issues with your code. I understand that you're a beginner, so it's fine. Let me give you some tips: 1. You can't cast an object reference to some arbitrary type, just to the same type or a supertype (plus a few more specific cases), otherwise you'll get a ClassCastException in runtime. E.g. // bt/cli/CliClient.java:119Files.getFileStore((Path) options); // => java.lang.ClassCastException: bt.cli.Options cannot be cast to java.nio.file.Path You might want to skim through the corresponding JLS Chapter 5. Conversions and Promotions <https://docs.oracle.com/javase/specs/jls/se7/html/jls-5.html>, specifically section *5.1.6. Narrowing Reference Conversion*. *Keep in mind that in most cases casting is bad and probably means that you're doing something wrong.* 1. Some nullity checks in FileSystemStorageUnit don't make sense, e.g. // bt/data/file/FileSystemStorageUnit.java:53if (Files.createFile(file) == null) { // => always false ... } If you look inside these methods, you'll see that they always return the first argument (Path to file) or throw an exception. 1. Another issue with casting: // bt/data/file/FileSystemStorageUnit.java:185public long size() { ... return (long)Files.size(file); // => casting is redundant, Files.size(Path) already returns `long` ... } 1. As a general rule you shouldn't (and per this issue don't need to) change public API. Some client code may depend on certain methods/constructors having specific signatures, and changing the signature means breaking the client code, e.g. /* * @SInCE 1.0 */ // bt/data/file/FileSystemStorage.java:62 public FileSystemStorage(File rootDirectory) { // => can't just change the parameter type to java.nio.file.Path ... } This one is especially bad: // bt/data/StorageUnit.java:42ByteBuffer readBlock(long offset, int length); // => wrapping a byte array into buffer does not make much sense Of course the API is not limited to methods only; it includes class and package names, thrown exceptions, behavior that is promised in the javadoc, even side effects. So in *most* cases it's preferable to only *add* new API without changing/removing the old one (if old API is obsolete, it should be marked as deprecated and noticed in upgrade notes, so that the client code can migrate safely before it's finally removed). 1. dependency-reduced-pom.xml is generated by maven-shade-plugin and should not be checked into git repository. It's my fault that I haven't added the corresponding rule to .gitignore, I've updated it in add949a <add949a> . *To sum up*: I suggest that you discard your changes and start over, rebasing onto the latest commit in master. Keeping the above comments in mind, you need to do just the following: - add a new constructor FileSystemStorage(Path). Old constructor FileSystemStorage(File) should delegate object creation to the new constructor by calling this(rootDirectory.toPath()) - update FileSystemStorageUnit internals to use java.nio.files interfaces - verify that all tests pass — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG0LT5QX5ZgaiYiDpxzSYCTp6zPNJThCks5sVGT7gaJpZM4Orqid> .

[atomashpolskiy]

By the way, it turns out FileSystemStorageUnit already uses java.io.RandomAccessFile and java.nio.channels.FileChannel, so the performance is not a problem. What this issue is really about is switching to using generic java.nio.files.Path interface instead of java.io.File, which will let users use FileSystemStorage as a traditional file storage or an in-memory storage (e.g. via jims), depending on their needs. We might also want to switch to using jimfs-backed version in bt-tests to speed up things and remove the need to cleanup the test files after each test (as a bonus this will probably fix the Windows build).

[ogregoire]

There is no need to use RandomAccessFile. If one uses nio everywhere but a RAF when it's written on the disk, all the performance gain is lost...

Here's how to use nio's RAF:

MappedByteChannel channel = FileChannel.open(file, READ, WRITE).map(READ_WRITE, 0, size);
channel.position(12345);
while (buffer.hasRemaining()) {
  channel.put(buffer);
}

You can get a Path from a File using File::toPath.

Finally regarding the interface(s), indeed, you need to keep the existing, but you can create another method next to it with a default behavior:

interface Abc {
  void doSomething(File f);
}

Will become:

interface Abc {
  @Deprecated default void doSomething(File f) {
    doSomething(f.toPath());
  }
  void doSomething(Path p);
}

Then, of course, you need to rewrite all the code that use Abc, but that's part of the job for this ticket.

If needed, I can work on this issue, but not before next week.

Ideally the only java.io imports ever needed are java.io.IOException and java.io.UncheckedIOException.

[atomashpolskiy]

There is no need to use RandomAccessFile. If one uses nio everywhere but a RAF when it's written on the disk, all the performance gain is lost...

We do want to get rid of RandomAccessFile for API reasons, but don't expect any performance gains, because it's already used in combination with FileChannel, like this:

raf.seek(offset);
raf.getChannel().write(buffer);

Finally regarding the interface(s), indeed, you need to keep the existing, but you can create another method next to it with a default behavior...

Valid point, but we were discussing constructors ¯_(ツ)_/¯

[ogregoire]

Yeah, but I was mistaken (it's been a long time I worked with io in general), there is no need for a FileChannel, but for a MappedByteChannel (I edited my previous comment). And for that, it's rather bad to always create such a channel: since the channel will be loaded from disk everytime and therefore the performance lost. Keep only a MappedByteChannel, which will load the appropriate region upon need properly all by itself.

[atomashpolskiy]

@ogregoire , makes sense now, but this behavior is not always beneficial, e.g. when streaming media I want the data to be written to disk immediately. So it should be explicitly configurable by the user (either by some flag or better by instantiating a different implementation class). Common parts can be extracted into an abstract supertype

[ogregoire]

You can simply use force() after each "write" command. That will flush the changes to the disk, but keep that region in memory. Really, it's that simple. Alternatively, you can open the FileChannel in DSYNC mode.

[atomashpolskiy]

I get that but my point was that whether flush is eventual or immediate should be conditional and based on user's preference and/or current mode of operation.

There is also a problem of how much memory can/should be used. We can't just map all data into memory, so there should be some centralized management of what is loaded and what is discarded at each given moment.
Finally, this is also a design issue because StorageUnit is supposed to be just that - a storage, not a flushable cache.

Initially I was thinking about buffering data at the I/O worker level, which has (indirect) access to the storage and can decorate it as needed. Additionally it has the statistics of when/how often each piece is requested by peers, so it can act like a full-fledged cache, re-using the same mapped buffers.

[ogregoire]

I guess there must be some misunderstanding here: it's entirely possible to have a storage that works as you expect using nio and channels and mappedbytebuffer. The thing is that currently you always go through a byte[], which is far from efficient. The idea is to get rid of this temporary buffer by using the buffers we get from the MappedByteBuffer.

Regarding memory, it's a mappedbytebuffer, so it's memory that's directly mapped to the drive. No system calls. I don't see why it should be a user-preference: the user will want it to be written to disk, according to the best way to handle the disk. This is what is done by default.

Know what? I'll fork, put my changes in, and show them so if you like that, let's merge, if not, maybe use it as a base? But as I said, I can't start before Monday at the earliest. Oh, and I can't guarantee I'll be able to answer anymore today or tomorrow.

[MorrisLaw]

@ogregoire If you already have a solution maybe you can complete the ticket Monday. Otherwise as far as @atomashpolskiy original suggestion, itseemed like something I can achieve sometime this weekend. But reading through these comments, it looks like it's going to be more involved.

Is this still the case:
What this issue is really about is switching to using generic java.nio.files.Path interface instead of java.io.File

Or is there a different issue or just different opinions on how to fix the same issue ?

[MorrisLaw]

Here's another crack at the enhancement. Is this acceptable ?
https://github.com/MorrisLaw/bt/tree/enhancement/storage-nio

[MorrisLaw]

Actually some tests failed, so I'm looking into that now. I only tested bt-core at first, then when running all of them it failed the build.

Tests run: 249, Failures: 0, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Bt ................................................. SUCCESS [  0.024 s]
[INFO] Bt BEncoding ....................................... SUCCESS [  1.427 s]
[INFO] Bt Core ............................................ SUCCESS [  1.115 s]
[INFO] Bt HTTP Tracker Client ............................. SUCCESS [  0.447 s]
[INFO] mldht - Bittorrent Mainline DHT library ............ SUCCESS [  2.690 s]
[INFO] Bt Tests ........................................... FAILURE [  1.932 s]
[INFO] Bt DHT ............................................. SKIPPED
[INFO] Bt libmldht adapter ................................ SKIPPED
[INFO] Bt CLI Launcher .................................... SKIPPED
[INFO] Examples ........................................... SKIPPED
[INFO] JaCoCo aggregate report ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.377 s
[INFO] Finished at: 2017-08-06T17:44:54-04:00
[INFO] Final Memory: 26M/276M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on project bt-tests: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/morrislaw/Projects/bt/bt-tests/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :bt-tests

[MorrisLaw]

Ahaha nvm, it looks like I ran mvn test in the directory without the pom file. Now it all passes.

Tests run: 249, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.016 s
[INFO] Finished at: 2017-08-06T17:50:55-04:00
[INFO] Final Memory: 14M/250M
[INFO] ------------------------------------------------------------------------

[MorrisLaw]

Still working on errors, it appears that I'm failing the CI tests.

[ogregoire]

@MorrisLaw Just a small note from a very superficial review: you used Files.createDirectory(parent) where you should actually use Files.createDirectories(parent). The difference between the two is that if you call the first method with a/b/c when only a exists, you'll get an error because b doesn't exist, where with the second method, you'll get c and all its missing parents created without troubles.

[MorrisLaw]

Thank you for spotting that. I will refactor the code.

…

On Aug 6, 2017 6:55 PM, "Olivier Grégoire" ***@***.***> wrote: @MorrisLaw <https://github.com/morrislaw> Just a small note from a very superficial review: you used Files.createDirectory(parent) where you should actually use Files.createDirectories(parent). The difference between the two is that if you call the first method with a/b/c when only a exists, you'll get an error because b doesn't exist, where with the second method, you'll get c and all its missing parents created without troubles. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG0LT8HQVr65b8AtWe4AZMzidAnn5w3aks5sVkRdgaJpZM4Orqid> .

[ogregoire]

Okay, I reviewed the code, and added several comments (see list with a comment icon).

I still do think it's better to use MappedByteBuffer for performance, (compared to a simple, unmapped FileChannel). But to be honest, that would require some refactoring in the StorageUnit interface to use it optimally. With the interface as is, what you wrote, @MorrisLaw is the best we can do.

[atomashpolskiy]

@ogregoire , I'm genuinely interested, what you have in mind, and I'm all for experimentation, as long as it's not the default behavior.

[ogregoire]

Well, basically there's currently a design flaw. It's a subtle one, so what you did was actually good, but in case of using ByteBuffers, it's a bit of a premature optimization.

So I would suggest using such an interface (comments removed for brievity):

public interface StorageUnit extends AutoCloseable {

	ByteBuffer readBlock(long offset, long size);

	void writeBlock(ByteBuffer buffer, long offset);

	long capacity();

	long size();
}

Obviously I deleted the byte[]. Why is that? Because it's trivial to wrap a byte[] into a ByteBuffer and it's also trivial to get a byte[] from a ByteBuffer.

Okay, now why in the world would I make the read operation return a ByteBuffer everytime instead of filling the same byte[] again and again? Well, this is where the flaw is in the original design: it was assumed that creating your own ByteBuffer and reusing it again and again will provide you with improved performance regarding memory. It makes a lot of sense... Until we know how ByteBuffer works internally, and more specifically the MappedByteBuffer returned by FileChannel.map.

Well, if we keep the same ByteBuffer for the whole application (or part of it), it basically mean that we will copy, and copy, and copy. While getting the ByteBuffer from the memory-mapped file will allow us to skip at the very least one copy which can be expensive in the long term.

The important part of ByteBuffer's documentation is the following (slightly edited, emphasis mine):

Direct vs. non-direct buffers

A byte buffer is either direct or non-direct. Given a direct byte buffer, the Java virtual machine will make a best effort to perform native I/O operations directly upon it. That is, it will attempt to avoid copying the buffer's content to (or from) an intermediate buffer before (or after) each invocation of one of the underlying operating system's native I/O operations.
...
A direct byte buffer may also be created by mapping a region of a file directly into memory. An implementation of the Java platform may optionally support the creation of direct byte buffers from native code via JNI. If an instance of one of these kinds of buffers refers to an inaccessible region of memory then an attempt to access that region will not change the buffer's content and will cause an unspecified exception to be thrown either at the time of the access or at some later time.

This means that if we use the byte buffers that we get from FileChannel.map, the performance can be boosted. Especially when dealing with large, long-open files. Oh, by the way, what is mostly shared through torrents? Yes: large files. And they're open for a long time when they're being downloaded or uploaded.

Concretely, this means that we will provide directly the mapped file-region to the network buffer. Meaning that we skip Java's heap entirely by not copying byte[] into other temporary byte[] more than absolutely necessary.

[atomashpolskiy]

Ok, we definitely need more details here. As already mentioned in this topic, FileChannel class has a map() method, that is full of black magic and trickery (e.g. trying to gracefully handle OutOfMemory errors by asking JVM to perform garbage collection to trigger deallocation of previously allocated native memory buffers... you get the idea), but really boils down to a native method and mmap system call.

What mmap does is create a file mapping in the calling process's address space, more specifically in the native memory of the process (off-Java-heap). Upon creation the entire mapping is initialized by kernel with page faults, each of which will be resolved upon first access (i.e. lazy-loading data from disk).

The FileChannel.map() method returns a MappedByteBuffer. It's not a real buffer per se, but rather an abstract super-type for native memory (off-Java-heap) buffers. It provides a few utility methods, most importantly isLoaded(), load() and force(), all of which are backed by corresponding native methods and boil down to mincore, madvise and msync system calls.

The actual type of the returned object is DirectByteBuffer, which is a flavor of Direct-X-Buffer. This is the only implementation of a native memory buffer (comes in different flavors, depending on the type of content, hence X in template name). It can be constructed in two ways: (1) either by requesting a buffer of a given capacity (e.g. by calling ByteBuffer#allocateDirect), which ends up in malloc, (2) or by providing an existing "buffer" in the form of a virtual address of a pre-allocated memory region (which is precisely what FileChannel does: return the address from the call to mmap).

Considering Java, there are two peculiarities to keep in mind when working with DirectByteBuffer:

1. It implements the Buffer type, which means its' capacity is restricted to Integer.MAX_VALUE (2^31, or ~2GB)
2. Allocated native memory can not be freed deterministically (i.e. on-demand), but only as part of the GC. With long-living large-sized buffer objects this can become a problem, because they are likely to be promoted to the parts of the heap, that are less frequently processed by GC.

Here are the implications:

1. Creating a file mapping for a file of size N requires equal amount of RAM. If there's less physical memory than N, then the kernel will manage what parts of the file are resident in memory at each given moment (based on what pages are accessed), performing page-ins for requested pages and discarding unused pages. I think it's safe to say that memory mapping makes more sense when the whole file fits into memory.
2. Given that, in general case, we don't know beforehand, what is the total size of torrent's data, it's not reasonable to map all data into memory. This will clearly cause OOM at some point.
3. While memory-mapping a file definitely makes sense when we already have the file (i.e. when seeding), it in fact can negatively affect the performance when we're downloading the file. This is because each write may trigger a page-in from disk, even if there is no data yet (precise ratio of writes to page-ins depends on the size of the block being written, the size of memory and disk pages and whether the corresponding pages have already been loaded).

Therefore, while it may be beneficial to naively memory-map all files in some cases, more specifically:

• when total size of files is not very large (which size is "large" is specific to every given environment);
• when there are no particularly large files (> 2GB);
• when we already have the data and are just seeding;

I don't think it's as simple as you want it to be in general case.

Brief write-up on copy vs mmap by Linus Torvalds.

[atomashpolskiy]

With regards to this:

Concretely, this means that we will provide directly the mapped file-region to the network buffer. Meaning that we skip Java's heap entirely by not copying byte[] into other temporary byte[] more than absolutely necessary.

You are being overly optimistic about the code :) There are several layers between storage and network, and some of them currently rely on passing each block as byte[].

[ogregoire]

There are several layers between storage and network, and some of them currently rely on passing each block as byte[].

I'm aware of that and I'm not saying the change is trivial or even simple. What I highlighted is only the root for all work.

You have several concerns and I'm aware of them. I was aware of them before you raised them because I had them in the past. as well

You're saying we're providing byte[] at various places in the code. I saw that. The Range interface is one example and I see it's used for different purposes, one of them is computing the hash. In that particular case, why is it transmitting a byte[] and not a ByteBuffer? Because the ByteBuffer must be reused? Well, I never see any use of the mark() and reset() methods which exist for that kind of use-cases, for instance.

Switching to nio is no trivial case and must be done application-wide. There are a lot of changes and perhaps it might be necessary to rethink some ways the library works at its core.

It might be good for a further major version, and not for minor one.

[atomashpolskiy]

why is it transmitting a byte[] and not a ByteBuffer

It won't be a problem to add methods that work with buffers rather than byte arrays. I think it was kind of an unconscious decision to use byte[], because messaging uses byte[] to represent a BitTorrent's piece. So, the problem is mostly in the messaging layer.

You're saying we're providing byte[] at various places in the code.

I like your attitude! No joking :)

Switching to nio is no trivial case and must be done application-wide. There are a lot of changes and perhaps it might be necessary to rethink some ways the library works at its core.

As I've said previously, networking already uses NIO... The problem is that working with storage (reads and writes) is asynchronous in respect to networking. In short, current design is based on the following:

1. Messaging is single-threaded; consuming and producing messages from/to peers is done in a single thread in MessageDispatcher
2. Storage I/O is async and can be parallelized; it uses a single-threaded executor for now, but I think it's safe to use multiple threads there

I agree that piping data between network <----> storage is non-trivial in such design. But it also has its' pros.

[atomashpolskiy]

The Range interface is one example and I see it's used for different purposes

Most importantly it provides a mapping overlay, that relieves us from thinking about files and instead provides a view of torrent's data in the form of a contiguous byte array.

[ogregoire]

Hmm, I guess we weren't on the same page: I'm still in ;)

I'll try to work on a prototype where I introduce those changes gradually, if that's okay. Probably in my own branch or something for quite a while. We'll see.

[atomashpolskiy]

Sure! I also have some other global tasks in mind, which would require research and quite possibly re-thinking approach and APIs. I think you could do a great job on these, so if you're interested, I'll try to dump my thoughts in a couple of tickets