magicdgs / jsr203-http Goto Github PK
View Code? Open in Web Editor NEW**Under development** HTTP/S FileSystem provider for Java NIO.2
License: BSD 3-Clause "New" or "Revised" License
**Under development** HTTP/S FileSystem provider for Java NIO.2
License: BSD 3-Clause "New" or "Revised" License
For both the javadoc and the log, the HttpFileSystem
methods should have a better documentation, because we are not going to support close yet.
If using git for Windows, the EOL is converted and thus tests relying on reading all bytes from a file are not working. There are two possibilities to fix this problem:
In addition, we should test this on the CI (see #38)
Once the first point release is out, we should implement some integration tests to work with:
java.nio.file.Paths
java.nio.file.Files
If Git for linux is configured to convert EOL to CRLR, then the tests in the current master branch fail (see #39), because some of the tests check at the byte level the content of the file and thus there is a mismatch between the hosted file in GitHub pages and the local one.
We should add recommendations to clone the repository in Windows to make tests not failing.
By now, all the FS will be always open and a no-op for closing. We should implement also the getPath
method for having a functional FS (requires #15).
Other methods will be implemented later.
Closing the FS might be useful to get rid of all connections with an HTTP/S host, but that requires that we track all the open connections and also remove from the map of the FSP.
Hi @magicDGS,
I hope things are going well! I'm looking at adding native support for URL's to GATK and I was going to use your library as a starting point. Do you have any interest/availability in continuing work on this? I'm going to fork the project and start doing some work on it. If you're interested I'm happy to work with you together on it, or I can just do the work on my own.
Requires #54
We should decide if extra checking is required for it, because most likely it won't be used out of the package.
Requires a functional HttpFileSystem
(#19 and #22). For this being functional, we need to implement two parts (might be two different PRs):
getFileSystem
/ newFileSystem
: should create a FileSystem
for each host in a Map and re-usegetPath
: should get the FS associated with the host from the map and construct the path from itnewByteChannel
returning a URLSeekableByteChannel
(and failing for a writable channel), using Path.toUri().toUrl()
checkAccess
for checking for existance of files (using a HEAD request and checking HttpURLConnection.HTTP_OK
.This is the minimal for working with http paths that it is required for a functional version from my point of view.
Maybe cache this values in a volatile variable (after #15) might be important for some users (e.g., printing Path in logs or using them in a map).
Instead of the URL
path (current draft), we should use URI
to be sure that this is not happening: http://blog.markfeeney.com/2010/11/java-uri-vs-url.html
It would be good if the library could be used with the Java NIO FileSystems API. See:
https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystems.html
please try following code with your library:
(after creating the usual "java.nio.file.spi.FileSystemProvider" file in META-INF)
final URI uri = new URI("http://localhost:8000/"); final Path rootPath = FileSystems.getFileSystem(uri).getPath("/test/test.txt");
it will crash with a "java.nio.file.FileSystemNotFoundException".
Could you please fix following code in your "AbstractHttpFileSystemProvider":
@Override public final HttpFileSystem getFileSystem(final URI uri) { final HttpFileSystem fs = fileSystems.get(checkUri(uri).getAuthority()); if (fs == null) { throw new FileSystemNotFoundException("URI: " + uri); } return fs; }
it should be like following to work:
@Override public final HttpFileSystem getFileSystem(final URI uri) { final HttpFileSystem fs = fileSystems.get(checkUri(uri).getAuthority()); if (fs == null) { try { return newFileSystem(uri, new HashMap<>()); } catch (final IOException e) { throw new RuntimeException(e); } } return fs; }
thanx!, greetings.
Including:
We should implement start/end with methods for other HTTP/S paths. We should also decide, although it can be later and moved to a different issue, if we want the following behavior for the String
version:
InvalidPathException
if the String cannot be converted into a Path
, but this does not allow to return true for HttpPath("http://example.com/file.txt").endsWITH("file.txt")
. Nevertheless, if using the string "file://file.txt" it should throw because there is a mismatch with the providers. I don't know how much strict we should follow the contract here...HttpPath("http://example.com/file.txt").startWith("file.txt")
, this method should fail with InvalidPathException
because the scheme and authority aren't provided and thus it cannot be tested. Maybe we can relax this by returning false by anything that cannot be converted into an HttpPath
with the instance FileSystem
to mimic the behaviour of endsWith described before.To avoid the NPE and throw a more informative exception (e.g., IllegalArgumentException
or IllegalStateException
.
With the following fields:
HttpFileSystem
originating the URL (to get the separator and the scheme from the provider)byte
array (to operate over the paths) with offsets array (int[]
)The implementation of the path part (byte[]
) should be similar to other implementations of java.nio.Path
(e.g., UnixPath or HadoopPath).
The methods implemented for this simple HttpPath
should be:
getFileSystem()
toUri()
toString()
/ hashCode
/ equals
/ compareTo
isAbsolute()
/ toAbsolutePath()
- should return always true
/this
in the first implementation and we shouldn't allow relative HTTP/S pathsThe rest should still throw UnsupportedOperationException
I should claim the com.github.magicdgs
id in Maven Central (this will be useful for other projects too) to upload the artifacts for v0.1.0.
More info here: https://central.sonatype.org/pages/ossrh-guide.html
We should implement a relative HttpPath
to get parts of the Path (file name, directory name, etc), and also iterate over paths. This support should include:
int[]
) for easier splitting the directory (as in other path implementations)isAbsolute()
returning false for this pathstoAbsolutePath()
should return an absolute path sited on the rootgetFileName
getParent
getNameCount
getName(int)
subpath
iterator
- using getName(int)
In addition, this relative Paths should fail for constructing an URI (no relative paths there), with a meaningful error.
This requires that the map holding the FS in the FSP should be thread-safe.
We should check if there is any exception when establishing a connection with an URL that throws when the file does not exists. Here is a tracker for that. By now, we only catch the UnknownHostException
, but it can be other reasons to say that the file does not exists.
It only remains for testing in #15
Currently java8
Instead of a FS for each authority, as currently designed, it will be nice to have one por page to being able to retrieve "root" directories, etc.
It requires:
HttpPath
without authority, but stored in the FS fieldFileSystem
for each authority (base URI
with only authority and protocol)FileSystemProvider
keeping all the generated FileSystems
.This might be better before going further in the direction of a single FS per protocol.
Currently, I think that it is using the default (java6)
This will be better for the community
And design an issue/pull request template
In #48, we change our plugin to git versioning to com.palantir.git-version
; nevertheless, this does not handle the tagging system, nor help with the release proces (thus, it requires some scripting). There is a plugin that could handle this: https://github.com/ajoberstar/reckon
We should explore that option and add it before we move forward to version 0.1.0
Using Circle-CI, where if I remember correctly I have a free account for public projects.
Although we are always normalizing duplicated slashes, we should also implement other non-trivial normalization. Some examples are in https://en.wikipedia.org/wiki/URL_normalization
If a path such http://example.com/foo/bar
is tested for endsWith("/bar")
, the contract of Path::endsWith says that it should return false; in our case, it returns true.
Could I get your email address or something to ask you about the Hapflk tool? I'm so sorry for issuing it here
Implement the following methods to be able to resolve paths and get an absolute path from a relative ones:
resolve
and resolveSibling
relativize
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.