tahoe-lafs / magic-folder Goto Github PK

Tahoe-LAFS-based file synchronization

License: Other

Python 99.80% Makefile 0.20%

linux macos networking privacy python27 security synchronization tahoe-lafs twisted windows

magic-folder's Introduction

Free and Open Decentralized Data Store

Tahoe-LAFS (Tahoe Least-Authority File Store) is the first free software / open-source storage technology that distributes your data across multiple servers. Even if some servers fail or are taken over by an attacker, the entire file store continues to function correctly, preserving your privacy and security.

Table of contents

About Tahoe-LAFS
Installation
Issues
Documentation
Community
Contributing
FAQ
License

💡 About Tahoe-LAFS

Tahoe-LAFS helps you to store files while granting confidentiality, integrity, and availability of your data.

How does it work? You run a client program on your computer, which talks to one or more storage servers on other computers. When you tell your client to store a file, it will encrypt that file, encode it into multiple pieces, then spread those pieces out among various servers. The pieces are all encrypted and protected against modifications. Later, when you ask your client to retrieve the file, it will find the necessary pieces, make sure they haven’t been corrupted, reassemble them, and decrypt the result.

The image is taken from meejah's blog post at Torproject.org.

The client creates pieces (“shares”) that have a configurable amount of redundancy, so even if some servers fail, you can still get your data back. Corrupt shares are detected and ignored so that the system can tolerate server-side hard-drive errors. All files are encrypted (with a unique key) before uploading, so even a malicious server operator cannot read your data. The only thing you ask of the servers is that they can (usually) provide the shares when you ask for them: you aren’t relying upon them for confidentiality, integrity, or absolute availability.

Tahoe-LAFS was first designed in 2007, following the "principle of least authority", a security best practice requiring system components to only have the privilege necessary to complete their intended function and not more.

Please read more about Tahoe-LAFS architecture here.

✅ Installation

For more detailed instructions, read Installing Tahoe-LAFS.

Once tahoe --version works, see How to Run Tahoe-LAFS to learn how to set up your first Tahoe-LAFS node.

🐍 Python 2

Python 3.8 or later is required. If you are still using Python 2.7, use Tahoe-LAFS version 1.17.1.

🤖 Issues

Tahoe-LAFS uses the Trac instance to track issues. Please email jean-paul plus tahoe-lafs at leastauthority dot com for an account.

📑 Documentation

You can find the full Tahoe-LAFS documentation at our documentation site.

💬 Community

Get involved with the Tahoe-LAFS community:

Chat with Tahoe-LAFS developers at #tahoe-lafs channel on libera.chat IRC network or Slack.
Join our weekly conference calls with core developers and interested community members.
Subscribe to the tahoe-dev mailing list, the community forum for discussion of Tahoe-LAFS design, implementation, and usage.

🤗 Contributing

As a community-driven open source project, Tahoe-LAFS welcomes contributions of any form:

Before authoring or reviewing a patch, please familiarize yourself with the Coding Standard and the Contributor Code of Conduct.

🥳 First Contribution?

If you are committing to Tahoe for the very first time, it's required that you add your name to our contributor list in CREDITS. Please ensure that this addition has it's own commit within your first contribution.

🤝 Supporters

We would like to thank Fosshost for supporting us with hosting services. If your open source project needs help, you can apply for their support.

We are grateful to Oregon State University Open Source Lab for hosting tahoe-dev mailing list.

❓ FAQ

Need more information? Please check our FAQ page.

📄 License

You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0, or at your choice, any later version. (You may choose to use this package under the terms of either license, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL for the terms of the Transitive Grace Period Public Licence, version 1.0.

See TGPPL.PDF for why the TGPPL exists, graphically illustrated on three slides.

magic-folder's People

Stargazers

Watchers

Forkers

vu3rdd meejah altendky miguelramosfdz sajith purplesparkle privatestorageio crwood

magic-folder's Issues

Set up macOS CI

Participant Restores Old Version

As a participant, I want to restore (for all participants) a version of the file (that still exists) as it existed at some point previously so that I can replace the current one.

Logs interfere with CLI tools

Whatever is happening with logging in magic-folder interfere with many of the "user oriented" command-line tools. E.g. here's the output from magic-folder status:

(venv) meejah@buyan:~/work/leastauthority/src/magic-folder$ magic-folder --node-directory ../tahoe-lafs/testgrid/alice/ status 
2020-04-02T12:59:27-0600 [twisted.application.runner._runner.Runner#info] Starting reactor...
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041668>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041668>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccfdda28>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccfdda28>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffb9b0>, <HostnameEndpoint 0.0.0.0:9889>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffb9b0>, <HostnameEndpoint 0.0.0.0:9889>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041578>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71cd041578>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffbd70>, <HostnameEndpoint 127.0.0.1:8890>)

Magic-folder status for 'default':

Local files:
  lumière.txt (625 B): good, version=0, created 37 days ago

Remote files:
  bob's remote:
  alice's remote:
  lumière.txt (625 B): good, version=0, created 37 days ago



2020-04-02T12:59:27-0600 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f71ccffbd70>, <HostnameEndpoint 127.0.0.1:8890>)
2020-04-02T12:59:27-0600 [-] Main loop terminated.

The "deprecations" CircleCI job fails

For example, on https://circleci.com/gh/LeastAuthority/magic-folder/3439, it encountered this exception:

Traceback (most recent call last):
  File "/tmp/magic-folder.tox/upcoming-deprecations/bin/flogtool", line 6, in <module>
    from foolscap.logging.cli import run_flogtool
  File "/tmp/magic-folder.tox/upcoming-deprecations/lib/python2.7/site-packages/foolscap/logging/cli.py", line 4, in <module>
    from twisted.python import usage
  File "/tmp/magic-folder.tox/upcoming-deprecations/lib/python2.7/site-packages/twisted/__init__.py", line 16, in <module>
    "This version of Twisted is not compatible with Python 3.4 "
Exception: This version of Twisted is not compatible with Python 3.4 or below.

Twisted trunk@HEAD has dropped Python 2.7 support entirely now. Thus, it will no longer be possible to test against development snapshots of Twisted to watch for deprecations.

The smallest change to fix this would be to remove the CI configuration that installs Twisted trunk@HEAD into this test configuration.

missing import in the status module

status module is missing the import of readBody function from twisted.web.client.

The documentation, particular for configuration, presents Magic-Folder as part of Tahoe-LAFS itself

It should be updated to reflect the move to a stand-alone project.

The towncrier configuration still refers to tahoe-lafs

The cli helper should return more than just stdout

In returning only stdout or raising an exception, it makes it more difficult to write more sophisticated tests for behaviors of the program under test.

add generated file paths into .gitignore

git status shows a lot of generated data that needs to be ignored for version control.

Make sure all test coverage from tahoe-lafs got imported into this repository

See discussion at https://github.com/tahoe-lafs/tahoe-lafs/pull/695/files/25edce62576b14bcc14e2d5f66aca23462c55a7c#diff-16ef285a0652ba76b1d929d902e180e6

The "PyPy" CI job fails

The job fails with the message below. Possible fix is to upgrade the openssl version?

pypy27-coverage inst: /tmp/magic-folder.tox/.tmp/package/1/magic_folder-0.0.0.post288.zip
ERROR: invocation failed (exit code 1), logfile: /tmp/magic-folder.tox/pypy27-coverage/log/pypy27-coverage-3.log
================================== log start ===================================
Traceback (most recent call last):
  File "/usr/local/lib-python/2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib-python/2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/__main__.py", line 16, in <module>
    from pip._internal import main as _main  # isort:skip # noqa
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/main_parser.py", line 12, in <module>
    from pip._internal.commands import (
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
    from pip._internal.commands.completion import CompletionCommand
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
    from pip._internal.cli.base_command import Command
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/cli/base_command.py", line 20, in <module>
    from pip._internal.download import PipSession
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_internal/download.py", line 15, in <module>
    from pip._vendor import requests, six, urllib3
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_vendor/requests/__init__.py", line 97, in <module>
    from pip._vendor.urllib3.contrib import pyopenssl
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/crypto.py", line 15, in <module>
    from OpenSSL._util import (
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 205, in <module>
    _verify_openssl_version(Binding.lib)
  File "/tmp/magic-folder.tox/pypy27-coverage/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 169, in _verify_openssl_version
    "You are linking against OpenSSL 1.0.1, which is no longer "
RuntimeError: You are linking against OpenSSL 1.0.1, which is no longer supported by the OpenSSL project. You need to upgrade to a newer version of OpenSSL.

Factor introducer/storage node setup into a reusable Fixture

ListMagicFolder does a bunch of work to set up an introducer and a client w/ storage enabled. This logic can be factored out of the TestCase into a Fixture which will be more easily reused by other TestCase subclasses (or which will at least take some clutter out of ListMagicFolder).

refactor leave command

Eliminate no_network from magic_folder.test.cli.test_magic_folder.CreateMagicFolder

child of #22

Enable CI

Re-arrange the repository to make more sense as a stand-alone

The repository structure continues to reflect the structure of the magic-folders implementation as it existed inside Tahoe-LAFS. Without all of the rest of Tahoe-LAFS around it, this structure makes less sense. Consolidate it somewhat, removing redundancies that exist in the path hierarchies and the extremely sparse directories which now contain only one or two files.

Files / modules which could use a better name (suggested "better name" too, but maybe there's even-better names):

magic_folder/scripts/magic_folder_cli.py -> magic_folder/cli.py (#49)
magic_folder/frontends/magic_folder.py -> magic_folder/magic_folder.py (#70)
magic_folder/web/magic_folder.py -> .magic_folder/web.py (#69)
others?

Sometimes Hypothesis tests fail on CI because of the Hypothesis deadline

The Hypothesis deadline: https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.deadline

The CI execution environment is shared. We can't make any guarantees about how long our tests will take run (more generally, none of Linux, macOS, and Windows are real-time operating systems and Python is not a real-time programming language so it's hard to argue that you can ever guarantee how long a test will take to run in any supported environment, but I digress...).

Fortunately the Hypothesis deadline can be disabled. Do this for all CI.

Set up Windows CI

The CircleCI badge image is broken

Capitalization counts

Eliminate no_network from magic_folder.test.cli.test_magic_folder.ListMagicFolder

Child of #22

Participant Deletes File

As a participant, when another participant deletes a file, I want that file to be removed from my folder without manual action on my part so that responsibility for cleaning up obsolete files in the folder can be shared across the entire team.

Integration test improvements

It would be useful to use some of the "fixtures" currently defined in conftest.py as helpers for individual tests; putting most of the "real" work into a non-conftest.py location can facilitate this.

Participant Sees All History of Restored Files

As a participant, after I restore a deleted file, I want to be able to see the history of that file, including the history prior to its deletion, so that a mistaken choice to delete a file does not prevent me from seeing history in all of the cases I would normally want to look at history.

This more or less falls out of the snapshot design and the list-all-snapshots API.

Refactor magic-folder cli command handling

It would be nice if each command like create, invite, join, status etc are split into multiple modules and moved closer to the main magic-folder module than live in the scripts directory. We use Tahoe-LAFS web API whereever possible and file tickets against Tahoe-LAFS for any missing APIs.

Splitting this into multiple tickets.

magic_folder.test loads allmydata.test which loads conflicting Hypothesis profiles

magic_folder.test defines and loads a Hypothesis profile. Our CI configuration specifies a good profile to use for CI conditions.

Unfortunately allmydata.test does the same thing and our CI configuration does not specify a good profile fir it to use. allmydata.test loads second so its choice wins.

A solution would be to stop loading allmydata.test. This is probably a good idea anyway. Fortunately we make very light use of it so this is probably feasible.

Related to #43

Participant Restores Deleted Files

As a participant, I want to restore (either for all participants or only for myself) a version of the deleted file as it existed at some point previously so that I can restore it.

Most likely same implementation tasks as for #92

cli cleanup: one way to specify node-dir

There should be just one way to specify the node-dir.

Refactor invite

Refactor invite command and move it closer to the main magic-folder module.

Part of #50

Coverage is only collected from allmydata package

Our code is in the magic_folder package, though.

macOS CI doesn't upload coverage results

It collects coverage but doesn't publish it. This results in a coverage report that makes all macOS-specific codepaths appear uncovered.

CentOS 7 builder fails

We need CentOS 8.

Test our own inotify module instead of the one from Tahoe-LAFS

magic_folder/test/test_inotify.py imports allmydata.frontend.magic_folder.get_inotify_module so it tests the inotify in Tahoe-LAFS instead of ours.

The NixOS CI job builds spends too much time building too many things

It builds all of our self-packaged dependencies on every build. Instead, it could cache them.

Add comprehensive magicpath tests

many unit-tests still running "tahoe magic-folder"

Several unit-tests are still running tahoe magic-folder * commands instead of magic-folder * commands and are now broken because of tahoe magic-folder's recent removal from tahoe.

"magic-folder invite" doesn't check for existing user

The only hint that you've already got a user if you e.g. re-run the same magic-folder invite command twice in a row is:

2020-04-02T16:57:25-0600 [stdout#info] magic-folder: You cannot overwrite a directory with a file

We should check if the new directory entry that will be created is already used before doing that work. Looking in the "_invite()" code it uses tahoe_mv() to do this (which seems .. odd .. but might just be a naming thing?)

`magic-folder run` doesn't exit on listen error

If the magic-folder daemon fails to listen on its web-port, an exception is printed but the program keeps running (and logs "successful startup"). Magic-folder should exit if this fails.

(Also, I'm not sure that the --web-port should be optional? It currently defaults to tcp:9889. Maybe a better default would be "whatever is in magic-folder.url if that file exists in the node-dir" or just not having a default at all (i.e. it's a required option).

tahoe changes break magic-folder

The "remove magic-folder" PR that got merged into Tahoe recently has I think broken us (because it removes eliotutil). See, e.g., https://circleci.com/gh/LeastAuthority/magic-folder/4687

Switch CI to Tahoe-LAFS master

The branch it was pointing to has been merged and deleted.

--web-port should be required

Currently, --web-port has a default value of tcp:9889. This option should be non-optional -- the user must think about where they want to listen (and anyway a public / non-local port isn't a good default). Another reason is that trying to run two copies on the same machine will fail (well, the second one will fail silently because of #67).

Refactor join command

Coverage isn't collected or reported to codecov

The test suite jobs are running the testenv environment and not the coverage environment. No coverage is collected or reported by them.

Eliminate no_network from magic_folder.test.cli.test_magic_folder.StatusMagicFolder

Child of #22

refactor create command

Make test case temporary paths beneath _trial_temp

Trial runs tests beneath a _trial_temp directory it creates. This conveniently collects test-generated artifacts in one place instead of smearing them across, say, the project source directory or your home directory or /tmp. As part of this trial's mktemp helper also creates temporary paths beneath this directory instead of in /tmp.

Make our mktemp-alike API do something similar.

magic_folder.test.no_network should no longer be required

The unit test suite doesn't need to set up a "no network" Tahoe-LAFS grid and the integration tests don't use this approach (preferring to set up a network-using grid).

This module should go and anything that is still using it is bogus and needs to be fixed.

This looks like it is too large to do all at once so here are some sub-tasks:

magic_folder.test.cli.test_magic_folder.ListMagicFolder - #30
magic_folder.test.cli.test_magic_folder.StatusMagicFolder - #33
magic_folder.test.cli.test_magic_folder.CreateMagicFolder - #47
magic_folder.test.test_magic_folder.RealTest
magic_folder.test.test_magic_folder.RealTestAliceBob
magic_folder.test.test_magic_folder.MockTest
magic_folder.test.test_magic_folder.MockTestAliceBob

Flaky test: magic_folder.test.test_status.StatusTests.test_unknown_magic_folder_name

Traceback (most recent call last):
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/twisted/internet/defer
.py", line 151, in maybeDeferred
    result = f(*args, **kw)                                                                                           
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/testtools/testcase.py"
, line 723, in _run_test_method                                                                                            return self._get_test_method()()                                                                                     File "/home/meejah/work/leastauthority/src/magic-folder/src/magic_folder/test/test_status.py", line 133, in test_unkn
own_magic_folder_name
    folder_names(),
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", l
ine 1071, in wrapped_test                                                                                             
    state.run_engine()
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", l
ine 789, in run_engine
    info.__expected_traceback,                             
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", line 656, in execute_once                                                                                               
    % (test.__name__, text_repr[0])                         
  File "/home/meejah/work/leastauthority/src/magic-folder/venv/local/lib/python2.7/site-packages/hypothesis/core.py", line 856, in __flaky
    raise Flaky(message)                                                       
hypothesis.errors.Flaky: Hypothesis test_unknown_magic_folder_name(self=<magic_folder.test.test_status.StatusTests.test_unknown_magic_folder_name id=0x7f024dca7e50>, folder_names=[u'\x0e\U000aea6e\x1d',                                   
 u'\U000cd527\x05',                                                    
 u'\U0003b33c&\u4bbe',
 u'\x07\x02\U000c4c63\x08',
 u'\U00087e93',                   
 u'\x17\U00038411',                                                                                                    
 u'\U0007195e+',                
 u'\U00015a2f\U000e8997\x02 \U001089eb\r\U0002b815#\U0008c5c4\u6588\U00077492\U000f2fcd\x03',                         
 u'%',                                                                                                                 
 u'\ue70f\n\x14\U000ba7d6\U000b9f5d("\ud428'], collective_dircap=u'URI:DIR2:fdis5qniqxtadltxmcnd6ta7im:3mvag25uktenhbb466jtqtediw67pdldynnhwvogxqpjqlte5kdq', upload_dircap=u'URI:DIR2:jul3r6faef4hnqc7n4mkszakdy:t33i2ahnpcsl6f67udrb5rwsw74h2tqzcxcma744hyz4cuanhdrq') produces unreliable results: Falsified on the first call but did not on a subsequent one

Consider unicode normalization

Unicode is used throughout the implementation but largely without consideration for how it is normalized. This means that a folder might be assigned a name normalized using one form and then later requested using a different normalization of that name. Currently this will result in the folder seeming not to exist. The two different normalizations of the text will look identical to the end user so this is a misleading result.

Until this is addressed, the Hypothesis strategies that generate text will be limited to generating text with a particular normalization so that everything appears to work.

Make non-ASCII paths work

There is some effort in place to handle non-ASCII paths. However, on platforms that decline to specify a filesystem encoding (of UTF-8), there are still problems. Mainly, FilePath naively passes unicode around and when something tries to encode it using ASCII, there can be problems.

For now, testing against non-ASCII paths is disabled. This is done by having magic_folder.test.strategies.path_segments only build ASCII unicode paths. It should be possible to have this build non-ASCII unicode paths too and any tests that fail should be fixed.

remove _get_json_for_fragment and other dead code

It looks like cli.py: _get_json_for_fragment() is no longer used after the status rewrite. This function and its possible dependencies can be deleted..