pimutils / vdirsyncer Goto Github PK

Vdirsyncer seems to have problems with ~ in the paths in configuration files. It creates directories ~ in my home directory while syncing. Replacing ~ with the full path solves it.

Hello,

I managed to sync the ics file from example.cfg (https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics ), and now I'm trying to sync this ics file : http://www.mozilla.org/projects/calendar/caldata/EnglishHolidays.ics.

Using "vdirsyncer --verbosity=DEBUG sync" with python2.7 I have the following error :

Traceback (most recent call last):
File "/usr/bin/vdirsyncer", line 9, in
load_entry_point('vdirsyncer==0.1.4', 'console_scripts', 'vdirsyncer')()
File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 135, in main
_main(env, cfg)
File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 245, in _main
app()
File "/usr/lib/python2.7/site-packages/argvard/init.py", line 252, in call
if not self.call_commands(context, argv):
File "/usr/lib/python2.7/site-packages/argvard/init.py", line 187, in call_commands
self.commands[argument](context, argv)
File "/usr/lib/python2.7/site-packages/argvard/init.py", line 283, in call
self.call_main(context, argv)
File "/usr/lib/python2.7/site-packages/argvard/init.py", line 200, in call_main
self.main_func(context, **arguments)
File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 237, in sync_main
map(_sync_collection, actions)
File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 249, in _sync_collection
return sync_collection(**x)
File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 263, in sync_collection
pair_options.get('conflict_resolution', None))
File "/usr/lib/python2.7/site-packages/vdirsyncer/sync.py", line 84, in sync
action(storages, status, conflict_resolution)
File "/usr/lib/python2.7/site-packages/vdirsyncer/sync.py", line 98, in inner
dest_href, dest_etag = dest_storage.upload(item)
File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/filesystem.py", line 124, in upload
href = self._get_href(item.uid)
File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/filesystem.py", line 102, in _get_href
return uid + self.fileext
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

It seems that the main difference between the two files is in the TZOFFSETTO and TSOFFSETFROM with a '+' for the offset.

When I synchronize this file https://mozorg.cdn.mozilla.net/media/caldata/FrenchHolidays.ics, I have trouble with the different accents present in the name of the holidays.

Using python 2.7 and vdirsyncer 0.1.4.r37.g2656ac7, the ics files produced in .vdir/calendars/holidays/ are encoded as follow (for instance) :
"SUMMARY:Lundi de Pâques"
"CATEGORIES:Jours fériés"

I'm trying to synchronize my zimbra calendar with owncloud.

First, I have to coment an assertion :

vdirsyncer/storage/dav.py", line 365, in list
assert href not in hrefs
AssertionError

Working on a set(), it is not a big problem.

But after, I am dealing with a true problem : some event contain a '@'. Sometimes, it is converted into '%40'.

I get something like :
File "vdirsyncer/storage/dav.py", line 186, in get_multi
.format(href, hrefs_left))
KeyError: "/owncloud/remote.php/caldav/calendars/xx/yy/myname%40mycompany-225654.ics doesn't exist in set([u'/owncloud/remote.php/caldav/calendars/xx/yy/[email protected]'])"

I tried a lot of things with urllib.unquote or urllib.quote or replace() but after I get other exceptions (KeyError uid or KeyError item).

Can you understand my problem and have you any ideas ?

vdirsyncer can rather unexpectedly (from a user perspective :) ) delete all events from a server:

1. Create a storage pair between an (empty) local and a remote storage.
2. Make an initial sync -> everything is fetched from the remote storage, great!
3. Change the local storage path in the config file and create that directory
4. Resync -> all events are deleted from the remote storage

Expected behaviour: Detect that the storage path was changed and fetch everything from the server again.

A very simple fix might be to detect that a storage directory is empty but a status file exists and to at least prompt the user if he is really sure to delete everything from the server.

The keyring pypi package can be used for this.

I also think netrc files should be supported by either vdirsyncer or that library (see https://bitbucket.org/kang/python-keyring-lib/issue/139/support-netrc-files)

/cc @geier

I can't really make a convincing argument for this, but i feel like leif is just too little code to be a separate package.

But still show errors.

I would expect a something like "20130812T121956Z-13466-1000-1-6@host" not "[email protected]"

I mean, just look at it.

Unable to initialize storage carddav.
Here are the required arguments for the storage:
['url']
Here are the optional arguments:
['username', 'verify', 'auth', 'collection', 'useragent', 'password']
And here are the ones you gave: 
['username', 'url', 'verify', 'collection', 'item_types', 'auth', 'password']

A really nice feature of vdirsyncer is its ability to sync a lot of collections (from the same host) with minimal configuration work, so it would be nice if vdirsyncer could create folders for those calendars locally.

Is this a useful enhancement or is it already feature creep? At least a nice error message would be nice.

Don't know if it's just something I'm just doing wrong but vdirsyncer doesn't pick up my config in ~/.vdirsyncer/config, even setting VDIRSYNCER_CONFIG to this directory fails. Though, just to clarify the VDIRSYNCER_CONFIG is set in the shell profile correct? Anyway, putting the configuration in this directory gives me

Unable to find general section. You should copy the example config from the repository and edit it.
https://github.com/untitaker/vdirsyncer

I just discovered that a lot of my older VCards (created with some version of OS X's addressbook, I haven't used it in several years) do not have a UID, only a X-ABUID and some without even that. For me this wasn't really a problem (I just fixed up the Item class and than used sed to replace X-ABUID with UID), but I guess this might be a showstopper for some people.

Do you/we want to support those broken VCards(I guess some VEvents/VTodos will follow) or should those people fix their VCards?

• get GuessAuth merged, see requests/toolbelt#28
• make a PR on vdirsyncer, see #57
• wait for a new release of requests_toolbelt
• move dependency from build.sh to setup.py
• merge PR

I have no idea what this is, i imported this from geier/pycarddav#66.

@geier, care to elaborate?

Formerly known as Need better error messages when user tries to modify HttpStorage

The hard way: All storages take a bool read_only param which sets an attribute with the same name and value. The sync function will respect this and use a different algorithm for one-way sync.

The easy way: A parameter for HTTP storage to not raise NotImplementedError, and just do a noop. This will cause the sync algorithm to detect a deletion on the next sync. Very inefficient.

I had an old version of requests installed on one machine (0.14) and which raised a TypeError with the current setup. This led to a rather misleading error message (on verbosity DEBUG):

[...]
Sending request...
error: caldav storage doesn't take the parameters: username, url, password, collection, verify

Therefore I suggest to print a proper stacktrace at least in the highest verbosity mode.

While testing the singlefilestorage, i see that there is missing check (and some niece exception) for storage type config value:

Traceback (most recent call last):
  File "/usr/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.5', 'console_scripts', 'vdirsyncer')()
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 171, in main
    _main(env, cfg)
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 287, in _main
    app()
  File "/usr/lib/python2.7/dist-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/usr/lib/python2.7/dist-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/usr/lib/python2.7/dist-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/usr/lib/python2.7/dist-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 277, in sync_main
    map(_sync_collection, actions)
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 294, in _sync_collection
    return sync_collection(**x)
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 303, in sync_collection
    a = storage_instance_from_config(config_a, collection_description)
  File "/usr/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 121, in storage_instance_from_config
    storage_name = config.pop('type')
KeyError: 'type'

regards

Currently it is very verbose, especially the repetition of the credentials is
madness

Writing to the abook.db file. This basically would give us the nice CLI interface abook has and make pycarddav obsolete.

http://www.ietf.org/rfc/rfc6578.txt

Helps avoiding fetching the whole list of items. Should be implemented with fallback to a normal query of all items, because server support is uncertain.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Much like khal does

I just noticed i forgot to add GuessAuth to README.rst. However, i think this issue could be solved more generally with some .. include tricks?

https://developers.google.com/google-apps/calendar/auth

Apparently we have to register the client at Google, not sure how we would keep the keys secret (if any) in an opensource project.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

leif is supposed to handle this. Not sure how to integrate this in vdirsyncer.

Maybe it is also time to rewrite the Storage API again such that a storage represents a whole vdir instead of just one collection. Otherwise we'd have to put caldav-specific code for autodiscovery into vdirsyncer.cli. OTOH this makes the whole configuration a bit unflexible.

When I set verify=False, I get the following traceback:

Traceback (most recent call last):
  File "/usr/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/usr/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __ca
ll__
    if not self.call_commands(context, argv):
  File "/usr/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call
_commands
    self.commands[argument](context, argv)
  File "/usr/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __ca
ll__
    self.call_main(context, argv)
  File "/usr/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call
_main
    self.main_func(context, **arguments)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 193, in sync_m
ain
    b = storage_instance_from_config(config_b)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli.py", line 77, in storage
_instance_from_config
    return cls(**config)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 259, i
n __init__
    super(CaldavStorage, self).__init__(**kwargs)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 47, in
 __init__
    super(DavStorage, self).__init__(**kwargs)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 87, i
n __init__
    'verify': prepare_verify(verify),
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 63, i
n prepare_verify
    return expand_path(verify)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/utils.py", line 30, in expan
d_path
    p = os.path.expanduser(p)
  File "/usr/lib64/python2.7/posixpath.py", line 252, in expanduser
    if not path.startswith('~'):
AttributeError: 'int' object has no attribute 'startswith'

It looks like this is because in parse_options in utils.py, you convert the bools to ints.

Then in prepare_verify in storage/http.py, you expand the path if it's not a bool.

modifying parse_options as follows works for me, but not sure if it will break anything else!

def parse_options(items):
    for key, value in items:
        if value.lower() in ('yes', 'true', 'on'):
            value = True
        elif value.lower() in ('no', 'false', 'off'):
            value = False
    else:
        try:
            value = int(value)
        except ValueError:
            pass
        yield key, value

Irrespective of which of the --verbosity levels I choose, the traceback is exactly the same.

So, vdirsyncer --verbosity DEBUG sync and any other option always produce the same message:

vdirsyncer --verbosity DEBUG sync Traceback (most recent call last): File "/home/jason/.local/bin/vdirsyncer", line 9, in <module> load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')() File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main _main(env, cfg) File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main app() File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__ if not self.call_commands(context, argv): File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands self.commands[argument](context, argv) File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__ self.call_main(context, argv) File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main self.main_func(context, **arguments) File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 193, in sync_main b = storage_instance_from_config(config_b) File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 77, in storage_instance_from_config return cls(**config) File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 259, in __init__ super(CaldavStorage, self).__init__(**kwargs) File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 58, in __init__ raise exceptions.StorageError('URL is not a collection') vdirsyncer.exceptions.StorageError: URL is not a collection

Using this basic config:

[general]
# A folder where vdirsyncer can store some metadata about each pair.
status_path = ~/.config/vdir/status/

# CALENDAR
[pair holidays]
a = holidays_local
b = holidays_remote
collections = holidays

[storage holidays_local]
type = filesystem
path = ~/.config/vdir/calendars/
fileext = .ics
create = true

[storage holidays_remote]
type = caldav
url = https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics
verify = false

With a different debug level, like CRITICAL, the same...

KeyboardInterrupt from the child processes gets lost somewhere.

I just noticed it is partially written in Markdown and uses italics where i prefer codeblocks. Unfortunately i am on mobile right now.

A large proportion of Travis builds is spent on compiling lxml, so caching
dependencies is a cheap shot. There is a documented way to cache directories,
but it's an unsolved question which directory to cache.

Currently it does one request each for x in ('VTODO', 'VEVENT').

As #5 shows, this needs to be done. Furthermore, radicale's error messages really suck.

Please, add option to sync into one (single) ics file, which then can be used as foreign file for Orage (the XFCE's light calendar) - there is a lack of CalDAV clients in Linux, if one don't want to use Thunderbird or Evolution, then working with shared calendars can be problematic ;)

I think initialy about merging synced ics files into one and this seems to be trivial task, but split them back can be more trickly...

Because Radicale's documentation suggests the usage of file extensions inside collection names (for guessing the collection type), the testsuite currently does that, qith the cost of weaker tests. However, i assumed this was absolutely necessary, but apparently it is not.

Long-running process that keeps connections open. Not sure if this would be that useful.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

I have a local branch for this. This is currently just a list of showstoppers that need to be resolved in some way.

owncloud/calendar#372

https://github.com/fruux/sabre-dav/issues/422

fruux/sabre-dav@89f8fcb#commitcomment-5794292

Currently it filters for VTODO

https://trac.calendarserver.org/browser/CalendarServer/trunk/doc/Extensions/caldav-ctag.txt

We'd have to figure out server support first, and especially whether we can detect non-support in Radicale.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

From the bug reports so far, it seems that the current config format doesn't make it obvious at all how to set up collections. The greatest confusion seems to come from the collections parameter, which was introduced to keep configs short.

/cc @ways

like the help text. On the other hand: couldn't sync be the default command, especially as long as it is the only one?

Hi.

This is my output:

$ vdirsyncer --verbosity DEBUG sync
object.__init__() takes no parameters

Unable to initialize storage carddav.
Here are the required arguments for the storage:
['url']
Here are the optional arguments:
['username', 'verify', 'auth', 'collection', 'useragent', 'password']
And here are the ones you gave: 
['username', 'url', 'verify', 'collection', 'item_types', 'auth', 'password']

I'm attempting to sync from an ownCloud server which is successfully syncing via DavDroid (https://github.com/rfc2822/davdroid).

Part of config:

# An example configuration for vdirsyncer.
# Optional parameters are commented out.

[general]
# A folder where vdirsyncer can store some metadata about each pair.
status_path = ~/.vdirsyncer/status/

# The amount of processes to use when synchronizing. If the CPU of your server
# is the bottleneck (which is plausible for small homeservers), set this to the
# number of cores * 2. Defaults to one process for each collection. The value
#0 will be ignored, the value 1 disables multiprocessing.
#processes =

# CONTACTS
[pair lars_contacts]
# Similar to accounts in OfflineIMAP.
a = lars_contacts_local
b = lars_contacts_remote
collections = private,coworkers  # collections = addressbooks in this case
# conflict_resolution = None  # abort when collisions occur (default)
# conflict_resolution = a wins  # assume a's items to be more up-to-date
conflict_resolution = b wins  # assume b's items to be more up-to-date

[storage lars_contacts_local]
# A storage references actual data on a remote server or on the local disk.
# Similar to repositories in OfflineIMAP.
type = filesystem
path = ~/.contacts/
fileext = .vcf
create = True  # create directory if it doesn't exist
encoding = utf-8

[storage lars_contacts_remote]
type = carddav
item_types = VTODO,VEVENT
url = https://owncloudserver/remote.php/carddav/addressbooks/lars/default/
auth = digest  # basic|digest
username = user # for http auth, if empty, no auth will be used
password = pass # can also be fetched from the system password storage or netrc

...

After sudo pip install vdirsyncer on my Debian Wheezy, vdirsyncer --help give :

Traceback (most recent call last):
  File "/usr/local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.4', 'console_scripts', 'vdirsyncer')()
  File "/usr/local/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 134, in main
    cfg = load_config(fname)
  File "/usr/local/lib/python2.7/dist-packages/vdirsyncer/cli.py", line 58, in load_config
    return general, pairs, storages
UnboundLocalError: local variable 'general' referenced before assignment

The patch to fix it in cli.py :

36a37
>     general = {}

Thank you for your work

Before i added CarddavStorage, i added a few tests to the storage test suite.

After i finished writing CarddavStorage and put all the common methods of CaldavStorage and CarddavStorage in a common superclass, i ran the tests and found them failing. It took me embarrassingly long to realize that they didn't break because of some error while doing the refactoring above, but because i added new tests. In other words, CaldavStorage was broken for radicale all the time.

Since i basically copypasted this from geier/khal, i think that the equivalent of these methods is also broken there. That means, only for radicale, AFAICT it still follows the RFC correctly and it seems to work for me on ownCloud.

One of these:

• User prompt
• One-side-wins (easiest)
• Line-wise (probably too much work)

I think vdirsyncer.sync would require a callback function for conflicts which
gets provided by the cli module.

Also add a CONTRIBUTORS file to the repo