Verbosity options produce the same output about vdirsyncer

[untitaker]

Yeah there is currently no extra information loggeg. The HTTP thing mentioned in the help used to be there, but was always a ugly hack which i removed at some point and forgot to update the help.

Your storage type needs to be http.

[jasonwryan]

Thanks. Changing storage type to http throws a different error:

Syncing holidays from holidays
Traceback (most recent call last):
  File "/home/jason/.local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 68, in sync
    list_b = dict(prepare_list(storage_b, b_href_to_uid))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 21, in prepare_list
    for href, etag in storage.list():
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 117, in list
    r.raise_for_status()
  File "/home/jason/.local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

This URL works under khal: why would it be failing here?

[untitaker]

Can you do

pip uninstall vdirsyncer
pip install git+https://github.com/untitaker/vdirsyncer.git#issue29

and see if that fixes it?

[geier]

works for me

[untitaker]

Okay, the issue with the http storage type should be fixed with 0c14b14 and the original issue of not actually displaying HTTP traffic in debug mode is fixed with 633a7de

@jasonwryan You can install vdirsyncer from git using

pip uninstall vdirsyncer
pip install git+https://github.com/untitaker/vdirsyncer.git

I will do a release tomorrow if all problems for you have been solved.

[jasonwryan]

Sorry: I get the same error. Did either of you use my config to test? I am trying to narrow this down, but am not having any luck.

I installed the git version with pip2 install --user git+https://github.com/untitaker/vdirsyncer.git:

The error:

Syncing holidays from holidays
Traceback (most recent call last):
  File "/home/jason/.local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 73, in sync
    list_b = dict(prepare_list(storage_b, b_href_to_uid))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 26, in prepare_list
    for href, etag in storage.list():
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 106, in list
    r.raise_for_status()
  File "/home/jason/.local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

[geier]

try changing url = https://mozorg.cdn.mozilla.net/media/caldata/ and collection = QueenslandHolidays.ics (this is probably a part were better documentation is still needed)

[untitaker]

Please uninstall old versions of vdirsyncer with pip uninstall vdirsyncer,
it is very likely pip won't do anything because it thinks it already has the
latest version.

[untitaker]

If in doubt, post output of pip here.

[jasonwryan]

OK. I pip uninstalled and deleted everyhting in .local/lib/ then pip2 install --user git+https://github.com/untitaker/vdirsyncer.git, changed the url and collections as Christian suggested and now the sync works.

Thanks for all the help.

[geier]

thanks for all the debugging :)

[untitaker]

Hmm, it actually should also work without the collection arg, and just the URL. Anyway, i'm glad you got it fixed :)

[jasonwryan]

I got it to work with my work calendar using the same schema: using the final /value from the URL as the collection name. Definitely worth updating the example config. :)

[untitaker]

Not sure if what you're describing makes sense...

This is from the init code:

if collection is not None:
    url = urlparse.urljoin(url, collection)

The reason the collection arg was added is so that you don't have to resupply
the full URL for all your calendars.

[untitaker]

I'd be very interested in getting a debug log when your setup doesn't work without the collection parameter, because it really should.

[jasonwryan]

Yes: that is correct: the collection value is appended to the end of the URL, so--for the URL to correctly resolve, collection must be that final value.

Debug output with the original values:

vdirsyncer --verbosity DEBUG sync                                                                                                                   [2/633]
Syncing holidays from holidays                                                                                                                          [1/633]
GET https://mozorg.cdn.mozilla.net/media/caldata/holidays
None
None
Sending request...
404
CaseInsensitiveDict({'content-length': '220', 'x-backend-server': 'bedrock3.webapp.scl3.mozilla.com', 'server': 'Apache', 'date': 'Fri, 11 Apr 2014 20:46:50 GMT', 'x-cache-info': 'caching', 'content-type': 'text/html; charset=iso-8859-1'})
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /media/caldata/holidays was not found on this server.</p>
</body></html>

Traceback (most recent call last):
  File "/home/jason/.local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 73, in sync
    list_b = dict(prepare_list(storage_b, b_href_to_uid))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 26, in prepare_list
    for href, etag in storage.list():
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 106, in list
    r.raise_for_status()
  File "/home/jason/.local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

[untitaker]

Sorry to bother you so much, but could you also post the output of the working configuration?

[untitaker]

To be precise, either of these should work:

[storage bob_calendar_remote]
type = http
url = https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics

[storage bob_calendar_remote]
type = http
collection = QueenslandHolidays.ics
url = https://mozorg.cdn.mozilla.net/media/caldata/

[jasonwryan]

No problem. There is a lot of output, so I've removed most of the middle section, which is just VEVENT entries:

vdirsyncer --verbosity DEBUG sync                                                                                                                         
Syncing QueenslandHolidays.ics from holidays
GET https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics
None
None
Sending request...
200
CaseInsensitiveDict({'content-length': '11873', 'x-cache': 'HIT', 'x-backend-server': 'bedrock4.webapp.scl3.mozilla.com', 'accept-ranges': 'bytes', 'expires': 'Fri, 18 Apr 2014 20:59:57 GMT', 'server': 'ECAcc (syd/EB8F)', 'last-modified': 'Wed, 05 Mar 2014 21:30:25 GMT', 'etag': '"4f3e2ba8ffe40"', 'cache-control': 'max-age=604800', 'date': 'Fri, 11 Apr 2014 20:59:57 GMT', 'x-cache-info': 'not cacheable; response code not cacheable', 'content-type': 'text/calendar'})
BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
BEGIN:VEVENT
CREATED:20111013T052821Z
LAST-MODIFIED:20111013T052949Z
DTSTAMP:20111013T052949Z
UID:8

<snip />

SUMMARY:Queen's Birthday
DTSTART;VALUE=DATE:20121001
DTEND;VALUE=DATE:20121002
TRANSP:TRANSPARENT
SEQUENCE:1
X-MOZ-GENERATION:1
END:VEVENT
END:VCALENDAR

Deleting item 637fa767-025f-4139-a5ee-e611848a21df from <HttpStorage(**{'username': '', 'url': 'https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics'})>
Traceback (most recent call last):
  File "/home/jason/.local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 91, in sync
    action(storages, status, conflict_resolution)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 145, in inner
    dest_storage.delete(dest_href, dest_etag)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/base.py", line 128, in delete
    raise NotImplementedError()
NotImplementedError

[jasonwryan]

The first does not, provided that there is a collections defined under the pair section and it is not the final value of the URL. If that makes sense.

[untitaker]

Ahh, what you're seeing here is HTTP resources being read-only. You changed something in your collection, and that change can't propagate because... well, how you're supposed to change the file on the mozilla server?

[jasonwryan]

No, I haven't changed anything in my local dirs... I understand that this sort of calendar is read only.

[untitaker]

Then vdirsyncer is looking at an empty or nonexistent directory :S

Assuming you want to have your ics files directly under ~/.config/vdir/calendars/holidays/:

[pair holidays]
a = holidays_local
b = holidays_remote

[storage holidays_local]
type = filesystem
path = ~/.config/vdir/calendars/holidays/
fileext = .ics
create = true

[storage holidays_remote]
type = http
url = https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics
verify = false

Then, for your private calendars, do this:

[pair my_calendar]
a = my_calendar_local
b = my_calendar_remote
collections = a,b,c,d

[storage my_calendar_local]
type = filesystem
path = ~/.config/vdir/calendars/
fileext = .ics
create = true

[storage my_calendar_remote]
type = caldav
url = https://owncloud.local/remote.php/caldav/calendars/bob/

And then point khal at each of the subdirectories in ~/.config/vdir/calendars/.

[untitaker]

Also i guess you now have some unwanted stuff in ~/.config/vdir/...

[untitaker]

I updated the example cfg.

[untitaker]

So yeah, your configuration is actually right... my bad :S

[jasonwryan]

Thanks for patiently working through it with me. :)

[untitaker]

No problem! Feel free to post in the issue tracker if you find problems (especially documentation problems) with vdirsyncer.

[jasonwryan]

Sadly, somewhere over the last four or five commits, http calendar retreival is broken again. Using the config from the example, I get:

vdirsyncer sync holidays
Syncing public-holidays-all.ics from holidays
Traceback (most recent call last):
  File "/usr/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 68, in sync
    list_b = dict(prepare_list(storage_b, b_href_to_uid))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 21, in prepare_list
    for href, etag in storage.list():
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 117, in list
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

[untitaker]

Meh. Can you post your config again?

[jasonwryan]

In addition to my config, I tested with the example.cfg and it throws the same error...

[untitaker]

Hmm, what i actually wanted to get was the URL vdirsyncer tries to use. Since debug mode should reveal this too, could you...?

[jasonwryan]

Happy to try: testing with head this morning, debug was deactivated again. I'll do a fresh pull in the morning on my work box and try again.

[untitaker]

Make sure to always uninstall the old version since pip thinks it doesn't have to do anything if it already has 0.1.2

[jasonwryan]

I was just cloning it directly from here: I'll check the git log when I get in to make sure it is the correct version/commit and give it another go. Cheers.

[untitaker]

I would actually be surprised if the latest five commits have any influence on vdirsyncers behavior, as they're mostly test and doc changes.

[jasonwryan]

Using this config:

[general]
# A folder where vdirsyncer can store some metadata about each pair.
status_path = /home/jason/.config/vdir/status/

# HTTP CALENDAR
[pair holidays]
a = holidays_local
b = holidays_remote

[storage holidays_local]
type = filesystem
path = ~/.config/vdir/calendars/holidays/
fileext = .ics

[storage holidays_remote]
type = http
url = https://mozorg.cdn.mozilla.net/media/caldata/QueenslandHolidays.ics
#auth = basic
#username =
#password =
#verify = True

I get this result (no debug info printed):

vdirsyncer --verbosity DEBUG sync                                                                                                                         
Syncing holidays
Traceback (most recent call last):
  File "/home/jason/.local/bin/vdirsyncer", line 9, in <module>
    load_entry_point('vdirsyncer==0.1.2', 'console_scripts', 'vdirsyncer')()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 101, in main
    _main(env, cfg)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 211, in _main
    app()
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 252, in __call__
    if not self.call_commands(context, argv):
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 187, in call_commands
    self.commands[argument](context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 283, in __call__
    self.call_main(context, argv)
  File "/home/jason/.local/lib/python2.7/site-packages/argvard/__init__.py", line 200, in call_main
    self.main_func(context, **arguments)
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 208, in sync_main
    action()
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/cli.py", line 203, in x
    pair_options.get('conflict_resolution', None))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 68, in sync
    list_b = dict(prepare_list(storage_b, b_href_to_uid))
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/sync.py", line 21, in prepare_list
    for href, etag in storage.list():
  File "/home/jason/.local/lib/python2.7/site-packages/vdirsyncer/storage/http.py", line 117, in list
    r.raise_for_status()
  File "/home/jason/.local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found

[untitaker]

I don't think you have the latest version installed, vdirsyncer shows a lot more information to me. Please do this:

pip uninstall vdirsyncer  # repeat until it errors?
pip install git://github.com/untitaker/vdirsyncer.git

and post the output.

[jasonwryan]

My bad: I had an old binary in ~/.local/bin/: sorry for the noise.

[untitaker]

No problem. Does it work now?

[jasonwryan]

Yes: thank you. Both my work and holiday calendars are working. One compication was that the schema for one of the holiday calendars was broken: changing to another one worked perfectly.