jimmxinu / fanficfare

FanFicFare is a tool for making eBooks from stories on fanfiction and other web sites.

FanFicFare

FanFicFare makes reading stories from various websites much easier by helping you download them to EBook files.

FanFicFare was previously known as FanFictionDownLoader (AKA FFDL, AKA fanficdownloader).

Main features:

• Download FanFiction stories from over 100 different sites. into ebooks.

• Update previously downloaded EPUB format ebooks, downloading only new chapters.

• Get Story URLs from Web Pages.

• Support for downloading images in the story text. (EPUB and HTML only -- download EPUB and convert to AZW3 for Kindle) More details on configuring images in stories and cover images can be found in the FAQs or this post in the old FFDL thread.

• Support for cover image. (EPUB only)

• Optionally keep an Update Log of past updates (EPUB only).

There's additional info in the project wiki pages.

There's also a FanFicFare maillist for discussion and announcements and a discussion thread for the Calibre plugin.

Getting FanFicFare

Official Releases

This program is available as:

• A Calibre plugin from within Calibre or directly from the plugin discussion thread, or;
• A Command Line Interface (CLI) Python package that you can install with:

pip install FanFicFare

• As of late November 2019, the web service version is shutdown. See the Wiki Home page for details.

Test Versions

FanFicFare is released roughly every month, but new test versions are posted more frequently as changes are made.

Test versions are available at:

• The test plugin is posted at MobileRead.
• The test version of CLI for pip install is uploaded to the testpypi repository and can be installed with:

pip install --extra-index-url https://testpypi.python.org/pypi --upgrade FanFicFare

Other Releases

Other versions may be available depending on your OS. I(JimmXinu) don't directly support these:

• Arch Linux: The latest CLI release can be obtained from the fanficfare AUR package. It will install the calibre plugin, if calibre is installed.

fanficfare's Issues

fanfiction.tenhawkpresents.com doesn't work - Problem on site.

Are you using the CLI, web service or calibre plugin?
Web service

What is the story URL?
http://fanfiction.tenhawkpresents.com/viewstory.php?sid=35

What is the output format you used?
Epub

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?


What steps will reproduce the problem?
1.repeating story download with provided address
2.
3.

What is the expected output? What do you see instead?
Epub
I get text:
Deadline exceeded while waiting for HTTP response from URL: 
http://fanfiction.tenhawkpresents.com/viewstory.php?sid=35&chapter=74&ageconsent
=ok&warning=3 Source


What version of the product are you using? On what operating system?
Latest webversion

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 15 Feb 2013 at 8:22

Provide an rc or init file

It would be nice to have an init/config file outside the codetree (for instance 
in ~/.config or ~/.fanficdownload.cfg) for default values like outputformat or 
output directory, things that are liable to be specific to the installation, 
but not usually change much.

That way you could keep the source under source control and update frequently, 
without losing local configurations/modifications

Original issue reported on code.google.com by [email protected] on 22 Dec 2010 at 11:43

Specific ffnet story not being downloaded

What steps will reproduce the problem?

Use the cli and try to download http://www.fanfiction.net/s/2362049/

Output is: 

DEBUG:downloader.py(100):reading ['defaults.ini'] config file(s), if present
DEBUG:__init__.py(82):site:www.fanfiction.net
DEBUG:adapter_fanfictionnet.py(72):URL: http://www.fanfiction.net/s/2362049/
=Trying newer chapter: http://www.fanfiction.net/s/2362049/36/
Traceback (most recent call last):
  File "./downloader.py", line 219, in <module>
    main()
  File "./downloader.py", line 199, in main
    output_filename=writeStory(config,adapter,options.format,options.metaonly)
  File "./downloader.py", line 43, in writeStory
    writer.writeStory(outstream=outstream,metaonly=metaonly)
  File "/Users/thomas/Documents/Devel/EbookLib/lib/fanficdownloader/fanficdownloader/writers/base_writer.py", line 198, in writeStory
    outfilename=self.getOutputFileName()
  File "/Users/thomas/Documents/Devel/EbookLib/lib/fanficdownloader/fanficdownloader/writers/base_writer.py", line 120, in getOutputFileName
    return self.getBaseFileName()
  File "/Users/thomas/Documents/Devel/EbookLib/lib/fanficdownloader/fanficdownloader/writers/base_writer.py", line 123, in getBaseFileName
    return self.formatFileName(self.getConfig('output_filename'))
  File "/Users/thomas/Documents/Devel/EbookLib/lib/fanficdownloader/fanficdownloader/writers/base_writer.py", line 140, in formatFileName
    return string.Template(template).substitute(values).encode('utf8')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/string.py", line 172, in substitute
    return self.pattern.sub(convert, self.template)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/string.py", line 162, in convert
    val = mapping[named]
KeyError: 'title'

That's with the latest version (4.3.2) on OS X 10.7.3, Python 2.7.1

Original issue reported on code.google.com by [email protected] on 25 Feb 2012 at 8:36

Can not show images

Images was not show that the html file contained.

Original issue reported on code.google.com by [email protected] on 1 Jul 2012 at 2:55

ff.net parser is broken

What steps will reproduce the problem?
1. $python downloader.py http://someff.neturlIdunno epub
2. Wait
3. script barfs and dies

FFnet has changed minor html.  Script does not fetch text, because it cannot 
find text, does not realize it has not fetched text until it attempts 
_removeEntities.  Should fail earlier.

I got it to work using the changes included in the attached patch.

Original issue reported on code.google.com by [email protected] on 15 Sep 2010 at 3:16

Attachments:

• patch

adapters.adapter_fictionalley does not deal with utf8 metadata correctly

What steps will reproduce the problem?
$ python2.5 downloader.py -m -f html 
http://www.fictionalley.org/authors/worth_12_of_malfoy/resistance.html

What is the expected output? What do you see instead?

In the metadata (handily given by the debug logging) I see:

'description': u'Hogwarts has changed. Severus Snape is Headmaster, Dark Arts 
is on the curriculum, and the shadow of Voldemort\xe2\u20ac\u2122s reign of 
terror hangs heavily over the remaining students. Faced with a choice between 
hope and despair, three students determine to fight back against the new 
regime.  Neville, Ginny and Luna rally the remainder of 
Dumbledore\xe2\u20ac\u2122s Army and form a resistance movement. But the stakes 
are high and they must fight not only the school\xe2\u20ac\u2122s 
administration but their own demons as they struggle to survive in a cruel new 
Hogwarts. This is \xe2\u20ac\u02dcDeathly Hallows\xe2\u20ac\u2122 from the 
perspective of those Harry left behind, who never lost their faith that one day 
he would return, and prepared to fight alongside him for the very future of 
their world.'

What I expect to see is 

'description': u'Hogwarts has changed. Severus Snape is Headmaster, Dark Arts 
is on the curriculum, and the shadow of Voldemort\u2019s reign of terror hangs 
heavily over the remaining students. Faced with a choice between hope and 
despair, three students determine to fight back against the new regime.  
Neville, Ginny and Luna rally the remainder of Dumbledore\u2019s Army and form 
a resistance movement. But the stakes are high and they must fight not only the 
school\u2019s administration but their own demons as they struggle to survive 
in a cruel new Hogwarts. This is \u2018Deathly Hallows\u2019 from the 
perspective of those Harry left behind, who never lost their faith that one day 
he would return, and prepared to fight alongside him for the very future of 
their world.'

- specifically, the curly quotes appear to be utf8 encoded inside the unicode 
string, rather than decoded from utf8 to start with.

What version of the product are you using? On what operating system?

hg HEAD on linux

Please provide any additional information below.

For the record, to get the properly encoded string I had to resort to:

unicode(story.metadata['description'].encode('cp1252'), 'utf8')

Original issue reported on code.google.com by [email protected] on 24 Aug 2011 at 3:00

Error on Unicode Upper char in story desc

Story http://www.fanfiction.net/s/5223465/1/A_Good_Year using
http://2-6-beta.latest.fanfictionloader.appspot.com (code version ff7c42463b25)

Error happened: Traceback (most recent call last): File "/base/data/
home/apps/fanfictionloader/2-6-beta.346492954846423634/main.py", line
228, in post data = loader.download() File "/base/data/home/apps/
fanfictionloader/2-6-beta.346492954846423634/fanficdownloader/
downloader.py", line 108, in download self.writer.finalise() File "/
base/data/home/apps/fanfictionloader/2-6-beta.346492954846423634/
fanficdownloader/output.py", line 346, in finalise description =
unicode(description) UnicodeDecodeError: 'ascii' codec can't decode
byte 0xe2 in position 96: ordinal not in range(128)

But only in appengine or appengine sdk:

App Engine SDK 
release: "1.3.7" 
timestamp: 1282688496 
api_versions: ['1'] 
Python 2.5.2 
wxPython 2.8.8.1 (msw-unicode) 

Works using same python install (2.5.2 in windows7) using commandline version.

Appears to be trying to decode unicode string that's already unicode.

I've checked in a work around similar to several other places in the code in 
code version 20d7d18355d8.

Original issue reported on code.google.com by [email protected] on 28 Nov 2010 at 2:53

won't download AO3 fics rated Mature

Are you using the CLI, web service or calibre plugin?
CLI

What is the story URL?
http://archiveofourown.org/works/114938

What is the output format you used?
html

What steps will reproduce the problem?
the usual downloading process

What is the expected output? What do you see instead?
Here is the error output:

@:~/ffddl/fanficdownloader-4.4.31$ python downloader.py -f html 
http://archiveofourown.org/works/114938
FFDL:INFO:adapter_archiveofourownorg.py(134):url: 
http://archiveofourown.org/works/114938/navigate
FFDL:INFO:adapter_archiveofourownorg.py(135):metaurl: 
http://archiveofourown.org/works/114938
FFDL:WARNING:base_adapter.py(179):Caught an exception reading URL: 
http://archiveofourown.org/works/114938/navigate  Exception <urlopen error 
[Errno 111] Connection refused>.
FFDL:WARNING:base_adapter.py(179):Caught an exception reading URL: 
http://archiveofourown.org/works/114938/navigate  Exception <urlopen error 
[Errno 111] Connection refused>.
FFDL:WARNING:base_adapter.py(179):Caught an exception reading URL: 
http://archiveofourown.org/works/114938/navigate  Exception <urlopen error 
[Errno 111] Connection refused>.
FFDL:WARNING:base_adapter.py(179):Caught an exception reading URL: 
http://archiveofourown.org/works/114938/navigate  Exception <urlopen error 
[Errno 111] Connection refused>.
FFDL:ERROR:base_adapter.py(181):Giving up on 
http://archiveofourown.org/works/114938/navigate
FFDL:ERROR:base_adapter.py(182):<urlopen error [Errno 111] Connection refused>
Traceback (most recent call last):
  File "/home//ffddl/fanficdownloader-4.4.31/fanficdownloader/adapters/base_adapter.py", line 176, in _fetchUrl
    return self._decode(self._fetchUrlRaw(url,parameters))
  File "/home//ffddl/fanficdownloader-4.4.31/fanficdownloader/adapters/base_adapter.py", line 165, in _fetchUrlRaw
    return self.opener.open(url.replace(' ','%20')).read()
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>
Traceback (most recent call last):
  File "downloader.py", line 246, in <module>
    main()
  File "downloader.py", line 180, in main
    adapter.getStoryMetadataOnly()
  File "/home//ffddl/fanficdownloader-4.4.31/fanficdownloader/adapters/base_adapter.py", line 238, in getStoryMetadataOnly
    self.extractChapterUrlsAndMetadata()
  File "/home//ffddl/fanficdownloader-4.4.31/fanficdownloader/adapters/adapter_archiveofourownorg.py", line 138, in extractChapterUrlsAndMetadata
    data = self._fetchUrl(url)
  File "/home//ffddl/fanficdownloader-4.4.31/fanficdownloader/adapters/base_adapter.py", line 183, in _fetchUrl
    raise(excpt)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>


What version of the product are you using? On what operating system?
fanficdownloader-4.4.31, ubuntu 12.04

Please provide any additional information below.
The error is positively perplexing, because 1) it wasn't there in the previous 
versions and 2) is_adult:true is there in both the default and personal.ini

Original issue reported on code.google.com by [email protected] on 1 Nov 2012 at 9:16

Epub files from thehexfiles not reading on ibooks ipad

Are you using the CLI, web service or calibre plugin?
Web service and calibre

What is the story URL?
http://www.thehexfiles.net/viewstory.php?sid=15313

What is the output format you used?
epub

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?
Yes

What steps will reproduce the problem?
Any stories from thehexfiles

What is the expected output? What do you see instead?
The files open just fine in ibook but after the summary page, there is no words 
except Chapter headings
I can still read the files normally with calibre viewer on laptop so it is just 
not working on ipad

What version of the product are you using? On what operating system?
Latest version of calibre on windows vista & web service

Please provide any additional information below.
I tried installing Kobo & Bluefire to open the files but the same problem occurs

Original issue reported on code.google.com by [email protected] on 3 Jul 2012 at 7:00

unexpected error downloading specific url

Are you using the CLI, web service or calibre plugin?
CLI, web service

What is the story URL?
http://www.hpfandom.net/eff/viewstory.php?sid=3140

What is the output format you used?
html

What steps will reproduce the problem?
normal download process

What is the expected output? What do you see instead?

error output:

/ffddl/fanficdownloader-4.4.29$ python downloader.py -f html 
http://www.hpfandom.net/eff/viewstory.php?sid=3140
Traceback (most recent call last):
  File "downloader.py", line 246, in <module>
    main()
  File "downloader.py", line 180, in main
    adapter.getStoryMetadataOnly()
  File "/home/ffddl/fanficdownloader-4.4.29/fanficdownloader/adapters/base_adapter.py", line 238, in getStoryMetadataOnly
    self.extractChapterUrlsAndMetadata()
  File "/home/ffddl/fanficdownloader-4.4.29/fanficdownloader/adapters/adapter_hpfandomnet.py", line 191, in extractChapterUrlsAndMetadata
    for char in value.split(','):
AttributeError: 'NoneType' object has no attribute 'split'


What version of the product are you using? On what operating system?
using version 4.4.29, but it was there in version 4.4.28 as well.

Please provide any additional information below.
The site itself is in the list of supported sites. In fact, I have downloaded 
urls from this site before, without any problems. 

compare this page that can be downloaded as normal: 
http://www.hpfandom.net/eff/viewstory.php?sid=3397

Original issue reported on code.google.com by [email protected] on 19 Oct 2012 at 5:46

EPUB output is invalid: XHTML has nested <p> elements.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Test it with epubcheck or http://threepress.org/document/epub-validate/. The 
most common error will likely be something that looks like: "ERROR: 
Five_Times_Ashlyn_Sniped_at_Boone.epub/OEBPS/MS4gdGhlbWUgMzogaG9wbGVzcw==.xhtml(
18): element "p" from namespace "http://www.w3.org/1999/xhtml" not allowed in 
this context".

What is the expected output? What do you see instead?
The EPUB output is invalid. While it may work on some devices, it may fail on 
others.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10), plus the 
patches from issue 6, issue 7 and issue 8 (which don't affect this issue).

Please provide any additional information below.
The use of BeautifulSoup to clean the HTML has the side effect of causing some 
tags to nest. For instance, using FanFiction.net, the body of each chapter is 
contained in a 'div' element, which itself contains a series of 'p' elements. 
However, when this outermost 'div' element is renamed to a 'p' element, it 
invalidates the syntax, because 'p' elements cannot nest directly inside each 
other.

Additionally, using BeautifulStoneSoup rather than BeautifulSoup causes the 
parser not to know that 'hr' and 'br' tags are self-closing (i.e., they 
shouldn't contain anything). It then extends each, e.g., 'hr' tag until the 
start of the next 'hr' tag. Later, when 'hr' elements are converted to 'p' 
elements, we get nested 'p' elements, and therefore invalid XHTML, which causes 
EPUB validation to fail. Using BeautifulSoup instead causes these tags to 
auto-close, which prevents that source of nested 'p' elements.

On the FanFiction.net examples that I tested, at least, changing the parser 
used from BeautifulStoneSoup to BeautifulSoup and commenting out the code that 
changed 'br', 'hr' and 'div' elements to 'p' elements led to valid markup. Is 
there a reason that those elements were being changed? (There's a note in issue 
3 asking the same thing.)

The attached patch makes these changes (it also makes "allPs" actually only 
refer to all 'p' elements); as I haven't validated a full testsuite, I can't 
guarantee that this doesn't mix something else up. I don't know why one would 
want to (for instance) get rid of all 'hr' elements, but if there's a reason, 
you might want to take this patch with a grain of salt.

Original issue reported on code.google.com by [email protected] on 16 Sep 2010 at 8:15

Attachments:

• nesting-p.patch

fanfiction.net problem: <td> with random width

Are you using the CLI, web service or calibre plugin?
CLI, web service

What is the story URL?
http://www.fanfiction.net/s/6461838/1/Color-Shift
http://www.fanfiction.net/s/6627054/1/Daybreak
http://www.fanfiction.net/s/7032323/1/-Forget
http://www.fanfiction.net/s/5909129/1/Gilded-Cage
http://www.fanfiction.net/s/1941315/1/Courage-of-the-Spirit
http://www.fanfiction.net/s/8264234/1/The-Leaf-s-Naruto

What is the output format you used?
html

What steps will reproduce the problem?
Simply downloading the links with CLI or webapp, nothing else.

What version of the product are you using? On what operating system?
Version 4.4.44, Ubuntu 12.04

What is the expected output? What do you see instead?
The <td> of the titlepage entries suddenly seem to have a random width, without 
the default.ini/personal.ini having been changed from what I usually use.

Please provide any additional information below.
If you download the links above, you will see each one has a different <td> 
with. The width in the link for "Daybreak" is a reasonable one. Admittedly, the 
<td> width has at least for some time now been varied in ffnet, but never this 
badly. Usually the td with you see in "The Leaf's Naruto" been more or less the 
standard one in ffnet downloads.

And this link: http://www.fanfiction.net/s/1941315/1/Courage-of-the-Spirit 
seems to have a horizontal scrollbar without there being any need for it. I 
couldn't find anything that 

Among the links above, the <td> width in this one is particularly ridiculous: 
http://www.fanfiction.net/s/7032323/1/Forget

I've attached the files for the links in question. I think it's just a 
fanfiction.net thing.  

Original issue reported on code.google.com by [email protected] on 25 Feb 2013 at 2:08

Attachments:

• files.zip

Fanfiction.net looses hr tag

What steps will reproduce the problem?
1. Downloaded latest code from repository.
2. ran command python downaloder.py 
http://www.fanfiction.net/s/4747195/1/Rise_To_The_Top epub
3. Open in epub viewer.

What is the expected output? What do you see instead?
Expect to see horizontal lines as they appear in the source. No lines.

What version of the product are you using? On what operating system?
Command line, Python 2.6.4, Ubuntu Linux 9.10

Please provide any additional information below.
hr tags do not show up in xhtml source code in epub files either.

Original issue reported on code.google.com by [email protected] on 23 Apr 2010 at 8:30

Attachments:

• [Rise To The Top - Blue Bongo.epub](https://storage.googleapis.com/google-code-attachments/fanficdownloader/issue-3/comment-0/Rise To The Top - Blue Bongo.epub)

Series Column on main Calibre screen Blank in the plugin version, at least for FF.NET stories

Are you using the CLI, web service or calibre plugin?

Calibre Plugin

What is the story URL?

No specific story URL.  But using FF.NET site story links.  Example:

url: http://www.fanfiction.net/s/1004376/1/

What is the output format you used?

Text

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

Overwrite Always

What is the expected output? What do you see instead?

In the Main Calibre screen, expected the series information to be displayed for 
stories downloaded in the series column.  It is in the Tags column, but then it 
is mixed in with all the other tags.

What version of the product are you using? On what operating system?

fanfictiondownloader plugin, the July 9th version installed.  Windows 7 64Bit 
SP1.

Original issue reported on code.google.com by [email protected] on 13 Jul 2013 at 5:41

EPUB output is invalid: XHTML has <u> elements.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Test it with epubcheck or http://threepress.org/document/epub-validate/. The 
following error will sometimes appear: "ERROR: 
Harry_Potter_and_the_Methods_of_Rationality.epub/OEBPS/MTEuIE9tYWtlIEZpbGVzIDEgY
W5kIDI=.xhtml(45): unknown element "u" from namespace 
"http://www.w3.org/1999/xhtml"".

What is the expected output? What do you see instead?
The EPUB output is invalid. While it may work on some devices, it may fail on 
others.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10), plus the 
patches from issue 6, issue 7, issue 8 and issue 9 (which don't affect this 
issue, though the patch is made on top of the patch for issue 9).

Please provide any additional information below.
The EPUB standard specifies a particular subset of XHTML 1.1 which is part of 
the standard. While readers may support more modules, only the modules 
specified in the standard are guaranteed to be present.

http://www.idpf.org/doc_library/epub/OPS_2.0.1_draft.htm#Section2.2

Specifically, the "u" element--underlined text--is not included. It needs to be 
replaced with a CSS element. The "stylesheet.css" defined in constants.py 
already includes a "u" class which sets "text-decoration: underline"; 
therefore, replacing "u" elements with "span class='u'" elements retains the 
meaning in a specification-compliant fashion.

The attached patch, which is written on top of the patch for issue 9 and may 
not apply without it, makes this modification.

Original issue reported on code.google.com by [email protected] on 20 Sep 2010 at 2:15

Attachments:

• u-to-span.patch

output css text-align:center not retained

Are you using the CLI, web service or calibre plugin?
CLI, web version

What is the story URL?
http://www.fanfiction.net/s/3589409/1/

What is the output format you used?
html

What steps will reproduce the problem?
normal downloading

What is the expected output? What do you see instead?
The parts of the text that are centered on ffnet are aligned left on the output 
html.

What version of the product are you using? On what operating system?
4.4.28, ubuntu 12.04

Please provide any additional information below.
I have been using fanficdownloader only for a few days, and I have one last 
question (hopefully) about it. I noticed that, unless specifically specified as 
such on the output_css, no text element retains their alignment (and everything 
is aligned left). Not even <h1> or <h2>. 

Is that supposed to be this way, or am I missing something? Is there some way 
to compensate for it? 

Thank you! 

Original issue reported on code.google.com by [email protected] on 13 Oct 2012 at 11:51

Can't download from ffnet

Are you using the CLI, web service or calibre plugin?
CLI

What is the story URL?
http://www.fanfiction.net/s/5742587


What is the output format you used?
html


What steps will reproduce the problem?
The usual way of downloading

What is the expected output? What do you see instead?
This is the error message:

FFDL:INFO:base_writer.py(208):Save directly to file: Little wooden men 
(WIP).html
Traceback (most recent call last):
  File "downloader.py", line 275, in <module>
    main(sys.argv[1:])
  File "downloader.py", line 255, in main
    output_filename=writeStory(configuration,adapter,options.format,options.metaonly)
  File "downloader.py", line 61, in writeStory
    writer.writeStory(outstream=outstream,metaonly=metaonly)
  File "/home/csad/archives/fanficdownloader-4.4.62/fanficdownloader/writers/base_writer.py", line 227, in writeStory
    self.story = self.adapter.getStory() # get full story
  File "/home/csad/archives/fanficdownloader-4.4.62/fanficdownloader/adapters/base_adapter.py", line 209, in getStory
    data = self.getChapterText(url)
  File "/home/csad/archives/fanficdownloader-4.4.62/fanficdownloader/adapters/adapter_fanfictionnet.py", line 280, in getChapterText
    data = data[data.index("<div class='storytextp"):]
ValueError: substring not found


What version of the product are you using? On what operating system?
Using the latest version, 4.4.62, ubuntu 12.04

Please provide any additional information below.
This error started today, as far as I know. I was using the version. 4.4.56. I 
have then downloaded the newest version, and I still see the same error. 

Original issue reported on code.google.com by [email protected] on 1 Jul 2013 at 9:04

EPUB output is invalid: item IDs and IDREFs in OEBPS/content.opf contain invalid characters.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Test it with epubcheck or http://threepress.org/document/epub-validate/. One 
of the errors will be: "ERROR: 
Harry_Potter_and_the_Methods_of_Rationality.epub/OEBPS/content.opf(17): bad 
value for attribute "id" ".

What is the expected output? What do you see instead?
The EPUB output is invalid. While it may work on some devices, it may fail on 
others.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10), plus the 
patch from issue 6 (which doesn't affect content.opf generation).

Please provide any additional information below.
There are several different issues causing validation to fail. This is one of 
them. See the appendix of the current OPF draft:

http://www.idpf.org/doc_library/epub/OPF_2.0.1_draft.htm#AppendixA

The "item" element inside the "manifest" element has an "id" attribute, which 
is of XML attribute type "ID". Additionally, the "itemref" element (inside the 
"spine" element) has an "idref" attribute, of XML attribute type "IDREF". See 
the XML 1.0 spec, section 3.3.1, and the description of the 'Name' production:

http://www.w3.org/TR/REC-xml/#sec-attribute-types
http://www.w3.org/TR/REC-xml/#NT-Name

In short, IDs and IDREFs cannot contain the '=' character, and cannot start 
with a number. The attached patch ensures that all IDs and IDREFs begin with 
'_', and ensures that only valid characters are used for the chapterId values. 
See also Wikipedia's note on encoding Base64 for use in XML. (The restriction 
on "Name" tokens described there is unnecessary here, as the first character of 
each token is always '_', which is always valid.)

http://en.wikipedia.org/wiki/Base64#XML

Additionally, I've changed constants.py to make it easier to read while 
changing things here.

Original issue reported on code.google.com by [email protected] on 16 Sep 2010 at 3:52

Attachments:

• ids-and-idrefs.patch

keep hr

Are you using the CLI, web service or calibre plugin?
CLI and web service

What is the story URL?
http://archiveofourown.org/works/41844

What is the output format you used?
html, epub, mobi


What steps will reproduce the problem?
simply typing in the url and downloading

What is the expected output? What do you see instead?
I want to keep the <hr>, or at least have some space between text and summary 
(especially chapter summaries), but there is none. I tried using replace_hr: 
false or even replace_hr: true, but it doesn't work, and I don't have the 
python knowledge to know what to do. Couldn't find it in FAQ. Is the output 
supposed to be without any horizontal line? 

What version of the product are you using? On what operating system?
fanficdownloader-4.4.28 , ubuntu 12.04


Thank you. 

Original issue reported on code.google.com by [email protected] on 13 Oct 2012 at 4:41

fanfiction.net; download failed on web service and calibre plugin

Are you using the CLI, web service or calibre plugin?

Error found using web service (http://fanfictiondownloader.appspot.com) on 
Google Chrome (v26.0.1410.43) and calibre plugin (v1.7.19)

What is the story URL?

http://www.fanfiction.net/s/8602587/1/Mask-Of-Indifference

What is the output format you used?

epub

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

Overwrite if Newer

What steps will reproduce the problem?
1. "Add new from URL(s)" in the calibre plugin
2. Copy the URL (if not in the clipboard)
3. Press OK

What is the expected output? What do you see instead?

Expected output: to download the epub file into calibre

Output in Calibre:
Status: Bad
Title: Unknown
Author: Unknoen
Comment: 'NoneType' object has no attribute 'find'
URL: (url of the story)

Output in web service:
Failed; 'NoneType' object has no attribute 'find'

What version of the product are you using? On what operating system?

Calibre: v0.9.25
Calibre plugin (fanfiction downloader plugin): v1.7.19

Please provide any additional information below.

N/A

Original issue reported on code.google.com by [email protected] on 1 Apr 2013 at 1:37

Current INI parse order in CLI seems backwards

My apologies if I misunderstand how this code should be behaving.  I assume 
that config fragments in ~/.fanficdownloader should override values set in the 
appdir/*.init files.  Appdir files are currently taking precedence.

What steps will reproduce the problem?
1. Create a ~/.fanficdownloader/personal.ini with an output_filename value 
different from the app default
2. Run the download
3. The output_filename will be the appdir/defaults.init value

This is what fixed it for me to behave like I expected:

diff -r 736b41f0b95c downloader.py
--- a/downloader.py Sun Jan 01 09:26:17 2012 -0600
+++ b/downloader.py Sun Jan 01 17:09:02 2012 -0500
@@ -84,14 +84,14 @@

    conflist = []
    homepath = join(expanduser("~"),".fanficdownloader")
+   if isfile("defaults.ini"):
+       conflist.append("defaults.ini")
+   if isfile("personal.ini"):
+       conflist.append("personal.ini")
    if isfile(join(homepath,"defaults.ini")):
        conflist.append(join(homepath,"defaults.ini"))
    if isfile(join(homepath,"personal.ini")):
        conflist.append(join(homepath,"personal.ini"))
-   if isfile("defaults.ini"):
-       conflist.append("defaults.ini")
-   if isfile("personal.ini"):
-       conflist.append("personal.ini")
    if options.configfile:
        conflist.extend(options.configfile)

Original issue reported on code.google.com by [email protected] on 1 Jan 2012 at 10:11

downloader.py mis-spelled

What steps will reproduce the problem?
1. N/A
2.
3.

What is the expected output? What do you see instead?
N/A

What version of the product are you using? On what operating system?
Current

Please provide any additional information below.

Kind of an small issue but it should probably be fixed before many more changes 
take place. Considering it's the "program" itself, it would be best if it's 
spelled correctly.

Original issue reported on code.google.com by [email protected] on 30 Oct 2010 at 12:59

At Fanfiction.net, parsing fail due to posible changes in the story page format

Are you using the CLI, web service or calibre plugin?
Error found using both the CLI (4.4.51) and the calibre plugin (1.7.18).

What is the story URL?
http://www.fanfiction.net/s/9051357/1/Substitute-Wizards
http://www.fanfiction.net/s/6448082/17/DADA-boycott
http://www.fanfiction.net/s/8915631/1/What-Happens-When-A-God-Gets-Bored
http://www.fanfiction.net/s/7645594/1/Stalker-las-cronicas-del-stalker-magico
(... and every other story there is, it seems)

What is the output format you used?
epub

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

Add New, and Update EPUB if New Chapters.

What steps will reproduce the problem?
1. "Add new from URL(s)" in the calibre plugin
2. Copy the URL (if not in the clipboard)
3. Press OK

What is the expected output? What do you see instead?
Expected: The story to be downloaded (and added to Calibre if using the plugin).

Output in Calibre:
Status: Bad
Title: Unknown
Author: Unknoen
Comment: 'NoneType' object has no attribute 'findAll'
URL: (url of the story)

Output in CLI:
Traceback (most recent call last):
  File "downloader.py", line 267, in <module>
    main(sys.argv[1:])
  File "downloader.py", line 203, in main
    adapter.getStoryMetadataOnly()
  File "/home/pola/Proyectos/fanficdownloader/fanficdownloader/adapters/base_adapter.py", line 238, in getStoryMetadataOnly
    self.extractChapterUrlsAndMetadata()
  File "/home/pola/Proyectos/fanficdownloader/fanficdownloader/adapters/adapter_fanfictionnet.py", line 163, in extractChapterUrlsAndMetadata
    for b in graydiv.findAll('button'):
AttributeError: 'NoneType' object has no attribute 'findAll'


What version of the product are you using? On what operating system?
CLI (4.4.51)
Calibre plugin (1.7.18) [Firefox 19.0.2]
SO: Ubuntu 12.10

Please provide any additional information below.
As seen in the CLI output, "graydiv", result of:
    "gui_table1i.find('div', {'class':'xgray'})"
is not a node. It seems that the format of the story page changed yet again.

Original issue reported on code.google.com by [email protected] on 29 Mar 2013 at 2:01

keep site-specific adapters in a subdirectory

It would be easier to maintain the growing number of site adapters in a 
subdirectory and source the whole directory via include mechanism. This way the 
main script does not need editing for each added adapter.

If a small tutorial for writing adapters and maybe a skeleton could be provided 
in the wiki, I guess the number of adapters could grow very quickly. It would 
also be advantageous for private adapters.

Original issue reported on code.google.com by [email protected] on 22 Dec 2010 at 11:47

Cannot download fics from fanfiction.net

What steps will reproduce the problem?
1.Find a story on fanfiction.net
2.Copy and paste the story URL in the download line
3.Select HTML and then press download 

What is the expected output? What do you see instead?
Story usually downloaded quickly. Now I see this Failed:argument of type 
'NoneType' is not iterable.

What version of the product are you using? On what operating system?
Newest version on http--ffd-retief-hrd.appspot. Newest version of IE.

Please provide any additional information below.
It was working fine a couple of hours ago, but now this error continues. 
Fanfiction.net has installed a new image manager app to their site. 

Original issue reported on code.google.com by [email protected] on 5 Jun 2012 at 2:49

Provide a "re-download" (or "update") link to update works-in-progress

An incomplete story gets new chapters over time, with varying frequency. It
would be nice if the stories on the "Recently downloaded" page had a way to
collect updates (or re-download the entire story) without cluttering up the
list with duplicates.

I suggest adding an "Update" link which would do the following:

1) Create a new download using the same URL as the original download
2) Remove the original download if step 1 succeeds

Original issue reported on code.google.com by [email protected] on 14 Apr 2010 at 7:05

Trouble Updating Existing Fanfic

Are you using the CLI, web service or calibre plugin?
Calibre plugin

What is the story URL?
http://www.fanfiction.net/s/5082954/1/
http://www.fanfiction.net/s/4230835/1/

What is the output format you used?
EPUB

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

Overwrite if Newer


What steps will reproduce the problem?
1. I selected one of the two books that have one new chapter since I last 
downloaded them using the calibre plug-in
2. Select Update Existing Fanfic Books
3. It tells me that it will not overwrite because it is not newer

What is the expected output? What do you see instead?
I expected it to overwrite and add one more chapter

What version of the product are you using? On what operating system?
calibre version 0.8.66
fanfictiondownloader v1.6.7
Mac OS 10.7.4

Please provide any additional information below.
The last time I download the following fanfic 
(http://www.fanfiction.net/s/5082954/1/) was on July 15, 2012 and there is 60 
chapters on the version I have downloaded.  Online it shows there is 61 
chapters.

The last time I download the following fanfic 
(http://www.fanfiction.net/s/4230835/1/) was on March 27, 2012 and there is 27 
chapters on the version I have downloaded.  Online it shows there is 28 
chapters.

Original issue reported on code.google.com by [email protected] on 29 Aug 2012 at 2:29

Attachments:

• [Hesitation - VerelLupin.epub](https://storage.googleapis.com/google-code-attachments/fanficdownloader/issue-32/comment-0/Hesitation - VerelLupin.epub)
• [Lynne's AU Ending to The Marriage Stone - lynned0101.epub](https://storage.googleapis.com/google-code-attachments/fanficdownloader/issue-32/comment-0/Lynne's AU Ending to The Marriage Stone - lynned0101.epub)

Unable to download from fanfiction.net using calibre plugin

What steps will reproduce the problem?
1.I copied the url from fanfiction.net website
2.Add to "Add new from URLs" option
3.Pressed okay

What is the expected output? What do you see instead?
It should have downloaded the fic in epub format. 
Nothing was downloaded and there is an error message "argument of type 
'NoneType' is not iterable.

What version of the product are you using? On what operating system?
Calibre 0.8.55
Fanficdownloader plugin 1.5.26

Please provide any additional information below.
I'm attaching the snip image of the error message.

Original issue reported on code.google.com by [email protected] on 10 Jun 2012 at 4:29

Attachments:

• ffdlincalibre.JPG

Story download from Fanfiction.net failed because " list index out of range"

Are you using the CLI, web service or calibre plugin?

calibre plugin


What is the story URL?

http://www.fanfiction.net/s/2650472/1/

What is the output format you used?

mobi

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

add new


What steps will reproduce the problem?
1. go to calibre plug in
2. select add new from URL(s)
3. select "mobi" output, Overwrite if newerm ok

What is the expected output? What do you see instead?
a mobi format story to download to calibre

FFDL download log says: 

None of the URLs can be downloaded

Status    Title          Author             Comment                   
Bad      Unknown    Unknown    list index out of range  

                URL
http://www.fanfiction.net/s/2650472/1/


What version of the product are you using? On what operating system?

v1.7.14 on OS X 10.6.8

Please provide any additional information below.

This happened for 13 story URLs by this author

Original issue reported on code.google.com by [email protected] on 25 Mar 2013 at 11:27

Fanfiction.net Broken Again

What steps will reproduce the problem?
1. Try to download a story from Fanfiction.net
2.
3.

What is the expected output? What do you see instead?

Fanfiction.net worked fine yesterday. Something changed on their end (again) I 
suspect. Here is the output today:

=Trying newer chapter: http://www.fanfiction.net/s/8120562/14/
Traceback (most recent call last):
  File "downloader.py", line 219, in <module>
    main()
  File "downloader.py", line 199, in main
    output_filename=writeStory(config,adapter,options.format,options.metaonly)
  File "downloader.py", line 41, in writeStory
    writer.writeStory(outstream=outstream,metaonly=metaonly)
  File "C:\Users\randalls\apps\fanficdownloader-4.0.3\fanficdownloader\writers\base_writer.py", line 188,
 in writeStory
    outfilename=self.getOutputFileName()
  File "C:\Users\randalls\apps\fanficdownloader-4.0.3\fanficdownloader\writers\base_writer.py", line 118,
 in getOutputFileName
    return self.getBaseFileName()
  File "C:\Users\randalls\apps\fanficdownloader-4.0.3\fanficdownloader\writers\base_writer.py", line 121,
 in getBaseFileName
    return self.story.formatFileName(self.getConfig('output_filename'),self.getConfig('allow_unsafe_filen
ame'))
  File "C:\Users\randalls\apps\fanficdownloader-4.0.3\fanficdownloader\story.py", line 303, in formatFile
Name
    return string.Template(template).substitute(values).encode('utf8')
  File "C:\Users\randalls\apps\python27\lib\string.py", line 172, in substitute
    return self.pattern.sub(convert, self.template)
  File "C:\Users\randalls\apps\python27\lib\string.py", line 162, in convert
    val = mapping[named]
KeyError: 'title'

What version of the product are you using? On what operating system?

4.4.16b on Win7 Home Premium 64-bit

Please provide any additional information below.

none.

Original issue reported on code.google.com by [email protected] on 28 Jun 2012 at 6:13

At Petulant Poetess site, fic downloads fail because of NoneType' object has no attribute 'findAll

Are you using the CLI, web service or calibre plugin?

calibre plugin


What is the story URL?

http://www.thepetulantpoetess.com/viewstory.php?sid=8666

What is the output format you used?

m0bi

If calibre plugin, did you use Add New, Update EPUB if New Chapters, Update
EPUB Always, Overwrite if Newer, Overwrite Always or Update Calibre
Metadata Only?

add new


What steps will reproduce the problem?
1. go to calibre plug in
2. select add new from URL(s)
3. select "mobi" output, Overwrite if newerm ok

What is the expected output? What do you see instead?
a mobi format story to download to calibre

FFDL download log says:

Status: Bad 
Title: Unknown
Author: Unknown
Comment: 'NoneType' object has no attribute 'findAll'
URL: http://www.thepetulantpoetess.com/viewstory.php?sid=8666&i=1


What version of the product are you using? On what operating system?

v1.7.14 on OS X 10.6.8

Please provide any additional information below.

This happens pretty often at this site

Original issue reported on code.google.com by [email protected] on 25 Mar 2013 at 11:17

Add a marker of sorts for WIPs?

I'm using the CLI and I've been wondering, is it doable to add a marker of 
sorts for WIPs? Like, adding (WIP) or chapter numbers like (1-5) or something 
like that into the title for fanfic without the metatag "complete"?

No idea if it's even possible, but I thought I'd ask.

Original issue reported on code.google.com by [email protected] on 23 Feb 2013 at 5:36

Crashes on fanfiction.net (probably only on chapters with EN DASH in their title)

What steps will reproduce the problem?
Run the command:
python downaloder.py 
http://www.fanfiction.net/s/2120109/1/Harry_Potter_and_the_Dormant_Power epub

What is the expected output? What do you see instead?
The above command gives the error:
Traceback (most recent call last):
  File "downaloder.py", line 112, in <module>
    loader.download()
  File "downaloder.py", line 50, in download
    urls = self.adapter.extractIndividualUrls()
  File "/Users/mitchell/Code/fanficdownloader/ffnet.py", line 102, in extractIndividualUrls
    u = l.decode('utf-8')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 
210: ordinal not in range(128)

What version of the product are you using? On what operating system?
Revision 54fc9b30ce on Mac OSX 10.6

Please provide any additional information below.
I am not sure if this is correct, but replacing line 102 of ffnet.py, currently:
u = l.decode('utf-8')
with this:
u = l
try:
  u = l.decode('utf-8')
except:
  pass

appears to fix the issue

Original issue reported on code.google.com by [email protected] on 17 Aug 2010 at 2:31

hexfiles format

Are you using the CLI, web service or calibre plugin?
CLI

What is the story URL?
http://thehexfiles.net/viewstory.php?sid=11941

What is the output format you used?
html

What steps will reproduce the problem?
normal downloading process 

What version of the product are you using? On what operating system?
fanficdownloader-4.4.31, ubuntu 12.04

What is the expected output? What do you see instead?
In fiction downloaded from hexfiles, the content table (you know, with the 
category, genre, summary, etc.) has this weird issue: The row with the summary 
has an additional <td> to it, which means the text of the summary is always 
displayed in the far right. 

For example in the url above:

<table class="full">

<tr><td><b>Category:</b></td><td>Harry Potter, One-Shot (Adult)</td></tr>

<tr><td><b>Genre:</b></td><td>Action/Adventure, Crack!Fic, Humour, One 
Shot</td></tr>

<tr><td><b>Characters:</b></td><td>Draco Malfoy, Harry Potter</td></tr>

<tr><td><b>Relationships:</b></td><td>Harry Potter/Draco Malfoy</td></tr>

<tr><td><b>Status:</b></td><td>Completed</td></tr>

<tr><td><b>Published:</b></td><td>2008-09-14</td></tr>

<tr><td><b>Updated:</b></td><td>2008-09-14</td></tr>

<tr><td><b>Rating:</b></td><td>PG-13</td></tr>

<tr><td><b>Warnings:</b></td><td>Adult Language, Sexual Content, 
Violence</td></tr>

<tr><td><b>Chapters:</b></td><td>1</td></tr>

<tr><td><b>Words:</b></td><td>4,233</td></tr>

<tr><td><b>Publisher:</b></td><td>thehexfiles.net</td></tr>

<tr><td><b>Summary:</b></td><td><td class="summary">
When Harry collapses at his birthday party, Ron just knows that Malfoy poisoned 
him. And he's going to find out how.
</td></td></tr>

</table>


Please provide any additional information below.

This issue has always been there even in the previous few versions. It's not a 
real annoyance if the summary is short, but it is when the summaries are long. 
I'm not really sure anything can be done about it, but I thought I'd report it 
anyway.

Original issue reported on code.google.com by [email protected] on 1 Nov 2012 at 11:39

include image

I'm using the CLI version (and also the web version), and I'm trying to include 
images into the downloaded page, but it doesn't seem to work. I added the 
following into the personal.ini, but it didn't work:

include_images:true 
nook_img_fix:true 
never_make_cover:true

I wrote those down both in [default] and in [html], but no difference.

I also tried to include it into default.ini and then in both on the off chance 
it helps, but it didn't work.

What am I doing wrong?

I want to include images inside the page, but not covers. Archiveofourown.org 
for example allows postings of fanart.

The pages I tried to download: 

http://archiveofourown.org/works/248963
http://archiveofourown.org/works/118762

Output is supposed to be html. Command used:
python downloader.py -f html http://archiveofourown.org/works/118762

version used: fanficdownloader-4.4.28, OS Ubuntu 12.04

Can anyone help me out here? I know next to nothing about python.

Original issue reported on code.google.com by [email protected] on 12 Oct 2012 at 11:35

Encoding problems for certain characters in HTML output

I am using Calibre version 0.8.49 with FFDL plugin 1.5.16 on Windows 7 64-bit. 
When download stories to HTML files certain characters are mistranslated. This 
does not happen to all stories only a few, I have listed a few below for 
reference. I have listed the characters below that are affected and some of the 
sites that this occurs on. It seems to be an encoding problem as notepad++ is 
able to see the correct content but when I open the page in Firefox the garbled 
characters are back. I have attached screenshots of the output I get. Please 
let me know if there is anything I have to do to fix this issue.

–
½
é
…
“
”
¾
'

http://www.fanfiction.net/s/6473889/1/
http://www.fanfiction.net/s/4246300/1/
http://www.fanfiction.net/s/3668356/1/
http://www.fanfiction.net/s/4199033/1/
http://www.ficwad.com/story/105190
http://www.fanfiction.net/s/4852650/1/
http://www.fanfiction.net/s/2227035/1/
http://www.fanfiction.net/s/3123793/1/

Original issue reported on code.google.com by [email protected] on 6 May 2012 at 5:48

Attachments:

• screenshots.zip

Prevent from running in older versions (was Syntax error preventing execution)

What steps will reproduce the problem?
1. Execute the program from command line.

What is the expected output? What do you see instead?
Expected output: file name. Actual output: syntax error, line 33 of zipdir.py

What version of the product are you using? On what operating system?
0.4

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 10 Mar 2011 at 8:45

fanfiktion.de

Using the Calibre Plugin (Latest on Calibre latest Version) and the Web Service 
I noticed I couldn't grab any fictions from : Fanfiktion.de
Here are URL's I used : 

http://www.fanfiktion.de/s/46ccbef30000616306614050
http://www.fanfiktion.de/s/4f32db090002392b06501b58
http://www.fanfiktion.de/s/4766f6890000688a0650f230

I tried to grab them in epub. But I got this message : 

'NoneType' object has no attribute 'find'

When you click on source : you are redirected to the story... weird. 

Original issue reported on code.google.com by [email protected] on 24 Oct 2012 at 7:31

Background is always white

What steps will reproduce the problem?

Any download from ff.net (only site I've used so far)


What is the expected output? What do you see instead?

I use a custom CSS in the Calibre Viewer.  This gives me a Dark background and 
light text, better for reading at night.  When I view a file downloaded with 
this plugin the background is white dispute the viewers custom CSS.


What version of the product are you using? On what operating system?

Calibre is 0.8.42, plugin is 1.5.9 (downloaded today)


Please provide any additional information below.

Custom CSS is included in screenshot below.

Original issue reported on code.google.com by [email protected] on 17 Mar 2012 at 2:12

Attachments:

• [Background problem.jpg](https://storage.googleapis.com/google-code-attachments/fanficdownloader/issue-24/comment-0/Background problem.jpg)

Provide a command line option to choose the output file

If this python script is used in another script (for example, to save the 
result to a device), then it would be helpful to choose the output directory.

Original issue reported on code.google.com by [email protected] on 26 Nov 2010 at 10:09

FEATURE REQUEST: return name of finished file or allow post-processing scripts

Hi.

Yet another feature request:

I need to postprocess epubs after the download process to add them to my 
library. Currently (I'm on OS X) I dothis with a folder action, (basically a 
watched directory), but it would be nicer, if I could manage a closer 
integration into the toolchain.

There's two possibilities: 
One: would be that the downloader returns the full filepath, so I can take that 
output and stuff it into my script.

Even better would be a config option where an arbitrary script could be defined 
as postprocessing script, and the downloader would call that script after 
writing the file, eg:

post_process_cmd: /usr/local/bin/addbook -f %epub%

or something similar.

If that's already possible or easily hackable somehow, let me know, please.

Thanks. 

Original issue reported on code.google.com by [email protected] on 2 Nov 2011 at 7:53

Italics formatting is not retained in EPUB output from FictionPress

What steps will reproduce the problem?
1. Entered this URL:  http://www.fictionpress.com/s/2590716/1/Tenebrism
2. Selected EPUB format
3. Downloaded EPUB file of story
4. Opened with Stanza on laptop 

What is the expected output? What do you see instead?
There are lots of italicized words in this story, but that formatting
didn't show up in the EPUB version - it was just standard text.  (I went
back and tried HTML, and it worked fine, italics and everything.)

What version of the product are you using? On what operating system?
version? um...the website? (http://fanfictionloader.appspot.com/)  
And Mac OS X

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 4 Apr 2010 at 8:49

Attachments:

• [Alyn Drasil - Tenebrism.epub](https://storage.googleapis.com/google-code-attachments/fanficdownloader/issue-1/comment-0/Alyn Drasil - Tenebrism.epub)

Question: file output folder

Are you using the CLI, web service or calibre plugin?
CLI

This may be a stupid question, but. I'm using the CLI, and I don't know how to 
change the file output folder. 

I checked if there was a such a command listed in python downloader.py --help 
but there isn't. 

Original issue reported on code.google.com by [email protected] on 23 Feb 2013 at 5:28

Using prettify() changes whitespace.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Examine the whitespace in, for instance, the early paragraphs of chapter 2.

What is the expected output? What do you see instead?
The original document contained the HTML '<p>"<i>Wingardium Leviosa.</i>"</p>'; 
this will be rendered in the output as '" Wingardium Leviosa. "', rather than 
'"Wingardium Leviosa."' (with italics, of course.) Spacing should be preserved.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10), plus the 
patches from issue 6, issue 7, issue 8, issue 9 and issue 10 (which don't 
affect this issue).

Please provide any additional information below.
Using the prettify() method causes whitespace to appear between tags. See the 
manual:

http://www.crummy.com/software/BeautifulSoup/documentation.html#Printing%20a%20D
ocument

"The prettify method adds strategic newlines and spacing to make the structure 
of the document obvious. It also strips out text nodes that contain only 
whitespace, which might change the meaning of an XML document. The str and 
unicode functions don't strip out text nodes that contain only whitespace, and 
they don't add any whitespace between nodes either."

It's the additional whitespace which is a concern here. It's not a validation 
problem, but it can and does make the output look a little odd when mysterious 
spaces appear between, for example, quote marks and their contents.

I believe this can be fixed by replacing "x.prettify()" with "str(x)" where it 
appears, but I'm not sure how many times it's required just yet. It makes the 
output somewhat more difficult to read, but this shouldn't be cause for 
changing the presrentation.

Original issue reported on code.google.com by [email protected] on 20 Sep 2010 at 2:44

CLI can't download from fanfiction.net

Are you using the CLI, web service or calibre plugin?
CLI

What is the story URL?
Example: www.fanfiction.net/s/6680379/1/Heresy
But all pages from fanfiction.net, really.

What is the output format you used?
html

What steps will reproduce the problem?
Trying to download as usual: 
python downloader.py -f html  http://www.fanfiction.net/s/6680379/1/Heresy


What is the expected output? What do you see instead?
Html. Error message.

What version of the product are you using? On what operating system?
version 4.4.50, Ubuntu 12.04

Please provide any additional information below.
I have no idea what happened. Only the day before it worked fine. Now this is 
the error message I get for any and all links I'm trying to download per CLI, 
while the web app works just fine as always:

Traceback (most recent call last):
  File "downloader.py", line 264, in <module>
    main(sys.argv[1:])
  File "downloader.py", line 200, in main
    adapter.getStoryMetadataOnly()
  File "/home/fanficdownloader-4.4.50/fanficdownloader/adapters/base_adapter.py", line 238, in getStoryMetadataOnly
    self.extractChapterUrlsAndMetadata()
  File "/home/fanficdownloader-4.4.50/fanficdownloader/adapters/adapter_fanfictionnet.py", line 163, in extractChapterUrlsAndMetadata
    for b in graydiv.findAll('button'):
AttributeError: 'NoneType' object has no attribute 'findAll'

Original issue reported on code.google.com by [email protected] on 29 Mar 2013 at 2:13

Feature request: use ini variables in default cover definition

It's easy for me to hack myself a little cover image script and use that as the 
URL for the cover. Unfortunately variables like ${title} are not expanded. 

If that's easy to fix, it would be nice to have.

Original issue reported on code.google.com by [email protected] on 17 Mar 2012 at 2:58

Request to read config(ini) from CLI option

I know I should fill out some sort of 'bug report', but this is just dumb. This 
program apparently reads two files in the current directory as the config.

There's no way to specify another config file on the command line, and there's 
no way for it to locate its config file if you, you know, happen to be running  
it from another directory.

I can't even imagine how this begins to make sense to anyone. How, exactly, is 
anyone imagining this program gets used? Are we supposed to chdir into the 
program directory before we use it? Are we supposed to stick personal.ini files 
in our epub storage directory? What exactly is the presumed process here?

I was writing a nice little interface to download things and automatically 
update them and mail them out when they change, and just require someone to put 
fanficdownloader somewhere and point to it...but apparently I'm going to have 
to include the entire thing along with my php script, and hack the 
config.read('personal.ini') line to actually _find_ the config files.

I hate to be so blunt, but, seriously, this is idiotic. At the _bare minimum_ 
give me the option to specify a config file on the command line, so I can 
override the crazy-ass default of 'let's look for the config files in the 
current directory, and only the current directory!'

Original issue reported on code.google.com by [email protected] on 18 Sep 2011 at 4:13

EPUB output is invalid: unique-identifier in OEBPS/content.opf doesn't point to an ID field.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Test it with epubcheck or http://threepress.org/document/epub-validate/. One 
of the errors will be: "ERROR: hprationality.epub/OEBPS/content.opf(3): bad 
value for attribute "unique-identifier"".

What is the expected output? What do you see instead?
The EPUB output is invalid. While it may work on some devices, it may fail on 
others.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10), plus the 
patches from issue 6 and issue 7 (which don't affect content.opf generation).

Please provide any additional information below.
There are several different issues causing validation to fail. This is one of 
them. See section 2.1 of the current OPF draft, and a lay explanation of what 
how the unique-identifier attribute is used:

http://www.idpf.org/doc_library/epub/OPF_2.0.1_draft.htm#Section2.1
http://netkingcol.blogspot.com/2010/01/closer-look-at-opf.html

The value of the "package" element's "unique-identifier" attribute is an IDREF, 
which points to an element with that ID. (It should, therefore, be constant for 
a format.) So, if we set it to "BookId", then the unique identifier for the 
EPUB will be the contents of an element setting an ID attribute to "BookId". 
(Previously, the unique-identifier was being randomly set, so it didn't point 
to anything; the element with 'BookId' as its ID had constant content.)

The attached patch does this, and changes the constants to fit.

Some resources suggest using an ISBN or the like for an ID, meaning that the 
unique-identifier is the same across all copies of a book. That's beyond the 
scope of this particular report, but Python's UUID module supports making UUIDs 
from URLs, which could very well be the easiest way to get a canonical UUID for 
a fic.

Original issue reported on code.google.com by [email protected] on 16 Sep 2010 at 4:30

Attachments:

• unique-identifier.patch

[Feature Request] Change background colour for HTML

Currently the background colour of downloaded HTML files is set to white.

I usually change this to light gray before I start reading, because reading 
continuously on a PC for a long time makes my eyes tired due to the bright 
white background.

Could there be a per-user setting for a preferred background colour? The 
default setting for all users will remain white.

http://www.ficsavers.com/savers/ has this feature, but unfortunately it doesn't 
support fictionally downloads.

Thanks for a very useful app!

Original issue reported on code.google.com by [email protected] on 14 Dec 2011 at 12:11

EPUB output is invalid: 'mimetype' is not the first file, and is compressed.

What steps will reproduce the problem?
1. Generate EPUB output, e.g., "python downaloder.py 
http://www.fanfiction.net/s/5782108/1/ epub".
2. Test it with epubcheck or http://threepress.org/document/epub-validate/. The 
first error will be "ERROR: Harry_Potter_and_the_Methods_of_Rationality.epub: 
length of first filename in archive must be 8, but was 22".

What is the expected output? What do you see instead?
The EPUB output is invalid. While it may work on some devices, it may fail on 
others.

What version of the product are you using? On what operating system?
I'm using current tip (26:54fc9b30ced5) on Python 2.6.4 (Ubuntu 9.10).

Please provide any additional information below.
There are several different issues causing validation to fail. This is the 
first one. See section 4 of the OCF standard (part of what makes up EPUB): 
http://www.idpf.org/ocf/ocf1.0/download/ocf10.htm

In short, the file 'mimetype' must appear first in the zipfile, it must be 
uncompressed and have no extended attributes, and it must contain precisely the 
text 'application/epub+zip'.

The attached patch fixes this--the list of files sent to the ZIP engine is now 
ordered, with 'mimetype' at the top--and the version of Python I have doesn't 
support setting per-item compression, so all files are now uncompressed. I'm 
not a Python coder, and this makes filesize significantly worse, since the 
content files are uncompressed. You may want to revisit my methods, to say the 
least.

Original issue reported on code.google.com by [email protected] on 15 Sep 2010 at 8:23

Attachments:

• mimetype-first-uncompressed.patch