menorki / decruft

Automatically exported from code.google.com/p/decruft

This is Menorki's github page

decruft's Issues

does not handle Asian characters correctly.

This patch is to convert the raw html content into unicode, then performs the 
clean_html operation. Otherwise, the non-ASCII characters in raw html file will 
be processed incorrectly.

And this patch also assumed the latest BeautifulSoup module is used, other than 
the one shipped in decruft.

Original issue reported on code.google.com by [email protected] on 31 Oct 2011 at 10:21

Attachments:

• cleaner.diff

Problem with extracted text for Swedish sites

What steps will reproduce the problem?

Use decruft to extract the summary from the following site:
http://svenska.yle.fi/nyheter/artikel.php?id=203724

The extracted text contains some incorrect characters, for example

" Assange måste betala 200 000 pund för sin frihet"

The expected one is "Assange måste betala 200 000 pund för sin frihet"

Original issue reported on code.google.com by [email protected] on 16 Dec 2010 at 7:12