topfunky / hpple Goto Github PK

Hi!

I want to use hpple in my little iPhone project at stumbled upon a problem. When parsing a table in a website, I noticed that text within a tag was not parsed. Following the example I tested with:






Un couple épatant

(2002)

The text (2002) should be content of the <td> tag but querying the respective element's content resulted in an empty string. I debugged and found the cause, although I am not sure whether it broke anything else, which is why I would kindly ask you to look into it.

The problem lies in XPathQuery's DictionaryForNode method:

if ([[resultForNode objectForKey:@"nodeName"] isEqual:@"text"] && parentResult)
{
[parentResult setObject:[currentNodeContent stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]] forKey:@"nodeContent"];
return nil;
}

If I got it right, the parser checks the currentNode's name first and if this is set to "text" (I assume libxml does that) it sets the parentNode's content to that string. The problem here is, it doesn't check whether there is already a string. Instead it replaces a potentially present string, even with an empty string. Here is my solution (again not really tested yet):

Please note that I don't remove newline characters. I'm not sure what's the best approach here, but a string without any seperators doesn't seem like a good idea to me.

I hope this really is a bug. Feedback is appreciated.

Cheers,
Chris

I have a problem, I parse html page and view freezing while its parsing. How to use this library for parsing asynchronously?

Hi, I am currently working on the blog reader and use HPPLe to parse HTML. I would like to use it to get a full structure of a blog post, so to know how different tags are following one after another. What I would like to accomplish in the end is - instead of revealing a blog post in UIWebView, get all its elements and show them in native views. The problem is that structure varies from post to post - so I need to figure out an algorithm that will scan each time post's HTML code and give me its clear structure. Hope you might help me with it. You can also check this question on stackoverflow - http://stackoverflow.com/questions/24041858/recreating-blog-post-structure-in-a-native-ios-view .

When I use this code to parse a website that has an embedded video, it doesn't include the video. Is this a problem with the parser?

I tried to parse Korean in HTML but it isn't working. I think it doesn't use NSString but cString on parsing. Any character must be in ASCII scope if it works
P.S. also cannot parse by unicode

Unable to parse when Used:
1> peekAtSearchWithXPathQuery:@"//div[@id = 'pager']"
2> searchWithXPathQuery:@"//div[@id = 'pager']"

Hi! So i'm am using this cool lib in my app to parse html. it works fine, but i found out a memory leak, each time i use searchWithXPathQuery. i've search all internet, but still have leaks.

in xcode Instruments in table with leaked objects :

Responsible library - libxml2.2.dylib
Responsible frame - xmlStrndup

I'm comparing the results of [rootElement raw] with the actual HTML, and the results are similar, but not the same. Specifically, the characters "]]" have been appended inside each "script" tag.

Um...this isn't correct behavior, is it? Just checking to make sure I understand how the "raw" method is supposed to work.

[Update: Okay, I see why: There's a "![CDATA[" that's been added at the beginning of each script tag. Which is causing the HTML to not behave properly.]

make it better

When I search for the span via simple query string, the following html which looks like this:
"3097"
"Designer: someDesigner"

Ends up coming out like this in the hippleElement.

code:

Basically it doesn't see anything between quotes. Seems like a c parsing issue to me but I'm too retarded to understand your code...lol.

libxml/tree can be find .
even pasted the ${SDKROOT}/usr/include/libxml2 to the header search

This query:

"//body/table[1]/tr/td[@align='center']/table[@cellspacing=0]/tr[@height=24]/td[@colspan=3]/table[@cellspacing=0]/tr/td[@align='right']/a[1]/@href"

on this page:

"http://www.phdcomics.com/comics/archive.php?comicid=1238"

should return the "href" attribute of the first "a" tag that matches the prefix. Instead, it seems to return the hrefs of all the "a" attributes that matches the prefix (4, in this case).

The library is not available through cocoapods. After adding it to Podfile if I run pod install it says [!] Unable to find a specification for Hpple.

How to use it according to parse gb2312 html? Thank you.

NSStringEncoding encoding = CFStringConvertEncodingToNSStringEncoding(kCFStringEncodingGB_18030_2000);
TFHpple *xpathParser = [[TFHpple alloc] initWithHTMLData:htmlData encoding:[NSString stringWithFormat:@"%lu",(unsigned long)encoding]];

And it doesn't help me .

It would be really nice to have a nice documentation for this library.

I have to use 'EUC-KR'
but hpple loves 'NSUTF8StringEncoding'

How can i make it possible to show EUC-KR(-2147481280)?

Cool lib TopFunky

Out of interest, I couldn't get it to work for this xml at all :(

Every XPath query I try returns an empty array. Tried debugging it too, can't see what's going on. Also tried stripping out the xmlns stuff. No joy. Response string looks good too, hpple just don't dig it :(

        ASIFormDataRequest *storeRequest;
    NSString *urlString = @"http://clients.multimap.com/API/search/1.2/asdanew?countryCode=GB&returnFields=wal_mart_no,name&routeModes=driving&lat=53.799720&lon=-1.549170&orderByFields=Distance&count=10";    

    storeRequest = [ASIHTTPRequest requestWithURL:[NSURL URLWithString:urlString]]; 
    [storeRequest startSynchronous];    
    NSError *error = [storeRequest error];
    if (!error) {       
        NSString *response = [storeRequest responseString];     
        NSData *data = [response dataUsingEncoding:NSUTF8StringEncoding];       

        //NOT WORKING - TFHpple Not Parsing At All :(
        TFHpple *doc = [[TFHpple alloc] initWithXMLData:data];
        NSArray *records = [doc search:@"//Field" ];
        NSLog(@"%@", records);
        }

Is it possible to get innerHTML?

I have some html like
< span>text< br>text2< /span>, and only text2 will show up on [element content].

So i dont know how to get the full content, i remember Hpricot offers something like innerHtml.

Thank you!

my html is
<html><body><div id='nativerich'>!;&amp;(*)</div></body></html>

when I use hpple parser my html

I got div raw string : <div id="nativerich">!;&amp;(*)</div>
but div content string: "!;&(*)"

how to make div raw string content be same as div content string ,parse & as &

Seeking for Help in Emergency

Hi,

I have tried various xpath expressions to get all images, within lists, divs, etc. But, not all images (or script tags) are found.

Not sure if its an xpath issue, but how can the entire tree be iterated through, searching for these tags ... while it may be slower, brute force sometimes works.

Thanks,

Peter

When I try and retrieve text from an html link, I can output it correctly in the NSLog but when I try and set a label's text to it, nothing shows up. This is confusing because I see the data in the log but I cannot make it appear in a label. It works if I put it in a text field which is strange but I want a label. Thanks for the support.

Trying to use a query that works in Chrome console but not in hpple

"//ul//li/span[normalize-space(text())='10']"

Any ideas on how to replicate this to work with hpple would be greatly appreciated.

The raw methods return a " " after every opening tag.

Example :

     <h4>&#13;Lorem Ipsum</h4>

Hi,

First thank you for your work on Hpple, it helps a lot.

In an unusual case in my code when the query don't match I've hit a memory leak, please see this diff.

-- seb

For Example:
div class="content noleft"
table class="single playlist"

how to solve this problem?

Many websites has text content of paras inside double quotes, instead of enclosing

HTML tags. It is not parse such sites.

For the following HTML :-

It is not possible to extract the text because TFHppleElement node with class 'article' has all the text without parent HTML tag in its content.

It has various child nodes with "a", "em" tags, the rest is in its content.

See http://appleinsider.com/articles/14/08/02/al-qaeda-prefers-android-over-apples-ios for example.

TFHppleElement * div = [[doc searchWithXPathQuery:@"//div[@Class='article']"] objectAtIndex:0];
po div.content (it won't print text which is hyperlinked or italicised.

add the libxml.2.2 the Xcode says can not find lixml/tree in the XPathQuery.h file

Looks like all the Testcases are out-dated. Its looking for the methods that doesn't exist any more. Shouldn't this be corrected?

I'm wondering if support for searchWithXPathQuery will be available on the TFHppleElement itself.

Let's say I parse a complete html document with Hppl and select a repetitive part like tr's on a table through xpath.
I then got an array of TFHppleElements for these tr's.
I then like to search in this subset for some child nodes with xpath to get more detailed information.

Will this be possible?

I'd like to see support for types other than XPATH_NODESET such as XPATH_STRING, XPATH_NUMBER, and XPATH_BOOLEAN. Perhaps a more general object wrapper could be created to store other data types and be returned instead of TFHppleElements only.

  dyld: Library not loaded: @rpath/Hpple.framework/Hpple
  Referenced from: /private/var/mobile/Containers/Bundle/Application/A40EF408-0A65-44AB-8AE0-    EF4E253836BE/ZXDev-iOS.app/ZXDev-iOS
  Reason: image not found

why?

The following code crashes:

NSData* data = [NSData dataWithContentsOfURL:[NSURL URLWithString:@"http://tvtropes.org/pmwiki/pmwiki.php/Comicbook/TheAvengers"]];
    TFHpple* hpple = [TFHpple hppleWithHTMLData:data];
    NSArray* array = [hpple searchWithXPathQuery:@"//div"];

with output:

'NSInvalidArgumentException', reason: '-[__NSCFDictionary setObject:forKey:]: attempt to insert nil value (key: nodeContent)'

Using queries like @"//table", @"//td", @"//tr" also crashes, while @"//span", @"//a" do not.

I've put this in a simple, separate class to isolate the issue. The search I'm actually trying to do is @"//div[@Class='indent']", which crashes on the above URL and many other TVTropes pages, but does work on for example http://tvtropes.org/pmwiki/pmwiki.php/Film/TheAvengers- but plain @"//div" doesn't work there either.

I’ve recently added hpple to the CocoaPods package manager repo.

CocoaPods is a tool for managing dependencies for OSX and iOS Xcode projects and provides a central repository for iOS/OSX libraries. This makes adding libraries to a project and updating them extremely easy and it will help users to resolve dependencies of the libraries they use.

However, hpple doesn't have any version tags. I’ve added the current HEAD as version 0.0.1, but a version tag will make dependency resolution much easier.

Semantic version tags (instead of plain commit hashes/revisions) allow for resolution of cross-dependencies.

In case you didn’t know this yet; you can tag the current HEAD as, for instance, version 1.0.0, like so:

$ git tag -a 1.0.0 -m "Tag release 1.0.0"
$ git push --tags

Could you bump the version so the latest code can be pulled in through CocoaPods?

If you no longer want to maintain this I can fork and push it into CocoaPods as a separate project. Just wanted to check what you want to do first.

Thanks.

Hello, you stated within the title that it can be used for HTML and XML.
Can I use it for XML now?
I did a text but it does not seems to work. HTML works well.

Please tell me, what's missing if it is possible to parse XML.

For one of my projects I have a lot of C++ code which requires me to specify the Apple LLVM 5.1 - Language as 'Objective-C++' rather than leave it as 'According to file type'.
This produces two compilation errors in XpathQuery.m
XPathQuery.m:192:11: No matching function for call to 'htmlReadMemory'
XPathQuery.m:218:11: No matching function for call to 'xmlReadMemory'
where the detail is (for the first one):
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.1.sdk/usr/include/libxml2/libxml/HTMLparser.h:206:3: Candidate function not viable: cannot convert argument of incomplete type 'const void *' to 'const char *'
NB when I am able to leave the compiler setting as 'According to file type' I get no errors.

Now that we can initialize a TFHpple with an encoding parameter, one has to search through the libxml2 code to find what literal values are allowed for the encoding. This is not in keeping with the way string encoding is handled throughout Objective-C. Instead hppleWithHTMLData: encoding: and similar methods should take an NSStringEncoding enum, such as NSUTF8StringEncoding or NSISOLatin1StringEncoding, and then pass the appropriate encoding string to libxml2.

Unfortunately this would break compatibility of the API for anyone already using the new with-encoding methods; perhaps still more methods like hppleWithHTMLData: enumEncoding: should be created?

I use Profile to check if my app has memory and think following code has memory leak bug:

TFHpple.m

• (TFHpple *) hppleWithData:(NSData *)theData isXML:(BOOL)isDataXML {
  return [[[self class] alloc] initWithData:theData isXML:isDataXML];
  }

should be:

• (TFHpple *) hppleWithData:(NSData *)theData isXML:(BOOL)isDataXML {
  return [[[[self class] alloc] initWithData:theData isXML:isDataXML] autorelease];
  }

TFHppleElement.m

• (TFHppleElement *) hppleElementWithNode:(NSDictionary *) theNode {
  return [[[self class] alloc] initWithNode:theNode];
  }

should be:

• (TFHppleElement *) hppleElementWithNode:(NSDictionary *) theNode {
  return [[[[self class] alloc] initWithNode:theNode] autorelease];
  }

This is what's used currently:

//
//  XPathQuery.h
//  FuelFinder
//
//  Created by Matt Gallagher on 4/08/08.
//  Copyright 2008 __MyCompanyName__. All rights reserved.
//

Original article: https://www.cocoawithlove.com/2008/10/using-libxml2-for-parsing-and-xpath.html

Here's the license specified: https://www.cocoawithlove.com/about/

// Copyright © 2008-2018 Matt Gallagher ( http://cocoawithlove.com ). All rights reserved.

// Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee
// is hereby granted, provided that the above copyright notice and this permission notice appear in
// all copies.

// THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
// AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
// FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
// NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH
// THE USE OR PERFORMANCE OF THIS SOFTWARE.

Dear topFunky, how should i ignore <<br />>?
the html file like this:
<div class="Fiche">
<p>Every Sunday, our Chef proposes a buffet high in color.<br />
<br />
A brunch <br />
<br />
Every <br />
<br />
Information/ reservations : (377) 98 06 03 60</p>

When i parse the text between<p> and</p>, what i only get is: Information/ reservations : (377) 98 06 03 60.
How should i ignore <br />, thanks a lot!

My code:
NSArray *array4Soustitre = [xpathParser search:@"//div [@Class='Fiche']/p"];
TFHppleElement *ele= [array4Soustitre objectAtIndex:0];
NSLog(@"content is %@ ",[ele content]);

How can I get the raw HTML of an element, not include the tags of that element? For example, if i were getting , and body looks like this:

How can I get only:

stuff in body

Currently the raw property includes the tags of the target element itself.

see also:
http://stackoverflow.com/questions/27198558/blocked-when-trying-to-get-the-content-of-html

my html code is like this:

• python1
• python2

And my objc code is:
TFHpple* hpple = [TFHpple hppleWithHTMLData:data];
NSArray* links = [hpple searchWithXPathQuery:@"//a[@href!='']"];
for (TFHppleElement* link in links) {
NSLog(@"link's parent: %@", link.parent);
}

The output shows that the link's parent is nil, while it should be

hi,
first thank you for hpple. It helps a lot.

Recently when I use hpple to parse a html file, the content, which should have something in it, is null.
Here is part of the html file:

        发信人:
            znslm
            (小白), 信区: Pictures 标 题: Re: 你十六岁喜欢的那个人怎么样了？发信站: 南京大学小百合站 (Mon Nov 7 21:49:51 2011)    小孩上小学了
            
            
            
            喜欢我的那位呢/ 我猜的 --
        
        

how can I get the image url in this string ?

See also: http://stackoverflow.com/questions/27820804/objective-c-xpath-get-multiple-tags-value

Cause I need to iterate several times to parse, it is not effectively.

Hi,

thx for this library, I was using it and I saw a memory leak while using it.
I had a quick look at it, please test the fix below (lines with "//RNR")

hope it helps, it corrected my issue :-)

NSDictionary *DictionaryForNode(xmlNodePtr currentNode, NSMutableDictionary *parentResult,BOOL parentContent)
{
NSMutableDictionary *resultForNode = [NSMutableDictionary dictionary];
if (currentNode->name) {
NSString *currentNodeContent = [NSString stringWithCString:(const char *)currentNode->name
encoding:NSUTF8StringEncoding];
resultForNode[@"nodeName"] = currentNodeContent;
}

xmlChar *nodeContent = xmlNodeGetContent(currentNode);
if (nodeContent != NULL) {
    NSString *currentNodeContent = [NSString stringWithCString:(const char *)nodeContent
                                                      encoding:NSUTF8StringEncoding];
    if ([resultForNode[@"nodeName"] isEqual:@"text"] && parentResult) {
        if (parentContent) {
            NSCharacterSet *charactersToTrim = [NSCharacterSet whitespaceAndNewlineCharacterSet];
            parentResult[@"nodeContent"] = [currentNodeContent stringByTrimmingCharactersInSet:charactersToTrim];
            xmlFree(nodeContent); //RNR
            return nil;
        }
        if (currentNodeContent != nil) {
            resultForNode[@"nodeContent"] = currentNodeContent;
        }
        xmlFree(nodeContent); //RNR
        return resultForNode;
    } else {
        resultForNode[@"nodeContent"] = currentNodeContent;
    }
    xmlFree(nodeContent);
}

Rachid

as title.

Hi
i use your class in my app to display image of a gallery in a blog under wordpress.com

In iPhone works perfectly , but when i use hpple on Apple Watch it's works on simulator but on fisical device it won't works !
When the parsing arrive on this line of code

TFHpple *parser = [TFHpple hppleWithHTMLData:appleWatchHtmlData];
NSLog(@"parser %@", parser);
NSString *tutorialsXpathQueryString = @"//div[@class='gallery galleryid-1461 gallery-columns-3 gallery-size-thumbnail']/dl/dt/a/img";

NSArray *imageFilesArray = [tutorialsParser searchWithXPathQuery:tutorialsXpathQueryString];
NSMutableArray *imagesArray = [[NSMutableArray alloc] initWithCapacity:0];

the nslog say
2015-10-25 18:19:13.939 WatchApp Extension[398:328494] parser <TFHpple: 0x1655d7e0>
2015-10-25 18:19:13.947 WatchApp Extension[398:328494] Unable to parse.

Perhaps it may be due to the limited apple watch resources, but how can i solve this issue ?

Thank you so much