flameish / novel-grabber Goto Github PK

Hello there!

First of all, I would like to thank you for the wonderful application and all the work you've put into it.

I'm facing a problem with WuxiaWorld on Novel Grabber that is the access to VIP-only chapters. For example, I Shall Seal The Heavens from chapter 101 and forward is locked to VIP users, and so they appear on the ebook generated by Novel Grabber like this

Is there any way to fix this? I tried using Webnovel, but although Webnovel lists 1611 chapters, Novel Grabber only identifies 40, since chapters 41 and forward require coins.

Only tried it on dreame but the rendered epub is only titled as unknown.epub. Also, no cover and metadata. Option for get images was checked.

Hello! I would like to request if we could just copy and paste a list of URL links and their chapter names in the automatic/manual tabs so that it would be easier to edit?
For example:
https://www.website.comChapter 1- Chapter Name

It's just that there are some websites that do not automatically add new chapters and adding them always every time I try to download is becoming time-consuming.

I would also like to request if https://foxaholic.com/ could be added to the list of websites supported? Thank you very much! :)

https://justlightnovels.com/

After trying to download novels from Comrademao this is what happens.

[INFO]Fetching novel info...
[ERROR]HTTP error fetching URL

Please add https://wuxiaworld.site/ it has up to date release of webnovel chapters

Thank you

When you manually scrape from a table of contents link and there is repeats in the link text (e.g. there is multiple "Chapter 1"s) the html file of these will all be named "Chapter-1.html" and therefore only the last-scraped "Chapter 1" will be used for all chapters called "Chapter 1". A situation where this commonly occurs is when the novel is split of into volumes and the link text does not include the volume and the chapter count is reset after every volume.

This issue only occurs starting from this commit: 1e8e89f

Before this commit each chapter has a unique, always incrementing 5 digit prefix (e.g. "00001-Chapter-01.html") and therefore does not have this bug.

The main issue seems to be that the grabber doesnt allow for javascript since the epub only shows
" Chapter loading... Retry Chapters cannot be loaded without javascript."

Any chance you can add a vutton to enable js or fix issues like this?

before when a chapter would download it would say 568-569 series name but now that is gone and there seems to be a bug when i set it to only title but it keeps moving to back to authors name series title

Right now I believe the only option is EPUB but the software i use cant read that, only PDF. So can there be an option to change what the output file format is?

When URL of the novel is provided in the Novel URL option and then "check availability of novel" It doesn't show the info. and gets stuck at:
[INFO]Fetching novel info...
I have tried it with both with and without headerless browser option and am using the latest 2.4.4 version.I have tried URL of several other supported sites but they work fine and show the info.of the novel after clicking "check availability of novel" but when I do this with Comrademao novels it just shows the black Loading spiral continuously even after a long time where the info. of the novel is supposed to appear.

Hello,
Is the temporary folder that was created when manually downloading novels no longer visible? It was useful when I wasn't sure if the chapter container that I entered was correct and it allowed me to quickly visualize the content of the html in case of just downloading an empty html body.

Thank you.

I tried the newest version it give me the JNI error has occurred ERROR but when i switch to the 2.3.4 version it works just fine

When executing the jar on Arch Linux with :
java version "14.0.2" 2020-07-14
Java(TM) SE Runtime Environment (build 14.0.2+12-46)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

The packages I installed for Java were:
jdk 14.0.2 (aur)
jre 140.2 (aur)

I get the following error (and an empty window opens):

Exception in thread "pool-1-thread-1" java.lang.NoClassDefFoundError: javax/activation/DataSource
	at library.LibrarySystem.pollLibrary(LibrarySystem.java:21)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 4 more

I've managed to use the application before on Ubuntu/Deb based systems, but no luck on Arch Linux. I'm guessing it is becausing I'm missing a package and online searching (Google/StackOverflow) leads me to believe the missing package was present in Java 8, but was removed in later versions.

This is due to an issue Where I originally had a series download to a network drive but in the library it set all future downloads instead to c/user/novel-grabber-series. I don't know why this happened or if I just forgot to do something. Now comes the issue: Even if I set up it correctly now every series up to till will be missing chapters in the original location. So I either need a way to re-download the entire series or the program needs to be able to scan a folder and see what is missing.

Also there seem to be some folder bug: when a new chapters were added it created to folders. Ex: for Martial God Asura there was one 'Martial" folder and a 'Martial God Asura" folder. Martial had the actual chapters and the other pictures. Overgeared did not have this issue so I assume its a bug with more than one word

I noticed that although all the chapters are downloaded, the content of the chapters are incomplete in Dreame. I tried setting a waiting period, but the result is the same.

There are also some strange characters in the file content like € ™ � â etc. The encoding in the file says it's utf-8.

Novel-Grabber can't get the chapter list from Webnovel, neither manually nor automatically. The manual grab just gets navigation links from the website, while the automatic checking doesn't finish.
Webnovel.com chapters scroll continuously, there is no "next chapter" button.

Tried to download some novels on novelfull, when I get to chapter 900-1000ish, it just stops.

Ran it with the console and got the following error code:
Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap space
at org.jsoup.parser.CharacterReader.(CharacterReader.java:36)
at org.jsoup.parser.CharacterReader.(CharacterReader.java:41)
at org.jsoup.parser.TreeBuilder.initialiseParse(TreeBuilder.java:38)
at org.jsoup.parser.HtmlTreeBuilder.initialiseParse(HtmlTreeBuilder.java:65)
at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:46)
at org.jsoup.parser.Parser.parseInput(Parser.java:35)
at org.jsoup.helper.DataUtil.parseInputStream(DataUtil.java:175)
at org.jsoup.helper.HttpConnection$Response.parse(HttpConnection.java:835)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:287)
at grabber.scripts.ChapterContentScripts.defaults(ChapterContentScripts.java:222)
at grabber.scripts.ChapterContentScripts.fetchContent(ChapterContentScripts.java:45)
at grabber.Chapter.saveChapter(Chapter.java:48)
at grabber.Novel.downloadChapters(Novel.java:137)
at gui.GUI.lambda$null$2(GUI.java:332)
at gui.GUI$$Lambda$67/20323159.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

It seems that Java ran out of memory. Is there a way to allocate more RAM to the program? Thanks for your work.

it would be nice to have ScribbleHub added.

If i leave it on and it was originally on chapter 105 it wont move from that spot until i restart the program. When it is restarted it wont remember what it already downloaded and now will have 2 copies of older updates along with the new chapters

Hi! I can't seem to download from CG lately. I can't fetch novel info. Can I output the epub files as what I put in the Title like with version 2.6.0? I think I would prefer that. Thank you! :)

can you please make the output in txt /pdf / html format and make chapter as list like this
chapter 1
chapter 2
chapter 3
chapter 4
chapter 5
.
.
.
.
.etc...
🙏🙏🙏🙏

Hello! I was hoping if maybe there could be a way to manually add missing chapters? Because sometimes the website does not include a chapter in their list of contents or forgot to link it. It would also be cool if there's a feature where we could rearrange some chapters and not just invert chapter order.

Thank you very much!

twice now, at chapters 930 and 1037 for two different series the program with now freeze and your forced to to go into task manager to shut it down

I click the button and nothing happens, I don't know if there is a bug or it the program is actually working on it. There seems to be a bug when i added a series form wuxiaworld to my favorites and it shows latest chapter 407 and newest chapter 0. I click "download chapters form last downloaded" and nothing happens, it seems unable to download anything through the button on the library tab

I tried using all the host sites to see if it was an issue with only one of them, but none of them load at all. i waited about 30 minutes for each host site, trying to fetch the novel info with a chapter range of 1-5 and absolutely nothing happened for any of them. Maybe I am at fault and should have waited an hour for it to fetch the info, but I just don't have that type of patience so that's on me.

It would be a nice if you could add An About Section & Table Of Contents that will be there after cover Image in the EPUB file.

In the end of the book, there could be a credit section where the Author & translator names can be added with source of DDL of the EPUB file.

Also characters Like "&" get missed out of name when Downloading Novels like Tales of Demons & Gods

Hi,
I would like to request a possibility of logging in on websites that have VIP/Paid stories. Also, if you can include booklat.com.ph as a source.

Do you have any plans on porting this as a telegram bot?

Thanks.

Hello! I can't seem to download anything from Flying Lines lately.

This is the error code from version 2.3.1

2749 [pool-1-thread-1] INFO io.github.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\pc.m2\repository\webdriver\chromedriver\win32........blahblah.......\chromedriver.exe
Starting ChromeDriver .........blahblah........... on port ............blahblah
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Jan 20, 2020 12:13:53 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
21549 [pool-2-thread-1] INFO io.github.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\pc.m2\repository\webdriver\chromedriver\win32.......blahblah..........\chromedriver.exe
Starting ChromeDriver ................blahblah.................. on port ............blahblah
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Jan 20, 2020 12:14:10 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C

This error code from version 2.3.2

799 [pool-1-thread-1] INFO io.github.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as C:\Users\pc.m2\repository\webdriver\chromedriver\win32.......blahblah.........\chromedriver.exe
Starting ChromeDriver .......................................... on port .........blahblah
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Jan 20, 2020 12:17:47 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Exception in thread "pool-1-thread-1" java.lang.NullPointerException
at java.io.StringReader.(Unknown Source)
at org.jsoup.parser.Parser.parse(Parser.java:107)
at org.jsoup.Jsoup.parse(Jsoup.java:58)
at grabber.AutoNovel.getChaptersHeaderless(AutoNovel.java:320)
at grabber.AutoNovel.getChapterList(AutoNovel.java:175)
at grabber.AutoNovel.(AutoNovel.java:94)
at gui.GUI.lambda$null$5(GUI.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I currently use version 2.3.1 because I find the previous one more stable for me. For version 2.3.2, Chrysanthemum G would not display the Title and Author....it's only left as Untitled and I prefer not to use a headerless browser.

This is my go-to novel downloader lately and I hope that this would be resolved soon. Thank you very much for your hard work!

Like FMD/FMD2 you search for a series then add it a list where every so often it will look for new chapters

The issue with this one is all the links go to google drives so does that make it impossible?

I tried several versions of the program and they all gave me the error "Unable to run jarfile novel-grabber.jar" when I tired to open them with CMD, trying without CMD gave me a JNI error but I uninstalled java and reinstalled and it did not fix. I finally got version 2.3.4 to work but the newer ones did't. I don't know why because the Error it gave does't really explain much. Just thought I'd leave this here since I'd like to use the never versions as they come out. Thank you for your time.

The thing is, I have realized that not all mtl is equal. As the translation quality varies a lot when comparing sites.

The site requested for adding to grabs.

https://www.mtlnovel.com/

Thank you.

I am trying to create an epub of a novel from Wuxiaworld (tried from Royalroad, too) on mac (java 8). If i export it as an epub then when i try to open it on iBooks it gives me an error because the file is damaged. If i try to export to calibre, when i convert it (like it is shown in the readme), the chapter are all out of order.
Any ideas?

Prints at console:

Exception in thread "pool-7-thread-1" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at grabber.Download.startAutoDownload(Download.java:165)
    at gui.GUI.lambda$null$3(GUI.java:238)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

App is somewhat less stable after - clicking the menu next to the site dropdown resulted in:

Exception in thread "AWT-EventQueue-0" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at gui.autoEditMetadata.createUIComponents(autoEditMetadata.java:135)
    at gui.autoEditMetadata.$$$setupUI$$$(autoEditMetadata.java)
    at gui.autoEditMetadata.<init>(autoEditMetadata.java:32)
    at gui.autoEditMetadata.main(autoEditMetadata.java:87)
    at gui.GUI.lambda$new$29(GUI.java:555)
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
    at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
    at java.awt.Component.processMouseEvent(Component.java:6539)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
    at java.awt.Component.processEvent(Component.java:6304)
    at java.awt.Container.processEvent(Container.java:2239)
    at java.awt.Component.dispatchEventImpl(Component.java:4889)
    at java.awt.Container.dispatchEventImpl(Container.java:2297)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4904)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4535)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4476)
    at java.awt.Container.dispatchEventImpl(Container.java:2283)
    at java.awt.Window.dispatchEventImpl(Window.java:2746)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
    at java.awt.EventQueue$4.run(EventQueue.java:733)
    at java.awt.EventQueue$4.run(EventQueue.java:731)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

I tried updating the program but nothing happened so then i tried restarting my computer and even deleting that folder but the program never pops up now and in task manager is at 0% for the java task. when trying to start either the old or new version of the program the library program was recreated but nothing else

i can't download unlocked chapters in webnovel (not the privilage the normal locked chapters). I can only grab the Free chapters so the epub won't really update

Trying to grab novels from comrade mao but they results in:

[ERROR]String must not be empty
null
Eg. Apotheosis, Yuan Zun and Etc! Please fix it when possible! Also can you give an example of what should be the next tag for Chapter to Chapter Manual Grabbing.

There's no problem with grabbing chapters and the "epub" is created according to the logs. But all files are like this

I tried other sites and there was no problem.

When I tried to download from flying lines, it would say:

[INFO]Starting headerless browser...
[INFO]Connecting...
Cannot find elements when the selector is null

I used the 2.3.0 version.

Hello! It seems like the latest release saves the epub titles wrong. Like for CG, I would name the epub <Title> but the resulting epub would be "Unknown" <Title>. It seems like Boxnovel also has this but with the author instead of "Unknown". I'm not sure if other sites also have this issue though. Thank you!

In CG, particularly in the "please visit ______.com" in every chapter, this line replaced the missing paragraphs. So because of this issue, I tried to run the previous releases, but I think because I deleted the config.json or perhaps because of the new code, I couldn't run them.

There are 2 main issues with it:

• The page novelupates links to is a prechapter messages.

• they have anti "copy" text in their text which the programm picks up too

Hello,

I am attempting to create epubs for several novels on CG and some are not coming out in a readable form. It works for some titles however titles such as this one:

https://chrysanthemumgarden.com/novel-tl/mpcd/

Have an issue with the epub. The reader I am using is ibooks on a 2018 ipad pro running the latest ios.

Novel Grabber puts Translator as Author.

I tried to open this using Firefox's EPubReader extension, but was met with this message: Unfortunately an unexpected error occurred opening the ebook. Please send the following error message to [email protected]. Thanks!: loadPage: TypeError: f.style is undefined

Then I take it into Sigil to edit it and I'm met with this message: This EPUB has HTML files that are not well formed. Sigil can attempt to automatically fix these files, although this can result in minor data loss. Do you want to automatically fix the files?

I'm not sure what was changed, but I allowed Sigil to fix it.

Seems like svg cannot be saved, was wondering if you could add support for saving svg images.

I tried making an epub for Chrysanthemum Garden but the output epub does not contain the content of the novel: only the title and the table of contents. There are blank pages only where the words should be. I hope this would be fixed in the next update! Thank you!

Something got changed with the website and now is not working