adenosie / lude

Ludicrously fast (censored) scraper

lude

This project is in very early stage; More awesome features will come soon. Keep watching me!

Todo List

• Provide decent networking
  • Bypass DPI censorships
  • Asynchronous, concurrent, parallel or whatever
• Downloading hentais from:
  • E-hentai
  • Exhentai
  • Hitomi
  • And more...

Examples

from here:

use tokio::time::{sleep, Duration};
use tokio::sync::mpsc;
use std::sync::Arc;
use std::fs;
use std::fs::File;
use std::io::Write;
use std::path::PathBuf;
use lude::ehentai::{Explorer, Article};

async fn parallel() {
    // load article infos 
    let url = String::from("https://e-hentai.org/g/1335995/ba04527f3d/");
    let explorer = Explorer::new();

    let mut article = explorer.article_from_path(url).await.unwrap();
    article.load_image_list().await.unwrap();
    let len = article.meta().length;

    // load images parallelly
    let (tx, mut rx) = mpsc::channel(len);

    let article = Arc::new(article);

    for i in 0..len {
        sleep(Duration::from_millis(100)).await;

        let tx = tx.clone();
        let article = Arc::clone(&article);

        tokio::spawn(async move {
            println!("downloading {}th image..", i);
            let image = article.load_image(i).await.unwrap();
            println!("saved {}th image!", i);
            tx.send((i, image)).await.unwrap();
        });
    }

    let mut path = PathBuf::from("./tests/parallel/");

    // clean the directory
    for entry in fs::read_dir(path.as_path()).unwrap() {
        let entry = entry.unwrap();
        fs::remove_file(entry.path()).unwrap();
    }

    println!("cleaned directory, waiting for images...");

    // save shits
    path.push("0.jpg");

    for _ in 0..len {
        let (i, image) = rx.recv().await.unwrap();
        println!("received {}th image", i);

        path.set_file_name(&format!("{}.jpg", i));
        let mut file = File::create(path.as_path()).unwrap();
        file.write_all(&image).unwrap();
    }
}

License

This project is under MPL 2.0. Unless you modify the code of this library, You can freely link this both statically and dynamically to your code, with no worry of your code being infected. See LICENSE for full license.

lude's Issues

TLS Fragmentation gets 'broken pipe' after sending 3-4 fragments

we should make FRAG_SIZE big enough(>150) so that there are no more than 4 fragments, in order to avoid the server to close connection.

Download images parallelly

It's hard since E-Hentai blocks those who does many(4? 8? I don't know exactly) requests at once. We should work around this by setting a limit of maximum concurrent requests.

Glue detour with hyper::Client

implement a connector so that we can use hyper::Client, which provides name resolving, thread pool, and other nice things.

Bypass school network

connected to school wifi and send client-hello to e-hentai.org:443 over TCP:

16, 03, 01, 02, 00, 01, 00, 01, FC, 03, 03, 72, 40, 60, D2, 8B, 70, A5, 8A, 01, F3, CD, 89, 6D, BF, 55, 66, 9B, 12, 45, A6, 61, 74, 73, EB, 68, E4, BA, 6B, 7D, 05, 65, 8C, 20, D4, 7C, 98, DF, 18, 3E, A5, EA, C2, E8, 30, 48, 66, 6A, 23, 6E, CF, 0C, 4C, 4D, 8A, EE, 66, 18, 22, 13, 59, DC, 6D, 13, D6, 89, 00, 3E, 13, 02, 13, 03, 13, 01, C0, 2C, C0, 30, 00, 9F, CC, A9, CC, A8, CC, AA, C0, 2B, C0, 2F, 00, 9E, C0, 24, C0, 28, 00, 6B, C0, 23, C0, 27, 00, 67, C0, 0A, C0, 14, 00, 39, C0, 09, C0, 13, 00, 33, 00, 9D, 00, 9C, 00, 3D, 00, 3C, 00, 35, 00, 2F, 00, FF, 01, 00, 01, 75, 00, 00, 00, 11, 00, 0F, 00, 00, 0C, 65, 2D, 68, 65, 6E, 74, 61, 69, 2E, 6F, 72, 67, 00, 0B, 00, 04, 03, 00, 01, 02, 00, 0A, 00, 0C, 00, 0A, 00, 1D, 00, 17, 00, 1E, 00, 19, 00, 18, 00, 23, 00, 00, 00, 16, 00, 00, 00, 17, 00, 00, 00, 0D, 00, 30, 00, 2E, 04, 03, 05, 03, 06, 03, 08, 07, 08, 08, 08, 09, 08, 0A, 08, 0B, 08, 04, 08, 05, 08, 06, 04, 01, 05, 01, 06, 01, 03, 03, 02, 03, 03, 01, 02, 01, 03, 02, 02, 02, 04, 02, 05, 02, 06, 02, 00, 2B, 00, 09, 08, 03, 04, 03, 03, 03, 02, 03, 01, 00, 2D, 00, 02, 01, 01, 00, 33, 00, 26, 00, 24, 00, 1D, 00, 20, 1E, 13, 19, 60, F0, 3D, 71, 22, 3F, 56, 3A, 1D, 0D, 01, D5, 00, 41, D5, AC, 7D, 86, E8, 43, 08, 75, 57, AC, 0F, 91, 45, 84, 53, 00, 15, 00, C7, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00

got response:

48, 54, 54, 50, 2F, 31, 2E, 31, 20, 32, 30, 30, 20, 4F, 4B, 0D, 0A, 43, 6F, 6E, 74, 65, 6E, 74, 2D, 54, 79, 70, 65, 3A, 20, 74, 65, 78, 74, 2F, 68, 74, 6D, 6C, 3B, 20, 63, 68, 61, 72, 73, 65, 74, 3D, 65, 75, 63, 2D, 6B, 72, 0D, 0A, 50, 72, 61, 67, 6D, 61, 3A, 20, 6E, 6F, 2D, 63, 61, 63, 68, 65, 0D, 0A, 43, 61, 63, 68, 65, 2D, 63, 6F, 6E, 74, 72, 6F, 6C, 3A, 20, 6E, 6F, 2D, 63, 61, 63, 68, 65, 2C, 20, 6E, 6F, 2D, 73, 74, 6F, 72, 65, 2C, 20, 6D, 75, 73, 74, 2D, 72, 65, 76, 61, 6C, 69, 64, 61, 74, 65, 0D, 0A, 0D, 0A, 3C, 68, 74, 6D, 6C, 3E, 3C, 68, 65, 61, 64, 3E, 3C, 74, 69, 74, 6C, 65, 3E, 62, 6C, 6F, 63, 6B, 3C, 2F, 74, 69, 74, 6C, 65, 3E, 3C, 6D, 65, 74, 61, 20, 68, 74, 74, 70, 2D, 65, 71, 75, 69, 76, 3D, 43, 6F, 6E, 74, 65, 6E, 74, 2D, 54, 79, 70, 65, 20, 63, 6F, 6E, 74, 65, 6E, 74, 3D, 74, 65, 78, 74, 2F, 68, 74, 6D, 6C, 3B, 20, 63, 68, 61, 72, 73, 65, 74, 3D, 65, 75, 63, 2D, 6B, 72, 3E, 3C, 2F, 68, 65, 61, 64, 3E, 3C, 62, 6F, 64, 79, 20, 74, 6F, 70, 6D, 61, 72, 67, 69, 6E, 3D, 30, 20, 6C, 65, 66, 74, 6D, 61, 72, 67, 69, 6E, 3D, 30, 3E, 3C, 74, 61, 62, 6C, 65, 20, 63, 65, 6C, 6C, 70, 61, 64, 64, 69, 6E, 67, 3D, 30, 20, 63, 65, 6C, 6C, 73, 70, 61, 63, 69, 6E, 67, 3D, 30, 20, 77, 69, 64, 74, 68, 3D, 31, 30, 30, 25, 20, 68, 65, 69, 67, 68, 74, 3D, 31, 30, 30, 25, 3E, 3C, 74, 72, 3E, 3C, 74, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 31, 38, 25, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 42, 36, 44, 32, 45, 33, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 74, 72, 3E, 3C, 74, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 33, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 46, 46, 46, 46, 46, 46, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 74, 72, 3E, 3C, 54, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 34, 30, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 46, 46, 30, 30, 30, 30, 20, 61, 6C, 69, 67, 6E, 3D, 63, 65, 6E, 74, 65, 72, 3E, 3C, 66, 6F, 6E, 74, 20, 63, 6F, 6C, 6F, 72, 3D, 23, 46, 46, 46, 46, 46, 46, 20, 73, 69, 7A, 65, 3D, 34, 3E, 3C, 62, 3E, 0D, 0A, BA, BB, 20, BB, E7, C0, CC, C6, AE, B4, C2, 20, C1, A2, B1, D9, C7, D2, 20, BC, F6, 20, BE, F8, BD, C0, B4, CF, B4, D9, 2E, 3C, 2F, 62, 3E, 3C, 2F, 66, 6F, 6E, 74, 3E, 3C, 2F, 54, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 74, 72, 3E, 3C, 74, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 33, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 46, 46, 46, 46, 46, 46, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 74, 72, 3E, 3C, 54, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 32, 30, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 45, 35, 46, 34, 46, 44, 3E, 3C, 74, 72, 3E, 3C, 54, 64, 20, 68, 65, 69, 67, 68, 74, 3D, 38, 30, 25, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 45, 35, 46, 34, 46, 44, 20, 61, 6C, 69, 67, 6E, 3D, 63, 65, 6E, 74, 65, 72, 20, 76, 61, 6C, 69, 67, 6E, 3D, 74, 6F, 70, 3E, 3C, 74, 61, 62, 6C, 65, 20, 63, 65, 6C, 6C, 70, 61, 64, 64, 69, 6E, 67, 3D, 30, 20, 63, 65, 6C, 6C, 73, 70, 61, 63, 69, 6E, 67, 3D, 31, 20, 77, 69, 64, 74, 68, 3D, 34, 30, 30, 20, 68, 65, 69, 67, 68, 74, 3D, 31, 30, 30, 3E, 3C, 74, 72, 3E, 3C, 74, 64, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 38, 35, 42, 30, 43, 43, 3E, 3C, 74, 61, 62, 6C, 65, 3E, 3C, 74, 72, 3E, 3C, 54, 64, 20, 62, 67, 63, 6F, 6C, 6F, 72, 3D, 23, 46, 46, 46, 46, 46, 46, 20, 61, 6C, 69, 67, 6E, 3D, 63, 65, 6E, 74, 65, 72, 20, 77, 69, 64, 74, 68, 3D, 34, 30, 30, 20, 68, 65, 69, 67, 68, 74, 3D, 37, 30, 3E, 3C, 66, 6F, 6E, 74, 20, 63, 6F, 6C, 6F, 72, 3D, 23, 31, 46, 34, 38, 38, 35, 20, 73, 69, 7A, 65, 3D, 32, 3E, 3C, 62, 3E, 0D, 0A, B1, B3, C0, B0, BA, CE, 20, C1, A4, BA, B8, BA, B8, BE, C8, 20, B1, E2, BA, BB, C1, F6, C4, A7, 20, C1, A6, 34, 37, C1, B6, 3C, 62, 72, 3E, 0D, 0A, 28, B0, D4, C0, D3, 2C, 20, C0, BD, B6, F5, 2C, 20, B5, B5, B9, DA, 20, B5, EE, 20, BE, F7, B9, AB, BF, CD, 20, B9, AB, B0, FC, C7, D1, 20, BB, E7, C0, CC, C6, AE, 29, BF, A1, 3C, 62, 72, 3E, 0D, 0A, C0, C7, B0, C5, 20, C0, CE, C5, CD, B3, DD, 20, C1, A2, BC, D3, 20, B0, FC, B8, AE, 20, BD, C3, BD, BA, C5, DB, BF, A1, 20, C0, C7, C7, CF, BF, A9, 3C, 62, 72, 3E, 0D, 0A, BB, E7, C0, CC, C6, AE, 20, C1, A2, BC, D3, C0, CC, 20, C2, F7, B4, DC, 20, B5, C7, BE, FA, C0, BD, C0, BB, 20, BE, CB, B7, C1, B5, E5, B8, B3, B4, CF, B4, D9, 2E, 3C, 62, 72, 3E, 3C, 62, 72, 3E, 0D, 0A, BA, BB, 20, BB, E7, C0, CC, C6, AE, 20, C1, A2, BC, D3, B0, FA, 20, B0, FC, B7, C3, B5, C8, 20, BB, E7, C7, D7, C0, BA, 20, C3, E6, C3, BB, B3, B2, B5, B5, B1, B3, C0, B0, C3, BB, BF, AC, B1, B8, C1, A4, BA, B8, BF, F8, 3C, 62, 72, 3E, 0D, 0A, 28, 30, 34, 31, 2D, 36, 34, 30, 2D, 31, 39, 32, 33, 29, C0, B8, B7, CE, 20, B9, AE, C0, C7, C7, CF, BF, A9, 20, C1, D6, BD, C3, B1, E2, 20, B9, D9, B6, F8, B4, CF, B4, D9, 2E, 3C, 2F, 62, 3E, 3C, 2F, 66, 6F, 6E, 74, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 74, 72, 3E, 3C, 74, 64, 20, 61, 6C, 69, 67, 6E, 3D, 63, 65, 6E, 74, 65, 72, 20, 68, 65, 69, 67, 68, 74, 3D, 33, 30, 3E, 3C, 66, 6F, 6E, 74, 20, 63, 6F, 6C, 6F, 72, 3D, 46, 46, 46, 46, 46, 46, 20, 73, 69, 7A, 65, 3D, 33, 3E, 3C, 2F, 66, 6F, 6E, 74, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 2F, 74, 61, 62, 6C, 65, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 2F, 74, 61, 62, 6C, 65, 3E, 3C, 2F, 74, 64, 3E, 3C, 2F, 74, 72, 3E, 3C, 2F, 74, 61, 62, 6C, 65, 3E, 3C, 2F, 62, 6F, 64, 79, 3E, 3C, 2F, 68, 74, 6D, 6C, 3E

...which is translated to this in euc-kr:

HTTP/1.1 200 OK
Content-Type: text/html; charset=euc-kr
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate

<html><head><title>block</title><meta http-equiv=Content-Type content=text/html; charset=euc-kr></head><body topmargin=0 leftmargin=0><table cellpadding=0 cellspacing=0 width=100% height=100%><tr><td height=18% bgcolor=#B6D2E3></td></tr><tr><td height=3 bgcolor=#FFFFFF></td></tr><tr><Td height=40 bgcolor=#FF0000 align=center><font color=#FFFFFF size=4><b>
본 사이트는 접근할 수 없습니다.</b></font></Td></tr><tr><td height=3 bgcolor=#FFFFFF></td></tr><tr><Td height=20 bgcolor=#E5F4FD><tr><Td height=80% bgcolor=#E5F4FD align=center valign=top><table cellpadding=0 cellspacing=1 width=400 height=100><tr><td bgcolor=#85B0CC><table><tr><Td bgcolor=#FFFFFF align=center width=400 height=70><font color=#1F4885 size=2><b>
교육부 정보보안 기본지침 제47조<br>
(게임, 음란, 도박 등 업무와 무관한 사이트)에<br>
의거 인터넷 접속 관리 시스템에 의하여<br>
사이트 접속이 차단 되었음을 알려드립니다.<br><br>
본 사이트 접속과 관련된 사항은 충청남도교육청연구정보원<br>
(041-640-1923)으로 문의하여 주시기 바랍니다.</b></font></td></tr><tr><td align=center height=30><font color=FFFFFF size=3></font></td></tr></table></td></tr></table></td></tr></table></body></html>

native-tls mistakes this for a tls record, tries to parse the 2-3rd byte, 54 54(TT of HTTP in ASCII) as TLS version number, and fails.
seems like the school network blocks by IP address and send the above message regardless of the port.

Make the design more elegant

Current project design is quite dirty; ehentai::Explorer is enormous, handling both networking and parsing, while ehentai::Page is splitted alone. It would be nice if we can load asynchronously from methods in some structs other than Explorer, so that codes like below is possible:

let explorer = Explorer::new().await;
let search = explorer.search("misc:tankoubon").take(5);

while let Some(page) = search.try_next().await? {
    for draft in page {
        let article = draft.try_load().await?;

        // struct Article will have nothing with networking, so that
        // external frontends have no worry of overheads

        save_article(article, "~/books/.manga/"); // some function from an external frontend
    }
}

But I have no idea how to make this possible. I face a problem in every direction to change the design. How can I make a progress?

Handle 'Offensive For Everyone'

Currently, when an article with 'Offensive For Everyone' flag (aka visibility flag) is requested, the parser fails. We should work around this, because no one wanna crash every time they request wrong article only because they were naive and .unwrap()'d the result.

The solution for this involves a cookie to be set, but we haven't yet done any feature involving cookies. So this would take a lot of work. But login session would need a cookie too, so this should be done someday.

Login

#7 related.

I don't know how login works, but it would need cookies. Future researches will go below.