Very slow to combine chunks about pyload

[DrekoDev]

Hi @GammaC0de , do you have any clue ? If you need more logs here they are

`

7	23.01.2024 18:05:25	DEBUG	ADDON UserAgentSwitcher: Use custom user-agent string Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
38	23.01.2024 18:05:25	DEBUG	HOSTER OneFichierCom[240]: Plugin version: 1.19
39	23.01.2024 18:05:25	DEBUG	HOSTER OneFichierCom[240]: Plugin status: testing
40	23.01.2024 18:05:25	WARNING	HOSTER OneFichierCom[240]: Plugin may be unstable
41	23.01.2024 18:05:25	INFO	ACCOUNT OneFichierCom: Grabbing account info for user dre*******...
42	23.01.2024 18:05:25	DEBUG	ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/console/abo.pl | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
43	23.01.2024 18:05:25	DEBUG	ACCOUNT OneFichierCom: Expire date: 2024-07-23
44	23.01.2024 18:05:25	DEBUG	ACCOUNT OneFichierCom: Account info for user dre*******: {'login': {'timestamp': 1706032894.382562, 'password': '***', 'stats': [2, 1706033009.192815], 'valid': True}, 'data': {'validuntil': 1721689200.0, 'plugin': , 'trafficleft': -1, 'maxtraffic': None, 'premium': True, 'login': 'dre', 'type': 'OneFichierCom', 'options': {'limitDL': ['0']}}}
45	23.01.2024 18:05:25	DEBUG	ACCOUNT OneFichierCom: Using account dre*******
46	23.01.2024 18:05:25	INFO	HOSTER OneFichierCom[240]: Processing url: https://1fichier.com/?x1pkrjifk6w5gezkcop0&af=4814702
47	23.01.2024 18:05:25	INFO	HOSTER OneFichierCom[240]: Looking for direct download link...
48	23.01.2024 18:05:25	DEBUG	HOSTER OneFichierCom[240]: LOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
49	23.01.2024 18:05:27	DEBUG	HOSTER OneFichierCom[240]: Redirect #1 to: https://c-4.1fichier.com/p947107115
50	23.01.2024 18:05:27	DEBUG	HOSTER OneFichierCom[240]: LOAD URL https://c-4.1fichier.com/p947107115 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
51	23.01.2024 18:05:28	INFO	HOSTER OneFichierCom[240]: Direct download link detected
52	23.01.2024 18:05:28	INFO	HOSTER OneFichierCom[240]: Downloading file...
53	23.01.2024 18:05:28	DEBUG	HOSTER OneFichierCom[240]: DOWNLOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | cookies=True | resume=None | get={} | disposition=True | chunks=None | post={} | ref=True
54	23.01.2024 18:05:29	DEBUG	Content-Disposition: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
55	23.01.2024 18:05:29	DEBUG	Chunked with range 0-
56	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: File saved
57	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: Checking download...
58	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: File is OK
59	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: Checking file (with built-in rules)...
60	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: Checking file (with custom rules)...
61	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: Checking for link errors...
62	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: No errors found
63	23.01.2024 18:12:05	INFO	HOSTER OneFichierCom[240]: No errors found
64	23.01.2024 18:12:05	DEBUG	ADDON ExternalScripts: No script found under folder download_processed
65	23.01.2024 18:12:05	DEBUG	ADDON ExternalScripts: No script found under folder package_processed
66	23.01.2024 18:12:05	INFO	Download finished: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
67	23.01.2024 18:12:05	DEBUG	ADDON ExternalScripts: No script found under folder download_finished
68	23.01.2024 18:12:05	INFO	Package finished: percy
69	23.01.2024 18:12:05	DEBUG	ADDON ExternalScripts: No script found under folder package_finished

`

[GammaC0de]

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy:

pyload/src/pyload/core/network/http/http_download.py

Lines 88 to 104 in ed850e8

	def _copy_chunks(self):
	init = self.info.get_chunk_filename(0) #: initial chunk name
	if self.info.get_count() > 1:
	with open(init, mode="rb+") as fo: #: first chunk file
	for i in range(1, self.info.get_count()):
	#: input file
	#: seek to beginning of chunk, to get rid of overlapping chunks
	fo.seek(self.info.get_chunk_range(i - 1)[1] + 1)
	fname = f"{self.filename}.chunk{i}"
	with open(fname, mode="rb") as fi:
	buffer_size = 32 << 10
	while True: #: copy in chunks, consumes less memory
	data = fi.read(buffer_size)
	if not data:
	break
	fo.write(data)

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

[DrekoDev]

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy:

pyload/src/pyload/core/network/http/http_download.py

Lines 88 to 104 in ed850e8

	def _copy_chunks(self):
	init = self.info.get_chunk_filename(0) #: initial chunk name
	if self.info.get_count() > 1:
	with open(init, mode="rb+") as fo: #: first chunk file
	for i in range(1, self.info.get_count()):
	#: input file
	#: seek to beginning of chunk, to get rid of overlapping chunks
	fo.seek(self.info.get_chunk_range(i - 1)[1] + 1)
	fname = f"{self.filename}.chunk{i}"
	with open(fname, mode="rb") as fi:
	buffer_size = 32 << 10
	while True: #: copy in chunks, consumes less memory
	data = fi.read(buffer_size)
	if not data:
	break
	fo.write(data)

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

Here is the information tab:

Python:2.7.18 (default, May 3 2020, 20:31:45) [GCC 9.2.0]OS:posix linux2 Linux 7facc51a3a80 3.10.0-1160.102.1.el7.x86_64 #1 SMP Tue Oct 17 15:42:21 UTC 2023 x86_64Version de pyLoad:0.4.20Dossier d'Installation:/app/pyloadDossier de Configuration:/configDossier de Téléchargement:/downloadsEspace libre :152.20 GiBLangue:enWebinterface Theme:modern by [GammaC0de](https://github.com/GammaC0de) based on work of [Marco Fernandes](https://github.com/xunil75) and Dogan BagciPort de l'Interface Web8000Port de l'Interface à Distance7227

Would you need an access to the pyload on my seedbox ?

[GammaC0de]

Would you need an access to the pyload on my seedbox

To get access just to the pyload on the seedbox would not help because this is a system and OS issue.

You can read here https://unix.stackexchange.com/questions/79751/is-there-a-faster-alternative-to-cp-for-copying-large-files-20-gb for possible reasons why this is happening.