Comments (13)
Ok so I was testing to see what the performance was like on my ryzen 7 1700x just to give an idea of the speed differences. This is in no way perfect testing as this is a machine used for serving files and media use but I tested about 2 times each and saw about a 2 to 10 second difference between runs.
I tested with a 3 hour and 40 minute audiobook saved as flac in mulitple tracks.
for --jobs=1 I got a merge time of 4 minutes and 30 seconds
for --jobs=4 I got a time of 1 minute and 30 seconds
for --jobs=8 I got a time of 54 seconds
for --jobs=16 i got a time of 47 seconds (this last one was the see how smt helped in this workload which it did what I expected.)
Thank you again so much for all the work put into this it appears to have decreased time by a large amount. I dont see this helping for large single files but most rips from cds are lots of smaller tracks being merged together which this will help alot for.
Have a great day
from m4b-tool.
Good news, i played around with the symfony process wrapper and found a dirty-hack-solution, that might at least work for file conversions (the most time consuming task). First experiments show huge performance improvements, if multiple files are converted (up to 90% better performance using a dual-core system while converting 2 files simultaneously). Tests with --ffmpeg-threads
did not result in a significant performance boost on my system.
The --jobs
parameter can be used, to define, how many files should be converted simultaneously.
Here is a (highly experimental) nightly build of m4b-tool.tar.gz.
Use e.g. --jobs=8
to run 8 instances of ffmpeg converting 8 files simultaneously. Less files will result in slower conversion... I would really appreciate feedback :-)
Hint: At the moment this does not work in combination with the
--audio-profile
parameter for high efficiency codecs.
from m4b-tool.
Glad I could help... was interesting to implement :-) There is still much room for improvement, but for now, I'm closing this ticket, as the biggest problem seems to be solved...
from m4b-tool.
Unfortunately PHP is very limited regarding multithreading, well, limited means it just is not supported without really dirty hacks and really extensive effort...
For the moment you can append the experimental parameter --ffmpeg-threads
to control how many threads should be used for encoding...
I'll leave this feature request open and will do some research, since performance is one of the main topics in my personal todo list...
from m4b-tool.
ah ok wasn't sure how much php was designed for multi threading very well. shoot me down if im wrong I don't know php that well or to be honest really at all aside from the small tinkering I've done in such things but what about pthreads?
from m4b-tool.
ah ok wasn't sure how much php was designed for multi threading very well
Since PHP is usually running on a web server, thread safety was not the main goal of PHP, which in my opinion leads to the conclusion, that PHP is not the right tool for the job (i thought about re-implementing m4b-tool
in go or rust, but for now it's not worth the time)
what about pthreads?
pthreads is integrated in PHP ZTS (thread safe), but by default, PHP install is the NTS (non thread safe). You could check it on your system:
$ php -v
PHP ... ( NTS )
// ....
BUT: Threadsafety is not the problem at all. Since what m4b-tool
mainly does is calling other binaries (mostly ffmpeg
and mp4v2
), multithreading could be implemented by using a non-blocking call of binaries approach. This is possible, since symfony already performs non-blocking calls, but to prevent problems, the calls are implemented as blocking at the moment
The thing is, that i have to verify, if this would be a real improvement over --ffmpeg-threads
, if i call multiple binaries at the same time, or if this is just MUCH effort that is hardly measurable...
Multithreading is hard - and not always a valuable improvement...
I have to finish some other tasks before releasing the 0.4.0... e.g. the --batch-pattern
feature, which allows to convert a bunch of audio-books without writing an extra shell script :-) But before the release the performance improvements are on my bucket list...
from m4b-tool.
would it be possible to write them as non blocking but check when the tasks have exited? then as the tasks close open new tasks for the next files?
When you are converting from inputFile to outputFile which is a .m4b in this case why not spawn either all the tasks wait for it to be done or spawn X number of tasks wait for some to complete spawn more until all the tasks r complete.
Unsure if u can tell if u use non blocking whether a task is complete or not
from m4b-tool.
Yes, that would be possible, but much effort to do this reliable in PHP. Also i have to VERIFY, that this would really improve the performance (lets say at least 5%).
Since ffmpeg
already is multithreaded and the conversion step is the most time consuming task, i'm not sure that it will help a lot. The most time consuming things are:
- Reading the exact duration of the single tracks, to generate valid chapters (either with
mp4info
orffmpeg
) - Converting each of the source files to tmp
m4b
files before merging is possible (ffmpeg
) - Merging the resulting m4b-files (
ffmpeg
) - Tagging / generating chapters (
mp4tags
,mp4chaps
orffmpeg
)
Even more time is needed if you use the following flags:
--adjust-for-ipod
means that each input file length has to be estimated, to check, if the desired sample rate is playable on ipods--max-chapter-length
and--desired-chapter-length
(only in the nightly) will result in a very time consuming silence detection, if chapters are longer than--max-chapter-length
to place the chapters marks where silences are
If you would like to check, what is the most time consuming thing in your case, you could add a:
-v --logfile=m4b-tool.log
which results in a detailed logfile with time entries for each task / command in ms... example:
NOTICE 0ms 1 match for pattern ./%g/%a/%s/%p - %n/
NOTICE 0ms ================================
NOTICE 2ms merge ./
NOTICE 3ms => ./001.m4b
NOTICE 4ms - name: ...
NOTICE 4ms - artist: ...
NOTICE 4ms - genre: ...
NOTICE 4ms - series: ...
NOTICE 5ms - series-part: ...
NOTICE 5ms
NOTICE 5ms ================================
NOTICE 5ms
NOTICE 6ms ================================
NOTICE 17ms 0 matches for pattern ...
NOTICE 17ms
NOTICE 18ms ================================
NOTICE 18ms processing ./
NOTICE 30ms reading metadata and streaminfo for file ./001.mp3
NOTICE 3560ms
NOTICE 3630ms searching for cover in ./
NOTICE 3630ms using cover ./cover.jpg
NOTICE 3631ms searching for description.txt in ./
NOTICE 3631ms success: found description.txt for import
NOTICE 3633ms ipod auto adjust active, getting track durations
NOTICE 3633ms load estimated duration for file ./001.mp3
....
But as I already said: There are some things to do before the performance tweaking. It's not that i don't know HOW to do it or how to do it the best way... its about IF the effort is valuable and it really improves the performance... Perhaps it is more valuable to improve the order of commands, implement a cache, remove redundant binary calls, use new ffmpeg
features to combine two steps to one, etc. etc... it will take some research... patience please ;)
from m4b-tool.
sorry i wasn't trying to be impatient I was just trying to throw some ideas at the fan to see what stuck. anyways please try to have a good day.
Just saw that ffmpeg was only utilizing one of my 8 cores so I figured maybe just throw my execution cycles since it seems to already convert from say flac or mp3 to m4b then from that string the files together. I may be wrong but anyways sorry to bother you so much
from m4b-tool.
I may be wrong but anyways sorry to bother you so much
I really like your ideas and your commitment - so every idea is a good idea, every tester is a good tester, feel free to keep suggesting :-) I just would like to state that it takes some time for the release and news to this topic, I don't think that I'm able to release or analyse this in the next 4 weeks.
Just saw that ffmpeg was only utilizing one of my 8 cores
I think, that ffmpeg
tries to automatically use the optimal number of cores since 2014. See this thread for more information: https://superuser.com/questions/155305/how-many-threads-does-ffmpeg-use-by-default
So it strongly depends on the codec, how many cores ffmpeg
is using. Referring this thread, ffmpeg
is not able to encode every codec fully in parallel, even if more cores would be available.
Did you already try the --ffmpeg-threads=8
param?
This should result in ffmpeg
utilize all the cores you have - unless it is not possible for the used codec... (m4b
is mp4
with aac
or alac
under the hood). Important is, that ffmpeg
has to be compiled with pthreads
, see https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu for reference.
For testing purposes, you could take a look at the docker section - i tried to optimize the docker image for performance and enable all the features needed to get the best audio quality...
from m4b-tool.
I will probably end up taking a look at the docker section thanks btw. I didn't mean for you to do it right at this minute I sorta expected it to be done either in 2 months or never.
However i am going to try compiling with pthreads for ffmpeg and seeing what happens hopefully it works well. I will let you know the result in the end. Thanks btw
EDIT:
Just tested with pthreads enabled and have no different in core utilization. ill try docker soon enough
from m4b-tool.
@popcorn9499 Ok, here is an improved version, where simultaneus merging should be fully supported. This should improve performance a lot when merging a set of files.
Provide the --jobs=8
parameter to run 8 simultaneous conversion processes...
Limits:
- Only the CONVERSION of files is possible simultaneously atm - other things like metadata reading or length detection are still limited to one call at a time
- If you have less files than cores (e.g. when you would like to merge 2 files but have 8 cores), the amount of files is the limit for the simultaneous encoding - so only 2 parallel encoding processes are started
I'll wait for testing feedback for 5 days, then I'll close this issue. Feel free do reopen it, if something is wrong with this feature.
from m4b-tool.
Sorry I took so long to get back that thing called life happened. Anyways I tested this abit and it seems to work extremely well seemed well in initial testing. I am going to send some numbers using time to see how long each command takes to execute on the same files both with some varying numbers of cores on the jobs parameter
Thank you so far its looking good
from m4b-tool.
Related Issues (20)
- Using --filename-template with --batch-pattern does not skip existing files HOT 1
- Corrupt files when merging many mp3 files HOT 5
- Difficult to intall HOT 3
- im running m4b-tool in a bash script on a macOS and its working really well, but --series-part is not working. Any idea why? HOT 5
- Cue sheets for reading metadata HOT 2
- Not adding metadata to the output file HOT 6
- How do you set tags as chapter names? HOT 2
- ffmpeg version 4.0.0 or higher is required HOT 9
- Implicit conversion from float to int loses precision (Parser/SilenceParser.php line 61) HOT 1
- Return type of M4bTool\Audio\Tag::offsetUnset($offset) should either be compatible with ArrayAccess::offsetUnset(mixed $offset): void, or the #[\ReturnTypeWillChange] attribute should be used HOT 6
- [FeatureRequest] Flag to copy all tags HOT 3
- Anyway to merge and use individual .mp3 filenames as chapter titles? HOT 2
- MusicBrainz HTTP 404 HOT 1
- podman permission denied error. suggested alias for running m4b-tool with podman HOT 1
- check --output-file is writeable before doing a processing job HOT 1
- Cannot build pre-release HOT 3
- mp4v2 HOT 2
- Chapter names are not being setup correctly. HOT 4
- Deprecation warnings running php v8? HOT 9
- Having images or text files in an intermediate directory for batch processing can lead to unexpected results HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from m4b-tool.