meyda / meyda

Audio feature extraction for JavaScript.

meyda

Meyda is a Javascript audio feature extraction library. Meyda supports both offline feature extraction as well as real-time feature extraction using the Web Audio API. We wrote a paper about it, which is available here.

Would you like your Meyda project to be featured in our user showcase? Please let us know!

Want to ask questions, or chat about Meyda? Check out our slack channel on the Web Audio Slack Team. You'll need to sign up here before you can join.

hughrawlinson | nevosegal | jakubfiala | 2xAA

Usage

Please see the documentation for setup and usage instructions.

Contributing

Please see our contributor guidelines for guidelines about contributing, and bear in mind that we adhere to the Contributor Covenant

meyda's Issues

Add Envelope Feature Extractor

Spectral Flatness is out of range, and incorrectly scaled

Spectral Flatness has an issue!

It seems to vary between 0 and 1.13ish. It needs to be between 0 and 1.

Remove feature info object

As this is not very useful (maybe just for visualisations) we should remove this from meyda.js, possibly to a separate file.

Add Mel Cepstrum Feature Extractor

You should give a sample source in your readme.md

It's not immediately clear to me how I hook in a source. Giving an example with a creative commons mp3 would help a lot :)

Consider using Flow

I can't believe I'm saying this, but... http://flowtype.org/ is a static type checker for Javascript. It could save us a lot of time writing unit tests. I suggest using weak mode for now, I'll check out how that might work this evening - I'm hoping to combine it with linting and testing.

Add Spectral Flatness Feature Extractor

Move to using Audio Workers

This needs to happen once browsers have implemented Audio Workers, but before ScriptProcessorNode is removed from browsers.

Add Spectral Variation Feature Extractor

Add Constant Q Transform

Bark/Mel scale should really be calculated only once

… as they're only dependent on bufferSize and sample rate, which wouldn't change unless a new Meyda is instantiated.

suggested solution:
the scales themselves should be stored as Float32Arrays, e.g. self.barkScale. They can be calculated once, before the onaudioprocess handler is attached to the SPN. There can also be conversion functions, as are now in the mfcc extractor. I think it'd be great to keep this all as public variables, so that people could for example have a mel converter built with Meyda, or visualize the bark scale bounds alongside some other things.

Website doesn't work for a bunch of features

It seems that anything not outputting numbers or arrays is broken, looks like the canvas ID set at https://github.com/hughrawlinson/meyda/blob/master/index.html#L213 is wrong, as on https://github.com/hughrawlinson/meyda/blob/master/index.html#L252 you request meyda.featureInfo[v].arrayNames["1"] instead of meyda.featureInfo[v]["1"]

I suspect the line on 213 should be accessing arrayNames instead.

Add Spectral Shape Statistics Feature Extractor

Add Spectral Flux Feature Extractor

Make a standalone 'ampSpectrum' extractor?

Alright so having attempted to use our MFCC extractor as a standalone function in node, I realised that actually I cannot use any spectral features unless the FFT is calculated internally by the Meyda object. We may say this is by design, but in order for the "standalone" extractors to work, we need the ability to calculate the amplitude spectrum without having to run an SPN/audioworker. Otherwise the simplicity of using Meyda in node is compromised by the fact that I first have to FFT my data with a non-meyda method, and then run the extractor on it.

This would be an easy thing to add, and I'll assign this issue to myself if approved. The only ugly part is that the proposed extractor is then basically a wrapper for Nick Jones' jsfft, but whatever.

Add Temporal Shape Statistics Feature Extractor

Add Perceptual Sharpness Feature Extractor

CLI Modularization

Any useful functions and transforms we come up with for the CLI should probably be split out into lib files and exposed for developers to use for their convenience, rather than having people reimplementing things. This is part of the next issue, to do with reconsidering the packaging structure.

Add Spectral Kurtosis Feature Extractor

taylor swift

Can you add a function that increases the amount of Taylor Swift by a factor of 2? This would be very much appropriated.

Much Love,
Alex xox

Add Loudness Feature Extractor

Add Spectral Decrease Feature Extractor

Add Spectral Centroid Feature Extractor

Modularisation

Hello,

After a quick chat with @jakubfiala I mentioned that modularisation would be great for my use-case, so I thought it would be a good idea to post an issue here.

I'm a big fan of commonJS, I've been using it in many different libraries and it has worked great so far, and it allows a really nice workflow…

I've gone ahead and did some basic separations here:

https://github.com/vectorsize/meyda/tree/commonJS

to see how it works and it seems to be working perfectly!

I also did a few performance optimisations. Everything is in a commonJS branch in case u guys like it and feel like incorporating it.

That said if you decide to go for other type of modules, I think what I did is a good starting point.
Also the small optimisations in the main function still make sense.

Let me know what you think.

Add Spectral Spread Feature Extractor

Automated Benchmarking

A script to test speed and accuracy vs other frameworks would be really useful in helping us gauge what use-cases meyda is appropriate for.

Add Amplitude Modulation Feature Extractor

Add Spectral Rolloff Feature Extractor

Add Octave Band Signal Intensity Ratio Feature Extractor

Add Zero Crossing Rate Feature Extractor

Add Perceptual Spread Feature Extractor

Add Octave Band Signal Intensity Feature Extractor

Add Linear Predictor Coefficients Feature Extractor

Add Mel-Frequencies Cepstrum Coefficients Feature Extractor

Multiple packages

This is bound to be controversial, but...

Javascript packages have been more and more moving towards small, core packages to reduce bloat in their dependent packages. Two examples of this are grunt, and babel, who have respectively split out functionality between ['grunt', 'grunt-cli', 'grunt-contrib-sass', 'gulp-contrib-whatever','etc'] and ['babel','babel-es2015','babel-jsx','etc']. I believe we should at least consider this approach for meyda, purely because there are many different use cases for our users, and a growing one-size-fits-all package might not be the best way to serve users. This isn't a case like C/C++, where a library is included but only the linked-to code remains in the binary after compilation, when a package depends on Meyda it must (some caveats, but usually does) deploy with the entire package, including files that will never get used in the production deployment.

Use cases for meyda include:

• Frontend developers who want to run meyda in the browser
• Node users who want to run meyda in realtime or "surrealtime" ✨
• Users who want to use the CLI
• People who want to contribute to Meyda

Cleary those who wish to contribute to Meyda should use the git repository, but the other usecases are not so straightforward. Frontend developers can use both npm or bower to install packages, so it's not entirely clear which files should be in the npm package. It's pretty clear that the bower package shouldn't include the node distribution files (for those who may be unfamiliar, we compile into a minified concatenated file for the web, and to unminified unconcatenated files with module loading for node), but should the node package contain the distribution files? Should we include instructions in the documentation for web developers using the npm distribution to link to the web distribution file instead of the node one, or should we point out that if they include via npm and therefore get the node distribution from require they should take care that browserify (or whatever equivalent tool) is properly requiring the nested files for their build process? Probably neither of those need to be distributed the source ES6 code or the tests, or the config files for the CI services/linter/etc, or the CLI. Users who want to use the CLI probably don't need the source code, just the distribution code, but users who want to use the node module don't necessarily need the cli.

All of this is fine, if meyda is only used infrequently on a small scale, but if meyda is ever included on a project with a very high deploy frequency, the extra cost of deploying the extraneous files could really add up. Given our new focus on offline, I don't think that scenario is too far fetched. In my opinion, we should probably discuss the above in order to make sure meyda serves the widest variety of use cases in the best way possible.

Add a build system to automatically generate the minified version

It'd be great to automatically grab a minified version of meyda including the dependencies (fftjs, complex_array) whenever we commit to master. Alternatively, if we just run the build system before we commit to master, that'd work, but a way to test that would be excellent.

Add Envelope Shape Statistics Feature Extractor

Make Meyda work on Chrome

cli.js exposes things to global scope

It shouldn't do that, in case someone mistakenly includes it somewhere for some reason.

Add Auto Correlation Feature Extractor

Add Spectral Crest Factor Per Band Feature Extractor

Add Spectral Flatness Per Band Feature Extractor

Add Chromagram to Feature Extractor

there's already an mfcc, so adding a chroma isn't too much further work.

Here's a good primer on chroma: http://labrosa.ee.columbia.edu/matlab/chroma-ansyn/

Add Frames Feature Extractor

Put the source of the WAC2015 paper into a branch

Lets clean up the source of the WAC2015 paper on Meyda and open source it either on a branch of this repo or on another repo. I'm not sure I have it anymore - was it in Overleaf or did we keep it somewhere else?

Offline `get` method

Following our discussion – related to #85 – I think the best course of action is what @hughrawlinson suggested.

This would be an offline version of the get method (couldn't think of a hot name for it yet), which would behave as follows:

• Argument no. 1 will be identical to the standard get, i.e. a String denoting the desired feature, or an Array of such strings. Argument no. 2 will be a Float32Array containing the signal buffer to analyze.
• It is IMO necessary to make sure that this method can be called even while the internal SPN/AudioWorker/Paul/whatever they're going to call it is running and doing its thing.
• The offline get would then:
  • Make a list of all the calculations needed to extract the specified features (this could be either the amplitude spectrum, complex spectrum, or none)
  • Perform the calculations (ideally in a separate AudioWorker, but for now it can be done on the JS thread)
  • Call the specified extractors, and pass them the signal/calculated FFT data
  • return the output the same way as the classic get would
• As a consequence, Meyda should be able to run without a specified AudioContext/source node

As discussed in #85 and #84 this would create a simple API for offline AFE, and also extend the functionality of a single Meyda object, as it could now in theory analyze two simultaneous audio streams.

Add Complex Domain Onset Detection Feature

Add Spectral Skewness Feature Extractor

Loudness is not working

Add Line Spectral Frequency Feature Extractor