Real-time spectrogram analysis about coast_audio

[ekuleshov]

@EliaTolin you should be able to connect the FftNode from coast_audio_fft into the audio graph as used in the audio_recorder' example. That will give you the FFT data for a given audio data buffer. The music_playerdoes that for selected audio file and shows the result withFftView` widget.

@SKKbySSK though I could not figure out how to process audio data without having EncoderNode or something like MabPlaybackDeviceNode at the end of the audio graph.

Also it is somewhat cumbersome to get the audio position from within processor nodes, such as FftNode. You have to go all the way to decoder to get the position, as MabAudioPlayer.position method does. It would be really handy if ProcessorNodeMixin.process() method also had the AudioTime parameter along with the AudioBuffer.

[SKKbySSK]

@ekuleshov
Sorry for my late response.

I could not figure out how to process audio data without having EncoderNode or something like MabPlaybackDeviceNode at the end of the audio graph.

Most of nodes have one outputBus and it has read function to process audio data.
So, you should be able to do that something like this.

final graphNode = GraphNode();
final sineNode = FunctionNode(function: const SineFunction(), format: format, frequency: freq);
final sineVolumeNode = VolumeNode(volume: 0.2);

graphNode.connect(sineNode.outputBus, sineVolumeNode.inputBus);
graphNode.connectEndpoint(sineVolumeNode.outputBus);

// Allocate 10 seconds audio buffer.
final frames = AllocatedAudioFrames(
  length: format.sampleRate * 10,
  format: format,
);

frames.acquireBuffer((buffer) {
  // Read the graph's output data
  final framesRead = graphNode.outputBus.read(buffer);
  final readBuffer = buffer.limit(framesRead);

  // do something on readBuffer
});

[SKKbySSK]

Also it is somewhat cumbersome to get the audio position from within processor nodes, such as FftNode. You have to go all the way to decoder to get the position, as MabAudioPlayer.position method does. It would be really handy if ProcessorNodeMixin.process() method also had the AudioTime parameter along with the AudioBuffer.

Since position has different implications depending on the context and the processor node has responsible for processing audio data, not for calculating position, I don't think it is a good solution.

[ekuleshov]

Also it is somewhat cumbersome to get the audio position from within processor nodes, such as FftNode. You have to go all the way to decoder to get the position, as MabAudioPlayer.position method does. It would be really handy if ProcessorNodeMixin.process() method also had the AudioTime parameter along with the AudioBuffer.

Since position has different implications depending on the context and the processor node has responsible for processing audio data, not for calculating position, I don't think it is a good solution.

Thank you for these pointers. But will that work, for example, with FftNode? And I'm still unclear how to link the audio position with result of processing, e.g. use FftCompletedCallback data to create an audio spectrogram similar to this.

[SKKbySSK]

FftResult has frames and format properties.
So, you can calculate the duration of the result like this.

FftResult result;
final durationSeconds = AudioTime.fromFrames(frames: result.frames, format: result.format).seconds;

And you should sum it in each callback.

var positionSeconds = 0.0;
FftNode(
  ...
  onFftCompleted: (result) {
    positionSeconds += AudioTime.fromFrames(frames: result.frames, format: result.format).seconds;
  },
);

[ekuleshov]

FftResult has frames and format properties. So, you can calculate the duration of the result like this... And you should sum it in each callback.

Unfortunately this approach does not work when upstream node rolled back position in the audio stream. E.g. when FftNode (only using it as an example) is hooked up in the music player and user rewinds the audio, that summed up position won't be correct anymore. I suppose I could also subscribe to that rewind event, but that is even more cumbersome.

[ekuleshov]

final framesRead = graphNode.outputBus.read(buffer);

@SKKbySSK in the above snippet it only reads up to the allocated 10 sec buffer out of the graphNode.outputBus.
But how can I continuously read the entire decoded audio stream, say in chunks of 256 frames?

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.