imclab / nupic.audio

Audio (analog, digital) experiments using NuPIC HTM/CLA

nupic.audio

 Gitter public chat channel

Auditory experiments using cortical learning algorithms (CLA) and hierarchical temporal memory (HTM).

Repositories of interest

• Numenta's nupic.critic Audio streaming
• NuMozart Digital (MIDI) streaming and composition
• HTMforGenreClassification Genre classification
• Hackathon scripts and data for Musenta

Note: These repositories currently are all work-in-progress.

Online videos of interest

Taken from the collection gathered via Gitter channel https://gitter.im/rcrowder/EncodingSpecificityPrinciple -

• From Ear to Primary Cortex
• Anatomy - Ear Overview
• Anatomy - Middle Ear
• Introduction to Biological Audition - Part 1
• Introduction to Biological Audition - Part 2
• Auditory perception in speech technology
• Auditory cortex 1 - Physiology and sound localization
• Auditory cortex 2 - Language; bats and echolocation

Online books and references

• https://jp.mathworks.com/matlabcentral/answers/uploaded_files/23580/index.pdf Spectral Envelopes in Sound Analysis and Synthesis By Diemo Schwarz, Diplomarbeit Nr. 1622, IRCAM (Institut de la Recherche et Coordination Acoustique/Musique)

• https://ccrma.stanford.edu/~jos/dft/
  Mathematics of the Discrete Fourier Transform (DFT) with audio appliccations
  By Julius O. Smith III, Center for Computer Research in Music and Acoustics (CCRMA)

• http://www.dspguide.com/
  The Scientist and Engineer's Guide to Digital Signal Processing
  By Steven W. Smith, Ph.D.

• http://www.eecs.qmul.ac.uk/~simond/pub/2012/PlumbleyDixon12-ima-tutorial-slides.pdf
  Tutorial: Music Signal Processing
  By Mark Plumbley and Simon Dixon, Centre for Digital Music (Queen Mary University of London)

Potential areas of investigation

• Genre and style classification
• Musical prediction and composition
• Acoustic correlation using canonical correlation analysis (CCA)
• Transient analysis (harmonic tracking)
• Motion derivative encoding (similar to optical flow)
• Echo location and spatial positioning (e.g. Anterior Ventral Cochlea Nucleus)
• Stream segmentation and seperation (includes selective attention)
• Cortical pathways and projections, 'What' and 'Where' pathways (belts?)
• Auditory nerve spike firing (e.g. IHC to CN GBC integrators)
• Dendritic micro-circuits and synaptic placement (temporal smoothing)
• Spike-timing dependent plasticity
• Acetylcholine inhibition enhancing discharge frequency but decreasing synaptic adaption
• Acoustic related cell, and dendrite, membrane properties (cascading conductances, shunting)

An alternative for the encoding of audio signals is the modelling of spike firing of auditory-nerve fibers. A collection of models can be found in the EarLab @ Boston University (http://earlab.bu.edu/ See Modelling -> Downloadable Models). If you plan to use these models, beware of their history and limitations. For example, early models lack some necessary non-linearity in their responses.