allthemusicllc / atm-cli Goto Github PK

When downloading the tar archive from S3

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>1DBB3BF7EEE016D6</RequestId>
<HostId>
Uw+R0ha8IHnuc2P8QBAzhdSdlp/PvUBvD+W20z+NQ/JIuwvacN8cznp6G1BPu9elA6pDVAkc/Ck=
</HostId>
</Error>

Can you add the utility to grep throughout generated archive to found desired or similar melody?

The installation instructions need to include --recurisive in the git clone line.

git clone --recursive https://github.com/allthemusicllc/atm-cli.git

Thank you for your efforts to prevent copyright madness!

After watching your TED talk, I wanted a way to simply listen to a bunch of melodies strung together in a single MIDI file. With the permissions issues and the way you compress multiple midi files it wasn't all that easy to simply play them. I decided to write my own utilities that allow a user to do the brute force generation, but store them in a single MIDI file rather than breaking each melody into a single file. It could be nice to see similar features added to atm-cli. My utilities are not very fleshed out with command line arguments, but hopefully they inspire something.

I understand that focus is placed on being able to store them efficiently, and it seems like batching several melodies into a single MIDI file could help with that. The headers in simplest MIDI file is 22 bytes. The track is 6x the number of notes plus 1 (assuming a running status of note on events is used for all notes). So in an 8 notes melody, 49 bytes. The header represents 30% of the file size. For a 12 note melody, the header would be 23% of the file size. If you string them together, the header could be negligible compared to the track size. Maybe compression handles this, but a more direct approach could be more accessible.

It seems like there could be a tangible way to store a minimal version of the melodies without needing to store the whole MIDI file as well (this is what the generateNotes and notes2Midi commands do). Simply storing a single byte per note could save a lot of space and the conversion to a MIDI file is easy enough. Less than a byte per note could be achieved with the right approach. This seems comparable to using compression, but could be much more efficient if the right strategy is used.

Anyway, just some thoughts. Love the idea behind All The Music!

The link in the README to download a set of samples at the end of the Getting Started section is read-protected. For this to be useful the file should be publicly available for download.

The specific link is:
https://allthemusic.s3.amazonaws.com/datasets/20190727.C4_D4_E4_F4_G4_A4_B4_C5.L10.tar

Hi guys,

I followed your instructions in your readme.md. The test call cargo run --release -- -h # show usage succeeded but the example below always throws an error. Is this a known issue and does your tool support Windows 10?

λ cargo run --release -- single -n 'C:4,D:4,E:4,F:4,G:4,A:4,B:4,C:5' -t test.mid
    Finished release [optimized] target(s) in 0.10s
     Running `target\release\atm.exe single -n 'C:4,D:4,E:4,F:4,G:4,A:4,B:4,C:5' -t test.mid`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ()', lib\libatm\src\lib.rs:197:26
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.40\src\backtrace\mod.rs:66
   1: std::sys_common::backtrace::_print_fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:77
   2: std::sys_common::backtrace::_print::{{impl}}::fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:59
   3: core::fmt::write
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libcore\fmt\mod.rs:1052
   4: std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\io\mod.rs:1426
   5: std::sys_common::backtrace::_print
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:62
   6: std::sys_common::backtrace::print
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\sys_common\backtrace.rs:49
   7: std::panicking::default_hook::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:204
   8: std::panicking::default_hook
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:224
   9: std::panicking::rust_panic_with_hook
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:472
  10: std::panicking::begin_panic_handler
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:380
  11: core::panicking::panic_fmt
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libcore\panicking.rs:85
  12: core::option::expect_none_failed
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libcore\option.rs:1199
  13: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter
  14: <libatm::MIDINoteSequence as core::str::FromStr>::from_str
  15: atm::directives::parse_sequence_argument
  16: <atm::directives::SingleDirectiveArgs as core::convert::From<&clap::args::arg_matches::ArgMatches>>::from
  17: atm::cli::Cli::run
  18: BITSET
  19: main
  20: std::rt::lang_start_internal::{{closure}}
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\rt.rs:52
  21: std::panicking::try::do_call<closure-0,i32>
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:305
  22: panic_unwind::__rust_maybe_catch_panic
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libpanic_unwind\lib.rs:86
  23: std::panicking::try
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panicking.rs:281
  24: std::panic::catch_unwind
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\panic.rs:394
  25: std::rt::lang_start_internal
             at /rustc/b8cedc00407a4c56a3bda1ed605c6fc166655447\/src\libstd\rt.rs:51
  26: main
  27: invoke_main
             at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  28: __scrt_common_main_seh
             at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  29: BaseThreadInitThunk
  30: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: process didn't exit successfully: `target\release\atm.exe single -n 'C:4,D:4,E:4,F:4,G:4,A:4,B:4,C:5' -t test.mid` (exit code: 101)

Hi guys, i just installed "atm", and i got some issues while trying to run it.
I just followed the instructions to install and run the examples to generate the melodies but my pc does recognize "atm" as an internal or external command.
I already tried to run on my CMD prompt as administrator and with GIT as administrator.

I hope that you can help me.

Thanks.

Dear author, I am a music lover and a software engineer.I have great interest in this project and hope to participate in it. I hope to get in touch with you and look forward to your reply ：）

I've followed all the directions and cloned the repository today but continue to receive the 'zsh: command not found: atm' error. I'm using a Macbook Pro running BigSur v11.1 (20C69). Any help will be much appreciated. I did see a similar issue posted and tried what was suggested, and that didn't work, either.

Thank you,

Anna

I'll be happy to contribute financially or with resources to pre-generate a "groove" dataset. Criteria for a groove dataset:

1. All note length variations from whole to sixteenth including syncopation and tuplets are included in 2-3 bar sequences as a single note
2. The above needs to be done for all possible meters up to 12/x starting from 2/4.

For starters, we could do 4/4, eh?

I think this is self-explanatory. Depends on #5 (just a tiny bit)

As in the CC FAQ, all CC licenses should not be used for software. You may wish to go with Apache instead.

The crate documentation is confusing: I'm not exactly sure what this is generating in batch mode?

The documentation says that it generates all "permutations" of a given sequence: the documentation for utils::gen_sequences() shows an example with two identical notes (C:4, C:4). On the other hand, gen_sequences() appears to generate a Cartesian Product of note sequences of the given length via itertools::Itertools::multi_cartesian_product(). The example result would then contain duplicate melody sequences…

My guess is that the intent is indeed to do what gen_sequences() currently does: generate the set of all sequences of the given length drawn from the given note alphabet set. I'd suggest:

• Correcting the documentation throughout not to refer to "permutation".
• Calling the input note alphabet a "set" rather than a "sequence" throughout.
• Fixing the example to not contain duplicate notes.
• Normalizing the input note alphabet at some point so that it is actually a set, possibly issuing a warning if it contained duplicate elements.

While I like the concept of being able to generate all possible melodies I think this is mathematically implausible. I will try to illustrate my point with specific numerical values. Note that where rounding occurs I round down (favouring this being a possible endeavour). Similarly I make some assumptions which tend to err on the side of shrinking the space and not growing it.

Assuming playing around a different key centre is equivalent we can divide the number of possible notes by 12. Let's assume only the keys on a piano keyboard (there exists other pitches) so we could call it 88/12 choices per a note (rounded down to 7). We ignore tempo, and swing, but similarly rhythm is part of a melody. Let's say we allow note lengths from 1/1 to 1/32 and triplets (12 options). We also care about rests; let's say we also have those 12 options for length, plus 0 length (no gap), i.e 13 possibilities. One could say that you have 12 notes and 11 (potentially absent) rests in the phrase, so we have:
7^12 * 12^12 * 13^11 = 2.21 * 10^35
We could reasonably say that a melody could be encoded in this way at several (<12) different time divisions (since we don't care about tempo) so we can divide the number by 12 to eliminate that form of redundancy:
2.21 * 10^35 / 12 = 1.84 * 10^34
In the best possible case you need enough bits to index into that list to encode a melody (densest possible encoding) meaning the base 2 logarithm of that number worth of bits:
ln(1.84 * 10^34) / ln(2) = 113.8 bits
Let's convert that to bytes then round it down: 14 bytes per melody... so to store all optimally encoded melodies we need:
1.84 * 10^34 * 14 bytes = 2.5 * 10^35 bytes = 2.5 * 10^11 yottabytes
This is many orders of magnitude more than the total amount of data storage in the world. In short, I do not believe it is possible to copyright (encode and store in persistent tangible media) every possible 12 note melody built in the western European musical tradition, nor will it ever be.

Create new directive titled split that will split an output archive into chunks. Users can specify either the number of output chunks, or the desired size of each output chunk. Data will be copied from the source archive, and thus the source will be left intact.

Hi,

When extracting the tar archives the permissions are all zero. This is a problem when opening the files.

tar --list -vf test.mid | less
---------- 0 0 0 398 Jan 1 1970 6060/6060/batch1.tar.gz
---------- 0 0 0 407 Jan 1 1970 6060/6060/batch2.tar.gz
---------- 0 0 0 408 Jan 1 1970 6060/6060/batch3.tar.gz
---------- 0 0 0 406 Jan 1 1970 6060/6060/batch4.tar.gz
---------- 0 0 0 408 Jan 1 1970 6060/6060/batch5.tar.gz
---------- 0 0 0 411 Jan 1 1970 6060/6060/batch6.tar.gz
---------- 0 0 0 409 Jan 1 1970 6060/6060/batch7.tar.gz
---------- 0 0 0 407 Jan 1 1970 6060/6060/batch8.tar.gz
---------- 0 0 0 412 Jan 1 1970 6060/6060/batch9.tar.gz
---------- 0 0 0 417 Jan 1 1970 6060/6060/batch10.tar.gz
---------- 0 0 0 406 Jan 1 1970 6060/6060/batch11.tar.gz

Could you set "rw" at least?

--HPS

Currently the length of the note sequence specified by -n in the batch directive must be less than or equal to the -L output sequence length parameter. This requirement should be removed to allow, as an example, all melodies of length 12 from a 13-note input.

From a legal standpoint, what is the advantage of enumerating all the sequences? I think there's a very good "compressed format" for storage of this collection of sequences: given a "melody number" one can quickly produce the sequence with that number from a given input note sequence. Does explicitly representing each sequence compressed much less well really do something legally valuable?

atm single -n 'C:4,D:4,E:4,F:4,G:4,A:4,B:4,C:5' -t test.mid
thread 'main' panicked at 'called Result::unwrap() on an Err value: ()', src\libcore\result.rs:1188:5
stack backtrace:
0: backtrace::backtrace::trace_unsynchronized
at C:\Users\VssAdministrator.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.40\src\backtrace\mod.rs:66
1: std::sys_common::backtrace::_print_fmt
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\sys_common\backtrace.rs:84
2: std::sys_common::backtrace::_print::{{impl}}::fmt
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\sys_common\backtrace.rs:61
3: core::fmt::write
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libcore\fmt\mod.rs:1025
4: std::io::Write::write_fmtstd::sys::windows::stdio::Stderr
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\io\mod.rs:1426
5: std::sys_common::backtrace::_print
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\sys_common\backtrace.rs:65
6: std::sys_common::backtrace::print
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\sys_common\backtrace.rs:50
7: std::panicking::default_hook::{{closure}}
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:193
8: std::panicking::default_hook
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:210
9: std::panicking::rust_panic_with_hook
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:471
10: std::panicking::begin_panic_handler
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:375
11: core::panicking::panic_fmt
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libcore\panicking.rs:84
12: core::result::unwrap_failed
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libcore\result.rs:1188
13: <alloc::vec::Vec as alloc::vec::SpecExtend<T,I>>::from_iter
14: <libatm::MIDINoteSequence as core::str::FromStr>::from_str
15: atm::directives::parse_sequence_argument
16: <atm::directives::SingleDirectiveArgs as core::convert::From<&clap::args::arg_matches::ArgMatches>>::from
17: atm::cli::Cli::run
18: core::ptr::real_drop_in_place
19: vtable
20: std::rt::lang_start_internal::{{closure}}
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\rt.rs:52
21: std::panicking::try::do_call<closure-0,i32>
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:292
22: panic_unwind::__rust_maybe_catch_panic
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libpanic_unwind\lib.rs:78
23: std::panicking::try
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panicking.rs:270
24: std::panic::catch_unwind
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\panic.rs:394
25: std::rt::lang_start_internal
at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src\libstd\rt.rs:51
26: main
27: invoke_main
at d:\agent_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
28: __scrt_common_main_seh
at d:\agent_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
29: BaseThreadInitThunk
30: RtlUserThreadStart
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

It would be nice to have pre-built binaries ready to go.
For a general drive-by "try it out" it would've been amazing to have pre-built releases, even though it took less then a minute to compile.

I do understand this may be in its' early stage, perhaps that's something to consider for future development. You may also be able to run cargo and generate changelog and binaries with GitHub Actions.

The project itself is amazing, great job!

Hello,

I was wondering if you know of anyone who has explored searching/parsing the midi dataset after its creation -- i.e. inputting a midi sequence and finding that sequence in the dataset. Could be useful for real-world applications? I'd love to help out if that seems like useful functionality that hasn't been done yet.

Hi guys, i just installed "atm", and i got some issues while trying to run it.
I just followed the instructions to install and run the examples to generate the melodies but my pc does recognize "atm" as an internal or external command.
I already tried to run on my CMD prompt as administrator and with GIT as administrator.

I hope that you can help me.

Thanks.

All code is released into the public domain via the Creative Commons Attribute 4.0 International License.

The Creative Commons Attribution (not "Attribute") is NOT for public domain.
The CC0 Public Domain Declaration is for public domain.

The two are quite different.

Provide a melody lookup tool for musicians,
to be able to search for whether a specific melody is included in your pool,
then calculate a note contextual merkle tree branch or subset number,
of the specific melody or groove, to be referenced, e.g. as metadata in digital publishing.
c-c-c-c-c-c-c-c-c-c-c-c
tree:0-0-0-0-0-0-0-0-0-0-0-0 (12 x same)
c-c-c-c-c-c-c-c-c-c-c-d
tree:0-0-0-0-0-0-0-0-0-0-0-1 (11 x same,1up)
https://en.bitcoinwiki.org/wiki/Merkle_tree
https://musicbrainz.org/doc/MusicBrainz_Identifier

The current installation instruction, when followed, give an error when building as rust is unable to find the dependency libatm, due to it being a submodule and not checked out when the repository is cloned. The installation instructions should be updated to include the lines:

$ git submodule init
$ git submodule update

between changing into the atm-cli directory and building.

Hi,

Is there room for alternative output formats?

Tar files need at least 512 bytes (one block) per file.

Further when extracting these small files to a real filesystem they need much more disk-space than 71-bytes worth of MIDI data, because disks are organized into 512-byte blocks typically.

You might be better off using one large text file as output format.

/* melody NNN */
C5
D5
E5

/* melody MMM */
C5
D5
E5B

For example in the format that Midi Player Pro (http://www.selasky.org/hans_petter/midistudio/) supports, or ABC midi for that sake.

--HPS

OS: Win 10 2004
The tool shows an error if we specify notes argument inside single quotes (like in help):

atm gen single 'C:4' test.mid

Shows: error: Invalid value for '<notes>': Invalid note at index 0
I suppose that it should be fixed or mentioned in README.
Also atm gen single C:4 test.mid and atm gen single "C:4" test.mid works, but my editors are not able to open the result midi (it's attached). Am I doing something wrong?
test.zip

error.log

Followed all the instructions to compile and run. Every combination of example parameters fails with the attached error. Tried on 2 different machines. Any ideas?

Hi guys, i just installed "atm", and i got some issues while trying to run it.
I just followed the instructions to install and run the examples to generate the melodies but my pc does recognize "atm" as an internal or external command.
I already tried to run on my CMD prompt as administrator and with GIT as administrator.

I hope that you can help me.

Thanks.