ropensci / fastmatmr

R bindings to fast_matrix_market for reading and writing .mtx files

fastMatMR

About

fastMatMR provides R bindings for reading and writing to Matrix Market files using the high-performance fast_matrix_market C++ library (version 1.7.4).

Why?

Matrix Market files are crucial to much of the data-science ecosystem. The fastMatMR package focuses on high-performance read and write operations for Matrix Market files, serving as a key tool for data extraction in computational and data science pipelines.

The target audience and scientific applications primarily include data scientists or researchers developing numerical methods who may wish to either test standard NIST (National Institute of Standards and Technology) which include:

comparative studies of algorithms for numerical linear algebra, featuring nearly 500 sparse matrices from a variety of applications, as well as matrix generation tools and services.

Additionally, being able to use the matrix market file format, means it is easier to interface R analysis with those in Python (e.g. SciPy uses the same underlying C++ library). These files can also be used with the Tensor Algebra Compiler (TACO).

Features

• Extended Support: fastMatMR supports standard R vectors, matrices, as well as Matrix sparse objects.

• Performance: The package is a thin wrapper around one of the fastest C++ libraries for reading and writing .mtx files.

• Correctness: Unlike Matrix, roundtripping with NA and NaN values works by coercing to NaN instead of to arbitrarily high numbers.

We have vignettes for both read and write operations to demonstrate the performance claims.

Alternatives and statement of need

• The Matrix package allows reading and writing sparse matrices in the .mtx (matrix market) format.
  • However, for .mtx files, it can only handles sparse matrices for writing and reading.
  • Round-tripping (writing and subsequently reading) data with NA and NaN values produces arbitrarily high numbers instead of preserving NaN / handling NA

Installation

CRAN

For the latest CRAN version:

install.packages("fastMatMR")

R-Universe

For the latest development version of fastMatMR:

install.packages("fastMatMR",
                 repos = "https://ropensci.r-universe.dev")

Development Git

For the latest commit, one can use:

# install.packages("devtools")
devtools::install_github("ropensci/fastMatMR")

Quick Example

library(fastMatMR)
spmat <- Matrix::Matrix(c(1, 0, 3, 2), nrow = 2, sparse = TRUE)
write_fmm(spmat, "sparse.mtx")
fmm_to_sparse_Matrix("sparse.mtx")

The resulting .mtx file is language agnostic, and can even be read back in python as an example:

pip install fast_matrix_market
python -c 'import fast_matrix_market as fmm; print(fmm.read_array_or_coo("sparse.mtx"))'
((array([1., 3., 2.]), (array([0, 0, 1], dtype=int32), array([0, 1, 1], dtype=int32))), (2, 2))
python -c 'import fast_matrix_market as fmm; print(fmm.read_array("sparse.mtx"))'
array([[1., 3.],
       [0., 2.]])

Similarly, fastMatMR supports writing and reading from other R objects (e.g. standard R vectors and matrices), as seen in the getting started vignette.

Contributing

Contributions are very welcome. Please see the Contribution Guide and our Code of Conduct.

License

This project is licensed under the MIT License.

Logo

The logo was generated via a non-commercial use prompt on hotpot.ai, both blue, and green, as a riff on the NIST Matrix Market logo. The text was added in a presentation software (WPS Presentation). Hexagonal cropping was accomplished in a hexb compatible design using hexsticker.

fastmatmr's Issues

TST: Add to coverage for new int based functions

See #57 for the features which lack tests, and #18 for the associated coverage failures.

Bonus: Figure out why this wasn't caught on the CI.

ENH: Support reading into `matrix`

The standard R variant.

TST: Refactor unit tests

the readability of read unit tests could be improved if all tests are self contained and independent of each other: e.g. you can run any test_that block in itself and it is a working test (no setup code before test_that blocks ideally)

I think there is no need to add library(testthat) to test files

consistency of indentation styles could be a bit improved, consider running styler::style_pkg, see for example https://github.com/HaoZeke/fastMatMR/blob/main/tests/testthat/test-write_fmm.R#L34

First noted here: ropensci/software-review#606 (comment).

ENH: Handle integers in input files

vec_to_fmm(1L, "vector.mtx")
#> Error: Invalid input type, expected 'double' actual 'integer'

However, I was wondering if it could actually accept integers as well?

First noted here: ropensci/software-review#606 (comment).

DOC: Add details on language interop

Maybe add a bit more context on why would I want to read back in Python? Possibly rather focus on being a language agnostic format?

First noted here: ropensci/software-review#606 (comment).

ENH: Support for character vectors

Should be possible

ENH: Support SPArse Matrix Package `spam`

https://search.r-project.org/CRAN/refmans/spam/html/spam-package.html

MAINT: Prepare to pass ROpenSci

To get reviews before CRAN :)

MAINT: Refactor build folders

possibly move rebuild-benchmarks to a tools sub directory or similar? https://r-pkgs.org/misc.html#sec-misc-tools.

First noted here: ropensci/software-review#606 (comment).

DOC: `readme` consistency (style)

In the readme: "we support / the package supports". Consider unifying these two wording styles

First noted here: ropensci/software-review#606 (comment).

DOC: API

Perhaps with pkgdown?

DOC: Reword `readme`

readme: "Unlike the Matrix package" : maybe rephrase? it is a bit off-putting for me to start the whole readme with a negative comparison, although
I understand that a core part of the motivation for this package is that Matrix does not support everything and not fast enough. Even changing the order of the two parts of the sentence would feel better for me but this is really personal

First noted here: ropensci/software-review#606 (comment).

MAINT: Better error messages for sparse vs non-sparse matrics

improve error message? It is a matrix, just sparse, not plain

mat_to_fmm(sparse_mat, "sparse_matrix.mtx")
#> Error in mat_to_fmm(sparse_mat, "sparse_matrix.mtx") : object is not a matrix

First noted here: ropensci/software-review#606 (comment).

DOC: Plot scales

Did you try out logarithmic scale for the benchmark plots? Would they possibly be more readable?

First noted here: ropensci/software-review#606 (comment).

DOC: Rename vignettes

rename basic_usage to getting started? The vignette title is getting started but you can see basic_usage in the url

First noted here: ropensci/software-review#606 (comment).

ENH: Better named functions

consider increased consistency of function names? e.g fmm_to_sparseMatrix vs sparse_to_fmm. sparse_to_fmm was a bit confusing to me in that if I understand correctly, the package is 'fast', not the target format itself (?). matrix and vector is abbreviated to spare a few characters, while sparse_matrix is not. A vague suggestion: possibly follow the pattern of fmm_write_vector, fmm_read_to_vector for all formats?

First noted here: ropensci/software-review#606 (comment).

This is a breaking change, so I'm not too sure, but then again, better now than any other time. I'm OK with fmm myself, since its the package name, but yeah consistency is a good target.

ENH: Handle `~` or be more explicit

it seems the lib cannot handle paths with ~ (?) If this cannot be made to work easily, some error message would be very helpful, I think it would be needed
to be added here: https://github.com/HaoZeke/fastMatMR/blob/main/src/to_file.cpp#L26 The corresponding read file returns an error message: 'cannot open file' which is more helpful.
(I am guessing the issue is related to ~ in path because write_fmm(vec, "~/vector.mtx") returned false for me while write_fmm(vec, "/home/ildi_home/vector.mtx") worked)

First noted here: ropensci/software-review#606 (comment).

Generally, since R needs to work on Windows as well, supporting ~ will be hard, but a better error message would be nicer.

MAINT: Possibly hide functions which shouldn't be called by users

If the users are not supposed to be calling the lower level functions, do they need to be exported, and thus being part of the public user interface of the package? It could be less confusing possibly if only those functions are exported which are meant to be called directly by users.

First noted here: ropensci/software-review#606 (comment)

ENH: Support for reading vectors

Standard R vectors (numeric).

ENH: Use Ryu

Or another floating point library. As is done upstream.

Stalling when reading sparse matrices W/O loading Matrix

Thank you for a fantastic tool! It has really sped up my day-to-day work when combining R and Python.

If I do

library(fastMatMR)
fmm_to_sparse_Matrix("/path/to/my/sparse/matrix.mtx")

it stalls. It seems to read in the matrix (activity on multiple CPUs, memory increases), but when it tries to write it as sparse in memory nothing happens (one core at 100%, no change in memory).

But, if I do

library(Matrix)
library(fastMatMR)
fmm_to_sparse_Matrix("/path/to/my/sparse/matrix.mtx")

It finishes in seconds, as expected.

> sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] fastMatMR_1.0.0.0 Matrix_1.6-1.1   

loaded via a namespace (and not attached):
[1] compiler_4.3.1 tools_4.3.1    grid_4.3.1     lattice_0.21-8

DOC: User facing

A vignette would be good.

MAINT: Get `R CMD check` to give less notes

I see that R CMD check seem to give a note about pixi files, it seems pixi.* might not
be the correct syntax in .Rbuildignore? (I do not know what the correct syntax would be,
but the check gives a note both in github actions and locally it seems. If all else fails, adding the files one by one instead of a wildcard syntax definitely works)

First noted here: ropensci/software-review#606 (comment).

pkgcheck results - main

Checks for fastMatMR (v0.0.1.0)

git hash: 7c7b96af

• ✔️ Package name is available
• ✔️ has a 'codemeta.json' file.
• ✔️ has a 'contributing' file.
• ✔️ uses 'roxygen2'.
• ✔️ 'DESCRIPTION' has a URL field.
• ✔️ 'DESCRIPTION' has a BugReports field.
• ✔️ Package has at least one HTML vignette
• ✔️ All functions have examples.
• ✔️ Package has continuous integration checks.
• ✔️ Package coverage is 90.5%.
• ✔️ R CMD check found no errors.
• ✔️ R CMD check found no warnings.

Package License: MIT + file LICENSE

MAINT: More scoped `nolint` rules

this is really overly pedantic, but you could consider specifying which rule to ignore in the #nolint comments instead of ignoring every rule (it is considered generally a best practice, but does not have real impact here). This is not needed if you would need to ignore several rules

First noted here: ropensci/software-review#606 (comment).

DOC: Vignette variables are overwritten

write_fmm example: sparse_mat variable is overwritten before it is written to disk so it is not easy to actually run both of the two variants included

First noted here: ropensci/software-review#606 (comment).

ENH: Handle `.mtx.gz`

future suggestion: support reading .mtx.gz files as I can download from for example https://math.nist.gov/MatrixMarket/data/NEP/quebec/qh1484.html ?

First noted here: ropensci/software-review#606 (comment).

DOC,MAINT: Idempotent examples

I did not tick Examples, because although the examples run, but only in the right order, i.e. if I first run the examples of the write
functions then the examples of the read functions run correctly but they are not self contained

First noted here: ropensci/software-review#606 (comment).

MAINT: Vignette accessibility

I am not sure how, but I think there is a relatively easy way to have the getting started vignette show up as a separate menu item in the packagedown website and not just among articles, that could improve its discoverability

First noted here: ropensci/software-review#606 (comment).

CI: Rename workflow

pre commit github workflow name is a bit misleading to me, I understand you could add the same functionality as a pre commit hook, but a github action is a post commit/push thing, maybe title it after what it does, e.g. style checking?

First noted here: ropensci/software-review#606 (comment).

DOC: Make `fmm_to*` runnable

I understand that the fmm_to_* functions need an mtx file to work, and I guess it’s therefore the examples with these functions are marked with \dontrun{}. However, wouldn’t it possible to define a character vector in R that contains the required input, and then feed this to fmm_to_* in order to have an example that can actually be run?

First noted here: ropensci/software-review#606 (comment)

DOC: Vignette reduction (getting started)

consider moving the vignette summary to the beginning/remove it? It felt to me like repeating basically the same information as the beginning of the vignette

First noted here: ropensci/software-review#606 (comment).

DOC: Function help details are incorrect

In several function documentation you write "This function has no return value." This is technically not true, all functions return NULL if they have no other return value but rather these functions are not called for their return value, but for their side effect? Maybe see how other packages handle write functions?

First noted here: ropensci/software-review#606 (comment).

MAINT: Better error messages for `sparse_to_fmm`

improve error message?

mat <- matrix(c(1, 2, 3, 4), nrow = 2)
sparse_to_fmm(mat, "matrix.mtx")
#> Error: Invalid input type, expected 'double' actual 'NULL'

First noted here: ropensci/software-review#606 (comment).

The wrong function is called but the error message is confusing.

DOC: Move addenum

clarify what the addendum is about in the getting started readme? If I understand correctly, it demonstrates the incorrect handling of NA by Matrix package?

First noted here: ropensci/software-review#606 (comment).

ENH: `SparseM` matrices

https://cran.r-project.org/web/packages/SparseM/index.html

MAINT: Acknowledge reviewers

Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

Make these changes before final changes!

BENCH: Add some

MAINT: Misc review comments

where is this included from? I only see cpp11.cpp https://github.com/HaoZeke/fastMatMR/blob/main/src/from_file.cpp#L10
My IDE underlined this with red, but it might be fine, I am not familiar with cpp bindings in R packages

First noted here: ropensci/software-review#606 (comment).

ENH: Explicitly list non-zero entries

Currently sparse_to_fmm uses a standard header, but the number of non-zero entries can be pre-computed.

TST: Add some

Both round tripping through R and also the python integration tests.

MAINT: Fix the versioning

towncrier works, but there are pkgdown issues with it, and there are better alternatives, like https://github.com/cynkra/fledge/

MAINT: Update `fast_matrix_market` to 1.7.4

Changelog:
https://github.com/alugowski/fast_matrix_market/releases

TST: Do not write to project root

Ideally, tests would not write to the project root in my opinion even if those files are added
to .gitignore, but rather write to a temp directory. https://testthat.r-lib.org/articles/test-fixtures.html might help find a more robust solution.

First noted here: ropensci/software-review#606 (comment).

ENH: Support reading into a sparse matrix

For starters Matrix seems like a good bet (CSC).

DOC: Fix news

Rendered incorrectly.

MAINT: Remove commented code

commented out code should be removed (https://github.com/HaoZeke/fastMatMR/blob/main/src/to_file.cpp#L20)

First noted here: ropensci/software-review#606 (comment).

ENH: Better error handling

what happens if cpp lib call fails for some reason? Will the file handle be closed properly? See https://github.com/HaoZeke/fastMatMR/blob/main/src/to_file.cpp#L30

First noted here: ropensci/software-review#606 (comment).

I'm not sure this can even happen though..

DOC: Consistent argument names

I did not tick function documentation because although they exist, argument names
(fname vs filename) are not always consistent between the documentation and the code
itself

First noted here: ropensci/software-review#606 (comment).