omicsml / dance Goto Github PK
View Code? Open in Web Editor NEWDANCE: a deep learning library and benchmark platform for single-cell analysis
Home Page: https://pydance.readthedocs.io
License: BSD 2-Clause "Simplified" License
DANCE: a deep learning library and benchmark platform for single-cell analysis
Home Page: https://pydance.readthedocs.io
License: BSD 2-Clause "Simplified" License
- [ ] TODO later: use the `download_unzip` utility instead of doing these system calls.
Originally posted by @RemyLau in #223 (comment)
log1p
normalized expression instead of the raw countsI don't see any place with documentation for how I actually go about using the models in the package. Does any documentation exist?
TODO in a future PR: refactor dataset object to use the dance abstract dataset class, see [`ClusteringDataset`](https://github.com/OmicsML/dance/blob/f3e0e83ef3dbf25339e1a6ffa10ffb18ac81d72d/dance/datasets/singlemodality.py#L235) for example
Originally posted by @RemyLau in #228 (comment)
fit(*args, **kwargs) -> None
predict(*args, **kwargs) -> Union[np.ndarray, torch.Tensor, other?]
fit_predict
shortcut function combining fit
and predict
score(*args, **kwargs) -> float
(determine the type of metric via metric: Union[str, Callable]
, default setting saved in class attr. Abstract class per type of tasks?)__repr__
(show model metadata and info)I followed the install.sh file to install dance in a conda env. Now I am following the tutorial. I am getting this error
"OSError: libcusparse.so.11: cannot open shared object file: No such file or directory"
when running
"from dance.datasets.singlemodality import ClusteringDataset"
I don't know where the issue is coming from. I appreciate any suggestions.
We need both the return types for feature_channels
and label_channels
..
I work in an HPC environment where conda installation is not possible, pip installation results in a ModuleNotFoundError
for dance.metadata. The directory appears to be missing its __init__.py
file. Cloning the repo, adding the file and pip installing from the local clone worked for me.
Hi,
I have found that Human PDAC (ST) for Spot Deconvolution is missing true_p.csv in the spot deconvolution module.
According to the link graph construction description from the paper, the graph should also contain interaction between real-spots.
However, in the current implementation (modified from the original implementation, see Su-informatics-lab/DSTG#16), only the cross interactions between pseudo-spots and real-spots are used.
Should we add the interactions between real-spots as described in the manuscript in the DANCE implementation? If so, it could be achieved quite easily by simply calling construct_link_graph
one more time by passing real_st_df
as both first two args and combine the edge list with the old graph.
Caution: be aware that one might need to modify the the new graph node indexes (or pass real_st_df
and another copy of real_st_df
with offsetted index)
This work is quite interesting. Do you have a plan for the code release date?
Hello,
I am trying to reproduce the scmogcn.py example from the DANCE documentation. I run this script:
in the same folder where scmogcn.py is located. I'm using MSU's HPCC and a conda environment. After submitting the job, my slurm file gives this error: "dgl._ffi.base.DGLError: [18:00:11] /opt/dgl/src/random/random.cc:36: Check failed: e == CURAND_STATUS_SUCCESS: CURAND Error: CURAND_STATUS_INITIALIZATION_FAILED at /opt/dgl/src/random/random.cc:36"
Do you know what I might be doing wrong or how I can fix this error so that I can successfully reproduce this example?
The official tutorial is not compatible with v1.0 release. Please refer to the issue here.
Hi, I intend to apply this model to different datasets rather than the competition datasets, and I wonder if you have any general loading data structure to load public datasets or not. Moreover, is it possible for me to use a lighter structure comparing the jointembedding structure if I have already processed the given dataset? Thanks.
Some users pointed out that fully installing the dance
package in a conda
environment may take up a lot of space (~12GB), and suggested using solutions like Singularity to containerize the package to take advantage of the scratch space on HPC. I think @JiayuanDing100 might have some experience with containerizing/virtualizing. Could you suggest some ways to approach this?
Use omegaconf DictConfig
object for storing and managing configurations. Update preprocessing pipelines construction with config parsing.
Currently, each model comes with a .score
function for evaluating the model performance. Most of them do the same thing, e.g., compute model accuracy for cell-type annotation tasks. Why not just have one generic function that calculates the accuracy (and similarly for other tasks)?
For the imputation model GraphSCI, I think the AE part is wrong, the size_factor is cell_specific, so it should have the same number as cells, but in your implementation the size_factor number is gene-specific. For the inputs of AE, it should be the transpose of the input of GNN, so the input's shape should be cell*gene.
Currently, there is no convention for which device the data is stored when a function, e.g., fit()
, is called. For example, even though the computation device to use is cuda
, the input graph may not be on the GPU and will need to be transferred later after subsampling the neighborhoods for mini-batch training. This inconsistency causes many issues for development.
One naive solution is to call tensor.to(device)
using the correct device every time a computation is being performed, which is certainly unsatisfactory and makes the code base not as clean.
cpu
fit()
, is called, perform any necessary device conversion inside that function.predict()
and score()
, which can be more flexible, since they will be used in various places with various configurations, e.g., training on GPU with mini-batch training and evaluating on CPU with full-batch.The following PR is a corresponding example of fixing the issue: #30
grep "\.to(" -r dance | awk -F":" '{print $1}' | sort -u
Currently, there are several different dataset objects specialized for each task and model (e.g., CellTypeDataset
, ClusteringDataset
), each of them takes a variety of specialized arguments that are not directly related to the underlying data, e.g., save path, processing scheme, choice of tissue. This complexity makes it quite hard to maintain the code base and implement new methods/datasets.
To improve this situation, we need to isolate raw dataset objects from transformation/processing methods.
AnnData
as an input and save it as a private attribute (read-only?).scanpy
(recall that now the base data object store an AnnData
object as a (private) attributeIt turns out that the random seed of the SVD solver impacts the PCA loadings (except for the top few) quite a bit, even when the tol
option is set to 0. To enable reproducibility, set the random_state
option.
When running the tuning examples recently introduced in #398 (and #406), there appears to be some random chance of having segfault
. The issue was later observed to be machine specific. I have only been getting this random segfault
on MSU ICER HPCC (Python 3.8.16). Running the same example script on papermachine
does not throw this segfault
.
Looking at the core dump file (using pystack), it appears that the issue was related to Python's threading
. More particularly when calling sklearn's randomized svd func (maybe some other similar packages as well). See detail core dump log below.
(dance) bash-4.2$ pystack core core.113134
Using executable found in the core file: /mnt/home/liurenmi/software/anaconda3/envs/dance/bin/python
Core file information:
state: D zombie: True niceness: 0
pid: 113134 ppid: 112816 sid: 112816
uid: 790872 gid: 2362 pgrp: 113134
executable: python arguments: python main.py
The process died due a segmentation fault accessing address: 0xffffffffffffff70
Traceback for thread 114928 [] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 300, in check_internal_messages
self._loop_check_status(
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
join_requested = self._join_event.wait(timeout=wait_time)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
gotit = waiter.acquire(True, timeout)
Traceback for thread 114927 [] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 268, in check_network_status
self._loop_check_status(
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
join_requested = self._join_event.wait(timeout=wait_time)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
gotit = waiter.acquire(True, timeout)
Traceback for thread 114926 [] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 286, in check_stop_status
self._loop_check_status(
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 251, in _loop_check_status
join_requested = self._join_event.wait(timeout=wait_time)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 306, in wait
gotit = waiter.acquire(True, timeout)
Traceback for thread 114874 [] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/interface/router.py", line 70, in message_loop
msg = self._read_message()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/interface/router_sock.py", line 27, in _read_message
resp = self._sock_client.read_server_response(timeout=1)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 285, in read_server_response
data = self._read_packet_bytes(timeout=timeout)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/sdk/lib/sock_client.py", line 255, in _read_packet_bytes
data = self._sock.recv(self._bufsize)
Traceback for thread 114845 [Has the GIL] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 298, in _run_job
self._function()
(Python) File "main.py", line 83, in evaluate_pipeline
preprocessing_pipeline(data)
(Python) File "/mnt/ufs18/home-026/liurenmi/repo/dance/dance/pipeline.py", line 238, in __call__
func(*args, **kwargs)
(Python) File "/mnt/ufs18/home-026/liurenmi/repo/dance/dance/transforms/cell_feature.py", line 56, in __call__
gene_feat = gene_pca.fit_transform(feat.T) # decompose into gene features
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/_set_output.py", line 157, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/base.py", line 1152, in wrapper
return fit_method(estimator, *args, **kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 460, in fit_transform
U, S, Vt = self._fit(X)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 512, in _fit
return self._fit_truncated(X, n_components, self._fit_svd_solver)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 616, in _fit_truncated
U, S, Vt = randomized_svd(
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/extmath.py", line 449, in randomized_svd
Q = randomized_range_finder(
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/sklearn/utils/extmath.py", line 277, in randomized_range_finder
Q, _ = linalg.lu(safe_sparse_dot(A, Q), permute_l=True)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/scipy/linalg/_decomp_lu.py", line 220, in lu
p, l, u, info = flu(a1, permute_l=permute_l, overwrite_a=overwrite_a)
Traceback for thread 114844 [] (most recent call last):
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 178, in _heartbeat
time.sleep(5)
Traceback for thread 113134 [] (most recent call last):
(Python) File "main.py", line 108, in <module>
wandb.agent(sweep_id, function=evaluate_pipeline, count=3)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/wandb_agent.py", line 581, in agent
return pyagent(sweep_id, function, entity, project, count)
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 348, in pyagent
agent.run()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 326, in run
self._run_jobs_from_queue()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/site-packages/wandb/agents/pyagent.py", line 220, in _run_jobs_from_queue
thread.join()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 1011, in join
self._wait_for_tstate_lock()
(Python) File "/mnt/home/liurenmi/software/anaconda3/envs/dance/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
More sysinfo below.
Machine that failed:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Machine that did not fail:
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Skipping for now but might come back later to fix this issue if it appears to be happening to more users other than myself.
Some current graph transforms
methods directly save the processed dgl
graphs in .uns
. The limitations for doing so are
dgl
vs. pyg
) to use in the downstream model..uns
is not a good place to store large datadgl
or pyg
graphs given raw feature, edge data, or adjacency matrix.fit
function and construct the graph using the appropriate framework within the fit
function.index
optionmask
option(base) [zhan2210@gateway-03 ~]$ conda activate dance-env
(dance-env) [zhan2210@gateway-03 ~]$ export LD_LIBRARY_PATH=/mnt/home/zhan2210/
ENTER/lib: $LD_LIBRARY_PATH
(dance-env) [zhan2210@gateway-03 ~]$ python
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12
.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dance.datasets.multimodality import Joint EmbeddingNIPSDataset
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/ufs18/home-249/zhan2210/dance/dance/datasets/__init__.py", line 1,
in <module>
from dance.datasets.multimodality import JointEmbeddingNIPSDataset, Modalit
yMatchingDataset, ModalityPrediction Dataset
File "/mnt/ufs18/home-249/zhan2210/dance/dance/datasets/multimodality.py", li
ne 12, in <module>
from dance import logger
ImportError: cannot import name 'logger' from 'dance' (unknown location)
>>>
I can't download your dataset because of VPN restrictions and can't access the content under 'https://www.dropbox.com/'.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.