open2c / bioframe Goto Github PK
View Code? Open in Web Editor NEWGenomic interval operations on Pandas DataFrames
License: MIT License
Genomic interval operations on Pandas DataFrames
License: MIT License
The current bedGraph schema only includes chrom,start,end. The official bedGraph schema also includes dataValue field.
Check if an interval gets reduced below 0 when providing a negative pad value in ops.expand
.
bioframe/bioframe/core/construction.py
Line 63 in 1e1cd8d
in bioframe.closest(a,b)
the result is usually the same length than a. However, if some chromosome is missing in b but is in a, the result is smaller. This is dangerous!
Related but different issue: why don't we index the result by default with the index of a? This will prevent so so so many bugs! (I just had two in a row!)
Correct me if I'm wrong, but (one of) the most likely use case of closest would proceed as follows:
result = bioframe.closest(mydf, other_df)
do-something-to-result
mydf["new_column"] = result["some_column"] # option 1 or ...
mydf["new_column"] = result["some_column"].values # option 2
Right now both options 1 and 2 would usually work, but sometimes silently fail for two reasons. One is missing chromosome (both fail). Second is if the result is indexed differently than mydf! (if mydf is a slice and is indexed with [1,2,4,6,9,14,15]). First will fail silently, but second will work unless there are missing chromosomes. And if it is a mostly-complete slice, when you display the .head(), you won't even notice the bug. I just experienced both versions...
In my experience, pandas preserves the original index and length with .join(), but not with .merge(). So .join-like operations should comply with .join: same length and original index, while .merge()-like operations should allow for different length and no original index.
That would also require tranfering rle functions from old gists into arrayops
Hi,
I'm getting errors connecting to UCSC for both fetching centromeres and gene density, after upgrading to the most recent version of bioframe today. Is there a new way to call these functions based on the recent updates?
print(bioframe.__version__)
0.1.0-dev
#from cooltools example https://cooltools.readthedocs.io/en/stable/examples/pileups-example.html
#chromosome arms!
#Also specifying just good chromosomes 4, 14, 17, 18, 20, and 21
cens = bioframe.fetch_centromeres('hg19')
cens.set_index('chrom', inplace=True)
cens = cens.mid
GOOD_CHROMS = ['chr4', 'chr14', 'chr17', 'chr18', 'chr20', 'chr21']
arms = [arm
for chrom in GOOD_CHROMS
for arm in ((chrom, 0, cens.get(chrom,0)),
(chrom, cens.get(chrom,0), hg19.get(chrom,0)))
]
armsdf = pd.DataFrame(arms, columns=['chrom','start', 'end'])
arms
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-10-6971b191691f> in <module>
3 #Also specifying just good chromosomes 4, 14, 17, 18, 20, and 21
4
----> 5 cens = bioframe.fetch_centromeres('hg19')
6 cens.set_index('chrom', inplace=True)
7 cens = cens.mid
~/bin/miniconda3/envs/cooler-env/lib/python3.6/site-packages/bioframe/resources.py in fetch_centromeres(db, provider, merge, verbose)
77 client = UCSCClient(db)
78 fetchers = [
---> 79 ('centromeres', client.fetch_centromeres),
80 ('cytoband', client.fetch_cytoband),
81 ('cytoband', partial(client.fetch_cytoband, ideo=True)),
AttributeError: 'UCSCClient' object has no attribute 'fetch_centromeres'
Also an issue with gene density
#gene density (whole genome)
#using new way of cooltools compartment calling using gene density instead of GC% to phase A vs B (in cooltools examples)
# Download and compute gene count per genomic bin
bins = cooler.binnify(hg19, binsize)
genecov = bioframe.tools.frac_gene_coverage(bins, 'hg19')
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-9-f5a95c5a96d8> in <module>
3 # Download and compute gene count per genomic bin
4 bins = cooler.binnify(hg19, binsize)
----> 5 genecov = bioframe.tools.frac_gene_coverage(bins, 'hg19')
AttributeError: module 'bioframe' has no attribute 'tools'
Getting a warning:
/home/s1529682/.local/lib/python3.7/site-packages/bioframe/io/formats.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
We need a function to synchronize the indices of two tables with almost identical intervals. This is typically needed to enable safe transferring of columns between these tables. Alternatively, we can have a function that transfers columns between two tables of almost identical intervals.
@nvictus , is this a good summary of your request?
e.g. bf.overlap(df1,df2['chrom','start','start']) returns ValueError: object too deep for desired array
Currently is_contained
allows for "containment" checking only between 2 dataframes (viewframes?) that both have "name"-column (or similar) and such "names" of the df
are cataloged in view_df
.
This might be limiting, as for example it prevents us from checking if a set of regions (e.g arms) is contained within chromosomes:
make_viewframe(arms_df, check_bounds=chromsizes_df)
fails with the "not cataloged" error
I was recently checking out the cooltools suite and found that some functions used parts of the bioframe package that are not part of the module that could be installed via pip (i.e. PyPi) version.
I double-checked the version numbers and I have the latest bioframe version.
In particular this affects the bioframe.bedslice
function which is used in the cooltools cooltools.eigdecomp.cooler_cis_eigs
function to get the part of the phasing track matching the current region under decomposition. There it is used like defined in the repo here and also works as expected when using the cooltools function with a signature of
bedslice(frame, region)
However, when trying to use the bedslice
directly it is a completely different function with a signature of
bedslice(frame, chrom, start, end)
Is there any explanation for this?###
When trying to import bioframe: SyntaxError: invalid syntax
@golobor was this suppose to be an f-string? (fixed it for me)
When I try to run frac_gc, I get the following error:
/home/sameer/miniconda3/lib/python3.6/site-packages/bioframe/tools.py in _each(chrom_group)
149 for _, bin in chrom_group.iterrows():
150 s = seq[bin.start:bin.end]
--> 151 g = s.count('G')
152 g += s.count('g')
153 c = s.count('C')
AttributeError: 'Sequence' object has no attribute 'count'
I think this is because the variable s is an OrderedDict() of <chrom_name> and associated FastaRecord. The fasta record needs to be a string variable before the count method can be used successfully.
TODO:
API
Cosmetic:
arg
for arguments in docstrings, to agree with https://numpydoc.readthedocs.io/en/latest/format.html#parameters. internal links can be added with func:myfunc()
Changes in the existing code code:
New code:
(chrom, start, end, on)
. This is currently handled on a function-by-function basis to avoid casting to float.bioframe.core.construction.sanitize_bioframe
Test:
Docs:
_verify_columns
and _verify_column_dtypes
to specs documentation (https://stackoverflow.com/a/7740295)Line 1193 in 2267519
@gfudenberg , I forgot if we discussed it, but is that really the desired behaviour? In the standard application, splitting of chromosomes into arms, this setting leads to an omission of chromosomes for which centromeres are not specified.
Overall, this seems counter-intuitive to me... I would suggest (a) adding an argument to retain non-split intervals and (b) setting it to True by default. What do you and others think?
In light of Anton's PR, just want to start a discussion about future possible functions for 2D interval operations for a later release.
complement
, and it seems ill defined anyway.Other thoughts?
Would be nice to add fractional expand, to e.g. double the length of each interval, even if they have different lengths.
bioframe.fetch_chromsizes
pulls a filtered subset of chromosomes (ignoring contigs and such) in a "natural" order - but it does not reset a DataFrame index - so it is out of order and non-contiguous e.g. chrM is ~249 for hg19 - should we reset index before returning a dataframe ?
!!! true - only when using as_bed=True
!!! - as_bed = False
yields a chrom
indexed DataFrame
would be as simple as adding .reset_index(drop=True)
at the end of this line:
bioframe/bioframe/io/formats.py
Line 138 in 94a9fcd
Right now make_viewframe
does not auto-generate name
column , unless specifically asked to (at least when input is a 3-column DataFrame) ...
It should, though, because viewframe
is defined as a 4-column "table"
cooler.bins() returns chrom column as categorical.
If one is then to filter chroms (e.g. bins = bins[bins["chrom"] != "chrM") then "chrM" actually is left in the categorical variable, and would be a part of the .groupby(), even though the corresponding group is length 0.
Maybe this https://github.com/mirnylab/bioframe/blob/develop/bioframe/ops.py#L833 and similar places need a check for whether the group is empty?
related to "define bioframe standards" item of the current roadmap: #48
in cooltools CLI (and when working with real data in a notebook) we have some reoccurring "read-from-file" patterns:
We clearly need those functions in cooltools, but the question is if they need to be defined in bioframe
?
... Probably not, as making "read-from-file" functions that check for compatibility with cooler should be outside the scope of "bioframe" ...
However, implementing functions like that would require "bioframe" defined standards and checks for sure, like:
Last point - "read-from-file" functions like these can maybe reside in bioframe
only if they check for cooler-compatibility indirectly - e.g. via "chrom-sizes" and/or "binsize" in case we want or check alignment to bins
This includes rlencode/rldecode etc...
Also, consider adding code from this repo: https://gist.github.com/nvictus/66627b580c13068589957d6ab0919e66
bioframe/bioframe/genomeops.py
Line 283 in 94a9fcd
I naively assumed that frac_gene was using unique genes, not all predicted and verified RNA isoforms.
minimal example:
x = pd.DataFrame({"chrom":["chr1","chr2"],"start":[100,400],"end":[110,410]})
x.index = [0,0]
bioframe.merge(x, min_dist=5)
"combinatorially" broken output - all chroms are paired with all the intervals:
chrom start end n_intervals
0 chr1 100 110 1
1 chr1 400 410 1
2 chr2 100 110 1
3 chr2 400 410 1
should have been:
chrom start end n_intervals
0 chr1 100 110 1
1 chr2 400 410 1
more real-life use case scenario would be:
df = pd.concat([peaks1,peaks2])
bioframe.merge(df,min_dist=1000)
resetting index of the input DataFrame fixes that ...
is currently incompatible with passing cols that differ from the default chrom, start, end
should be an easy fix
@gfudenberg , you should probably know, do we really need check_exact when we compare pyranges vs bioframe via assert_frame_equal?
https://github.com/mirnylab/bioframe/blob/0ee7a471b998c36e22254cb2ef98c336d7f2d770/tests/test_ops.py#L131
Our tests on github actions fail for some reason b/c of this flag. On my own machine, same tests pass :-/. Not sure why there is such a discrepancy, but setting check_exact=False could probably solve it.
only able to fetch chromsizes for organisms with Arabic numbers (i.e. hg19) not Roman numeral (i.e. ce10)
I'm getting errors when calling cooltools functions for calculating eigenvectors/eigenvalues. For example slice_bed does not exist or no tools module. Can the cooltools package be updated to the proper bioframe functions?
it would be great to enhance the functionality of bioframe.EncodeClient
Upcoming breaking change in pandas default:
FutureWarning: The signature of `Series.to_csv` was aligned to that of `DataFrame.to_csv`, and argument 'header' will change its default value from False to True: please pass an explicit value to suppress this warning.
I am getting an error today trying to convert bedgraph formated pandas dataframes to bigwig - this code has worked previously (about 1 month ago), but today is giving an error. I have updated cooltools and bioframe to try to fix the issue, but it does not seem to have helped.
Here is the code:
#save to file
for cond in conditions:
bioframe.to_bigwig(insul[cond], chromsizes,
f'data/{long_names[cond]}.{binsize//1000}kb.insul_score_{window_bp}.bw',
f'log2_insulation_score_{window_bp}')
Here is the error:
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-5-d6f9cc84595f> in <module>
5 bioframe.to_bigwig(insul[cond], chromsizes,
6 f'data/{long_names[cond]}.{binsize//1000}kb.insul_score_{window_bp}.bw',
----> 7 f'log2_insulation_score_{window_bp}')
~/bin/miniconda3/envs/cooler-env/lib/python3.6/site-packages/bioframe/io/formats.py in to_bigwig(df, chromsizes, outpath, value_field)
433
434 run(['bedGraphToBigWig', f.name, cs.name, outpath],
--> 435 print_cmd=True)
436
437
~/bin/miniconda3/envs/cooler-env/lib/python3.6/site-packages/bioframe/io/process.py in run(cmd, input, raises, print_cmd, max_msg_len)
46 if len(out) > max_msg_len:
47 out = out[:max_msg_len] + b'... [truncated]'
---> 48 raise OSError("process failed: %d\n%s\n%s" % (p.returncode, out.decode('utf-8'), err.decode('utf-8')))
49
50 return out.decode('utf-8')
OSError: process failed: 255
Expecting 2 words line 1 of /tmp/tmpb8jcb5z7.chrom.sizes got 1
Here is the first few lines of the insulation file:
chrom start end is_bad_bin log2_insulation_score_480000 n_valid_pixels_480000
0 chr1 0 40000 True NaN 0.0
1 chr1 40000 80000 True NaN 0.0
2 chr1 80000 120000 True NaN 0.0
3 chr1 120000 160000 True NaN 0.0
4 chr1 160000 200000 True NaN 0.0
Here is what I am using for chromsizes - it is from bioframe.fetch_chromsizes
chr1 249250621
chr2 243199373
chr3 198022430
chr4 191154276
chr5 180915260
chr6 171115067
chr7 159138663
chr8 146364022
chr9 141213431
chr10 135534747
chr11 135006516
chr12 133851895
chr13 115169878
chr14 107349540
chr15 102531392
chr16 90354753
chr17 81195210
chr18 78077248
chr19 59128983
chr20 63025520
chr21 48129895
chr22 51304566
chrX 155270560
chrY 59373566
chrM 16571
Name: length, dtype: int64
pyranges has an extensive test suite is based on generative testing with hypothesis.
It basically generates random genomic data and ensures that the pyranges solution is equal to the bedtools solution. Would you be interested in a similar test suite for bioframe?
I only run a hundred iterations on CI, but on my home server I often let it run maany examples for each test.
Hi,
Thanks for your developed tool. Recently, I met a problem about the fetch_centromeres. When I tried the command 'bioframe.fetch_centromeres('mm9')', it reported an error :
##########
Traceback (most recent call last):
File "", line 1, in
File "/home/dell/softwares/anaconda2/envs/py36/lib/python3.6/site-packages/bioframe/io/resources.py", line 122, in fetch_centromeres
raise ConnectionError("No internet connection!")
ConnectionError: No internet connection!
##########
Could you give me a hand?
Thanks
I am just wondering why/if you did not find pyranges satisfactory for some reason :)
Several cooltools need to know the locations of centromeres. Thus, we need a function that would automatically fetch them and bioframe seems like the best library for it.
The biggest obstacle is that there is no standard file in goldenPath whose purpose would be to store centromeres. Several files may or may have centromeres (cytoband, gap, agp), but there is never a dedicated file that would work for all assemblies. Moreover, for some assemblies, like sacCer3, we'd need to use other resources besides UCSC.
So, the solution that I propose is to query centromeres in three steps:
(1) include centromeres for the model organisms (humans, mice (acro), zebrafish, drosophila (none), c.elegance, baker's and fission yeast, arabidopsis) into bioframe itself.
(2) if (1) fails, fetch and parse cytoband file from UCSC
(3) if (2) fails, fetch and parse gap
Also, since centromeres are extended regions, store them as (chrom, start, end, mid).
ping @nvictus
it'd be super nice to have a function that saves a bedframe into a bed file and resorts columns, potentially drops those not compatible with the bed format and formats the comment line.
clustering isn't only for overlapping intervals, right? I was wondering at first what's different between merge
and cluster
Line 457 in 9af1b45
add somethink like "limit by chromosome or region", or maybe a quick inline example would suffice, - otherwise not 100% obvious from the docstring that it has to be like:`{chr1: (0,100000), chr2: (400,8000000),...}
Line 112 in 9af1b45
"indepdendently" :))
Line 367 in 9af1b45
has been addressed by #67
As a reminder, once Anton's PR is merged we should see how/whether to implement operations with signal tracks, such as average profiles, at least as an example notebook at first.
How would I cite bioframe?
I am writing up my Ph.D. and discussing the merits/downsides of pyranges and bioframe.
I know you do not have a paper on it but it would be nice to have something to cite. Is it correct to say that bioframe was originally made to support cooler and other Hi-C software? Then I can cite cooler.
in an effort to unify handling of regions throughout cooltools/bioframe we should have something like to_ucsc
formatter somewhere , shoudn't we ?
e.g. to avoid code like that: regions.apply(lambda x: "{}:{}-{}".format(*x), axis=1)
...
There is something in cooler i believe but should it be in bioframe instead ?
a little snippet below is an example of my issues with bioframe.ops.overlap
:
pd.NA
to float
chrom,start,end
became object
and pd.Categorical
somehow - which made groupby
without observed=True
impossibleI'm not sure what of that is expected behavior and what isn't ...
I'll fill in this example with actual data later
from bioframe import io
timfname = "filename"
bins = clr.bins().fetch(chrom).copy()
val_rt = io.read_table(timfname, schema=bioframe.schemas.BEDGRAPH_FIELDS, skiprows=1)
val_rt = val_rt.rename({0:"chrom",1:"start",2:"end",3:"rt"}, axis=1)
# overlap RT signal with our bins ...
binned_rt = bioframe.ops.overlap(bins, val_rt, suffixes=['', '_rt'])
# freaking pd.NA !? can't cast them to float ... why ? why are they there in the first place ?!
binned_rt["value_rt"] = binned_rt["value_rt"].apply(lambda x: np.nan if x is pd.NA else x)
binned_rt = binned_rt \
.drop(labels=["chrom_rt","start_rt","end_rt"],axis=1) \
.astype({"start":int,"end":int,"weight":float})
# finally :
binned_rt = binned_rt.groupby(["chrom","start","end"],observed=True,sort=False).mean()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.