apriha / snps Goto Github PK
View Code? Open in Web Editor NEWtools for reading, writing, merging, and remapping SNPs
License: BSD 3-Clause "New" or "Revised" License
tools for reading, writing, merging, and remapping SNPs
License: BSD 3-Clause "New" or "Revised" License
Hi,
I am trying to read and output a VCF file using the SNPs library. I would like to do this to check that the VCF is valid and remap if the file is not from the GRCh37 reference genome.
However, I am getting the error: no data to save...
when I try and save the my SNPs object to a file.
# download small VCF data
wget https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/modules/beagle5/test/subsampled/FR07961001.pass.recode.subsampled.vcf.gz
gzip -d FR07961001.pass.recode.subsampled.vcf.gz
from snps import SNPs
import logging, sys
# for logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.StreamHandler(sys.stdout))
input_file = 'FR07961001.pass.recode.subsampled.vcf'
snps = SNPs(input_file)
snps.save_snps()
no data to save...
The snps object does seem to be empty:
df = snps.snps
df.columns.values
# array(['chrom', 'pos', 'genotype'], dtype=object)
len(df)
# 0
However, I also tried reading the input VCF line by line with SNPs using the following:
with open(input_file, 'r') as f:
snps = SNPs(f.read().encode('utf-8'))
And then when I outputted this object it seemed to have the data:
But I'm not sure if there are any issues with reading the format of the VCF:
Any ideas what might be going wrong. Any help would be much appreciated.
Many thanks in advance,
Phil
Duplicate SNPs can be reported for a particular locus in a file (e.g., two SNPs with the same position on the same chromosome, potentially with different genotypes). In one of the example files, there are 234 SNPs where another SNP has the same locus:
import pandas as pd
from snps import SNPs
s = SNPs("resources/662.23andme.340.txt.gz")
df = pd.DataFrame()
for chrom in s.snps.chrom.unique():
temp = s.snps.loc[s.snps.chrom == chrom]
df = df.append(temp.loc[temp.pos.duplicated(keep=False)])
print(len(df))
Several of these SNPs have "internal IDs", but there are cases where two different rsids are reported for the same locus. Additionally, the genotype reported is not always the same for a particular locus.
To remedy this, I suggest adding a deduplicate_loci
parameter to the SNPs
object and renaming the deduplicate
parameter to deduplicate_rsids
.
Then, if deduplicate_loci=True
(default), deduplicate the loci in this manner:
Chromosome by chromosome, find SNPs with duplicate positions. Then, for each duplicate position, sort the SNP rsids / internal IDs and keep the last in the normalized SNPs dataframe; this will ensure internal IDs are replaced with rsids (if available) and also keep IDs with higher numbers (potentially newer). In this process, add the kept rsid as a key to a _duplicate_loci
dict
, where the value is a list
of any deduplicated loci (str
rsids). Additionally, merge the genotype (updating any nans) and identifying discrepant genotypes as discrepant_loci
(similar to discrepant_XY
) and mark the genotype as nan.
Finally, loci should optionally be deduplicated after each merge (add deduplicate_loci
parameter to merge
as well).
Merging snps from different samples and saving them as a multi-sample VCF.
Hi, would that be possible?
Thanks!
Rename several SNPs
methods and properties to be less redundant, e.g., remove _snps
suffix from remap_snps
, discrepant_XY_snps
, etc. For now, retain an alias to these methods and properties and raise DeprecationWarning
.
To simplify merging SNPs, add a merge
method to the SNPs
class, and in the process obsolete the SNPsCollection
class. The merge method should accept an iterable of SNPs
objects. In this manner, SNPs
objects can be merged after the fact, vs. requiring a SNPsCollection
from the beginning to merge SNPs.
Similar functionality to SNPsCollection
should be provided, such as sorting SNPs after the merge and identifying discrepant genotypes and positions.
Properties of the SNPs
objects will need to be merged and behavior defined (e.g., source
, build
, discrepant_XY_snps
, etc.).
dask would help with reading large VCFs and could potentially increase the speed of calculations throughout (e.g., remapping and writing VCFs).
With the X chromosome, if only one allele is reported for a SNP, the allele is doubled such that the X chromosome genotype
will always be a string of length 2. Some data sources report two X chromosome alleles for all X chromosome SNPs, even if the individual is a male. So, by doubling single alleles, this makes all X chromosome data consistent.
However, instead of doubling alleles, it'd be better to analyze the X chromosome SNPs to determine if only one allele should be reported (i.e., if an individual is male). Any heterozygous SNPs not in the pseudoautosomal region could then be flagged as a discrepant SNP.
The thresholds and methods for determining the sex of a genotype should be reinvestigated. See comments regarding misidentification of Female genotypes as Male at #48.
To fix this, I propose investigating the optimal thresholds for these parameters using data from openSNP and updating determine_sex
as necessary.
Moreover, the script documenting the analysis and its results should be added to the repo as an example.
Update the dtype
of rsid
, chrom
, and genotype
columns to be pandas.StringDtype
as recommended here.
Also require pandas>1.0.0
.
The readme says that there the package is able to read VCF, but there is no example how to read them (only write). Trying to read VCFv4.1
fn_struct = 'my.vcf'
sc = SNPs(fn_struct)
sc.build
# 0
returns 0 records, though pyVCF works OK:
vcf_reader = vcf.Reader(filename =fn_struct)
sum((1 for record in vcf_reader))
# 184353
I think the error could be handled when trying to save invalid genotype files to VCF.
For example, if I run:
# create invalid empty file
touch file.txt
python
>>> from snps import SNPs
>>> s = SNPs('file.txt', output_dir='.')
# error message is handled nicely when vcf is false
>>> saved_snps = s.save_snps('test.txt', vcf=False)
no data to save...
# error message is not handled when vcf is true
>>> saved_snps = s.save_snps('test.vcf', vcf=True)
Then I get the following error message:
Traceback (most recent call last):
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'genotype'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/__init__.py", line 394, in save_snps
snps=self, filename=filename, vcf=vcf, atomic=atomic, **kwargs
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 951, in write_file
return w()
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 924, in __call__
return self._write_vcf()
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 1060, in _write_vcf
df["genotype"].notnull()
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/frame.py", line 2995, in __getitem__
indexer = self.columns.get_loc(key)
File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'genotype'
Any ideas how this could be improved? Perhaps catching the error here?
Thanks in advance
Per the Ensembl README (ftp://ftp.ensembl.org/pub/grch37/current/fasta/homo_sapiens/dna/README):
Human has sequenced Y chromosomes and the pseudoautosomal region (PAR)
on the Y is annotated. By definition the PAR region is identical on the
X and Y chromosome. The Y chromosome file contains the Y chromsome
minus these repeated PAR regions i.e. the unique portion of Y.
If writing a VCF that contains Y chromosome PAR SNPs, the full Y chromosome reference sequence needs to be reconstructed using the PAR regions from X in order to determine the reference alleles.
Add ability to load mtDNA FASTA data (e.g., from FTDNA) and parse out mtDNA SNPs (see #9).
I'm using aws ec2 ubuntu. It does not allow me to create an individual.
user662 = l.create_individual('User662', '/home/ubuntu/myprojectdir/AaronAzuma.zip')
Traceback (most recent call last):
File "", line 1, in
File "/home/ubuntu/myprojectdir/venv/lib/python3.8/site-packages/lineage/init.py", line 96, in create_individual
return Individual(name, raw_data, self._output_dir, **kwargs)
AttributeError: 'str' object has no attribute '_output_dir'
Validate by attempting to recognize assemblies of openSNP files.
I have been using the detect_build
functionality and I have noticed that snps
does not correct detect our build 38 Exome VCFs. This is likely because Exome data is not technically within scope of the genotype data that snps originally was intended to support.
It seems that the rsids that are listed here are not present in my Exome file. I am looking to find some more examples of rsids that are included in common Exome files so that this functionality is extended in the same way that you have originally implemented it.
https://github.com/apriha/snps/blob/master/src/snps/__init__.py#L398
Let me know your thoughts!
Will
It seems that openSNP has changed their downloads, a single zip file is no longer available. This breaks some of the analysis code, which is where I found this problem as I want to investigate and compare some alternative sex detection mechanisms. I'll take a look and see if I can work out a solution in a pull request.
Merge the Y chromosome SNP data for two closely related males, identifying discrepant SNPs in the process.
Add a phased
property to SNPs
to indicate if the genotype
is phased.
I'm getting errors across the whole test suite when running them locally on my Mac OSX Catalina 10.15.4 machine.
os.makedirs(path, exist_ok=True)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:213: in makedirs
makedirs(head, exist_ok=exist_ok)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:223: in makedirs
mkdir(name, mode)
E FileNotFoundError: [Errno 2] No such file or directory: '../../../../var/folders/h3/k5sgzmp17cvbz1mm459fkf200000gn/T/tmp9ixj7r4i/fasta'
Looking around, I think this is something to do with the permissions on the /var/folders/
directories
Is there a way we can specify what temporary to use here?
Currently, for file types that use a generic parser, their source is identified as "generic". Suggest changing the source to "unknown".
Currently, there is a single general Reader class with various methods for different sources.
I wonder if a series of specific classes for different filetypes be better e.g. Reader23andMe, ReaderAncestry, etc. Still keeping the single Reader class overall, but delegating a lot of the details to the specific classes for implementation.
By grouping everything for a particular format together it makes it easier to read, and to add extra sources. Also it would allow subclasses (e.g. for Global Screening Array based sources) to abstract functionality more easily.
Hello
I'm trying to convert a .csv from FTDNA to .vcf
I get load my .csv (the assembly and build are OK) and I set the ouptut dir throught:
snps.SNPs(output_dir='C:Users\\Desktop\\')
.
The execution return:
SNPs('')
when I save the .csv throught:
s = SNPs('C:\\Users\\PC\\Desktop\\genome.csv')
s.save_snps(vcf=True)
I get
'output\\FTDNA_GRCh37.vcf'
But not generates any file
What I'm doing wrong?
Thanks in advance
Replace print
statements with calls to a logger.
Along the same lines as #108 has it been considered to use the Pandas nullable integer datatype (pd.Int64Dtype()
) for pos
? More details here. We've seen a number of files that fail to parse because the position information for a small number of rows is missing (for example, on a RSID with multiple possible locations).
Hi,
Thanks for a great tool.
Is there support (or plans to support) the following:
Thanks in advance,
Phil
Currently, snps
normalizes data into a dataframe with four columns named rsid
, chrom
, pos
, and genotype
.
genotype
can either be np.nan
or a string of length 1 or 2. For autosomal SNPs and the X chromosome, the genotype is always a length 2 string. See #43.
The Y chromosome and mtDNA alleles are often strings of length 1; however, SNPs in the pseudoautosomal region on the X and Y chromosomes often have two alleles reported.
So, to better handle various numbers of alleles reported for the X, Y, and mtDNA chromosomes, consider refactoring the genotype
column into allele1
and allele2
.
Note that this would also more naturally support phased genotypes (vs. indexing a length 2 string genotype), wherein allele1
could be alleles on one chromosome, and allele2
could be alleles on the other. See #44.
With pandas
1.1.0, pd.testing.assert_frame_equal(..., check_exact=True)
unexpectedly raises an AssertionError
, which causes several tests to fail. See pandas-dev/pandas#35446
Therefore, update snps
to not use pandas
1.1.0.
If you have a (23AndMe-like) file produced by snps, eg:
# Generated by snps v0.1.1.post243+ge018ea4, https://pypi.org/project/snps/
# Generated at 2020-01-20 22:55:53 UTC
# Source(s): 23andMe
# Assembly: GRCh37
# Phased: False
# SNPs: 638463
# Chromosomes: 1-22, X, Y, MT
rsid chromosome position genotype
rs548049170 1 69869 TT
rs13328684 1 74792 --
rs9283150 1 565508 AA
i713426 1 726912 --
rs116587930 1 727841 GG
rs3131972 1 752721 AG
rs12184325 1 754105 CC
rs12567639 1 756268 AA
And then you try reading this file into snps you get the following error Too many columns specified: expected 4 and found 1
:
Traceback (most recent call last):
File \"/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py\", line 2897, in get_loc
return self._engine.get_loc(key)
File \"pandas/_libs/index.pyx\", line 107, in pandas._libs.index.IndexEngine.get_loc
File \"pandas/_libs/index.pyx\", line 131, in pandas._libs.index.IndexEngine.get_loc
File \"pandas/_libs/hashtable_class_helper.pxi\", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File \"pandas/_libs/hashtable_class_helper.pxi\", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'genotype'
This was discovered by @willgdjones who noted that the issue occurs because:
It has ‘snps’ in the header and so the snps package thinks that it is from lineage dna which uses comma-separated values as opposed to tab-separated values
How can this issue be fixed? As long as it's not too difficult or time-consuming I'd be happy to work on a PR to fix this
Access the GWAS Catalog via the REST API to retrieve extensive data related to SNPs: https://www.ebi.ac.uk/gwas/docs/programmatic-access . Various reports could be generated, but this type of data would probably be best interacted with via a web front-end (see #11).
Add a chrom
parameter to SNPs.merge
to merge only the SNPs for the specified chromosome, identifying discrepant SNPs in the process.
Add ability to compute additional summary statistics:
Related to #29
I'd like to suggest adding an option to save files in a tabular rather than csv format for easier use in other tools and databases, such as GEDmatch. I've found that a simple replacement of all commas with tabs allows my test files to be used in GEDmatch, so it may be as simple as that.
Apologies in advance if this is already an option and that I just didn't look carefully enough at the docs!
-Gerard.
Currently tests save output files to a directory named output
at the repository root. It would be better to save output files to a temporary directory created by tempfile.TemporaryDirectory
.
Utilize NCBI's Variation Services API to populate missing RSIDs.
An idea for how this could be performed... For all SNPs in a dataset, lookup RSIDs in batches of 50k SNPs (1 request / second) using the VCF endpoint. Then, use the resulting information to populate any missing RSIDs. However, note that this method would require the REF
allele and a valid ALT
allele.
As an example, one row for this query (using GRCh38
) could be constructed as follows:
1 817186 . G A,C,G,T
Query:
curl -X POST "https://api.ncbi.nlm.nih.gov/variation/v0/vcf/file/set_rsids?assembly=GCF_000001405.38" -H "accept: text/plain; charset=utf-8" -H "Content-Type: text/plain; charset=utf-8" -d "1\t817186\t.\tG\tA,C,G,T"
Which would return:
1 817186 rs3094315 G A,C,G,T
This could also be used to verify SNP positions and help with #6.
Thanks to @gedankenstuecke for helping develop the idea!
Some newer Family Tree DNA famfinder files seem to have SNPs assigned to chromosome 0. Similar to SNPs._assign_par_snps
, use the RefSNP API to assign each of these SNPs to a chromosome and update the SNP's position to be consistent with the assembly.
Ancestry and 23andme appear to have discrepant mapping of indel SNPs, causing them to be discarded by SNPs, and sometimes causing the program to fail.
Indel SNPs positions where the allele is named D (deletion) or I (insertion) and can be deletions or insertions of several bases, or more complex multi-nucleotide changes (so technically, not all are actually SNPs). In my test case, Ancestry covers 8073 indel SNPs, and 23andme (v5) has 4828, of which 2143 are found on both platforms (by rs#). However, only 1328 of the 2143 report the same chromosomal location, so the remaining 815 to be rejected as discrepant.
Most of the location discrepancies are tiny - 475 of them are 1 base, 174 are 2, 50 are 3, 48 are 4, 33 are 5, 16 are 6, and a further 18 are 17 or less. In almost all cases, the difference is the length of the indel, and the ancestry location is less than the 23andme location. This seems to be due to ancestry reading the position where the deletion started, and 23andme reading the position where it ended, but there are some additional odd cases, and neither ancestry nor 23andme is always consistent with dbSNP.
In practical terms, I'd like to suggest two possible solutions
Loosen the criterion for merged SNPs to have identical location, and if an ancestry/23andme merge, retain the 23andme location. Having the same rs# and being within 20 bases of each other would seem a reasonably strict criterion
Could also just ignore all discrepant indels. The vast majority of them are homozygous in most people (i.e. are rare and usually disease-associated alleles), so they are not very informative for genealogy.
I also wanted to note that I was unable to get my two genotype files to merge by altering the parameter discrepant_genotypes_threshold to 1000000, but only by manually deleting all indel SNPs from the two input files. I haven't looked further into this issue, but may be worth checking out.
Update to use f-strings for creating dynamic strings (vs. .format
method).
With pandas
1.0.0, using na_rep
with to_csv
causes position values in output files to be truncated (e.g., files created via save_df_as_csv
). See pandas-dev/pandas#31447
In the meantime, a workaround would be to pin the version of pandas
to <=0.25.3.
Building on #107, consolidate several issues (e.g., duplicate_rsid
, discrepant_XY
) into one dataframe with the following columns / dtypes:
Column | pandas dtype |
---|---|
rsid |
pd.StringDtype() |
chrom |
pd.CategoricalDtype() |
pos |
pd.UInt32Dtype() |
genotype |
pd.CategoricalDtype() |
duplicate_rsid |
pd.BooleanDtype() |
discrepant_loci |
pd.BooleanDtype() |
discrepant_XY |
pd.BooleanDtype() |
heterozygous_MT |
pd.BooleanDtype() |
discrepant_vcf_position |
pd.BooleanDtype() |
discrepant_merge_position |
pd.BooleanDtype() |
discrepant_merge_genotype |
pd.BooleanDtype() |
Multiple issue columns could take on the value of True
, and getting SNPs with issues (e.g., discrepant_XY
) could be handled by filtering the issues dataframe.
rsid
s could appear more than once in this dataframe. However, if an rsid
has two or more rows that are equivalent (same values for chrom
, pos
, and genotype
), their issues should be consolidated into one row, with the issue columns flagging the issue(s).
The current implementation can take a long time to read VCFs. Consider using CyVCF, or pandas.read_csv to read VCFs. The pandas
implementation would likely be similar to the existing snps.io._write_vcf
.
The latest version of pandas
officially supports Python 3.6.1 and above. Similarly, I recommend the following changes to follow suit:
pandas
to be != 1.0.0 (avoids issue in #59)Currently, the Reader
and Writer
classes within io
read and write files from the filesystem. E.g.
Lines 71 to 73 in 7e5d622
I have a use case where it would be useful to be able to read and write from buffered IO.
I'm happy to implement this in the coming days and submit a PR, but I wanted to ask what your opinions on this would be.
Thanks!
Will
Ancestry reports mitochondrial (MT/Chr26) SNPs as diploid (e.g. "GG", with two identical alleles), even though there is just one mitochondrial genome, with multiple copies. 23andme reports as haploid (e.g. "G"). When 23andme and Ancestry files are merged in lineage, the csv output has a single allele for all 23andme SNPs but two alleles for any that are present on Ancestry (including SNPs present on both).
Suggest that, for consistency, all MT SNPs be reported with a single allele, and any heterozygosity in Ancestry calls (due to heteroplasmy, which I've never yet seen) be reported as an error.
Hi. Am trying out the codes. Understand that the raw files here mean genotype files from DTC companies. However, what I have is only vcf and gvcf files from in-house sequencing platform. May I know can these be used? How? I have zero knowledge in coding. Thanks a lot!
Hi @apriha,
I noticed throughout the codebase that Exceptions are caught and not raised to the user. This creates some challenges in tracking down the cause of bugs when they crop up.
I would propose either:
It would be great to get your thoughts!
Thanks,
Will
Merge the mtDNA SNP data for two closely related individuals that share the same mtDNA, identifying discrepant SNPs in the process.
The mapmygenome file has additional column GC.score, that gives the confidence level of the SNP.
It will be good to filer out SNP rows that has lower confidence (say < 0.75).
Requesting for adding this feature of filtering rows based on GC.score into your module.
The file format of mapymygenome has changed.
After reading the file, the data frame reports zero rows.
Request to update the reader.
The new header is :
SNP.Name Sample.ID Allele1...Top Allele2...Top GC.Score Sample.Name Sample.Group Sample.Index SNP.Index SNP.Aux Allele1...Forward Allele2...Forward Allele1...Design Allele2...Design Allele1...AB Allele2...AB Allele1...Plus Allele2...Plus Chr Position GT.Score Cluster.Sep SNP ILMN.Strand Customer.Strand Top.Genomic.Sequence Plus.Minus.Strand Theta R X Y X.Raw Y.Raw B.Allele.Freq Log.R.Ratio CNV.Value CNV.Confidence
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.