apriha / snps Goto Github PK

Hi,

I am trying to read and output a VCF file using the SNPs library. I would like to do this to check that the VCF is valid and remap if the file is not from the GRCh37 reference genome.

However, I am getting the error: no data to save... when I try and save the my SNPs object to a file.

To replicate issue

Download data:

# download small VCF data
wget https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/modules/beagle5/test/subsampled/FR07961001.pass.recode.subsampled.vcf.gz
gzip -d FR07961001.pass.recode.subsampled.vcf.gz

Run SNPs:

from snps import SNPs
import logging, sys

# for logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.StreamHandler(sys.stdout))

input_file = 'FR07961001.pass.recode.subsampled.vcf'
snps = SNPs(input_file)
snps.save_snps()

Output:

no data to save...

The snps object does seem to be empty:

df = snps.snps
df.columns.values
# array(['chrom', 'pos', 'genotype'], dtype=object)
len(df)
# 0

However, I also tried reading the input VCF line by line with SNPs using the following:

with open(input_file, 'r') as f:
  snps = SNPs(f.read().encode('utf-8'))

And then when I outputted this object it seemed to have the data:

But I'm not sure if there are any issues with reading the format of the VCF:

Any ideas what might be going wrong. Any help would be much appreciated.

Many thanks in advance,
Phil

Duplicate SNPs can be reported for a particular locus in a file (e.g., two SNPs with the same position on the same chromosome, potentially with different genotypes). In one of the example files, there are 234 SNPs where another SNP has the same locus:

import pandas as pd
from snps import SNPs
s = SNPs("resources/662.23andme.340.txt.gz")
df = pd.DataFrame()
for chrom in s.snps.chrom.unique():
    temp = s.snps.loc[s.snps.chrom == chrom]
    df = df.append(temp.loc[temp.pos.duplicated(keep=False)])
print(len(df))

Several of these SNPs have "internal IDs", but there are cases where two different rsids are reported for the same locus. Additionally, the genotype reported is not always the same for a particular locus.

To remedy this, I suggest adding a deduplicate_loci parameter to the SNPs object and renaming the deduplicate parameter to deduplicate_rsids.

Then, if deduplicate_loci=True (default), deduplicate the loci in this manner:

Chromosome by chromosome, find SNPs with duplicate positions. Then, for each duplicate position, sort the SNP rsids / internal IDs and keep the last in the normalized SNPs dataframe; this will ensure internal IDs are replaced with rsids (if available) and also keep IDs with higher numbers (potentially newer). In this process, add the kept rsid as a key to a _duplicate_loci dict, where the value is a list of any deduplicated loci (str rsids). Additionally, merge the genotype (updating any nans) and identifying discrepant genotypes as discrepant_loci (similar to discrepant_XY) and mark the genotype as nan.

Finally, loci should optionally be deduplicated after each merge (add deduplicate_loci parameter to merge as well).

Merging snps from different samples and saving them as a multi-sample VCF.
Hi, would that be possible?

Thanks!

Rename several SNPs methods and properties to be less redundant, e.g., remove _snps suffix from remap_snps, discrepant_XY_snps, etc. For now, retain an alias to these methods and properties and raise DeprecationWarning.

To simplify merging SNPs, add a merge method to the SNPs class, and in the process obsolete the SNPsCollection class. The merge method should accept an iterable of SNPs objects. In this manner, SNPs objects can be merged after the fact, vs. requiring a SNPsCollection from the beginning to merge SNPs.

Similar functionality to SNPsCollection should be provided, such as sorting SNPs after the merge and identifying discrepant genotypes and positions.

Properties of the SNPs objects will need to be merged and behavior defined (e.g., source, build, discrepant_XY_snps, etc.).

dask would help with reading large VCFs and could potentially increase the speed of calculations throughout (e.g., remapping and writing VCFs).

With the X chromosome, if only one allele is reported for a SNP, the allele is doubled such that the X chromosome genotype will always be a string of length 2. Some data sources report two X chromosome alleles for all X chromosome SNPs, even if the individual is a male. So, by doubling single alleles, this makes all X chromosome data consistent.

However, instead of doubling alleles, it'd be better to analyze the X chromosome SNPs to determine if only one allele should be reported (i.e., if an individual is male). Any heterozygous SNPs not in the pseudoautosomal region could then be flagged as a discrepant SNP.

The thresholds and methods for determining the sex of a genotype should be reinvestigated. See comments regarding misidentification of Female genotypes as Male at #48.

To fix this, I propose investigating the optimal thresholds for these parameters using data from openSNP and updating determine_sex as necessary.

Moreover, the script documenting the analysis and its results should be added to the repo as an example.

Update the dtype of rsid, chrom, and genotype columns to be pandas.StringDtype as recommended here.

Also require pandas>1.0.0.

The readme says that there the package is able to read VCF, but there is no example how to read them (only write). Trying to read VCFv4.1

fn_struct = 'my.vcf'
sc = SNPs(fn_struct)
sc.build
# 0

returns 0 records, though pyVCF works OK:

vcf_reader = vcf.Reader(filename =fn_struct)
sum((1 for record in vcf_reader))
# 184353

I think the error could be handled when trying to save invalid genotype files to VCF.

To reproduce

For example, if I run:

# create invalid empty file
touch file.txt
python

>>> from snps import SNPs
>>> s = SNPs('file.txt', output_dir='.')
# error message is handled nicely when vcf is false
>>> saved_snps = s.save_snps('test.txt', vcf=False)
no data to save...
# error message is not handled when vcf is true
>>> saved_snps = s.save_snps('test.vcf', vcf=True)

Then I get the following error message:

Traceback (most recent call last):
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'genotype'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/__init__.py", line 394, in save_snps
    snps=self, filename=filename, vcf=vcf, atomic=atomic, **kwargs
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 951, in write_file
    return w()
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 924, in __call__
    return self._write_vcf()
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/snps/io.py", line 1060, in _write_vcf
    df["genotype"].notnull()
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'genotype'

Any ideas how this could be improved? Perhaps catching the error here?

Thanks in advance

Per the Ensembl README (ftp://ftp.ensembl.org/pub/grch37/current/fasta/homo_sapiens/dna/README):

Human has sequenced Y chromosomes and the pseudoautosomal region (PAR)
on the Y is annotated. By definition the PAR region is identical on the
X and Y chromosome. The Y chromosome file contains the Y chromsome
minus these repeated PAR regions i.e. the unique portion of Y.

If writing a VCF that contains Y chromosome PAR SNPs, the full Y chromosome reference sequence needs to be reconstructed using the PAR regions from X in order to determine the reference alleles.

Add ability to load mtDNA FASTA data (e.g., from FTDNA) and parse out mtDNA SNPs (see #9).

I'm using aws ec2 ubuntu. It does not allow me to create an individual.

user662 = l.create_individual('User662', '/home/ubuntu/myprojectdir/AaronAzuma.zip')
Traceback (most recent call last):
File "", line 1, in
File "/home/ubuntu/myprojectdir/venv/lib/python3.8/site-packages/lineage/init.py", line 96, in create_individual
return Individual(name, raw_data, self._output_dir, **kwargs)
AttributeError: 'str' object has no attribute '_output_dir'

• Recognize assembly of newer 23andMe files
  • Generically, consider using dbSNP / RefSNP API
• Try to identify minor revision of assembly / build
  • Use SNPs that vary in location between minor revisions
    • What SNPs?
  • Do any APIs provide this information?

Validate by attempting to recognize assemblies of openSNP files.

I have been using the detect_build functionality and I have noticed that snps does not correct detect our build 38 Exome VCFs. This is likely because Exome data is not technically within scope of the genotype data that snps originally was intended to support.

It seems that the rsids that are listed here are not present in my Exome file. I am looking to find some more examples of rsids that are included in common Exome files so that this functionality is extended in the same way that you have originally implemented it.

https://github.com/apriha/snps/blob/master/src/snps/__init__.py#L398

Let me know your thoughts!

Will

It seems that openSNP has changed their downloads, a single zip file is no longer available. This breaks some of the analysis code, which is where I found this problem as I want to investigate and compare some alternative sex detection mechanisms. I'll take a look and see if I can work out a solution in a pull request.

Merge the Y chromosome SNP data for two closely related males, identifying discrepant SNPs in the process.

Add a phased property to SNPs to indicate if the genotype is phased.

I'm getting errors across the whole test suite when running them locally on my Mac OSX Catalina 10.15.4 machine.

    os.makedirs(path, exist_ok=True)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:213: in makedirs
    makedirs(head, exist_ok=exist_ok)
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py:223: in makedirs
    mkdir(name, mode)
E   FileNotFoundError: [Errno 2] No such file or directory: '../../../../var/folders/h3/k5sgzmp17cvbz1mm459fkf200000gn/T/tmp9ixj7r4i/fasta'

Looking around, I think this is something to do with the permissions on the /var/folders/ directories

Is there a way we can specify what temporary to use here?

Currently, for file types that use a generic parser, their source is identified as "generic". Suggest changing the source to "unknown".

Currently, there is a single general Reader class with various methods for different sources.

I wonder if a series of specific classes for different filetypes be better e.g. Reader23andMe, ReaderAncestry, etc. Still keeping the single Reader class overall, but delegating a lot of the details to the specific classes for implementation.

By grouping everything for a particular format together it makes it easier to read, and to add extra sources. Also it would allow subclasses (e.g. for Global Screening Array based sources) to abstract functionality more easily.

Hello

I'm trying to convert a .csv from FTDNA to .vcf

I get load my .csv (the assembly and build are OK) and I set the ouptut dir throught:

snps.SNPs(output_dir='C:Users\\Desktop\\').

The execution return:

SNPs('')

when I save the .csv throught:

s = SNPs('C:\\Users\\PC\\Desktop\\genome.csv')
s.save_snps(vcf=True)

I get
'output\\FTDNA_GRCh37.vcf'

But not generates any file

What I'm doing wrong?

Thanks in advance

Replace print statements with calls to a logger.

Along the same lines as #108 has it been considered to use the Pandas nullable integer datatype (pd.Int64Dtype()) for pos? More details here. We've seen a number of files that fail to parse because the position information for a small number of rows is missing (for example, on a RSID with multiple possible locations).

Hi,

Thanks for a great tool.

Is there support (or plans to support) the following:

1. Annotation of missing rsids in a VCF file? (Similiar to how liftover is currently performed. Just seen #19 which seems the same)
2. Compressed VCF input files? (Eg with bgzip or gzip)
3. PLINK binary input files?

Thanks in advance,
Phil

Currently, snps normalizes data into a dataframe with four columns named rsid, chrom, pos, and genotype.

genotype can either be np.nan or a string of length 1 or 2. For autosomal SNPs and the X chromosome, the genotype is always a length 2 string. See #43.

The Y chromosome and mtDNA alleles are often strings of length 1; however, SNPs in the pseudoautosomal region on the X and Y chromosomes often have two alleles reported.

So, to better handle various numbers of alleles reported for the X, Y, and mtDNA chromosomes, consider refactoring the genotype column into allele1 and allele2.

Note that this would also more naturally support phased genotypes (vs. indexing a length 2 string genotype), wherein allele1 could be alleles on one chromosome, and allele2 could be alleles on the other. See #44.

With pandas 1.1.0, pd.testing.assert_frame_equal(..., check_exact=True) unexpectedly raises an AssertionError, which causes several tests to fail. See pandas-dev/pandas#35446

Therefore, update snps to not use pandas 1.1.0.

If you have a (23AndMe-like) file produced by snps, eg:

# Generated by snps v0.1.1.post243+ge018ea4, https://pypi.org/project/snps/
# Generated at 2020-01-20 22:55:53 UTC
# Source(s): 23andMe
# Assembly: GRCh37
# Phased: False
# SNPs: 638463
# Chromosomes: 1-22, X, Y, MT
rsid	chromosome	position	genotype
rs548049170	1	69869	TT
rs13328684	1	74792	--
rs9283150	1	565508	AA
i713426	1	726912	--
rs116587930	1	727841	GG
rs3131972	1	752721	AG
rs12184325	1	754105	CC
rs12567639	1	756268	AA

And then you try reading this file into snps you get the following error Too many columns specified: expected 4 and found 1:

  Traceback (most recent call last):
    File \"/opt/conda/envs/common-latest-geno/lib/python3.7/site-packages/pandas/core/indexes/base.py\", line 2897, in get_loc
      return self._engine.get_loc(key)
    File \"pandas/_libs/index.pyx\", line 107, in pandas._libs.index.IndexEngine.get_loc
    File \"pandas/_libs/index.pyx\", line 131, in pandas._libs.index.IndexEngine.get_loc
    File \"pandas/_libs/hashtable_class_helper.pxi\", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
    File \"pandas/_libs/hashtable_class_helper.pxi\", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  KeyError: 'genotype'

This was discovered by @willgdjones who noted that the issue occurs because:

It has ‘snps’ in the header and so the snps package thinks that it is from lineage dna which uses comma-separated values as opposed to tab-separated values

How can this issue be fixed? As long as it's not too difficult or time-consuming I'd be happy to work on a PR to fix this

Access the GWAS Catalog via the REST API to retrieve extensive data related to SNPs: https://www.ebi.ac.uk/gwas/docs/programmatic-access . Various reports could be generated, but this type of data would probably be best interacted with via a web front-end (see #11).

Add a chrom parameter to SNPs.merge to merge only the SNPs for the specified chromosome, identifying discrepant SNPs in the process.

Add ability to compute additional summary statistics:

• Percentage NaN (autosomal, X, Y, MT)
• Percentage homozygous and heterozygous (autosomal, X)
• GC content (autosomal, X, Y, MT)

Related to #29

I'd like to suggest adding an option to save files in a tabular rather than csv format for easier use in other tools and databases, such as GEDmatch. I've found that a simple replacement of all commas with tabs allows my test files to be used in GEDmatch, so it may be as simple as that.

Apologies in advance if this is already an option and that I just didn't look carefully enough at the docs!

-Gerard.

Currently tests save output files to a directory named output at the repository root. It would be better to save output files to a temporary directory created by tempfile.TemporaryDirectory.

Utilize NCBI's Variation Services API to populate missing RSIDs.

An idea for how this could be performed... For all SNPs in a dataset, lookup RSIDs in batches of 50k SNPs (1 request / second) using the VCF endpoint. Then, use the resulting information to populate any missing RSIDs. However, note that this method would require the REF allele and a valid ALT allele.

As an example, one row for this query (using GRCh38) could be constructed as follows:

1 817186 . G A,C,G,T

Query:

curl -X POST "https://api.ncbi.nlm.nih.gov/variation/v0/vcf/file/set_rsids?assembly=GCF_000001405.38" -H "accept: text/plain; charset=utf-8" -H "Content-Type: text/plain; charset=utf-8" -d "1\t817186\t.\tG\tA,C,G,T"

Which would return:

1 817186 rs3094315 G A,C,G,T

This could also be used to verify SNP positions and help with #6.

Thanks to @gedankenstuecke for helping develop the idea!

Some newer Family Tree DNA famfinder files seem to have SNPs assigned to chromosome 0. Similar to SNPs._assign_par_snps, use the RefSNP API to assign each of these SNPs to a chromosome and update the SNP's position to be consistent with the assembly.

Ancestry and 23andme appear to have discrepant mapping of indel SNPs, causing them to be discarded by SNPs, and sometimes causing the program to fail.

Indel SNPs positions where the allele is named D (deletion) or I (insertion) and can be deletions or insertions of several bases, or more complex multi-nucleotide changes (so technically, not all are actually SNPs). In my test case, Ancestry covers 8073 indel SNPs, and 23andme (v5) has 4828, of which 2143 are found on both platforms (by rs#). However, only 1328 of the 2143 report the same chromosomal location, so the remaining 815 to be rejected as discrepant.

Most of the location discrepancies are tiny - 475 of them are 1 base, 174 are 2, 50 are 3, 48 are 4, 33 are 5, 16 are 6, and a further 18 are 17 or less. In almost all cases, the difference is the length of the indel, and the ancestry location is less than the 23andme location. This seems to be due to ancestry reading the position where the deletion started, and 23andme reading the position where it ended, but there are some additional odd cases, and neither ancestry nor 23andme is always consistent with dbSNP.

In practical terms, I'd like to suggest two possible solutions

1. Loosen the criterion for merged SNPs to have identical location, and if an ancestry/23andme merge, retain the 23andme location. Having the same rs# and being within 20 bases of each other would seem a reasonably strict criterion

2. Could also just ignore all discrepant indels. The vast majority of them are homozygous in most people (i.e. are rare and usually disease-associated alleles), so they are not very informative for genealogy.

I also wanted to note that I was unable to get my two genotype files to merge by altering the parameter discrepant_genotypes_threshold to 1000000, but only by manually deleting all indel SNPs from the two input files. I haven't looked further into this issue, but may be worth checking out.

Update to use f-strings for creating dynamic strings (vs. .format method).

With pandas 1.0.0, using na_rep with to_csv causes position values in output files to be truncated (e.g., files created via save_df_as_csv). See pandas-dev/pandas#31447

In the meantime, a workaround would be to pin the version of pandas to <=0.25.3.

Building on #107, consolidate several issues (e.g., duplicate_rsid, discrepant_XY) into one dataframe with the following columns / dtypes:

Multiple issue columns could take on the value of True, and getting SNPs with issues (e.g., discrepant_XY) could be handled by filtering the issues dataframe.

rsids could appear more than once in this dataframe. However, if an rsid has two or more rows that are equivalent (same values for chrom, pos, and genotype), their issues should be consolidated into one row, with the issue columns flagging the issue(s).

Create a new Resolver class to resolve SNP issues. Refactor SNPs._assign_par_snps to this class. Also, this class could be used to implement solutions to #13 and #19 .

The current implementation can take a long time to read VCFs. Consider using CyVCF, or pandas.read_csv to read VCFs. The pandas implementation would likely be similar to the existing snps.io._write_vcf.

The latest version of pandas officially supports Python 3.6.1 and above. Similarly, I recommend the following changes to follow suit:

• Make Python 3.6.1 the minimum version of Python supported
• Restrict pandas to be != 1.0.0 (avoids issue in #59)

Currently, the Reader and Writer classes within io read and write files from the filesystem. E.g.

I have a use case where it would be useful to be able to read and write from buffered IO.

I'm happy to implement this in the coming days and submit a PR, but I wanted to ask what your opinions on this would be.

Thanks!

Will

Ancestry reports mitochondrial (MT/Chr26) SNPs as diploid (e.g. "GG", with two identical alleles), even though there is just one mitochondrial genome, with multiple copies. 23andme reports as haploid (e.g. "G"). When 23andme and Ancestry files are merged in lineage, the csv output has a single allele for all 23andme SNPs but two alleles for any that are present on Ancestry (including SNPs present on both).

Suggest that, for consistency, all MT SNPs be reported with a single allele, and any heterozygosity in Ancestry calls (due to heteroplasmy, which I've never yet seen) be reported as an error.

Hi. Am trying out the codes. Understand that the raw files here mean genotype files from DTC companies. However, what I have is only vcf and gvcf files from in-house sequencing platform. May I know can these be used? How? I have zero knowledge in coding. Thanks a lot!

Hi @apriha,

I noticed throughout the codebase that Exceptions are caught and not raised to the user. This creates some challenges in tracking down the cause of bugs when they crop up.

I would propose either:

1. Logging the Exception message and the offending like that caused the Exception.
2. Allowing the Exception to be raised without catching it and allowing execution to continue.

It would be great to get your thoughts!

Thanks,
Will

Merge the mtDNA SNP data for two closely related individuals that share the same mtDNA, identifying discrepant SNPs in the process.

The mapmygenome file has additional column GC.score, that gives the confidence level of the SNP.
It will be good to filer out SNP rows that has lower confidence (say < 0.75).

Requesting for adding this feature of filtering rows based on GC.score into your module.

The file format of mapymygenome has changed.
After reading the file, the data frame reports zero rows.

Request to update the reader.

The new header is :
SNP.Name Sample.ID Allele1...Top Allele2...Top GC.Score Sample.Name Sample.Group Sample.Index SNP.Index SNP.Aux Allele1...Forward Allele2...Forward Allele1...Design Allele2...Design Allele1...AB Allele2...AB Allele1...Plus Allele2...Plus Chr Position GT.Score Cluster.Sep SNP ILMN.Strand Customer.Strand Top.Genomic.Sequence Plus.Minus.Strand Theta R X Y X.Raw Y.Raw B.Allele.Freq Log.R.Ratio CNV.Value CNV.Confidence