fmalmeida / gff-toolbox

Gff-toolbox is a toolbox of commands that enables one to get the gist of their GFF annotation files, as well as to analyse them in different ways.

Hello 😁 👋

Hello there, my name is Felipe Almeida, a brazilian scientist, bioinformatician, pipeline developer and problem solver. My main interests are: Bioinformatics, genomic surveillance, precision medicine, and microbial genomics. You can also find me on twitter @fmarquesalmeida, stackoverflow and linkedin.

Academic info

I'm a PhD student at the University of Brasilia, at the CompGen (Computational Genomics) laboratory with academic guidance from PhD. Prof. Georgios J. Pappas Jr.

Some of my favourite tools:

My stats

gff-toolbox's Issues

add a new plot (ideogram)

Generate a ideogram plot (with karyoploteR) using a feature type or attribute key selected by the user.

Obs: It would require a .fai (samtools faidx) or genome file as input to get the chr sizes.

add a "liftover-like" module

Add a module to receive a file containing two columns in order to fix the GFF chr names.

It gets in column 1 the names of chr we want to update and in column 2 the pattern for substitution, for "lift-over"

gff-toolbox plot enchancement

Would it be possible that, if requested, not plot the gene labels above in the features?

For instance, when plotting multiple GFFs at once to create groups with different colors, we may like to show the labels of only one group.

add a venn-like diagram to compare GFFs

Add a venn-like diagram to compare two GFFs to compare two GFF files:

1. Finds intersection between features (selected feature type)
2. Plots intersections in venn-like diagram

Problem with "gff3_ID_generator.py"

Hello,
So I have a gff3 file where there are lines missing ID (column 9). Those lines mainly concerns CDS' and stop codons. I tried to use "gff3_ID_generator.py" with the command :
"python3 gff3_ID_generator.py -g ../../../UK0001.gff3 -og UK0001Mo.gff3" .
With UK0001.gff3 being my gff file with missing IDs and UK0001Mo.gff3 being my desired output.

I get this error as the output:

INFO Reading input gff3 file: (../../../UK0001.gff3)
INFO Generate new ID for features in (../../../UK0001.gff3)
Traceback (most recent call last):
File "/home/mestiri/GFF3toolkit/gff3tool/lib/gff3_ID_generator.py", line 333, in
main(in_gff=args.gff, merge_report=args.merge_report, out_merge_report=args.out_merge_report, out_gff=args.output_gff, uuid_on=args.universally_unique_identifier, prefix=args.idprefix, digitlen=args.digitlen, report=args.report, alias=args.alias)
File "/home/mestiri/GFF3toolkit/gff3tool/lib/gff3_ID_generator.py", line 260, in main
if descend['attributes']['ID'] not in ID_dict:
KeyError: 'ID'

I don't understand what it means, to be honest. Is there something that I misunderstood concerning this python script ?

Thanks for your help ! Have a nice day !

Add new attributes (annotations) to a mongodb collection

Motivation

The gff-toolbox convert module is capable of converting a GFF to a mongo database, however it seems that we can't manipulate the gff information stored in database without relying on pure mongo commands. A useful routine task done in many analysis is the insertion of new information into a GFF, i.e annotate a gene/transcript. This task could be done using many tools, such as gffutils, BCBio or even bash/other language script, through the inclusion of new attributes into the raw GFF column 9 using as input a file telling which set of annotations (e.g. GO, PFAM, EC number) correlate with each gene. However, the same annotation task can also be done in a different way involving the conversion of GFF to mongodb, inclusion of annotations to mongodb corresponding collection and, further, if desired, conversion back to GFF. I've been wondering that despite it may seem a more difficult procedure than annotate a raw GFF and spend more computer resources, it has some advantages:

• Possibility to include a description, link or other metadata related to an annotation. GFF format spec declares fields Ontology_terms and Dbxref in column 9 to accommodate, respectively, annotations from GO/ontology servers and other databases (e.g. PFAM, PANTHER, EC). Despite this, I lack a description field for each term annotated in a GFF. A thing that can be easily done in mongodb. Indeed, it can be included in description field of GFF, but it brings to my next GFF issue: noisy/polluted GFF;
• Clean visualization/reading of gene attributes in gffs/genomes having enormous quantity of annotations; and
• Go biond GFFs. In some situations, we would like to gather the information contatined in GFF into a different format. For instance, higlass visualization tool requires refseq format to display genes. Therefore, generate a file with refseq specs from a stored mongodb is easier than manipulate a raw gff.

Proposed solution

Probably the list of advantages and disadvantages of using a mongodb as intermediate to accomodate annotations is bigger than I could think of, but I see this way as a facilitator. Hence, I propose a new gff-toolbox module to perform this task, i.e. annotate a mongodb collection created by gfftool-box convert. In the following i will try to explain the main architecture of this module, that at first hand I nominated as ingest.

We would like the ingest module to receive a set of annotations and include them in corresponding gene/transcript entry in mongodb. Thus, assume that the mongodb was created by gff-toolbox convert module - parameters XXX; XXX; - and that we also have a txt/tsv file tab-separated with annotations such as the following:

##ID	Id	IdType	description
gene-KPHS_00170	PTHR30520:SF0	PANTHER	TRANSPORTER-RELATED
gene-KPHS_00170	GO:0006810	GO	transport
gene-KPHS_00170	3.4.16.2	EC	Lysosomal Pro-Xaa carboxypeptidase
gene-KPHS_00170	GO:0005215	GO	transporter activity
gene-KPHS_02590	GO:0003735	GO	structural constituent of ribosome
gene-KPHS_02590	PTHR36029	PANTHER	

Inspecting the mongodb collection that correponds to gene-KPHS_00170 we can retrieve the json listing it's information:

{'_id': ObjectId('612e788a94ee11baab643fb0'),
  'recid': 'NC_016845.1',
  'source': 'RefSeq',
  'type': 'gene',
  'start': '22533',
  'end': '22802',
  'score': '.',
  'strand': '+',
  'phase': '.',
  'attributes': {'ID': 'gene-KPHS_00170',
   'Dbxref': 'GeneID:11844995',
   'Name': 'KPHS_00170',
   'gbkey': 'Gene',
   'gene_biotype': 'protein_coding',
   'locus_tag': 'KPHS_00170'}

The aim of the proposed gff-toolbox ingest module is to insert the annotations into corresponding gene in mongodb. After this procedure, we would like to have mongodb entry for gene-KPHS_00170 stored as:

{'_id': ObjectId('612e788a94ee11baab643fb0'),
  'recid': 'NC_016845.1',
  'source': 'RefSeq',
  'type': 'gene',
  'start': '22533',
  'end': '22802',
  'score': '.',
  'strand': '+',
  'phase': '.',
  'attributes': {'ID': 'gene-KPHS_00170',
   'Dbxref': 'GeneID:11844995',

   'Dbxref': [  'GeneID:11844995' ,
                    {'DBTAG': 'PANTHER', 'ID': 'PTHR30520:SF0', 'Description': 'FORMATE TRANSPORTER-RELATED'},
                    {'DBTAG': 'PANTHER', 'ID': 'PTHR30520', 'Description': 'FORMATE TRANSPORTER-RELATED'},
                    {'DBTAG': 'PFAM', 'ID': 'PF01226', 'Description': 'Formate/nitrite transporter'}
                    ],
   'Ontology_term': [ {'DBTAG': 'GO', 'ID': 'GO:0006810', 'Description': 'transport'}, 
                    {'DBTAG': 'GO', 'ID': 'GO:0016020', 'Description': 'membrane'},
                    {'DBTAG': 'GO', 'ID': 'GO:0005215', 'Description': 'transporter activity'}
                    ],

   'Name': 'KPHS_00170',
   'gbkey': 'Gene',
   'gene_biotype': 'protein_coding',
   'locus_tag': 'KPHS_00170'}

According to GFF spec, "two reserved attributes, Ontology_term and Dbxref, can be used to establish links between a GFF3 feature and a data record contained in another database" (i.e. annotations). Also, "the value of both Ontology_term and Dbxref is the ID of the cross referenced object in the form "DBTAG:ID". The DBTAG indicates which database the referenced object can be found in, and ID indicates the identifier of the object within that database". Therefore, in mongodb schema we include an object for each annotation declaring DBTAG, ID and optional fields such as Description. Unfortunately, this is not the json schema declared in gff-toolbox convert module: the Dbxref entries generated after parsing a GFF to mongodb do not separate the DBTAG and ID fields. We can fix this, by simply adjusting the code to separate those fields before parsing the json into mongodb collection. I propose to fix this, but I have to know if this can bring any problem in other gff-toolbox modules. @fmalmeida, can it?

Another suggestion, I would like your opinion, if we should decouple the "ingestion" of annotations to mongodb - that is the proposed solution in this issue - and the "digestion" of a mongodb collection into a GFF/other file format. I think, another gff-toolbox module or even the gff-toolbox convert could be an answer to this question.

@fmalmeida, let me know what you think about it and if I can submit the pull request - I kind of have a code that can be adjusted to become the aforementioned gff-toolbox ingest module.

convertion to JSON

GFF convertion to JSON is not working properly ... the output is empty