API key of NCBI about cenote-taker2

[mtisza1]

Hi,

Thanks for checking out the tool and opening this issue. I also saw the other issue earlier this week. Sorry for the slow reply, I was out of the office earlier this week.

I might imagine that the issue #17 and this issue are related. I have experienced this "PLEASE REQUEST AN API_KEY FROM NCBI" error when I've submitted too many queries in a very short time to NCBI via Eutils efetch. Are you running a lot of Cenote-Taker 2 jobs in parallel?

If so, I can add an argument to my tool to add your API key.

Let me know,

Mike

[neptuneyt]

Thanks for your reply in time. It was indeed caused by the parallel and I rerun the failed jobs with single model, and it was work finally . So, I suppose that an argument to add user API key is a best way to resolve the problem.
Looking forward your updating.

[neptuneyt]

Dear CT2 team, I had a problem with "PLEASE REQUEST AN API_KEY FROM NCBI" when running CT2, it seems that casused by eutils. does CT2 parse the taxonomic rank information by this way? How could I fix the problem? Looking forward your reply. Thank a lot.

(cenote-taker2) [yut@amms-sugon8401@006]$ cat 350.log
00000000000000000000000000
00000000000000000000000000
000000000^^^^^^^^000000000
000000^^^^^^^^^^^^^^000000
00000^^^^^CENOTE^^^^^00000
00000^^^^^TAKER!^^^^^00000
00000^^^^^^^^^^^^^^^^00000
000000^^^^^^^^^^^^^^000000
000000000^^^^^^^^000000000
00000000000000000000000000
00000000000000000000000000

Version 2.1.3

@@@@@@@@@@@@@@@@@@@@@@@@@
Your specified arguments:
original contigs:                  /mnt/data/share/database/Human_gut_virome_database/Public_Merged_Human_Virome_Database/SPLITS_8_PART/006/PMHVD_DREP_rep_seq.part_006.part_350.fasta
forward reads:                     /mnt/data/share/database/Human_gut_virome_database/Public_Merged_Human_Virome_Database/PMHVD_DREP_REP_SEQ_CT2_OUT/006/no_reads
reverse reads:                     /mnt/data/share/database/Human_gut_virome_database/Public_Merged_Human_Virome_Database/PMHVD_DREP_REP_SEQ_CT2_OUT/006/no_reads
title of this run:                 350_out
Isolate source:                    unknown
collection date:                   unknown
metagenome_type:                   unknown
SRA run number:                    unknown
SRA experiment number:             unknown
SRA sample number:                 unknown
Bioproject number:                 unknown
template file:                     /mnt/data/share/software/Cenote-Taker2/dummy_template.sbt
minimum circular contig length:    1000
minimum linear contig length:      1
virus domain database:             standard
min. viral hallmarks for linear:   0
min. viral hallmarks for circular: 0
handle known seqs:                 do_not_check_knowns
contig assembler:                  unknown_assembler
DNA or RNA:                        DNA
HHsuite tool:                      hhblits
original or TPA:                   original
Do BLASTP?:                        no_blastp
Do Prophage Pruning?:              True
Filter out plasmids?:              True
Run BLASTN against nt?             none
Location of Cenote scripts:        /mnt/data/share/software/Cenote-Taker2
Location of scratch directory:     none
GB of memory:                      50
number of CPUs available for run:  10
Annotation mode?                   True
@@@@@@@@@@@@@@@@@@@@@@@@@
scratch space will not be used in this run
HHsuite database locations:
/mnt/data/share/software/Cenote-Taker2/NCBI_CD/NCBI_CD
/mnt/data/share/software/Cenote-Taker2/pfam_32_db/pfam
/mnt/data/share/software/Cenote-Taker2/pdb70/pdb70
no CRISPR file given
Prophage pruning requires --lin_minimum_hallmark_genes >= 1. changing to:
--lin_minimum_hallmark_genes 1
time update: locating inputs:  11-18-21---21:44:16
/mnt/data/share/database/Human_gut_virome_database/Public_Merged_Human_Virome_Database/PMHVD_DREP_REP_SEQ_CT2_OUT/006/PMHVD_DREP_rep_seq.part_006.part_350.fasta
File with .fasta extension detected, attempting to keep contigs over 1 nt and find circular sequences with apc.pl
No circular contigs detected.
no reads provided or reads not found
No circular fasta files detected.
time update: running IRF for ITRs in non-circular contigs 11-18-21---21:44:18
time update: running prodigal on linear contigs  11-18-21---21:44:18
time update: running linear contigs with hmmscan against virus hallmark gene database: standard  11-18-21---21:46:01
 Starting pruning of non-DTR/circular contigs with viral domains
pruning script opened
fna files found
mv: cannot stat './350_out7.AA.sorted.fasta': No such file or directory
cut: ./350_out7.AA.hmmscan.sort.out: No such file or directory
mv: cannot stat './350_out8.AA.sorted.fasta': No such file or directory
cut: ./350_out8.AA.hmmscan.sort.out: No such file or directory
mv: cannot stat './350_out18.AA.sorted.fasta': No such file or directory
cut: ./350_out18.AA.hmmscan.sort.out: No such file or directory
mv: cannot stat './350_out20.AA.sorted.fasta': No such file or directory
cut: ./350_out20.AA.hmmscan.sort.out: No such file or directory
mv: cannot stat './350_out23.AA.sorted.fasta': No such file or directory
cut: ./350_out23.AA.hmmscan.sort.out: No such file or directory
time update: HMMSCAN of common viral domains beginning 11-18-21---21:46:56
time update: making tables for hmmscan and rpsblast outputs  11-18-21---21:49:57
time update: running RPSBLAST on each sequence  11-18-21---21:50:05
/mnt/data/share/database/Human_gut_virome_database/Public_Merged_Human_Virome_Database/PMHVD_DREP_REP_SEQ_CT2_OUT/006/350_out/no_end_contigs_with_viral_domain/COMBINED_RESULTS_PRUNE.AA.rpsblast.out
time update: parsing tables into virus_signal.seq files for hmmscan and rpsblast outputs  11-18-21---21:50:27
time update: Identifying virus chunks, chromosomal junctions, and pruning contigs as necessary  11-18-21---21:50:50
Running file: 350_out10.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out11.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out12.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out13.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out14.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out15.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out16.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out17.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out19.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out1.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out21.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out22.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out24.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out25.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out26.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out2.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out3.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out4.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out5.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out6.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
Running file: 350_out9.virus_signal.seq
   Window +/- to the right  ...  Chunk_end  Window midpoint
0       1                +  ...       none             2500
0       1                +  ...       none             2500

[2 rows x 6 columns]
time update: Making prophage table  11-18-21---21:53:10
 FINISHED PRUNING CONTIGS WITH AT LEAST 1 VIRAL DOMAIN(S)
 Grabbing ORFs wihout RPS-BLAST hits and separating them into individual files for HHsearch
time update: running HHsearch or HHblits  11-18-21---21:53:11
 Combining tbl files from all search results AND fix overlapping ORF module
No ITR contigs with minimum hallmark genes found.
Annotating linear contigs
time update: running BLASTX, annotate linear contigs  11-18-21---21:53:11
429 Too Many Requests
PLEASE REQUEST AN API_KEY FROM NCBI
No do_post output returned from 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=taxonomy&id=35344&rettype=full&retmode=xml&edirect_os=linux&edirect=13.3&tool=edirect&email=yut@amms-sugon8401'
Result of do_post http request is
$VAR1 = bless( {
                 '_headers' => bless( {
                                        'referrer-policy' => 'origin-when-cross-origin',
                                        'client-response-num' => 1,
                                        'content-length' => '87',
                                        'client-peer' => '130.14.29.110:443',
                                        'x-xss-protection' => '1; mode=block',
                                        'access-control-expose-headers' => 'X-RateLimit-Limit,X-RateLimit-Remaining,Retry-After',
                                        'x-ua-compatible' => 'IE=Edge',
                                        'x-ratelimit-remaining' => '0',
                                        'content-type' => 'application/json',
                                        'x-test-test' => 'test42',
                                        'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                        'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'vary' => 'Accept-Encoding',
                                        'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1',
                                        'date' => 'Thu, 18 Nov 2021 13:54:47 GMT',
                                        'client-date' => 'Thu, 18 Nov 2021 13:54:48 GMT',
                                        'server' => 'Finatra',
                                        'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                        'x-ratelimit-limit' => '3',
                                        '::std_case' => {
                                                          'x-test-test' => 'X-Test-Test',
                                                          'x-ratelimit-remaining' => 'X-RateLimit-Remaining',
                                                          'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                          'x-ua-compatible' => 'X-UA-Compatible',
                                                          'access-control-expose-headers' => 'Access-Control-Expose-Headers',
                                                          'client-peer' => 'Client-Peer',
                                                          'referrer-policy' => 'Referrer-Policy',
                                                          'client-response-num' => 'Client-Response-Num',
                                                          'x-xss-protection' => 'X-XSS-Protection',
                                                          'content-security-policy' => 'Content-Security-Policy',
                                                          'x-ratelimit-limit' => 'X-RateLimit-Limit',
                                                          'strict-transport-security' => 'Strict-Transport-Security',
                                                          'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                          'client-date' => 'Client-Date',
                                                          'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                          'client-ssl-cipher' => 'Client-SSL-Cipher'
                                                        },
                                        'client-ssl-socket-class' => 'IO::Socket::SSL',
                                        'retry-after' => '2',
                                        'content-security-policy' => 'upgrade-insecure-requests',
                                        'connection' => 'close'
                                      }, 'HTTP::Headers' ),
                 '_request' => bless( {
                                        '_method' => 'POST',
                                        '_headers' => bless( {
                                                               'content-type' => 'application/x-www-form-urlencoded',
                                                               'user-agent' => 'libwww-perl/6.39',
                                                               '::std_case' => {
                                                                                 'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                                               }
                                                             }, 'HTTP::Headers' ),
                                        '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                        '_content' => 'db=taxonomy&id=35344&rettype=full&retmode=xml&edirect_os=linux&edirect=13.3&tool=edirect&email=yut@amms-sugon8401',
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'}
                                      }, 'HTTP::Request' ),
                 '_msg' => 'Too Many Requests',
                 '_protocol' => 'HTTP/1.1',
                 '_content' => '{"error":"API rate limit exceeded","api-key":"218.241.250.70","count":"4","limit":"3"}
',
                 '_rc' => 429
               }, 'HTTP::Response' );

time update: running PHANOTATE, annotate linear contigs  11-18-21---21:56:39

[mtisza1]

Hi,

A simple fix for your problem can be achieved without making changes to Cenote-Taker 2. Please let me know if this is suitable for your needs, or you'd like me to push a quick patch for your use. (I will include the API key option in the next update regardless).

First, you need to get an API key from NCBI. I found the instructions to be pretty easy: here

Then you can just export the API key variable before your Cenote-Taker 2 run, using the key they give you, e.g.:

export NCBI_API_KEY="my_api_key_12345" ; python /path/to/Cenote-Taker2/run_cenote-taker2.py -c MY_CONTIGS.fasta -r my_contigs1_ct -m 32 -t 32 -p true -db virion

Now, this only gives you 10 requests per second instead of the 3 requests per second. So, if you are running a lot of jobs in parallel, it could still fail. If you have many people in your lab with NCBI accounts, you could perhaps get API keys for each of them. Further, NCBI suggests that you reach out to then to get access to more requests per second. I haven't tried this.

Good luck, and please let me know if you think you still need a quick patch pushed for you to do this.

Mike