Mercurial > repos > artbio > fetch_fasta_from_ncbi

annotate fetch_fasta_from_NCBI.py @ 1:7e41bbb94159 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
author artbio
date Sun, 15 Oct 2017 13:56:26 -0400
parents
children 50f5ef3313bb
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
1 #!/usr/bin/env python
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
2 # -*- coding: utf-8 -*-
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
3 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
4 From a taxonomy ID retrieves all the nucleotide sequences
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
5 It returns a multiFASTA nuc/prot file
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
6
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
7 Entrez Database UID common name E-utility Database Name
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
8 Nucleotide GI number nuccore
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
9 Protein GI number protein
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
10
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
11 Retrieve strategy:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
12
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
13 esearch to get total number of UIDs (count)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
14 esearch to get UIDs in batches
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
15 loop untile end of UIDs list:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
16 epost to put a batch of UIDs in the history server
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
17 efetch to retrieve info from previous post
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
18
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
19 retmax of efetch is 1/10 of declared value from NCBI
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
20
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
21 queries are 1 sec delayed, to satisfy NCBI guidelines
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
22 (more than what they request)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
23 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
24 import httplib
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
25 import logging
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
26 import optparse
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
27 import re
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
28 import sys
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
29 import time
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
30 import urllib
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
31 import urllib2
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
32
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
33
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
34 class QueryException(Exception):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
35 pass
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
36
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
37
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
38 class Eutils:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
39
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
40 def __init__(self, options, logger):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
41 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
42 Initialize retrieval parameters
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
43 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
44 self.logger = logger
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
45 self.base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
46 self.query_string = options.query_string
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
47 self.dbname = options.dbname
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
48 if options.outname:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
49 self.outname = options.outname
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
50 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
51 self.outname = 'NCBI_download' + '.' + self.dbname + '.fasta'
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
52 self.ids = []
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
53 self.retmax_esearch = 100000
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
54 self.retmax_efetch = 500
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
55 self.count = 0
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
56 self.webenv = ""
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
57 self.query_key = ""
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
58 self.datetype = options.datetype
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
59 if options.reldate:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
60 self.reldate = options.reldate
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
61 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
62 self.reldate = ''
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
63 if options.mindate:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
64 self.mindate = options.mindate
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
65 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
66 self.mindate = ''
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
67 if options.maxdate:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
68 self.maxdate = options.maxdate
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
69 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
70 self.maxdate = ''
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
71
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
72 def retrieve(self):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
73 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
74 Retrieve the fasta sequences corresponding to the query
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
75 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
76 self.get_count_value()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
77
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
78 # If no UIDs are found exit script
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
79 if self.count > 0:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
80 self.get_uids_list()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
81 try:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
82 self.get_sequences()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
83 except QueryException as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
84 self.logger.error("Exiting script.")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
85 raise e
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
86 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
87 self.logger.error("No UIDs were found. Exiting script.")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
88 raise Exception("")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
89
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
90 def get_count_value(self):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
91 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
92 just to retrieve Count (number of UIDs)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
93 Total number of UIDs from the retrieved set to be shown in the XML
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
94 output (default=20). By default, ESearch only includes the first 20
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
95 UIDs retrieved in the XML output. If usehistory is set to 'y',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
96 the remainder of the retrieved set will be stored on the History server
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
97
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
98 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
99 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
100 self.logger.info("retrieving data from %s" % self.base)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
101 self.logger.info("for Query: %s and database: %s" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
102 (self.query_string, self.dbname))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
103 querylog = self.esearch(self.dbname, self.query_string, '', '',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
104 "count", self.datetype, self.reldate,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
105 self.mindate, self.maxdate)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
106 self.logger.debug("Query response:")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
107 for line in querylog:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
108 self.logger.debug(line.rstrip())
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
109 if '</Count>' in line:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
110 self.count = int(line[line.find('<Count>')+len('<Count>'):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
111 line.find('</Count>')])
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
112 self.logger.info("Found %d UIDs" % self.count)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
113
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
114 def get_uids_list(self):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
115 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
116 Increasing retmax allows more of the retrieved UIDs to be included in
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
117 the XML output, up to a maximum of 100,000 records.
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
118 from http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
119 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
120 retmax = self.retmax_esearch
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
121 if (self.count > retmax):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
122 num_batches = (self.count / retmax) + 1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
123 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
124 num_batches = 1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
125 self.logger.info("Batch size for esearch action: %d UIDs" % retmax)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
126 self.logger.info("Number of batches for esearch action: %d " %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
127 num_batches)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
128 for n in range(num_batches):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
129 querylog = self.esearch(self.dbname, self.query_string, n*retmax,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
130 retmax, '', self.datetype, self.reldate,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
131 self.mindate, self.maxdate)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
132 for line in querylog:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
133 if '<Id>' in line and '</Id>' in line:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
134 uid = (line[line.find('<Id>')+len('<Id>'):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
135 line.find('</Id>')])
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
136 self.ids.append(uid)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
137 self.logger.info("Retrieved %d UIDs" % len(self.ids))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
138
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
139 def esearch(self, db, term, retstart, retmax, rettype, datetype, reldate,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
140 mindate, maxdate):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
141 url = self.base + "esearch.fcgi"
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
142 self.logger.debug("url: %s" % url)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
143 values = {'db': db,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
144 'term': term,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
145 'rettype': rettype,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
146 'retstart': retstart,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
147 'retmax': retmax,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
148 'datetype': datetype,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
149 'reldate': reldate,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
150 'mindate': mindate,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
151 'maxdate': maxdate}
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
152 data = urllib.urlencode(values)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
153 self.logger.debug("data: %s" % str(data))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
154 req = urllib2.Request(url, data)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
155 response = urllib2.urlopen(req)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
156 querylog = response.readlines()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
157 response.close()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
158 time.sleep(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
159 return querylog
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
160
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
161 def epost(self, db, ids):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
162 url = self.base + "epost.fcgi"
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
163 self.logger.debug("url_epost: %s" % url)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
164 values = {'db': db,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
165 'id': ids}
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
166 data = urllib.urlencode(values)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
167 req = urllib2.Request(url, data)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
168 serverResponse = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
169 nb_trials = 0
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
170 while not serverResponse:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
171 nb_trials += 1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
172 try:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
173 self.logger.debug("Try number %s for opening and readin URL %s"
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
174 % (nb_trials, url+data))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
175 response = urllib2.urlopen(req)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
176 querylog = response.readlines()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
177 response.close()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
178 serverResponse = True
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
179 except urllib2.HTTPError as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
180 self.logger.info("urlopen error:%s, %s" % (e.code, e.read()))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
181 self.logger.info("Retrying in 1 sec")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
182 serverResponse = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
183 time.sleep(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
184 except urllib2.URLError as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
185 self.logger.info("urlopen error: Failed to reach a server")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
186 self.logger.info("Reason :%s" % (e.reason))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
187 self.logger.info("Retrying in 1 sec")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
188 serverResponse = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
189 time.sleep(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
190 except httplib.IncompleteRead as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
191 self.logger.info("IncompleteRead error: %s" % (e.partial))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
192 self.logger.info("Retrying in 1 sec")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
193 serverResponse = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
194 time.sleep(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
195 self.logger.debug("query response:")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
196 for line in querylog:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
197 self.logger.debug(line.rstrip())
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
198 if '</QueryKey>' in line:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
199 self.query_key = str(line[line.find('<QueryKey>') +
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
200 len('<QueryKey>'):line.find('</QueryKey>')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
201 ])
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
202 if '</WebEnv>' in line:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
203 self.webenv = str(line[line.find('<WebEnv>')+len('<WebEnv>'):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
204 line.find('</WebEnv>')])
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
205 self.logger.debug("*** epost action ***")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
206 self.logger.debug("query_key: %s" % self.query_key)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
207 self.logger.debug("webenv: %s" % self.webenv)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
208 time.sleep(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
209
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
210 def efetch(self, db, query_key, webenv):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
211 url = self.base + "efetch.fcgi"
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
212 self.logger.debug("url_efetch: %s" % url)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
213 values = {'db': db,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
214 'query_key': query_key,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
215 'webenv': webenv,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
216 'rettype': "fasta",
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
217 'retmode': "text"}
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
218 data = urllib.urlencode(values)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
219 req = urllib2.Request(url, data)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
220 self.logger.debug("data: %s" % str(data))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
221 serverTransaction = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
222 counter = 0
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
223 response_code = 0
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
224 while not serverTransaction:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
225 counter += 1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
226 self.logger.info("Server Transaction Trial: %s" % (counter))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
227 try:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
228 response = urllib2.urlopen(req)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
229 response_code = response.getcode()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
230 fasta = response.read()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
231 response.close()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
232 if((response_code != 200) or
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
233 ("Resource temporarily unavailable" in fasta) or
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
234 ("Error" in fasta) or (not fasta.startswith(">"))):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
235 serverTransaction = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
236 if (response_code != 200):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
237 self.logger.info("urlopen error: Response code is not\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
238 200")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
239 elif ("Resource temporarily unavailable" in fasta):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
240 self.logger.info("Ressource temporarily unavailable")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
241 elif ("Error" in fasta):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
242 self.logger.info("Error in fasta")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
243 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
244 self.logger.info("Fasta doesn't start with '>'")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
245 else:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
246 serverTransaction = True
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
247 except urllib2.HTTPError as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
248 serverTransaction = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
249 self.logger.info("urlopen error:%s, %s" % (e.code, e.read()))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
250 except urllib2.URLError as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
251 serverTransaction = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
252 self.logger.info("urlopen error: Failed to reach a server")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
253 self.logger.info("Reason :%s" % (e.reason))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
254 except httplib.IncompleteRead as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
255 serverTransaction = False
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
256 self.logger.info("IncompleteRead error: %s" % (e.partial))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
257 if (counter > 500):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
258 serverTransaction = True
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
259 if (counter > 500):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
260 raise QueryException({"message":
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
261 "500 Server Transaction Trials attempted for\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
262 this batch. Aborting."})
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
263 fasta = self.sanitiser(self.dbname, fasta)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
264 time.sleep(0.1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
265 return fasta
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
266
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
267 def sanitiser(self, db, fastaseq):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
268 if(db not in "nuccore protein"):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
269 return fastaseq
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
270 regex = re.compile(r"[ACDEFGHIKLMNPQRSTVWYBZ]{49,}")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
271 sane_seqlist = []
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
272 seqlist = fastaseq.split("\n\n")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
273 for seq in seqlist[:-1]:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
274 fastalines = seq.split("\n")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
275 if len(fastalines) < 2:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
276 self.logger.info("Empty sequence for %s" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
277 ("|".join(fastalines[0].split("|")[:4])))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
278 self.logger.info("%s download is skipped" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
279 ("|".join(fastalines[0].split("|")[:4])))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
280 continue
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
281 if db == "nuccore":
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
282 badnuc = 0
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
283 for nucleotide in fastalines[1]:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
284 if nucleotide not in "ATGC":
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
285 badnuc += 1
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
286 if float(badnuc)/len(fastalines[1]) > 0.4:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
287 self.logger.info("%s ambiguous nucleotides in %s\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
288 or download interrupted at this offset\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
289 | %s" % (float(badnuc)/len(fastalines[1]),
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
290 "|".join(fastalines[0].split("|")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
291 [:4]),
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
292 fastalines[1]))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
293 self.logger.info("%s download is skipped" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
294 (fastalines[0].split("|")[:4]))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
295 continue
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
296 """ remove spaces and trim the header to 100 chars """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
297 fastalines[0] = fastalines[0].replace(" ", "_")[:100]
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
298 cleanseq = "\n".join(fastalines)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
299 sane_seqlist.append(cleanseq)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
300 elif db == "protein":
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
301 fastalines[0] = fastalines[0][0:100]
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
302 fastalines[0] = fastalines[0].replace(" ", "_")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
303 fastalines[0] = fastalines[0].replace("[", "_")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
304 fastalines[0] = fastalines[0].replace("]", "_")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
305 fastalines[0] = fastalines[0].replace("=", "_")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
306 """ because blast makedb doesn't like it """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
307 fastalines[0] = fastalines[0].rstrip("_")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
308 fastalines[0] = re.sub(regex, "_", fastalines[0])
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
309 cleanseq = "\n".join(fastalines)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
310 sane_seqlist.append(cleanseq)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
311 self.logger.info("clean sequences appended: %d" % (len(sane_seqlist)))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
312 return "\n".join(sane_seqlist)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
313
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
314 def get_sequences(self):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
315 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
316 Total number of records from the input set to be retrieved,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
317 up to a maximum of 10,000. Optionally, for a large set the value of
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
318 retstart can be iterated while holding retmax constant, thereby
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
319 downloading the entire set in batches of size retmax.
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
320 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
321 """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
322 batch_size = self.retmax_efetch
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
323 count = self.count
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
324 uids_list = self.ids
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
325 self.logger.info("Batch size for efetch action: %d" % batch_size)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
326 self.logger.info("Number of batches for efetch action: %d" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
327 ((count / batch_size) + 1))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
328 with open(self.outname, 'w') as out:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
329 for start in range(0, count, batch_size):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
330 end = min(count, start+batch_size)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
331 batch = uids_list[start:end]
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
332 if self.epost(self.dbname, ",".join(batch)) != -1:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
333 mfasta = ''
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
334 while not mfasta:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
335 self.logger.info("retrieving batch %d" %
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
336 ((start / batch_size) + 1))
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
337 try:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
338 mfasta = self.efetch(self.dbname, self.query_key,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
339 self.webenv)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
340 out.write(mfasta + '\n')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
341 except QueryException as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
342 self.logger.error("%s" % e.message)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
343 raise e
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
344
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
345
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
346 LOG_FORMAT = '%(asctime)s|%(levelname)-8s|%(message)s'
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
347 LOG_DATEFMT = '%Y-%m-%d %H:%M:%S'
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
348 LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
349
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
350
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
351 def __main__():
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
352 """ main function """
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
353 parser = optparse.OptionParser(description='Retrieve data from NCBI')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
354 parser.add_option('-i', dest='query_string', help='NCBI Query String')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
355 parser.add_option('-o', dest='outname', help='output file name')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
356 parser.add_option('-d', dest='dbname', help='database type')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
357 parser.add_option('-l', '--logfile', help='log file (default=stderr)')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
358 parser.add_option('--datetype', dest='datetype',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
359 choices=['mdat', 'pdat'],
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
360 help='Type of date used to limit a search.\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
361 [ mdat(modification date), pdat(publication date)]\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
362 (default=pdat)', default='pdat')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
363 parser.add_option('--reldate', dest='reldate',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
364 help='When reldate is set to an integer n, the search\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
365 returns only those items that have a date\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
366 specified by datetype within the last n days.')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
367 parser.add_option('--maxdate', dest='maxdate',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
368 help='Date range used to limit a search result by the\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
369 date specified by datetype. These two parameters\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
370 (mindate, maxdate) must be used together to\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
371 specify an arbitrary date range. The general date\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
372 format is YYYY/MM/DD, and these variants are also\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
373 allowed: YYYY, YYYY/MM.')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
374 parser.add_option('--mindate', dest='mindate',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
375 help='Date range used to limit a search result by the\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
376 date specified by datetype. These two parameters\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
377 (mindate, maxdate) must be used together to\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
378 specify an arbitrary date range. The general date\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
379 format is YYYY/MM/DD, and these variants are also\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
380 allowed: YYYY, YYYY/MM.')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
381 parser.add_option('--loglevel', choices=LOG_LEVELS, default='INFO',
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
382 help='logging level (default: INFO)')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
383 (options, args) = parser.parse_args()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
384 if len(args) > 0:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
385 parser.error('Wrong number of arguments')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
386 if((options.reldate and options.maxdate) or
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
387 (options.reldate and options.mindate)):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
388 parser.error("You can't mix 'reldate' and 'maxdate', 'mindate'\
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
389 parameters")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
390 if((options.mindate and not options.maxdate) or
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
391 (options.maxdate and not options.mindate)):
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
392 parser.error("mindate and maxdate must be used together")
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
393
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
394 log_level = getattr(logging, options.loglevel)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
395 kwargs = {'format': LOG_FORMAT,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
396 'datefmt': LOG_DATEFMT,
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
397 'level': log_level}
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
398 if options.logfile:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
399 kwargs['filename'] = options.logfile
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
400 logging.basicConfig(**kwargs)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
401 logger = logging.getLogger('data_from_NCBI')
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
402
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
403 E = Eutils(options, logger)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
404 try:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
405 E.retrieve()
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
406 except Exception as e:
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
407 sys.exit(1)
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
408
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
409
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
410 if __name__ == "__main__":
7e41bbb94159 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 6008aafac37eec1916d6b72c05d9cfcb002b8095
artbio
parents:
diff changeset
411 __main__()