Mercurial > repos > drosofff > fetch_fasta_from_ncbi

annotate retrieve_fasta_from_NCBI.py @ 4:64f45c5e94a0 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
author drosofff
date Mon, 15 May 2017 03:10:11 -0400
parents a9d8f69d59fb
children c6de5c7b4ae3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
1 #!/usr/bin/env python
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
2 # -*- coding: utf-8 -*-
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
3 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
4 From a taxonomy ID retrieves all the nucleotide sequences
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
5 It returns a multiFASTA nuc/prot file
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
6
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
7 Entrez Database UID common name E-utility Database Name
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
8 Nucleotide GI number nuccore
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
9 Protein GI number protein
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
10
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
11 Retrieve strategy:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
12
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
13 esearch to get total number of UIDs (count)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
14 esearch to get UIDs in batches
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
15 loop untile end of UIDs list:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
16 epost to put a batch of UIDs in the history server
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
17 efetch to retrieve info from previous post
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
18
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
19 retmax of efetch is 1/10 of declared value from NCBI
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
20
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
21 queries are 1 sec delayed, to satisfy NCBI guidelines (more than what they request)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
22
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
23
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
24 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
25 import sys
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
26 import logging
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
27 import optparse
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
28 import time
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
29 import urllib
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
30 import urllib2
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
31 import httplib
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
32 import re
3
a9d8f69d59fb planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 1
diff changeset
33
a9d8f69d59fb planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 1
diff changeset
34
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
35 class Eutils:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
36
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
37 def __init__(self, options, logger):
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
38 """
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
39 Initialize retrieval parameters
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
40 """
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
41 self.logger = logger
3
a9d8f69d59fb planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 1
diff changeset
42 self.base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
43 self.query_string = options.query_string
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
44 self.dbname = options.dbname
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
45 if options.outname:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
46 self.outname = options.outname
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
47 else:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
48 self.outname = 'NCBI_download' + '.' + self.dbname + '.fasta'
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
49 self.ids = []
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
50 self.retmax_esearch = 100000
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
51 self.retmax_efetch = 500
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
52 self.count = 0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
53 self.webenv = ""
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
54 self.query_key = ""
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
55
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
56 def retrieve(self):
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
57 """
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
58 Retrieve the fasta sequences corresponding to the query
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
59 """
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
60 self.get_count_value()
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
61
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
62 # If no UIDs are found exit script
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
63 if self.count > 0:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
64 self.get_uids_list()
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
65 self.get_sequences()
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
66 else:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
67 self.logger.info("No UIDs were found. Exiting script.")
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
68
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
69 def get_count_value(self):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
70 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
71 just to retrieve Count (number of UIDs)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
72 Total number of UIDs from the retrieved set to be shown in the XML
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
73 output (default=20). By default, ESearch only includes the first 20
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
74 UIDs retrieved in the XML output. If usehistory is set to 'y',
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
75 the remainder of the retrieved set will be stored on the History server;
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
76
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
77 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
78 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
79 self.logger.info("retrieving data from %s" % self.base)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
80 self.logger.info("for Query: %s and database: %s" %
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
81 (self.query_string, self.dbname))
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
82 querylog = self.esearch(self.dbname, self.query_string, '', '', "count")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
83 self.logger.debug("Query response:")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
84 for line in querylog:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
85 self.logger.debug(line.rstrip())
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
86 if '</Count>' in line:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
87 self.count = int(line[line.find('<Count>')+len('<Count>') : line.find('</Count>')])
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
88 self.logger.info("Found %d UIDs" % self.count)
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
89
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
90 def get_uids_list(self):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
91 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
92 Increasing retmax allows more of the retrieved UIDs to be included in the XML output,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
93 up to a maximum of 100,000 records.
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
94 from http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
95 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
96 retmax = self.retmax_esearch
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
97 if (self.count > retmax):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
98 num_batches = (self.count / retmax) + 1
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
99 else:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
100 num_batches = 1
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
101 self.logger.info("Batch size for esearch action: %d UIDs" % retmax)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
102 self.logger.info("Number of batches for esearch action: %d " % num_batches)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
103 for n in range(num_batches):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
104 querylog = self.esearch(self.dbname, self.query_string, n*retmax, retmax, '')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
105 for line in querylog:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
106 if '<Id>' in line and '</Id>' in line:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
107 uid = (line[line.find('<Id>')+len('<Id>') : line.find('</Id>')])
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
108 self.ids.append(uid)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
109 self.logger.info("Retrieved %d UIDs" % len(self.ids))
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
110
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
111 def esearch(self, db, term, retstart, retmax, rettype):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
112 url = self.base + "esearch.fcgi"
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
113 self.logger.debug("url: %s" % url)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
114 values = {'db': db,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
115 'term': term,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
116 'rettype': rettype,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
117 'retstart': retstart,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
118 'retmax': retmax}
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
119 data = urllib.urlencode(values)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
120 self.logger.debug("data: %s" % str(data))
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
121 req = urllib2.Request(url, data)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
122 response = urllib2.urlopen(req)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
123 querylog = response.readlines()
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
124 response.close()
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
125 time.sleep(1)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
126 return querylog
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
127
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
128 def epost(self, db, ids):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
129 url = self.base + "epost.fcgi"
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
130 self.logger.debug("url_epost: %s" % url)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
131 values = {'db': db,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
132 'id': ids}
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
133 data = urllib.urlencode(values)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
134 req = urllib2.Request(url, data)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
135 serverResponse = False
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
136 nb_trials = 0
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
137 while not serverResponse:
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
138 nb_trials += 1
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
139 try:
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
140 self.logger.debug("Try number %s for opening and readin URL %s" % ( nb_trials, url+data ))
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
141 response = urllib2.urlopen(req)
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
142 querylog = response.readlines()
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
143 response.close()
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
144 serverResponse = True
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
145 except urllib2.HTTPError as e:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
146 self.logger.info("urlopen error:%s, %s" % (e.code, e.read() ) )
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
147 self.logger.info("Retrying in 1 sec")
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
148 serverResponse = False
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
149 time.sleep(1)
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
150 except urllib2.URLError as e:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
151 self.logger.info("urlopen error: Failed to reach a server")
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
152 self.logger.info("Reason :%s" % ( e.reason ) )
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
153 self.logger.info("Retrying in 1 sec")
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
154 serverResponse = False
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
155 time.sleep(1)
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
156 except httplib.IncompleteRead as e:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
157 self.logger.info("IncompleteRead error: %s" % ( e.partial ) )
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
158 self.logger.info("Retrying in 1 sec")
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
159 serverResponse = False
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
160 time.sleep(1)
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
161 self.logger.debug("query response:")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
162 for line in querylog:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
163 self.logger.debug(line.rstrip())
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
164 if '</QueryKey>' in line:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
165 self.query_key = str(line[line.find('<QueryKey>')+len('<QueryKey>'):line.find('</QueryKey>')])
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
166 if '</WebEnv>' in line:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
167 self.webenv = str(line[line.find('<WebEnv>')+len('<WebEnv>'):line.find('</WebEnv>')])
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
168 self.logger.debug("*** epost action ***")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
169 self.logger.debug("query_key: %s" % self.query_key)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
170 self.logger.debug("webenv: %s" % self.webenv)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
171 time.sleep(1)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
172
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
173 def efetch(self, db, query_key, webenv):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
174 url = self.base + "efetch.fcgi"
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
175 self.logger.debug("url_efetch: %s" % url)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
176 values = {'db': db,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
177 'query_key': query_key,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
178 'webenv': webenv,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
179 'rettype': "fasta",
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
180 'retmode': "text"}
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
181 data = urllib.urlencode(values)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
182 req = urllib2.Request(url, data)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
183 self.logger.debug("data: %s" % str(data))
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
184 serverTransaction = False
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
185 counter = 0
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
186 response_code = 0
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
187 while not serverTransaction:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
188 counter += 1
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
189 self.logger.info("Server Transaction Trial: %s" % ( counter ) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
190 try:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
191 response = urllib2.urlopen(req)
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
192 response_code = response.getcode()
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
193 fasta = response.read()
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
194 response.close()
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
195 if ( (response_code != 200) or ("Resource temporarily unavailable" in fasta)
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
196 or ("Error" in fasta) or (not fasta.startswith(">") ) ):
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
197 serverTransaction = False
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
198 else:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
199 serverTransaction = True
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
200 except urllib2.HTTPError as e:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
201 serverTransaction = False
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
202 self.logger.info("urlopen error:%s, %s" % (e.code, e.read() ) )
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
203 except urllib2.URLError as e:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
204 serverTransaction = False
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
205 self.logger.info("urlopen error: Failed to reach a server")
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
206 self.logger.info("Reason :%s" % ( e.reason ) )
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
207 except httplib.IncompleteRead as e:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
208 serverTransaction = False
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
209 self.logger.info("IncompleteRead error: %s" % ( e.partial ) )
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
210 fasta = self.sanitiser(self.dbname, fasta)
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
211 time.sleep(0.1)
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
212 return fasta
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
213
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
214 def sanitiser(self, db, fastaseq):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
215 if db not in "nuccore protein" : return fastaseq
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
216 regex = re.compile(r"[ACDEFGHIKLMNPQRSTVWYBZ]{49,}")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
217 sane_seqlist = []
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
218 seqlist = fastaseq.split("\n\n")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
219 for seq in seqlist[:-1]:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
220 fastalines = seq.split("\n")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
221 if len(fastalines) < 2:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
222 self.logger.info("Empty sequence for %s" % ("|".join(fastalines[0].split("|")[:4]) ) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
223 self.logger.info("%s download is skipped" % ("|".join(fastalines[0].split("|")[:4]) ) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
224 continue
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
225 if db == "nuccore":
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
226 badnuc = 0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
227 for nucleotide in fastalines[1]:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
228 if nucleotide not in "ATGC":
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
229 badnuc += 1
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
230 if float(badnuc)/len(fastalines[1]) > 0.4:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
231 self.logger.info("%s ambiguous nucleotides in %s or download interrupted at this offset | %s" % ( float(badnuc)/len(fastalines[1]), "|".join(fastalines[0].split("|")[:4]), fastalines[1]) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
232 self.logger.info("%s download is skipped" % (fastalines[0].split("|")[:4]) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
233 continue
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
234 fastalines[0] = fastalines[0].replace(" ","_")[:100] # remove spaces and trim the header to 100 chars
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
235 cleanseq = "\n".join(fastalines)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
236 sane_seqlist.append(cleanseq)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
237 elif db == "protein":
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
238 fastalines[0] = fastalines[0][0:100]
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
239 fastalines[0] = fastalines[0].replace(" ", "_")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
240 fastalines[0] = fastalines[0].replace("[", "_")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
241 fastalines[0] = fastalines[0].replace("]", "_")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
242 fastalines[0] = fastalines[0].replace("=", "_")
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
243 fastalines[0] = fastalines[0].rstrip("_") # because blast makedb doesn't like it
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
244 fastalines[0] = re.sub(regex, "_", fastalines[0])
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
245 cleanseq = "\n".join(fastalines)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
246 sane_seqlist.append(cleanseq)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
247 self.logger.info("clean sequences appended: %d" % (len(sane_seqlist) ) )
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
248 return "\n".join(sane_seqlist)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
249
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
250 def get_sequences(self):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
251 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
252 Total number of records from the input set to be retrieved, up to a maximum
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
253 of 10,000. Optionally, for a large set the value of retstart can be iterated
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
254 while holding retmax constant, thereby downloading the entire set in batches
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
255 of size retmax.
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
256
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
257 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
258
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
259 """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
260 batch_size = self.retmax_efetch
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
261 count = self.count
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
262 uids_list = self.ids
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
263 self.logger.info("Batch size for efetch action: %d" % batch_size)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
264 self.logger.info("Number of batches for efetch action: %d" % ((count / batch_size) + 1))
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
265 with open(self.outname, 'w') as out:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
266 for start in range(0, count, batch_size):
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
267 end = min(count, start+batch_size)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
268 batch = uids_list[start:end]
4
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
269 if self.epost(self.dbname, ",".join(batch)) != -1:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
270 mfasta = ''
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
271 while not mfasta:
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
272 self.logger.info("retrieving batch %d" % ((start / batch_size) + 1))
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
273 mfasta = self.efetch(self.dbname, self.query_key, self.webenv)
64f45c5e94a0 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 3
diff changeset
274 out.write(mfasta + '\n')
0
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
275
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
276
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
277 LOG_FORMAT = '%(asctime)s|%(levelname)-8s|%(message)s'
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
278 LOG_DATEFMT = '%Y-%m-%d %H:%M:%S'
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
279 LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
280
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
281
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
282 def __main__():
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
283 """ main function """
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
284 parser = optparse.OptionParser(description='Retrieve data from NCBI')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
285 parser.add_option('-i', dest='query_string', help='NCBI Query String')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
286 parser.add_option('-o', dest='outname', help='output file name')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
287 parser.add_option('-l', '--logfile', help='log file (default=stderr)')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
288 parser.add_option('--loglevel', choices=LOG_LEVELS, default='INFO', help='logging level (default: INFO)')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
289 parser.add_option('-d', dest='dbname', help='database type')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
290 (options, args) = parser.parse_args()
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
291 if len(args) > 0:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
292 parser.error('Wrong number of arguments')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
293
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
294 log_level = getattr(logging, options.loglevel)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
295 kwargs = {'format': LOG_FORMAT,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
296 'datefmt': LOG_DATEFMT,
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
297 'level': log_level}
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
298 if options.logfile:
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
299 kwargs['filename'] = options.logfile
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
300 logging.basicConfig(**kwargs)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
301 logger = logging.getLogger('data_from_NCBI')
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
302
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
303 E = Eutils(options, logger)
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
304 E.retrieve()
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
305
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
306
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
307 if __name__ == "__main__":
0bdc5a73c8d1 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents:
diff changeset
308 __main__()