annotate BeautifulSoup.py @ 28:184d14e4270d

Update to Miller Lab devshed revision 4ede22dd5500
author Richard Burhans <burhans@bx.psu.edu>
date Wed, 17 Jul 2013 12:46:46 -0400
parents 2c498d40ecde
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1 """Beautiful Soup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2 Elixir and Tonic
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
3 "The Screen-Scraper's Friend"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
4 http://www.crummy.com/software/BeautifulSoup/
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
5
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
6 Beautiful Soup parses a (possibly invalid) XML or HTML document into a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
7 tree representation. It provides methods and Pythonic idioms that make
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
8 it easy to navigate, search, and modify the tree.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
9
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
10 A well-formed XML/HTML document yields a well-formed data
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
11 structure. An ill-formed XML/HTML document yields a correspondingly
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
12 ill-formed data structure. If your document is only locally
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
13 well-formed, you can use this library to find and process the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
14 well-formed part of it.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
15
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
16 Beautiful Soup works with Python 2.2 and up. It has no external
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
17 dependencies, but you'll have more success at converting data to UTF-8
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
18 if you also install these three packages:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
19
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
20 * chardet, for auto-detecting character encodings
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
21 http://chardet.feedparser.org/
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
22 * cjkcodecs and iconv_codec, which add more encodings to the ones supported
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
23 by stock Python.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
24 http://cjkpython.i18n.org/
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
25
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
26 Beautiful Soup defines classes for two main parsing strategies:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
27
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
28 * BeautifulStoneSoup, for parsing XML, SGML, or your domain-specific
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
29 language that kind of looks like XML.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
30
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
31 * BeautifulSoup, for parsing run-of-the-mill HTML code, be it valid
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
32 or invalid. This class has web browser-like heuristics for
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
33 obtaining a sensible parse tree in the face of common HTML errors.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
34
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
35 Beautiful Soup also defines a class (UnicodeDammit) for autodetecting
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
36 the encoding of an HTML or XML document, and converting it to
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
37 Unicode. Much of this code is taken from Mark Pilgrim's Universal Feed Parser.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
38
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
39 For more than you ever wanted to know about Beautiful Soup, see the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
40 documentation:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
41 http://www.crummy.com/software/BeautifulSoup/documentation.html
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
42
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
43 Here, have some legalese:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
44
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
45 Copyright (c) 2004-2010, Leonard Richardson
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
46
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
47 All rights reserved.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
48
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
49 Redistribution and use in source and binary forms, with or without
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
50 modification, are permitted provided that the following conditions are
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
51 met:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
52
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
53 * Redistributions of source code must retain the above copyright
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
54 notice, this list of conditions and the following disclaimer.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
55
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
56 * Redistributions in binary form must reproduce the above
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
57 copyright notice, this list of conditions and the following
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
58 disclaimer in the documentation and/or other materials provided
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
59 with the distribution.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
60
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
61 * Neither the name of the the Beautiful Soup Consortium and All
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
62 Night Kosher Bakery nor the names of its contributors may be
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
63 used to endorse or promote products derived from this software
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
64 without specific prior written permission.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
65
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
66 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
67 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
68 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
69 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
70 CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
71 EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
72 PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
73 PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
74 LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
75 NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
76 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE, DAMMIT.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
77
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
78 """
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
79 from __future__ import generators
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
80
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
81 __author__ = "Leonard Richardson (leonardr@segfault.org)"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
82 __version__ = "3.2.0"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
83 __copyright__ = "Copyright (c) 2004-2010 Leonard Richardson"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
84 __license__ = "New-style BSD"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
85
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
86 from sgmllib import SGMLParser, SGMLParseError
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
87 import codecs
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
88 import markupbase
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
89 import types
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
90 import re
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
91 import sgmllib
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
92 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
93 from htmlentitydefs import name2codepoint
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
94 except ImportError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
95 name2codepoint = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
96 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
97 set
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
98 except NameError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
99 from sets import Set as set
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
100
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
101 #These hacks make Beautiful Soup able to parse XML with namespaces
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
102 sgmllib.tagfind = re.compile('[a-zA-Z][-_.:a-zA-Z0-9]*')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
103 markupbase._declname_match = re.compile(r'[a-zA-Z][-_.:a-zA-Z0-9]*\s*').match
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
104
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
105 DEFAULT_OUTPUT_ENCODING = "utf-8"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
106
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
107 def _match_css_class(str):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
108 """Build a RE to match the given CSS class."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
109 return re.compile(r"(^|.*\s)%s($|\s)" % str)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
110
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
111 # First, the classes that represent markup elements.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
112
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
113 class PageElement(object):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
114 """Contains the navigational information for some part of the page
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
115 (either a tag or a piece of text)"""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
116
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
117 def setup(self, parent=None, previous=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
118 """Sets up the initial relations between this element and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
119 other elements."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
120 self.parent = parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
121 self.previous = previous
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
122 self.next = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
123 self.previousSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
124 self.nextSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
125 if self.parent and self.parent.contents:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
126 self.previousSibling = self.parent.contents[-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
127 self.previousSibling.nextSibling = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
128
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
129 def replaceWith(self, replaceWith):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
130 oldParent = self.parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
131 myIndex = self.parent.index(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
132 if hasattr(replaceWith, "parent")\
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
133 and replaceWith.parent is self.parent:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
134 # We're replacing this element with one of its siblings.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
135 index = replaceWith.parent.index(replaceWith)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
136 if index and index < myIndex:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
137 # Furthermore, it comes before this element. That
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
138 # means that when we extract it, the index of this
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
139 # element will change.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
140 myIndex = myIndex - 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
141 self.extract()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
142 oldParent.insert(myIndex, replaceWith)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
143
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
144 def replaceWithChildren(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
145 myParent = self.parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
146 myIndex = self.parent.index(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
147 self.extract()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
148 reversedChildren = list(self.contents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
149 reversedChildren.reverse()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
150 for child in reversedChildren:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
151 myParent.insert(myIndex, child)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
152
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
153 def extract(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
154 """Destructively rips this element out of the tree."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
155 if self.parent:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
156 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
157 del self.parent.contents[self.parent.index(self)]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
158 except ValueError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
159 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
160
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
161 #Find the two elements that would be next to each other if
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
162 #this element (and any children) hadn't been parsed. Connect
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
163 #the two.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
164 lastChild = self._lastRecursiveChild()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
165 nextElement = lastChild.next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
166
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
167 if self.previous:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
168 self.previous.next = nextElement
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
169 if nextElement:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
170 nextElement.previous = self.previous
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
171 self.previous = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
172 lastChild.next = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
173
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
174 self.parent = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
175 if self.previousSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
176 self.previousSibling.nextSibling = self.nextSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
177 if self.nextSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
178 self.nextSibling.previousSibling = self.previousSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
179 self.previousSibling = self.nextSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
180 return self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
181
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
182 def _lastRecursiveChild(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
183 "Finds the last element beneath this object to be parsed."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
184 lastChild = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
185 while hasattr(lastChild, 'contents') and lastChild.contents:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
186 lastChild = lastChild.contents[-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
187 return lastChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
188
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
189 def insert(self, position, newChild):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
190 if isinstance(newChild, basestring) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
191 and not isinstance(newChild, NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
192 newChild = NavigableString(newChild)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
193
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
194 position = min(position, len(self.contents))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
195 if hasattr(newChild, 'parent') and newChild.parent is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
196 # We're 'inserting' an element that's already one
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
197 # of this object's children.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
198 if newChild.parent is self:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
199 index = self.index(newChild)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
200 if index > position:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
201 # Furthermore we're moving it further down the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
202 # list of this object's children. That means that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
203 # when we extract this element, our target index
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
204 # will jump down one.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
205 position = position - 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
206 newChild.extract()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
207
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
208 newChild.parent = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
209 previousChild = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
210 if position == 0:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
211 newChild.previousSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
212 newChild.previous = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
213 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
214 previousChild = self.contents[position-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
215 newChild.previousSibling = previousChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
216 newChild.previousSibling.nextSibling = newChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
217 newChild.previous = previousChild._lastRecursiveChild()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
218 if newChild.previous:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
219 newChild.previous.next = newChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
220
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
221 newChildsLastElement = newChild._lastRecursiveChild()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
222
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
223 if position >= len(self.contents):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
224 newChild.nextSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
225
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
226 parent = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
227 parentsNextSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
228 while not parentsNextSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
229 parentsNextSibling = parent.nextSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
230 parent = parent.parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
231 if not parent: # This is the last element in the document.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
232 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
233 if parentsNextSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
234 newChildsLastElement.next = parentsNextSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
235 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
236 newChildsLastElement.next = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
237 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
238 nextChild = self.contents[position]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
239 newChild.nextSibling = nextChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
240 if newChild.nextSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
241 newChild.nextSibling.previousSibling = newChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
242 newChildsLastElement.next = nextChild
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
243
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
244 if newChildsLastElement.next:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
245 newChildsLastElement.next.previous = newChildsLastElement
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
246 self.contents.insert(position, newChild)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
247
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
248 def append(self, tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
249 """Appends the given tag to the contents of this tag."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
250 self.insert(len(self.contents), tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
251
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
252 def findNext(self, name=None, attrs={}, text=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
253 """Returns the first item that matches the given criteria and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
254 appears after this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
255 return self._findOne(self.findAllNext, name, attrs, text, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
256
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
257 def findAllNext(self, name=None, attrs={}, text=None, limit=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
258 **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
259 """Returns all items that match the given criteria and appear
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
260 after this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
261 return self._findAll(name, attrs, text, limit, self.nextGenerator,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
262 **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
263
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
264 def findNextSibling(self, name=None, attrs={}, text=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
265 """Returns the closest sibling to this Tag that matches the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
266 given criteria and appears after this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
267 return self._findOne(self.findNextSiblings, name, attrs, text,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
268 **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
269
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
270 def findNextSiblings(self, name=None, attrs={}, text=None, limit=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
271 **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
272 """Returns the siblings of this Tag that match the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
273 criteria and appear after this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
274 return self._findAll(name, attrs, text, limit,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
275 self.nextSiblingGenerator, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
276 fetchNextSiblings = findNextSiblings # Compatibility with pre-3.x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
277
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
278 def findPrevious(self, name=None, attrs={}, text=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
279 """Returns the first item that matches the given criteria and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
280 appears before this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
281 return self._findOne(self.findAllPrevious, name, attrs, text, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
282
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
283 def findAllPrevious(self, name=None, attrs={}, text=None, limit=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
284 **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
285 """Returns all items that match the given criteria and appear
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
286 before this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
287 return self._findAll(name, attrs, text, limit, self.previousGenerator,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
288 **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
289 fetchPrevious = findAllPrevious # Compatibility with pre-3.x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
290
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
291 def findPreviousSibling(self, name=None, attrs={}, text=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
292 """Returns the closest sibling to this Tag that matches the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
293 given criteria and appears before this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
294 return self._findOne(self.findPreviousSiblings, name, attrs, text,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
295 **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
296
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
297 def findPreviousSiblings(self, name=None, attrs={}, text=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
298 limit=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
299 """Returns the siblings of this Tag that match the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
300 criteria and appear before this Tag in the document."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
301 return self._findAll(name, attrs, text, limit,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
302 self.previousSiblingGenerator, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
303 fetchPreviousSiblings = findPreviousSiblings # Compatibility with pre-3.x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
304
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
305 def findParent(self, name=None, attrs={}, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
306 """Returns the closest parent of this Tag that matches the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
307 criteria."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
308 # NOTE: We can't use _findOne because findParents takes a different
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
309 # set of arguments.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
310 r = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
311 l = self.findParents(name, attrs, 1)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
312 if l:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
313 r = l[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
314 return r
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
315
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
316 def findParents(self, name=None, attrs={}, limit=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
317 """Returns the parents of this Tag that match the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
318 criteria."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
319
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
320 return self._findAll(name, attrs, None, limit, self.parentGenerator,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
321 **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
322 fetchParents = findParents # Compatibility with pre-3.x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
323
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
324 #These methods do the real heavy lifting.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
325
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
326 def _findOne(self, method, name, attrs, text, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
327 r = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
328 l = method(name, attrs, text, 1, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
329 if l:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
330 r = l[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
331 return r
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
332
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
333 def _findAll(self, name, attrs, text, limit, generator, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
334 "Iterates over a generator looking for things that match."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
335
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
336 if isinstance(name, SoupStrainer):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
337 strainer = name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
338 # (Possibly) special case some findAll*(...) searches
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
339 elif text is None and not limit and not attrs and not kwargs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
340 # findAll*(True)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
341 if name is True:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
342 return [element for element in generator()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
343 if isinstance(element, Tag)]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
344 # findAll*('tag-name')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
345 elif isinstance(name, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
346 return [element for element in generator()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
347 if isinstance(element, Tag) and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
348 element.name == name]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
349 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
350 strainer = SoupStrainer(name, attrs, text, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
351 # Build a SoupStrainer
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
352 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
353 strainer = SoupStrainer(name, attrs, text, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
354 results = ResultSet(strainer)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
355 g = generator()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
356 while True:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
357 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
358 i = g.next()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
359 except StopIteration:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
360 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
361 if i:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
362 found = strainer.search(i)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
363 if found:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
364 results.append(found)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
365 if limit and len(results) >= limit:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
366 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
367 return results
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
368
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
369 #These Generators can be used to navigate starting from both
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
370 #NavigableStrings and Tags.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
371 def nextGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
372 i = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
373 while i is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
374 i = i.next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
375 yield i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
376
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
377 def nextSiblingGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
378 i = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
379 while i is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
380 i = i.nextSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
381 yield i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
382
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
383 def previousGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
384 i = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
385 while i is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
386 i = i.previous
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
387 yield i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
388
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
389 def previousSiblingGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
390 i = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
391 while i is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
392 i = i.previousSibling
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
393 yield i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
394
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
395 def parentGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
396 i = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
397 while i is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
398 i = i.parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
399 yield i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
400
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
401 # Utility methods
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
402 def substituteEncoding(self, str, encoding=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
403 encoding = encoding or "utf-8"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
404 return str.replace("%SOUP-ENCODING%", encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
405
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
406 def toEncoding(self, s, encoding=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
407 """Encodes an object to a string in some encoding, or to Unicode.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
408 ."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
409 if isinstance(s, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
410 if encoding:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
411 s = s.encode(encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
412 elif isinstance(s, str):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
413 if encoding:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
414 s = s.encode(encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
415 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
416 s = unicode(s)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
417 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
418 if encoding:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
419 s = self.toEncoding(str(s), encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
420 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
421 s = unicode(s)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
422 return s
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
423
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
424 class NavigableString(unicode, PageElement):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
425
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
426 def __new__(cls, value):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
427 """Create a new NavigableString.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
428
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
429 When unpickling a NavigableString, this method is called with
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
430 the string in DEFAULT_OUTPUT_ENCODING. That encoding needs to be
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
431 passed in to the superclass's __new__ or the superclass won't know
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
432 how to handle non-ASCII characters.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
433 """
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
434 if isinstance(value, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
435 return unicode.__new__(cls, value)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
436 return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
437
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
438 def __getnewargs__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
439 return (NavigableString.__str__(self),)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
440
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
441 def __getattr__(self, attr):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
442 """text.string gives you text. This is for backwards
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
443 compatibility for Navigable*String, but for CData* it lets you
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
444 get the string without the CData wrapper."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
445 if attr == 'string':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
446 return self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
447 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
448 raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__.__name__, attr)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
449
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
450 def __unicode__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
451 return str(self).decode(DEFAULT_OUTPUT_ENCODING)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
452
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
453 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
454 if encoding:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
455 return self.encode(encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
456 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
457 return self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
458
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
459 class CData(NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
460
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
461 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
462 return "<![CDATA[%s]]>" % NavigableString.__str__(self, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
463
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
464 class ProcessingInstruction(NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
465 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
466 output = self
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
467 if "%SOUP-ENCODING%" in output:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
468 output = self.substituteEncoding(output, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
469 return "<?%s?>" % self.toEncoding(output, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
470
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
471 class Comment(NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
472 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
473 return "<!--%s-->" % NavigableString.__str__(self, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
474
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
475 class Declaration(NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
476 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
477 return "<!%s>" % NavigableString.__str__(self, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
478
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
479 class Tag(PageElement):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
480
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
481 """Represents a found HTML tag with its attributes and contents."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
482
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
483 def _invert(h):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
484 "Cheap function to invert a hash."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
485 i = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
486 for k,v in h.items():
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
487 i[v] = k
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
488 return i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
489
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
490 XML_ENTITIES_TO_SPECIAL_CHARS = { "apos" : "'",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
491 "quot" : '"',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
492 "amp" : "&",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
493 "lt" : "<",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
494 "gt" : ">" }
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
495
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
496 XML_SPECIAL_CHARS_TO_ENTITIES = _invert(XML_ENTITIES_TO_SPECIAL_CHARS)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
497
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
498 def _convertEntities(self, match):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
499 """Used in a call to re.sub to replace HTML, XML, and numeric
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
500 entities with the appropriate Unicode characters. If HTML
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
501 entities are being converted, any unrecognized entities are
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
502 escaped."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
503 x = match.group(1)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
504 if self.convertHTMLEntities and x in name2codepoint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
505 return unichr(name2codepoint[x])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
506 elif x in self.XML_ENTITIES_TO_SPECIAL_CHARS:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
507 if self.convertXMLEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
508 return self.XML_ENTITIES_TO_SPECIAL_CHARS[x]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
509 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
510 return u'&%s;' % x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
511 elif len(x) > 0 and x[0] == '#':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
512 # Handle numeric entities
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
513 if len(x) > 1 and x[1] == 'x':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
514 return unichr(int(x[2:], 16))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
515 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
516 return unichr(int(x[1:]))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
517
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
518 elif self.escapeUnrecognizedEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
519 return u'&amp;%s;' % x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
520 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
521 return u'&%s;' % x
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
522
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
523 def __init__(self, parser, name, attrs=None, parent=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
524 previous=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
525 "Basic constructor."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
526
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
527 # We don't actually store the parser object: that lets extracted
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
528 # chunks be garbage-collected
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
529 self.parserClass = parser.__class__
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
530 self.isSelfClosing = parser.isSelfClosingTag(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
531 self.name = name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
532 if attrs is None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
533 attrs = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
534 elif isinstance(attrs, dict):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
535 attrs = attrs.items()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
536 self.attrs = attrs
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
537 self.contents = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
538 self.setup(parent, previous)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
539 self.hidden = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
540 self.containsSubstitutions = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
541 self.convertHTMLEntities = parser.convertHTMLEntities
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
542 self.convertXMLEntities = parser.convertXMLEntities
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
543 self.escapeUnrecognizedEntities = parser.escapeUnrecognizedEntities
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
544
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
545 # Convert any HTML, XML, or numeric entities in the attribute values.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
546 convert = lambda(k, val): (k,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
547 re.sub("&(#\d+|#x[0-9a-fA-F]+|\w+);",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
548 self._convertEntities,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
549 val))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
550 self.attrs = map(convert, self.attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
551
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
552 def getString(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
553 if (len(self.contents) == 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
554 and isinstance(self.contents[0], NavigableString)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
555 return self.contents[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
556
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
557 def setString(self, string):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
558 """Replace the contents of the tag with a string"""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
559 self.clear()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
560 self.append(string)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
561
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
562 string = property(getString, setString)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
563
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
564 def getText(self, separator=u""):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
565 if not len(self.contents):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
566 return u""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
567 stopNode = self._lastRecursiveChild().next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
568 strings = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
569 current = self.contents[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
570 while current is not stopNode:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
571 if isinstance(current, NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
572 strings.append(current.strip())
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
573 current = current.next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
574 return separator.join(strings)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
575
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
576 text = property(getText)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
577
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
578 def get(self, key, default=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
579 """Returns the value of the 'key' attribute for the tag, or
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
580 the value given for 'default' if it doesn't have that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
581 attribute."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
582 return self._getAttrMap().get(key, default)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
583
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
584 def clear(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
585 """Extract all children."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
586 for child in self.contents[:]:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
587 child.extract()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
588
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
589 def index(self, element):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
590 for i, child in enumerate(self.contents):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
591 if child is element:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
592 return i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
593 raise ValueError("Tag.index: element not in tag")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
594
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
595 def has_key(self, key):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
596 return self._getAttrMap().has_key(key)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
597
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
598 def __getitem__(self, key):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
599 """tag[key] returns the value of the 'key' attribute for the tag,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
600 and throws an exception if it's not there."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
601 return self._getAttrMap()[key]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
602
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
603 def __iter__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
604 "Iterating over a tag iterates over its contents."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
605 return iter(self.contents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
606
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
607 def __len__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
608 "The length of a tag is the length of its list of contents."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
609 return len(self.contents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
610
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
611 def __contains__(self, x):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
612 return x in self.contents
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
613
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
614 def __nonzero__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
615 "A tag is non-None even if it has no contents."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
616 return True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
617
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
618 def __setitem__(self, key, value):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
619 """Setting tag[key] sets the value of the 'key' attribute for the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
620 tag."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
621 self._getAttrMap()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
622 self.attrMap[key] = value
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
623 found = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
624 for i in range(0, len(self.attrs)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
625 if self.attrs[i][0] == key:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
626 self.attrs[i] = (key, value)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
627 found = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
628 if not found:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
629 self.attrs.append((key, value))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
630 self._getAttrMap()[key] = value
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
631
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
632 def __delitem__(self, key):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
633 "Deleting tag[key] deletes all 'key' attributes for the tag."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
634 for item in self.attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
635 if item[0] == key:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
636 self.attrs.remove(item)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
637 #We don't break because bad HTML can define the same
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
638 #attribute multiple times.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
639 self._getAttrMap()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
640 if self.attrMap.has_key(key):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
641 del self.attrMap[key]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
642
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
643 def __call__(self, *args, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
644 """Calling a tag like a function is the same as calling its
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
645 findAll() method. Eg. tag('a') returns a list of all the A tags
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
646 found within this tag."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
647 return apply(self.findAll, args, kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
648
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
649 def __getattr__(self, tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
650 #print "Getattr %s.%s" % (self.__class__, tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
651 if len(tag) > 3 and tag.rfind('Tag') == len(tag)-3:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
652 return self.find(tag[:-3])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
653 elif tag.find('__') != 0:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
654 return self.find(tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
655 raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__, tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
656
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
657 def __eq__(self, other):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
658 """Returns true iff this tag has the same name, the same attributes,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
659 and the same contents (recursively) as the given tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
660
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
661 NOTE: right now this will return false if two tags have the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
662 same attributes in a different order. Should this be fixed?"""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
663 if other is self:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
664 return True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
665 if not hasattr(other, 'name') or not hasattr(other, 'attrs') or not hasattr(other, 'contents') or self.name != other.name or self.attrs != other.attrs or len(self) != len(other):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
666 return False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
667 for i in range(0, len(self.contents)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
668 if self.contents[i] != other.contents[i]:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
669 return False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
670 return True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
671
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
672 def __ne__(self, other):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
673 """Returns true iff this tag is not identical to the other tag,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
674 as defined in __eq__."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
675 return not self == other
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
676
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
677 def __repr__(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
678 """Renders this tag as a string."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
679 return self.__str__(encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
680
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
681 def __unicode__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
682 return self.__str__(None)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
683
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
684 BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
685 + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
686 + ")")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
687
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
688 def _sub_entity(self, x):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
689 """Used with a regular expression to substitute the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
690 appropriate XML entity for an XML special character."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
691 return "&" + self.XML_SPECIAL_CHARS_TO_ENTITIES[x.group(0)[0]] + ";"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
692
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
693 def __str__(self, encoding=DEFAULT_OUTPUT_ENCODING,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
694 prettyPrint=False, indentLevel=0):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
695 """Returns a string or Unicode representation of this tag and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
696 its contents. To get Unicode, pass None for encoding.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
697
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
698 NOTE: since Python's HTML parser consumes whitespace, this
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
699 method is not certain to reproduce the whitespace present in
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
700 the original string."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
701
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
702 encodedName = self.toEncoding(self.name, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
703
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
704 attrs = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
705 if self.attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
706 for key, val in self.attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
707 fmt = '%s="%s"'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
708 if isinstance(val, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
709 if self.containsSubstitutions and '%SOUP-ENCODING%' in val:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
710 val = self.substituteEncoding(val, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
711
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
712 # The attribute value either:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
713 #
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
714 # * Contains no embedded double quotes or single quotes.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
715 # No problem: we enclose it in double quotes.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
716 # * Contains embedded single quotes. No problem:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
717 # double quotes work here too.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
718 # * Contains embedded double quotes. No problem:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
719 # we enclose it in single quotes.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
720 # * Embeds both single _and_ double quotes. This
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
721 # can't happen naturally, but it can happen if
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
722 # you modify an attribute value after parsing
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
723 # the document. Now we have a bit of a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
724 # problem. We solve it by enclosing the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
725 # attribute in single quotes, and escaping any
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
726 # embedded single quotes to XML entities.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
727 if '"' in val:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
728 fmt = "%s='%s'"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
729 if "'" in val:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
730 # TODO: replace with apos when
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
731 # appropriate.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
732 val = val.replace("'", "&squot;")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
733
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
734 # Now we're okay w/r/t quotes. But the attribute
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
735 # value might also contain angle brackets, or
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
736 # ampersands that aren't part of entities. We need
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
737 # to escape those to XML entities too.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
738 val = self.BARE_AMPERSAND_OR_BRACKET.sub(self._sub_entity, val)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
739
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
740 attrs.append(fmt % (self.toEncoding(key, encoding),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
741 self.toEncoding(val, encoding)))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
742 close = ''
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
743 closeTag = ''
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
744 if self.isSelfClosing:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
745 close = ' /'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
746 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
747 closeTag = '</%s>' % encodedName
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
748
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
749 indentTag, indentContents = 0, 0
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
750 if prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
751 indentTag = indentLevel
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
752 space = (' ' * (indentTag-1))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
753 indentContents = indentTag + 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
754 contents = self.renderContents(encoding, prettyPrint, indentContents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
755 if self.hidden:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
756 s = contents
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
757 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
758 s = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
759 attributeString = ''
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
760 if attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
761 attributeString = ' ' + ' '.join(attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
762 if prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
763 s.append(space)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
764 s.append('<%s%s%s>' % (encodedName, attributeString, close))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
765 if prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
766 s.append("\n")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
767 s.append(contents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
768 if prettyPrint and contents and contents[-1] != "\n":
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
769 s.append("\n")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
770 if prettyPrint and closeTag:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
771 s.append(space)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
772 s.append(closeTag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
773 if prettyPrint and closeTag and self.nextSibling:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
774 s.append("\n")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
775 s = ''.join(s)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
776 return s
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
777
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
778 def decompose(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
779 """Recursively destroys the contents of this tree."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
780 self.extract()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
781 if len(self.contents) == 0:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
782 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
783 current = self.contents[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
784 while current is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
785 next = current.next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
786 if isinstance(current, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
787 del current.contents[:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
788 current.parent = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
789 current.previous = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
790 current.previousSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
791 current.next = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
792 current.nextSibling = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
793 current = next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
794
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
795 def prettify(self, encoding=DEFAULT_OUTPUT_ENCODING):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
796 return self.__str__(encoding, True)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
797
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
798 def renderContents(self, encoding=DEFAULT_OUTPUT_ENCODING,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
799 prettyPrint=False, indentLevel=0):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
800 """Renders the contents of this tag as a string in the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
801 encoding. If encoding is None, returns a Unicode string.."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
802 s=[]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
803 for c in self:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
804 text = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
805 if isinstance(c, NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
806 text = c.__str__(encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
807 elif isinstance(c, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
808 s.append(c.__str__(encoding, prettyPrint, indentLevel))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
809 if text and prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
810 text = text.strip()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
811 if text:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
812 if prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
813 s.append(" " * (indentLevel-1))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
814 s.append(text)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
815 if prettyPrint:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
816 s.append("\n")
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
817 return ''.join(s)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
818
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
819 #Soup methods
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
820
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
821 def find(self, name=None, attrs={}, recursive=True, text=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
822 **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
823 """Return only the first child of this Tag matching the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
824 criteria."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
825 r = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
826 l = self.findAll(name, attrs, recursive, text, 1, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
827 if l:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
828 r = l[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
829 return r
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
830 findChild = find
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
831
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
832 def findAll(self, name=None, attrs={}, recursive=True, text=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
833 limit=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
834 """Extracts a list of Tag objects that match the given
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
835 criteria. You can specify the name of the Tag and any
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
836 attributes you want the Tag to have.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
837
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
838 The value of a key-value pair in the 'attrs' map can be a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
839 string, a list of strings, a regular expression object, or a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
840 callable that takes a string and returns whether or not the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
841 string matches for some custom definition of 'matches'. The
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
842 same is true of the tag name."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
843 generator = self.recursiveChildGenerator
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
844 if not recursive:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
845 generator = self.childGenerator
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
846 return self._findAll(name, attrs, text, limit, generator, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
847 findChildren = findAll
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
848
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
849 # Pre-3.x compatibility methods
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
850 first = find
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
851 fetch = findAll
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
852
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
853 def fetchText(self, text=None, recursive=True, limit=None):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
854 return self.findAll(text=text, recursive=recursive, limit=limit)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
855
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
856 def firstText(self, text=None, recursive=True):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
857 return self.find(text=text, recursive=recursive)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
858
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
859 #Private methods
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
860
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
861 def _getAttrMap(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
862 """Initializes a map representation of this tag's attributes,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
863 if not already initialized."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
864 if not getattr(self, 'attrMap'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
865 self.attrMap = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
866 for (key, value) in self.attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
867 self.attrMap[key] = value
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
868 return self.attrMap
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
869
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
870 #Generator methods
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
871 def childGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
872 # Just use the iterator from the contents
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
873 return iter(self.contents)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
874
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
875 def recursiveChildGenerator(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
876 if not len(self.contents):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
877 raise StopIteration
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
878 stopNode = self._lastRecursiveChild().next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
879 current = self.contents[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
880 while current is not stopNode:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
881 yield current
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
882 current = current.next
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
883
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
884
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
885 # Next, a couple classes to represent queries and their results.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
886 class SoupStrainer:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
887 """Encapsulates a number of ways of matching a markup element (tag or
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
888 text)."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
889
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
890 def __init__(self, name=None, attrs={}, text=None, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
891 self.name = name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
892 if isinstance(attrs, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
893 kwargs['class'] = _match_css_class(attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
894 attrs = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
895 if kwargs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
896 if attrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
897 attrs = attrs.copy()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
898 attrs.update(kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
899 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
900 attrs = kwargs
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
901 self.attrs = attrs
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
902 self.text = text
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
903
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
904 def __str__(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
905 if self.text:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
906 return self.text
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
907 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
908 return "%s|%s" % (self.name, self.attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
909
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
910 def searchTag(self, markupName=None, markupAttrs={}):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
911 found = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
912 markup = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
913 if isinstance(markupName, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
914 markup = markupName
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
915 markupAttrs = markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
916 callFunctionWithTagData = callable(self.name) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
917 and not isinstance(markupName, Tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
918
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
919 if (not self.name) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
920 or callFunctionWithTagData \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
921 or (markup and self._matches(markup, self.name)) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
922 or (not markup and self._matches(markupName, self.name)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
923 if callFunctionWithTagData:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
924 match = self.name(markupName, markupAttrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
925 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
926 match = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
927 markupAttrMap = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
928 for attr, matchAgainst in self.attrs.items():
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
929 if not markupAttrMap:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
930 if hasattr(markupAttrs, 'get'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
931 markupAttrMap = markupAttrs
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
932 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
933 markupAttrMap = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
934 for k,v in markupAttrs:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
935 markupAttrMap[k] = v
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
936 attrValue = markupAttrMap.get(attr)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
937 if not self._matches(attrValue, matchAgainst):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
938 match = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
939 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
940 if match:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
941 if markup:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
942 found = markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
943 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
944 found = markupName
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
945 return found
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
946
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
947 def search(self, markup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
948 #print 'looking for %s in %s' % (self, markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
949 found = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
950 # If given a list of items, scan it for a text element that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
951 # matches.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
952 if hasattr(markup, "__iter__") \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
953 and not isinstance(markup, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
954 for element in markup:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
955 if isinstance(element, NavigableString) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
956 and self.search(element):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
957 found = element
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
958 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
959 # If it's a Tag, make sure its name or attributes match.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
960 # Don't bother with Tags if we're searching for text.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
961 elif isinstance(markup, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
962 if not self.text:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
963 found = self.searchTag(markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
964 # If it's text, make sure the text matches.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
965 elif isinstance(markup, NavigableString) or \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
966 isinstance(markup, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
967 if self._matches(markup, self.text):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
968 found = markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
969 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
970 raise Exception, "I don't know how to match against a %s" \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
971 % markup.__class__
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
972 return found
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
973
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
974 def _matches(self, markup, matchAgainst):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
975 #print "Matching %s against %s" % (markup, matchAgainst)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
976 result = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
977 if matchAgainst is True:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
978 result = markup is not None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
979 elif callable(matchAgainst):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
980 result = matchAgainst(markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
981 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
982 #Custom match methods take the tag as an argument, but all
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
983 #other ways of matching match the tag name as a string.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
984 if isinstance(markup, Tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
985 markup = markup.name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
986 if markup and not isinstance(markup, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
987 markup = unicode(markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
988 #Now we know that chunk is either a string, or None.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
989 if hasattr(matchAgainst, 'match'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
990 # It's a regexp object.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
991 result = markup and matchAgainst.search(markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
992 elif hasattr(matchAgainst, '__iter__'): # list-like
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
993 result = markup in matchAgainst
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
994 elif hasattr(matchAgainst, 'items'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
995 result = markup.has_key(matchAgainst)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
996 elif matchAgainst and isinstance(markup, basestring):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
997 if isinstance(markup, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
998 matchAgainst = unicode(matchAgainst)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
999 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1000 matchAgainst = str(matchAgainst)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1001
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1002 if not result:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1003 result = matchAgainst == markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1004 return result
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1005
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1006 class ResultSet(list):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1007 """A ResultSet is just a list that keeps track of the SoupStrainer
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1008 that created it."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1009 def __init__(self, source):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1010 list.__init__([])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1011 self.source = source
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1012
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1013 # Now, some helper functions.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1014
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1015 def buildTagMap(default, *args):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1016 """Turns a list of maps, lists, or scalars into a single map.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1017 Used to build the SELF_CLOSING_TAGS, NESTABLE_TAGS, and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1018 NESTING_RESET_TAGS maps out of lists and partial maps."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1019 built = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1020 for portion in args:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1021 if hasattr(portion, 'items'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1022 #It's a map. Merge it.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1023 for k,v in portion.items():
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1024 built[k] = v
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1025 elif hasattr(portion, '__iter__'): # is a list
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1026 #It's a list. Map each item to the default.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1027 for k in portion:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1028 built[k] = default
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1029 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1030 #It's a scalar. Map it to the default.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1031 built[portion] = default
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1032 return built
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1033
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1034 # Now, the parser classes.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1035
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1036 class BeautifulStoneSoup(Tag, SGMLParser):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1037
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1038 """This class contains the basic parser and search code. It defines
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1039 a parser that knows nothing about tag behavior except for the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1040 following:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1041
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1042 You can't close a tag without closing all the tags it encloses.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1043 That is, "<foo><bar></foo>" actually means
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1044 "<foo><bar></bar></foo>".
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1045
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1046 [Another possible explanation is "<foo><bar /></foo>", but since
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1047 this class defines no SELF_CLOSING_TAGS, it will never use that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1048 explanation.]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1049
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1050 This class is useful for parsing XML or made-up markup languages,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1051 or when BeautifulSoup makes an assumption counter to what you were
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1052 expecting."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1053
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1054 SELF_CLOSING_TAGS = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1055 NESTABLE_TAGS = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1056 RESET_NESTING_TAGS = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1057 QUOTE_TAGS = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1058 PRESERVE_WHITESPACE_TAGS = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1059
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1060 MARKUP_MASSAGE = [(re.compile('(<[^<>]*)/>'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1061 lambda x: x.group(1) + ' />'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1062 (re.compile('<!\s+([^<>]*)>'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1063 lambda x: '<!' + x.group(1) + '>')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1064 ]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1065
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1066 ROOT_TAG_NAME = u'[document]'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1067
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1068 HTML_ENTITIES = "html"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1069 XML_ENTITIES = "xml"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1070 XHTML_ENTITIES = "xhtml"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1071 # TODO: This only exists for backwards-compatibility
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1072 ALL_ENTITIES = XHTML_ENTITIES
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1073
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1074 # Used when determining whether a text node is all whitespace and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1075 # can be replaced with a single space. A text node that contains
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1076 # fancy Unicode spaces (usually non-breaking) should be left
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1077 # alone.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1078 STRIP_ASCII_SPACES = { 9: None, 10: None, 12: None, 13: None, 32: None, }
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1079
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1080 def __init__(self, markup="", parseOnlyThese=None, fromEncoding=None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1081 markupMassage=True, smartQuotesTo=XML_ENTITIES,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1082 convertEntities=None, selfClosingTags=None, isHTML=False):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1083 """The Soup object is initialized as the 'root tag', and the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1084 provided markup (which can be a string or a file-like object)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1085 is fed into the underlying parser.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1086
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1087 sgmllib will process most bad HTML, and the BeautifulSoup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1088 class has some tricks for dealing with some HTML that kills
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1089 sgmllib, but Beautiful Soup can nonetheless choke or lose data
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1090 if your data uses self-closing tags or declarations
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1091 incorrectly.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1092
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1093 By default, Beautiful Soup uses regexes to sanitize input,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1094 avoiding the vast majority of these problems. If the problems
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1095 don't apply to you, pass in False for markupMassage, and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1096 you'll get better performance.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1097
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1098 The default parser massage techniques fix the two most common
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1099 instances of invalid HTML that choke sgmllib:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1100
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1101 <br/> (No space between name of closing tag and tag close)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1102 <! --Comment--> (Extraneous whitespace in declaration)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1103
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1104 You can pass in a custom list of (RE object, replace method)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1105 tuples to get Beautiful Soup to scrub your input the way you
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1106 want."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1107
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1108 self.parseOnlyThese = parseOnlyThese
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1109 self.fromEncoding = fromEncoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1110 self.smartQuotesTo = smartQuotesTo
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1111 self.convertEntities = convertEntities
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1112 # Set the rules for how we'll deal with the entities we
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1113 # encounter
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1114 if self.convertEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1115 # It doesn't make sense to convert encoded characters to
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1116 # entities even while you're converting entities to Unicode.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1117 # Just convert it all to Unicode.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1118 self.smartQuotesTo = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1119 if convertEntities == self.HTML_ENTITIES:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1120 self.convertXMLEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1121 self.convertHTMLEntities = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1122 self.escapeUnrecognizedEntities = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1123 elif convertEntities == self.XHTML_ENTITIES:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1124 self.convertXMLEntities = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1125 self.convertHTMLEntities = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1126 self.escapeUnrecognizedEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1127 elif convertEntities == self.XML_ENTITIES:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1128 self.convertXMLEntities = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1129 self.convertHTMLEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1130 self.escapeUnrecognizedEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1131 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1132 self.convertXMLEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1133 self.convertHTMLEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1134 self.escapeUnrecognizedEntities = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1135
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1136 self.instanceSelfClosingTags = buildTagMap(None, selfClosingTags)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1137 SGMLParser.__init__(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1138
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1139 if hasattr(markup, 'read'): # It's a file-type object.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1140 markup = markup.read()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1141 self.markup = markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1142 self.markupMassage = markupMassage
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1143 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1144 self._feed(isHTML=isHTML)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1145 except StopParsing:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1146 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1147 self.markup = None # The markup can now be GCed
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1148
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1149 def convert_charref(self, name):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1150 """This method fixes a bug in Python's SGMLParser."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1151 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1152 n = int(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1153 except ValueError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1154 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1155 if not 0 <= n <= 127 : # ASCII ends at 127, not 255
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1156 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1157 return self.convert_codepoint(n)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1158
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1159 def _feed(self, inDocumentEncoding=None, isHTML=False):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1160 # Convert the document to Unicode.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1161 markup = self.markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1162 if isinstance(markup, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1163 if not hasattr(self, 'originalEncoding'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1164 self.originalEncoding = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1165 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1166 dammit = UnicodeDammit\
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1167 (markup, [self.fromEncoding, inDocumentEncoding],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1168 smartQuotesTo=self.smartQuotesTo, isHTML=isHTML)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1169 markup = dammit.unicode
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1170 self.originalEncoding = dammit.originalEncoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1171 self.declaredHTMLEncoding = dammit.declaredHTMLEncoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1172 if markup:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1173 if self.markupMassage:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1174 if not hasattr(self.markupMassage, "__iter__"):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1175 self.markupMassage = self.MARKUP_MASSAGE
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1176 for fix, m in self.markupMassage:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1177 markup = fix.sub(m, markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1178 # TODO: We get rid of markupMassage so that the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1179 # soup object can be deepcopied later on. Some
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1180 # Python installations can't copy regexes. If anyone
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1181 # was relying on the existence of markupMassage, this
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1182 # might cause problems.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1183 del(self.markupMassage)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1184 self.reset()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1185
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1186 SGMLParser.feed(self, markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1187 # Close out any unfinished strings and close all the open tags.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1188 self.endData()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1189 while self.currentTag.name != self.ROOT_TAG_NAME:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1190 self.popTag()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1191
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1192 def __getattr__(self, methodName):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1193 """This method routes method call requests to either the SGMLParser
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1194 superclass or the Tag superclass, depending on the method name."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1195 #print "__getattr__ called on %s.%s" % (self.__class__, methodName)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1196
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1197 if methodName.startswith('start_') or methodName.startswith('end_') \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1198 or methodName.startswith('do_'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1199 return SGMLParser.__getattr__(self, methodName)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1200 elif not methodName.startswith('__'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1201 return Tag.__getattr__(self, methodName)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1202 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1203 raise AttributeError
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1204
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1205 def isSelfClosingTag(self, name):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1206 """Returns true iff the given string is the name of a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1207 self-closing tag according to this parser."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1208 return self.SELF_CLOSING_TAGS.has_key(name) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1209 or self.instanceSelfClosingTags.has_key(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1210
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1211 def reset(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1212 Tag.__init__(self, self, self.ROOT_TAG_NAME)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1213 self.hidden = 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1214 SGMLParser.reset(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1215 self.currentData = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1216 self.currentTag = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1217 self.tagStack = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1218 self.quoteStack = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1219 self.pushTag(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1220
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1221 def popTag(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1222 tag = self.tagStack.pop()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1223
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1224 #print "Pop", tag.name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1225 if self.tagStack:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1226 self.currentTag = self.tagStack[-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1227 return self.currentTag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1228
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1229 def pushTag(self, tag):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1230 #print "Push", tag.name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1231 if self.currentTag:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1232 self.currentTag.contents.append(tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1233 self.tagStack.append(tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1234 self.currentTag = self.tagStack[-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1235
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1236 def endData(self, containerClass=NavigableString):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1237 if self.currentData:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1238 currentData = u''.join(self.currentData)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1239 if (currentData.translate(self.STRIP_ASCII_SPACES) == '' and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1240 not set([tag.name for tag in self.tagStack]).intersection(
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1241 self.PRESERVE_WHITESPACE_TAGS)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1242 if '\n' in currentData:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1243 currentData = '\n'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1244 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1245 currentData = ' '
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1246 self.currentData = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1247 if self.parseOnlyThese and len(self.tagStack) <= 1 and \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1248 (not self.parseOnlyThese.text or \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1249 not self.parseOnlyThese.search(currentData)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1250 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1251 o = containerClass(currentData)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1252 o.setup(self.currentTag, self.previous)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1253 if self.previous:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1254 self.previous.next = o
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1255 self.previous = o
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1256 self.currentTag.contents.append(o)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1257
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1258
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1259 def _popToTag(self, name, inclusivePop=True):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1260 """Pops the tag stack up to and including the most recent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1261 instance of the given tag. If inclusivePop is false, pops the tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1262 stack up to but *not* including the most recent instqance of
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1263 the given tag."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1264 #print "Popping to %s" % name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1265 if name == self.ROOT_TAG_NAME:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1266 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1267
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1268 numPops = 0
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1269 mostRecentTag = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1270 for i in range(len(self.tagStack)-1, 0, -1):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1271 if name == self.tagStack[i].name:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1272 numPops = len(self.tagStack)-i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1273 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1274 if not inclusivePop:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1275 numPops = numPops - 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1276
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1277 for i in range(0, numPops):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1278 mostRecentTag = self.popTag()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1279 return mostRecentTag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1280
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1281 def _smartPop(self, name):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1282
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1283 """We need to pop up to the previous tag of this type, unless
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1284 one of this tag's nesting reset triggers comes between this
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1285 tag and the previous tag of this type, OR unless this tag is a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1286 generic nesting trigger and another generic nesting trigger
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1287 comes between this tag and the previous tag of this type.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1288
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1289 Examples:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1290 <p>Foo<b>Bar *<p>* should pop to 'p', not 'b'.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1291 <p>Foo<table>Bar *<p>* should pop to 'table', not 'p'.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1292 <p>Foo<table><tr>Bar *<p>* should pop to 'tr', not 'p'.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1293
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1294 <li><ul><li> *<li>* should pop to 'ul', not the first 'li'.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1295 <tr><table><tr> *<tr>* should pop to 'table', not the first 'tr'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1296 <td><tr><td> *<td>* should pop to 'tr', not the first 'td'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1297 """
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1298
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1299 nestingResetTriggers = self.NESTABLE_TAGS.get(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1300 isNestable = nestingResetTriggers != None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1301 isResetNesting = self.RESET_NESTING_TAGS.has_key(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1302 popTo = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1303 inclusive = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1304 for i in range(len(self.tagStack)-1, 0, -1):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1305 p = self.tagStack[i]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1306 if (not p or p.name == name) and not isNestable:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1307 #Non-nestable tags get popped to the top or to their
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1308 #last occurance.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1309 popTo = name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1310 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1311 if (nestingResetTriggers is not None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1312 and p.name in nestingResetTriggers) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1313 or (nestingResetTriggers is None and isResetNesting
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1314 and self.RESET_NESTING_TAGS.has_key(p.name)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1315
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1316 #If we encounter one of the nesting reset triggers
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1317 #peculiar to this tag, or we encounter another tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1318 #that causes nesting to reset, pop up to but not
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1319 #including that tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1320 popTo = p.name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1321 inclusive = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1322 break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1323 p = p.parent
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1324 if popTo:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1325 self._popToTag(popTo, inclusive)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1326
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1327 def unknown_starttag(self, name, attrs, selfClosing=0):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1328 #print "Start tag %s: %s" % (name, attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1329 if self.quoteStack:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1330 #This is not a real tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1331 #print "<%s> is not real!" % name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1332 attrs = ''.join([' %s="%s"' % (x, y) for x, y in attrs])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1333 self.handle_data('<%s%s>' % (name, attrs))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1334 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1335 self.endData()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1336
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1337 if not self.isSelfClosingTag(name) and not selfClosing:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1338 self._smartPop(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1339
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1340 if self.parseOnlyThese and len(self.tagStack) <= 1 \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1341 and (self.parseOnlyThese.text or not self.parseOnlyThese.searchTag(name, attrs)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1342 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1343
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1344 tag = Tag(self, name, attrs, self.currentTag, self.previous)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1345 if self.previous:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1346 self.previous.next = tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1347 self.previous = tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1348 self.pushTag(tag)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1349 if selfClosing or self.isSelfClosingTag(name):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1350 self.popTag()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1351 if name in self.QUOTE_TAGS:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1352 #print "Beginning quote (%s)" % name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1353 self.quoteStack.append(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1354 self.literal = 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1355 return tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1356
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1357 def unknown_endtag(self, name):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1358 #print "End tag %s" % name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1359 if self.quoteStack and self.quoteStack[-1] != name:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1360 #This is not a real end tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1361 #print "</%s> is not real!" % name
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1362 self.handle_data('</%s>' % name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1363 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1364 self.endData()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1365 self._popToTag(name)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1366 if self.quoteStack and self.quoteStack[-1] == name:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1367 self.quoteStack.pop()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1368 self.literal = (len(self.quoteStack) > 0)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1369
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1370 def handle_data(self, data):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1371 self.currentData.append(data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1372
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1373 def _toStringSubclass(self, text, subclass):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1374 """Adds a certain piece of text to the tree as a NavigableString
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1375 subclass."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1376 self.endData()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1377 self.handle_data(text)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1378 self.endData(subclass)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1379
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1380 def handle_pi(self, text):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1381 """Handle a processing instruction as a ProcessingInstruction
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1382 object, possibly one with a %SOUP-ENCODING% slot into which an
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1383 encoding will be plugged later."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1384 if text[:3] == "xml":
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1385 text = u"xml version='1.0' encoding='%SOUP-ENCODING%'"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1386 self._toStringSubclass(text, ProcessingInstruction)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1387
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1388 def handle_comment(self, text):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1389 "Handle comments as Comment objects."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1390 self._toStringSubclass(text, Comment)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1391
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1392 def handle_charref(self, ref):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1393 "Handle character references as data."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1394 if self.convertEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1395 data = unichr(int(ref))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1396 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1397 data = '&#%s;' % ref
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1398 self.handle_data(data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1399
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1400 def handle_entityref(self, ref):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1401 """Handle entity references as data, possibly converting known
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1402 HTML and/or XML entity references to the corresponding Unicode
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1403 characters."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1404 data = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1405 if self.convertHTMLEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1406 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1407 data = unichr(name2codepoint[ref])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1408 except KeyError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1409 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1410
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1411 if not data and self.convertXMLEntities:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1412 data = self.XML_ENTITIES_TO_SPECIAL_CHARS.get(ref)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1413
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1414 if not data and self.convertHTMLEntities and \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1415 not self.XML_ENTITIES_TO_SPECIAL_CHARS.get(ref):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1416 # TODO: We've got a problem here. We're told this is
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1417 # an entity reference, but it's not an XML entity
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1418 # reference or an HTML entity reference. Nonetheless,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1419 # the logical thing to do is to pass it through as an
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1420 # unrecognized entity reference.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1421 #
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1422 # Except: when the input is "&carol;" this function
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1423 # will be called with input "carol". When the input is
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1424 # "AT&T", this function will be called with input
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1425 # "T". We have no way of knowing whether a semicolon
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1426 # was present originally, so we don't know whether
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1427 # this is an unknown entity or just a misplaced
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1428 # ampersand.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1429 #
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1430 # The more common case is a misplaced ampersand, so I
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1431 # escape the ampersand and omit the trailing semicolon.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1432 data = "&amp;%s" % ref
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1433 if not data:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1434 # This case is different from the one above, because we
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1435 # haven't already gone through a supposedly comprehensive
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1436 # mapping of entities to Unicode characters. We might not
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1437 # have gone through any mapping at all. So the chances are
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1438 # very high that this is a real entity, and not a
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1439 # misplaced ampersand.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1440 data = "&%s;" % ref
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1441 self.handle_data(data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1442
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1443 def handle_decl(self, data):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1444 "Handle DOCTYPEs and the like as Declaration objects."
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1445 self._toStringSubclass(data, Declaration)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1446
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1447 def parse_declaration(self, i):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1448 """Treat a bogus SGML declaration as raw data. Treat a CDATA
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1449 declaration as a CData object."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1450 j = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1451 if self.rawdata[i:i+9] == '<![CDATA[':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1452 k = self.rawdata.find(']]>', i)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1453 if k == -1:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1454 k = len(self.rawdata)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1455 data = self.rawdata[i+9:k]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1456 j = k+3
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1457 self._toStringSubclass(data, CData)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1458 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1459 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1460 j = SGMLParser.parse_declaration(self, i)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1461 except SGMLParseError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1462 toHandle = self.rawdata[i:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1463 self.handle_data(toHandle)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1464 j = i + len(toHandle)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1465 return j
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1466
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1467 class BeautifulSoup(BeautifulStoneSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1468
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1469 """This parser knows the following facts about HTML:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1470
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1471 * Some tags have no closing tag and should be interpreted as being
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1472 closed as soon as they are encountered.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1473
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1474 * The text inside some tags (ie. 'script') may contain tags which
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1475 are not really part of the document and which should be parsed
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1476 as text, not tags. If you want to parse the text as tags, you can
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1477 always fetch it and parse it explicitly.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1478
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1479 * Tag nesting rules:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1480
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1481 Most tags can't be nested at all. For instance, the occurance of
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1482 a <p> tag should implicitly close the previous <p> tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1483
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1484 <p>Para1<p>Para2
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1485 should be transformed into:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1486 <p>Para1</p><p>Para2
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1487
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1488 Some tags can be nested arbitrarily. For instance, the occurance
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1489 of a <blockquote> tag should _not_ implicitly close the previous
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1490 <blockquote> tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1491
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1492 Alice said: <blockquote>Bob said: <blockquote>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1493 should NOT be transformed into:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1494 Alice said: <blockquote>Bob said: </blockquote><blockquote>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1495
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1496 Some tags can be nested, but the nesting is reset by the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1497 interposition of other tags. For instance, a <tr> tag should
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1498 implicitly close the previous <tr> tag within the same <table>,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1499 but not close a <tr> tag in another table.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1500
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1501 <table><tr>Blah<tr>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1502 should be transformed into:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1503 <table><tr>Blah</tr><tr>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1504 but,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1505 <tr>Blah<table><tr>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1506 should NOT be transformed into
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1507 <tr>Blah<table></tr><tr>Blah
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1508
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1509 Differing assumptions about tag nesting rules are a major source
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1510 of problems with the BeautifulSoup class. If BeautifulSoup is not
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1511 treating as nestable a tag your page author treats as nestable,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1512 try ICantBelieveItsBeautifulSoup, MinimalSoup, or
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1513 BeautifulStoneSoup before writing your own subclass."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1514
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1515 def __init__(self, *args, **kwargs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1516 if not kwargs.has_key('smartQuotesTo'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1517 kwargs['smartQuotesTo'] = self.HTML_ENTITIES
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1518 kwargs['isHTML'] = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1519 BeautifulStoneSoup.__init__(self, *args, **kwargs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1520
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1521 SELF_CLOSING_TAGS = buildTagMap(None,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1522 ('br' , 'hr', 'input', 'img', 'meta',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1523 'spacer', 'link', 'frame', 'base', 'col'))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1524
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1525 PRESERVE_WHITESPACE_TAGS = set(['pre', 'textarea'])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1526
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1527 QUOTE_TAGS = {'script' : None, 'textarea' : None}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1528
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1529 #According to the HTML standard, each of these inline tags can
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1530 #contain another tag of the same type. Furthermore, it's common
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1531 #to actually use these tags this way.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1532 NESTABLE_INLINE_TAGS = ('span', 'font', 'q', 'object', 'bdo', 'sub', 'sup',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1533 'center')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1534
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1535 #According to the HTML standard, these block tags can contain
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1536 #another tag of the same type. Furthermore, it's common
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1537 #to actually use these tags this way.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1538 NESTABLE_BLOCK_TAGS = ('blockquote', 'div', 'fieldset', 'ins', 'del')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1539
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1540 #Lists can contain other lists, but there are restrictions.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1541 NESTABLE_LIST_TAGS = { 'ol' : [],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1542 'ul' : [],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1543 'li' : ['ul', 'ol'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1544 'dl' : [],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1545 'dd' : ['dl'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1546 'dt' : ['dl'] }
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1547
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1548 #Tables can contain other tables, but there are restrictions.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1549 NESTABLE_TABLE_TAGS = {'table' : [],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1550 'tr' : ['table', 'tbody', 'tfoot', 'thead'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1551 'td' : ['tr'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1552 'th' : ['tr'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1553 'thead' : ['table'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1554 'tbody' : ['table'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1555 'tfoot' : ['table'],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1556 }
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1557
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1558 NON_NESTABLE_BLOCK_TAGS = ('address', 'form', 'p', 'pre')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1559
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1560 #If one of these tags is encountered, all tags up to the next tag of
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1561 #this type are popped.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1562 RESET_NESTING_TAGS = buildTagMap(None, NESTABLE_BLOCK_TAGS, 'noscript',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1563 NON_NESTABLE_BLOCK_TAGS,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1564 NESTABLE_LIST_TAGS,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1565 NESTABLE_TABLE_TAGS)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1566
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1567 NESTABLE_TAGS = buildTagMap([], NESTABLE_INLINE_TAGS, NESTABLE_BLOCK_TAGS,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1568 NESTABLE_LIST_TAGS, NESTABLE_TABLE_TAGS)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1569
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1570 # Used to detect the charset in a META tag; see start_meta
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1571 CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1572
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1573 def start_meta(self, attrs):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1574 """Beautiful Soup can detect a charset included in a META tag,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1575 try to convert the document to that charset, and re-parse the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1576 document from the beginning."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1577 httpEquiv = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1578 contentType = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1579 contentTypeIndex = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1580 tagNeedsEncodingSubstitution = False
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1581
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1582 for i in range(0, len(attrs)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1583 key, value = attrs[i]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1584 key = key.lower()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1585 if key == 'http-equiv':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1586 httpEquiv = value
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1587 elif key == 'content':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1588 contentType = value
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1589 contentTypeIndex = i
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1590
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1591 if httpEquiv and contentType: # It's an interesting meta tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1592 match = self.CHARSET_RE.search(contentType)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1593 if match:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1594 if (self.declaredHTMLEncoding is not None or
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1595 self.originalEncoding == self.fromEncoding):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1596 # An HTML encoding was sniffed while converting
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1597 # the document to Unicode, or an HTML encoding was
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1598 # sniffed during a previous pass through the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1599 # document, or an encoding was specified
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1600 # explicitly and it worked. Rewrite the meta tag.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1601 def rewrite(match):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1602 return match.group(1) + "%SOUP-ENCODING%"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1603 newAttr = self.CHARSET_RE.sub(rewrite, contentType)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1604 attrs[contentTypeIndex] = (attrs[contentTypeIndex][0],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1605 newAttr)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1606 tagNeedsEncodingSubstitution = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1607 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1608 # This is our first pass through the document.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1609 # Go through it again with the encoding information.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1610 newCharset = match.group(3)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1611 if newCharset and newCharset != self.originalEncoding:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1612 self.declaredHTMLEncoding = newCharset
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1613 self._feed(self.declaredHTMLEncoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1614 raise StopParsing
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1615 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1616 tag = self.unknown_starttag("meta", attrs)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1617 if tag and tagNeedsEncodingSubstitution:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1618 tag.containsSubstitutions = True
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1619
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1620 class StopParsing(Exception):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1621 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1622
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1623 class ICantBelieveItsBeautifulSoup(BeautifulSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1624
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1625 """The BeautifulSoup class is oriented towards skipping over
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1626 common HTML errors like unclosed tags. However, sometimes it makes
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1627 errors of its own. For instance, consider this fragment:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1628
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1629 <b>Foo<b>Bar</b></b>
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1630
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1631 This is perfectly valid (if bizarre) HTML. However, the
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1632 BeautifulSoup class will implicitly close the first b tag when it
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1633 encounters the second 'b'. It will think the author wrote
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1634 "<b>Foo<b>Bar", and didn't close the first 'b' tag, because
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1635 there's no real-world reason to bold something that's already
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1636 bold. When it encounters '</b></b>' it will close two more 'b'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1637 tags, for a grand total of three tags closed instead of two. This
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1638 can throw off the rest of your document structure. The same is
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1639 true of a number of other tags, listed below.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1640
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1641 It's much more common for someone to forget to close a 'b' tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1642 than to actually use nested 'b' tags, and the BeautifulSoup class
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1643 handles the common case. This class handles the not-co-common
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1644 case: where you can't believe someone wrote what they did, but
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1645 it's valid HTML and BeautifulSoup screwed up by assuming it
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1646 wouldn't be."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1647
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1648 I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGS = \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1649 ('em', 'big', 'i', 'small', 'tt', 'abbr', 'acronym', 'strong',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1650 'cite', 'code', 'dfn', 'kbd', 'samp', 'strong', 'var', 'b',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1651 'big')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1652
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1653 I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGS = ('noscript',)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1654
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1655 NESTABLE_TAGS = buildTagMap([], BeautifulSoup.NESTABLE_TAGS,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1656 I_CANT_BELIEVE_THEYRE_NESTABLE_BLOCK_TAGS,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1657 I_CANT_BELIEVE_THEYRE_NESTABLE_INLINE_TAGS)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1658
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1659 class MinimalSoup(BeautifulSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1660 """The MinimalSoup class is for parsing HTML that contains
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1661 pathologically bad markup. It makes no assumptions about tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1662 nesting, but it does know which tags are self-closing, that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1663 <script> tags contain Javascript and should not be parsed, that
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1664 META tags may contain encoding information, and so on.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1665
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1666 This also makes it better for subclassing than BeautifulStoneSoup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1667 or BeautifulSoup."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1668
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1669 RESET_NESTING_TAGS = buildTagMap('noscript')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1670 NESTABLE_TAGS = {}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1671
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1672 class BeautifulSOAP(BeautifulStoneSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1673 """This class will push a tag with only a single string child into
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1674 the tag's parent as an attribute. The attribute's name is the tag
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1675 name, and the value is the string child. An example should give
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1676 the flavor of the change:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1677
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1678 <foo><bar>baz</bar></foo>
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1679 =>
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1680 <foo bar="baz"><bar>baz</bar></foo>
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1681
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1682 You can then access fooTag['bar'] instead of fooTag.barTag.string.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1683
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1684 This is, of course, useful for scraping structures that tend to
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1685 use subelements instead of attributes, such as SOAP messages. Note
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1686 that it modifies its input, so don't print the modified version
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1687 out.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1688
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1689 I'm not sure how many people really want to use this class; let me
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1690 know if you do. Mainly I like the name."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1691
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1692 def popTag(self):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1693 if len(self.tagStack) > 1:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1694 tag = self.tagStack[-1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1695 parent = self.tagStack[-2]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1696 parent._getAttrMap()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1697 if (isinstance(tag, Tag) and len(tag.contents) == 1 and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1698 isinstance(tag.contents[0], NavigableString) and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1699 not parent.attrMap.has_key(tag.name)):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1700 parent[tag.name] = tag.contents[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1701 BeautifulStoneSoup.popTag(self)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1702
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1703 #Enterprise class names! It has come to our attention that some people
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1704 #think the names of the Beautiful Soup parser classes are too silly
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1705 #and "unprofessional" for use in enterprise screen-scraping. We feel
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1706 #your pain! For such-minded folk, the Beautiful Soup Consortium And
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1707 #All-Night Kosher Bakery recommends renaming this file to
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1708 #"RobustParser.py" (or, in cases of extreme enterprisiness,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1709 #"RobustParserBeanInterface.class") and using the following
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1710 #enterprise-friendly class aliases:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1711 class RobustXMLParser(BeautifulStoneSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1712 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1713 class RobustHTMLParser(BeautifulSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1714 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1715 class RobustWackAssHTMLParser(ICantBelieveItsBeautifulSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1716 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1717 class RobustInsanelyWackAssHTMLParser(MinimalSoup):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1718 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1719 class SimplifyingSOAPParser(BeautifulSOAP):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1720 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1721
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1722 ######################################################
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1723 #
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1724 # Bonus library: Unicode, Dammit
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1725 #
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1726 # This class forces XML data into a standard format (usually to UTF-8
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1727 # or Unicode). It is heavily based on code from Mark Pilgrim's
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1728 # Universal Feed Parser. It does not rewrite the XML or HTML to
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1729 # reflect a new encoding: that happens in BeautifulStoneSoup.handle_pi
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1730 # (XML) and BeautifulSoup.start_meta (HTML).
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1731
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1732 # Autodetects character encodings.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1733 # Download from http://chardet.feedparser.org/
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1734 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1735 import chardet
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1736 # import chardet.constants
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1737 # chardet.constants._debug = 1
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1738 except ImportError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1739 chardet = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1740
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1741 # cjkcodecs and iconv_codec make Python know about more character encodings.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1742 # Both are available from http://cjkpython.i18n.org/
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1743 # They're built in if you use Python 2.4.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1744 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1745 import cjkcodecs.aliases
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1746 except ImportError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1747 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1748 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1749 import iconv_codec
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1750 except ImportError:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1751 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1752
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1753 class UnicodeDammit:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1754 """A class for detecting the encoding of a *ML document and
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1755 converting it to a Unicode string. If the source encoding is
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1756 windows-1252, can replace MS smart quotes with their HTML or XML
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1757 equivalents."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1758
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1759 # This dictionary maps commonly seen values for "charset" in HTML
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1760 # meta tags to the corresponding Python codec names. It only covers
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1761 # values that aren't in Python's aliases and can't be determined
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1762 # by the heuristics in find_codec.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1763 CHARSET_ALIASES = { "macintosh" : "mac-roman",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1764 "x-sjis" : "shift-jis" }
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1765
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1766 def __init__(self, markup, overrideEncodings=[],
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1767 smartQuotesTo='xml', isHTML=False):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1768 self.declaredHTMLEncoding = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1769 self.markup, documentEncoding, sniffedEncoding = \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1770 self._detectEncoding(markup, isHTML)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1771 self.smartQuotesTo = smartQuotesTo
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1772 self.triedEncodings = []
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1773 if markup == '' or isinstance(markup, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1774 self.originalEncoding = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1775 self.unicode = unicode(markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1776 return
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1777
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1778 u = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1779 for proposedEncoding in overrideEncodings:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1780 u = self._convertFrom(proposedEncoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1781 if u: break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1782 if not u:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1783 for proposedEncoding in (documentEncoding, sniffedEncoding):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1784 u = self._convertFrom(proposedEncoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1785 if u: break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1786
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1787 # If no luck and we have auto-detection library, try that:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1788 if not u and chardet and not isinstance(self.markup, unicode):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1789 u = self._convertFrom(chardet.detect(self.markup)['encoding'])
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1790
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1791 # As a last resort, try utf-8 and windows-1252:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1792 if not u:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1793 for proposed_encoding in ("utf-8", "windows-1252"):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1794 u = self._convertFrom(proposed_encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1795 if u: break
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1796
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1797 self.unicode = u
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1798 if not u: self.originalEncoding = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1799
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1800 def _subMSChar(self, orig):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1801 """Changes a MS smart quote character to an XML or HTML
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1802 entity."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1803 sub = self.MS_CHARS.get(orig)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1804 if isinstance(sub, tuple):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1805 if self.smartQuotesTo == 'xml':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1806 sub = '&#x%s;' % sub[1]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1807 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1808 sub = '&%s;' % sub[0]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1809 return sub
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1810
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1811 def _convertFrom(self, proposed):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1812 proposed = self.find_codec(proposed)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1813 if not proposed or proposed in self.triedEncodings:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1814 return None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1815 self.triedEncodings.append(proposed)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1816 markup = self.markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1817
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1818 # Convert smart quotes to HTML if coming from an encoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1819 # that might have them.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1820 if self.smartQuotesTo and proposed.lower() in("windows-1252",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1821 "iso-8859-1",
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1822 "iso-8859-2"):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1823 markup = re.compile("([\x80-\x9f])").sub \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1824 (lambda(x): self._subMSChar(x.group(1)),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1825 markup)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1826
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1827 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1828 # print "Trying to convert document to %s" % proposed
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1829 u = self._toUnicode(markup, proposed)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1830 self.markup = u
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1831 self.originalEncoding = proposed
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1832 except Exception, e:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1833 # print "That didn't work!"
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1834 # print e
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1835 return None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1836 #print "Correct encoding: %s" % proposed
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1837 return self.markup
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1838
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1839 def _toUnicode(self, data, encoding):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1840 '''Given a string and its encoding, decodes the string into Unicode.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1841 %encoding is a string recognized by encodings.aliases'''
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1842
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1843 # strip Byte Order Mark (if present)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1844 if (len(data) >= 4) and (data[:2] == '\xfe\xff') \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1845 and (data[2:4] != '\x00\x00'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1846 encoding = 'utf-16be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1847 data = data[2:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1848 elif (len(data) >= 4) and (data[:2] == '\xff\xfe') \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1849 and (data[2:4] != '\x00\x00'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1850 encoding = 'utf-16le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1851 data = data[2:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1852 elif data[:3] == '\xef\xbb\xbf':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1853 encoding = 'utf-8'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1854 data = data[3:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1855 elif data[:4] == '\x00\x00\xfe\xff':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1856 encoding = 'utf-32be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1857 data = data[4:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1858 elif data[:4] == '\xff\xfe\x00\x00':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1859 encoding = 'utf-32le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1860 data = data[4:]
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1861 newdata = unicode(data, encoding)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1862 return newdata
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1863
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1864 def _detectEncoding(self, xml_data, isHTML=False):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1865 """Given a document, tries to detect its XML encoding."""
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1866 xml_encoding = sniffed_xml_encoding = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1867 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1868 if xml_data[:4] == '\x4c\x6f\xa7\x94':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1869 # EBCDIC
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1870 xml_data = self._ebcdic_to_ascii(xml_data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1871 elif xml_data[:4] == '\x00\x3c\x00\x3f':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1872 # UTF-16BE
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1873 sniffed_xml_encoding = 'utf-16be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1874 xml_data = unicode(xml_data, 'utf-16be').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1875 elif (len(xml_data) >= 4) and (xml_data[:2] == '\xfe\xff') \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1876 and (xml_data[2:4] != '\x00\x00'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1877 # UTF-16BE with BOM
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1878 sniffed_xml_encoding = 'utf-16be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1879 xml_data = unicode(xml_data[2:], 'utf-16be').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1880 elif xml_data[:4] == '\x3c\x00\x3f\x00':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1881 # UTF-16LE
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1882 sniffed_xml_encoding = 'utf-16le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1883 xml_data = unicode(xml_data, 'utf-16le').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1884 elif (len(xml_data) >= 4) and (xml_data[:2] == '\xff\xfe') and \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1885 (xml_data[2:4] != '\x00\x00'):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1886 # UTF-16LE with BOM
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1887 sniffed_xml_encoding = 'utf-16le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1888 xml_data = unicode(xml_data[2:], 'utf-16le').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1889 elif xml_data[:4] == '\x00\x00\x00\x3c':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1890 # UTF-32BE
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1891 sniffed_xml_encoding = 'utf-32be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1892 xml_data = unicode(xml_data, 'utf-32be').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1893 elif xml_data[:4] == '\x3c\x00\x00\x00':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1894 # UTF-32LE
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1895 sniffed_xml_encoding = 'utf-32le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1896 xml_data = unicode(xml_data, 'utf-32le').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1897 elif xml_data[:4] == '\x00\x00\xfe\xff':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1898 # UTF-32BE with BOM
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1899 sniffed_xml_encoding = 'utf-32be'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1900 xml_data = unicode(xml_data[4:], 'utf-32be').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1901 elif xml_data[:4] == '\xff\xfe\x00\x00':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1902 # UTF-32LE with BOM
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1903 sniffed_xml_encoding = 'utf-32le'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1904 xml_data = unicode(xml_data[4:], 'utf-32le').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1905 elif xml_data[:3] == '\xef\xbb\xbf':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1906 # UTF-8 with BOM
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1907 sniffed_xml_encoding = 'utf-8'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1908 xml_data = unicode(xml_data[3:], 'utf-8').encode('utf-8')
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1909 else:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1910 sniffed_xml_encoding = 'ascii'
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1911 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1912 except:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1913 xml_encoding_match = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1914 xml_encoding_match = re.compile(
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1915 '^<\?.*encoding=[\'"](.*?)[\'"].*\?>').match(xml_data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1916 if not xml_encoding_match and isHTML:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1917 regexp = re.compile('<\s*meta[^>]+charset=([^>]*?)[;\'">]', re.I)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1918 xml_encoding_match = regexp.search(xml_data)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1919 if xml_encoding_match is not None:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1920 xml_encoding = xml_encoding_match.groups()[0].lower()
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1921 if isHTML:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1922 self.declaredHTMLEncoding = xml_encoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1923 if sniffed_xml_encoding and \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1924 (xml_encoding in ('iso-10646-ucs-2', 'ucs-2', 'csunicode',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1925 'iso-10646-ucs-4', 'ucs-4', 'csucs4',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1926 'utf-16', 'utf-32', 'utf_16', 'utf_32',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1927 'utf16', 'u16')):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1928 xml_encoding = sniffed_xml_encoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1929 return xml_data, xml_encoding, sniffed_xml_encoding
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1930
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1931
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1932 def find_codec(self, charset):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1933 return self._codec(self.CHARSET_ALIASES.get(charset, charset)) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1934 or (charset and self._codec(charset.replace("-", ""))) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1935 or (charset and self._codec(charset.replace("-", "_"))) \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1936 or charset
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1937
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1938 def _codec(self, charset):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1939 if not charset: return charset
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1940 codec = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1941 try:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1942 codecs.lookup(charset)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1943 codec = charset
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1944 except (LookupError, ValueError):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1945 pass
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1946 return codec
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1947
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1948 EBCDIC_TO_ASCII_MAP = None
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1949 def _ebcdic_to_ascii(self, s):
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1950 c = self.__class__
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1951 if not c.EBCDIC_TO_ASCII_MAP:
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1952 emap = (0,1,2,3,156,9,134,127,151,141,142,11,12,13,14,15,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1953 16,17,18,19,157,133,8,135,24,25,146,143,28,29,30,31,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1954 128,129,130,131,132,10,23,27,136,137,138,139,140,5,6,7,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1955 144,145,22,147,148,149,150,4,152,153,154,155,20,21,158,26,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1956 32,160,161,162,163,164,165,166,167,168,91,46,60,40,43,33,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1957 38,169,170,171,172,173,174,175,176,177,93,36,42,41,59,94,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1958 45,47,178,179,180,181,182,183,184,185,124,44,37,95,62,63,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1959 186,187,188,189,190,191,192,193,194,96,58,35,64,39,61,34,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1960 195,97,98,99,100,101,102,103,104,105,196,197,198,199,200,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1961 201,202,106,107,108,109,110,111,112,113,114,203,204,205,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1962 206,207,208,209,126,115,116,117,118,119,120,121,122,210,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1963 211,212,213,214,215,216,217,218,219,220,221,222,223,224,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1964 225,226,227,228,229,230,231,123,65,66,67,68,69,70,71,72,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1965 73,232,233,234,235,236,237,125,74,75,76,77,78,79,80,81,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1966 82,238,239,240,241,242,243,92,159,83,84,85,86,87,88,89,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1967 90,244,245,246,247,248,249,48,49,50,51,52,53,54,55,56,57,
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1968 250,251,252,253,254,255)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1969 import string
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1970 c.EBCDIC_TO_ASCII_MAP = string.maketrans( \
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1971 ''.join(map(chr, range(256))), ''.join(map(chr, emap)))
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1972 return s.translate(c.EBCDIC_TO_ASCII_MAP)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1973
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1974 MS_CHARS = { '\x80' : ('euro', '20AC'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1975 '\x81' : ' ',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1976 '\x82' : ('sbquo', '201A'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1977 '\x83' : ('fnof', '192'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1978 '\x84' : ('bdquo', '201E'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1979 '\x85' : ('hellip', '2026'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1980 '\x86' : ('dagger', '2020'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1981 '\x87' : ('Dagger', '2021'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1982 '\x88' : ('circ', '2C6'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1983 '\x89' : ('permil', '2030'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1984 '\x8A' : ('Scaron', '160'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1985 '\x8B' : ('lsaquo', '2039'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1986 '\x8C' : ('OElig', '152'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1987 '\x8D' : '?',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1988 '\x8E' : ('#x17D', '17D'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1989 '\x8F' : '?',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1990 '\x90' : '?',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1991 '\x91' : ('lsquo', '2018'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1992 '\x92' : ('rsquo', '2019'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1993 '\x93' : ('ldquo', '201C'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1994 '\x94' : ('rdquo', '201D'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1995 '\x95' : ('bull', '2022'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1996 '\x96' : ('ndash', '2013'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1997 '\x97' : ('mdash', '2014'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1998 '\x98' : ('tilde', '2DC'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
1999 '\x99' : ('trade', '2122'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2000 '\x9a' : ('scaron', '161'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2001 '\x9b' : ('rsaquo', '203A'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2002 '\x9c' : ('oelig', '153'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2003 '\x9d' : '?',
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2004 '\x9e' : ('#x17E', '17E'),
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2005 '\x9f' : ('Yuml', ''),}
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2006
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2007 #######################################################################
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2008
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2009
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2010 #By default, act as an HTML pretty-printer.
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2011 if __name__ == '__main__':
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2012 import sys
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2013 soup = BeautifulSoup(sys.stdin)
2c498d40ecde Uploaded
miller-lab
parents:
diff changeset
2014 print soup.prettify()