Bs4 Library In Python|| Import Error: No Module Named ‘Bs4’ || Python3 | From Bs4 Import Beautifulsoup 7 Câu Trả Lời Chính Xác Nhất

Python Programming Tutorials

Web scraping and parsing with Beautiful Soup 4 Introduction

Welcome to a tutorial on web scraping with Beautiful Soup 4. Beautiful Soup is a Python library aimed at helping programmers who are trying to scrape data from websites.

To use beautiful soup, you need to install it: $ pip install beautifulsoup4 . Beautiful Soup also relies on a parser, the default is lxml . You may already have it, but you should check (open IDLE and attempt to import lxml). If not, do: $ pip install lxml or $ apt-get install python-lxml .

To begin, we need HTML. I have created an example page for us to work with.

To begin, we need to import Beautiful Soup and urllib, and grab source code:

import bs4 as bs import urllib.request source = urllib.request.urlopen(‘https://pythonprogramming.net/parsememcparseface/’).read()

Then, we create the “soup.” This is a beautiful soup object:

soup = bs.BeautifulSoup(source,’lxml’)

If you do print(soup) and print(source) , it looks the same, but the source is just plain the response data, and the soup is an object that we can actually interact with, by tag, now, like so:

# title of the page print(soup.title) # get attributes: print(soup.title.name) # get values: print(soup.title.string) # beginning navigation: print(soup.title.parent.name) # getting specific values: print(soup.p)

Finding paragraph tags

is a fairly common task. In the case above, we’re just finding the first one. What if we wanted to find them all?

print(soup.find_all(‘p’))

We can also iterate through them:

for paragraph in soup.find_all(‘p’): print(paragraph.string) print(str(paragraph.text))

The difference between string and text is that string produces a NavigableString object, and text is just typical unicode text. Notice that, if there are child tags in the paragraph item that we’re attempting to use .string on, we will get None returned.

Another common task is to grab links. For example:

for url in soup.find_all(‘a’): print(url.get(‘href’))

In this case, if we just grabbed the .text from the tag, you’d get the anchor text, but we actually want the link itself. That’s why we’re using .get(‘href’) to get the true URL.

Finally, you may just want to grab text. You can use .get_text() on a Beautiful Soup object, including the full soup:

print(soup.get_text())

This concludes the introduction to Beautiful Soup. In the next tutorial, we’re going cover navigating a page’s elements to get more specifically what you want.

Beautifulsoup Installation

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work. The latest Version of Beautifulsoup is v4.9.3 as of now.

Prerequisites

How to install Beautifulsoup?

To install Beautifulsoup on Windows, Linux, or any operating system, one would need pip package. To check how to install pip on your operating system, check out – PIP Installation – Windows || Linux.

Now, run a simple command,

pip install beautifulsoup4

Wait and relax, Beautifulsoup would be installed shortly.

Install Beautifulsoup4 using Source code

One can install beautifulsoup, using source code directly, install beautifulsoup tarball from here – download the Beautiful Soup 4 source tarball

after downloading cd into the directory and run,

Python setup.py install

Verifying Installation

To check whether the installation is complete or not, let’s try implementing it using python

What is the difference beautifulsoup and bs4

I’m new to python and I tried to parse some XML files in order to add some new tags and store that new XML file.

python-beautifulsoup seams to be the right package for that. Searching around the web for tutorials, how to add an new tag to XML parsed by BeautifulSoup, i found out, that the package python-bs4 is used.

Looking at the package description, both packages have the same title:

python-bs4 – error-tolerant HTML parser for Python python-beautifulsoup – error-tolerant HTML parser for Python

So my question: what is the difference?

Getting Started with Beautiful Soup [Book]

Get full access to Getting Started with Beautiful Soup and 60K+ other titles, with free 10-day trial of O’Reilly.

There’s also live online events, interactive content, certification prep materials, and more.

Beautiful Soup – Installation

Beautiful Soup – Installation

Advertisements

As BeautifulSoup is not a standard python library, we need to install it first. We are going to install the BeautifulSoup 4 library (also known as BS4), which is the latest one.

To isolate our working environment so as not to disturb the existing setup, let us first create a virtual environment.

Creating a virtual environment (optional)

A virtual environment allows us to create an isolated working copy of python for a specific project without affecting the outside setup.

Best way to install any python package machine is using pip, however, if pip is not installed already (you can check it using – “pip –version” in your command or shell prompt), you can install by giving below command −

Linux environment

$sudo apt-get install python-pip

Windows environment

To install pip in windows, do the following −

Download the get-pip.py from https://bootstrap.pypa.io/get-pip.py or from the github to your computer.

Open the command prompt and navigate to the folder containing get-pip.py file.

Run the following command −

>python get-pip.py

That’s it, pip is now installed in your windows machine.

You can verify your pip installed by running below command −

>pip –version pip 19.2.3 from c:\users\yadur\appdata\local\programs\python\python37\lib\site-packages\pip (python 3.7)

Installing virtual environment

Run the below command in your command prompt −

>pip install virtualenv

After running, you will see the below screenshot −

Below command will create a virtual environment (“myEnv”) in your current directory −

>virtualenv myEnv

Screenshot

To activate your virtual environment, run the following command −

>myEnv\Scripts\activate

In the above screenshot, you can see we have “myEnv” as prefix which tells us that we are under virtual environment “myEnv”.

To come out of virtual environment, run deactivate.

(myEnv) C:\Users\yadur>deactivate C:\Users\yadur>

As our virtual environment is ready, now let us install beautifulsoup.

Installing BeautifulSoup

As BeautifulSoup is not a standard library, we need to install it. We are going to use the BeautifulSoup 4 package (known as bs4).

Linux Machine

To install bs4 on Debian or Ubuntu linux using system package manager, run the below command −

$sudo apt-get install python-bs4 (for python 2.x) $sudo apt-get install python3-bs4 (for python 3.x)

You can install bs4 using easy_install or pip (in case you find problem in installing using system packager).

$easy_install beautifulsoup4 $pip install beautifulsoup4

(You may need to use easy_install3 or pip3 respectively if you’re using python3)

Windows Machine

To install beautifulsoup4 in windows is very simple, especially if you have pip already installed.

>pip install beautifulsoup4

So now beautifulsoup4 is installed in our machine. Let us talk about some problems encountered after installation.

Problems after installation

On windows machine you might encounter, wrong version being installed error mainly through −

error: ImportError “No module named HTMLParser” , then you must be running python 2 version of the code under Python 3.

error: ImportError “No module named html.parser” error, then you must be running Python 3 version of the code under Python 2.

Best way to get out of above two situations is to re-install the BeautifulSoup again, completely removing existing installation.

If you get the SyntaxError “Invalid syntax” on the line ROOT_TAG_NAME = u’[document]’, then you need to convert the python 2 code to python 3, just by either installing the package −

$ python3 setup.py install

or by manually running python’s 2 to 3 conversion script on the bs4 directory −

$ 2to3-3.2 -w bs4

Installing a Parser

By default, Beautiful Soup supports the HTML parser included in Python’s standard library, however it also supports many external third party python parsers like lxml parser or html5lib parser.

To install lxml or html5lib parser, use the command −

Linux Machine

$apt-get install python-lxml $apt-get insall python-html5lib

Windows Machine

$pip install lxml $pip install html5lib

Generally, users use lxml for speed and it is recommended to use lxml or html5lib parser if you are using older version of python 2 (before 2.7.3 version) or python 3 (before 3.2.2) as python’s built-in HTML parser is not very good in handling older version.

Running Beautiful Soup

It is time to test our Beautiful Soup package in one of the html pages (taking web page – https://www.tutorialspoint.com/index.htm, you can choose any-other web page you want) and extract some information from it.

In the below code, we are trying to extract the title from the webpage −

from bs4 import BeautifulSoup import requests url = “https://www.tutorialspoint.com/index.htm” req = requests.get(url) soup = BeautifulSoup(req.text, “html.parser”) print(soup.title)

Output

One common task is to extract all the URLs within a webpage. For that we just need to add the below line of code −

for link in soup.find_all(‘a’): print(link.get(‘href’))

Output

https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/about/about_careers.htm https://www.tutorialspoint.com/questions/index.php https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/codingground.htm https://www.tutorialspoint.com/current_affairs.htm https://www.tutorialspoint.com/upsc_ias_exams.htm https://www.tutorialspoint.com/tutor_connect/index.php https://www.tutorialspoint.com/whiteboard.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/tutorialslibrary.htm https://www.tutorialspoint.com/videotutorials/index.php https://store.tutorialspoint.com https://www.tutorialspoint.com/gate_exams_tutorials.htm https://www.tutorialspoint.com/html_online_training/index.asp https://www.tutorialspoint.com/css_online_training/index.asp https://www.tutorialspoint.com/3d_animation_online_training/index.asp https://www.tutorialspoint.com/swift_4_online_training/index.asp https://www.tutorialspoint.com/blockchain_online_training/index.asp https://www.tutorialspoint.com/reactjs_online_training/index.asp https://www.tutorix.com https://www.tutorialspoint.com/videotutorials/top-courses.php https://www.tutorialspoint.com/the_full_stack_web_development/index.asp …. …. https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/free_web_graphics.htm https://www.tutorialspoint.com/online_file_conversion.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/free_online_whiteboard.htm http://www.tutorialspoint.com https://www.facebook.com/tutorialspointindia https://plus.google.com/u/0/+tutorialspoint http://www.twitter.com/tutorialspoint http://www.linkedin.com/company/tutorialspoint https://www.youtube.com/channel/UCVLbzhxVTiTLiVKeGV7WEBg https://www.tutorialspoint.com/index.htm /about/about_privacy.htm#cookies /about/faq.htm /about/about_helping.htm /about/contact_us.htm

Similarly, we can extract useful information using beautifulsoup4.

Now let us understand more about “soup” in above example.

Beautiful Soup Documentation — Beautiful Soup 4.9.0 documentation

Quick Start¶ Here’s an HTML document I’ll be using as an example throughout this document. It’s part of a story from Alice in Wonderland : html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” Running the “three sisters” document through Beautiful Soup gives us a BeautifulSoup object, which represents the document as a nested data structure: from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) print ( soup . prettify ()) # # # # # #

# # The Dormouse’s story # #

#

# Once upon a time there were three little sisters; and their names were # # Elsie # # , # # Lacie # # and # # Tillie # # ; and they lived at the bottom of a well. #

#

# … #

# # Here are some simple ways to navigate that data structure: soup . title # soup . title . name # u’title’ soup . title . string # u’The Dormouse’s story’ soup . title . parent . name # u’head’ soup . p #

The Dormouse’s story

soup . p [ ‘class’ ] # u’title’ soup . a # Elsie soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] soup . find ( id = “link3” ) # Tillie One common task is extracting all the URLs found within a page’s tags: for link in soup . find_all ( ‘a’ ): print ( link . get ( ‘href’ )) # http://example.com/elsie # http://example.com/lacie # http://example.com/tillie Another common task is extracting all the text from a page: print ( soup . get_text ()) # The Dormouse’s story # # The Dormouse’s story # # Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well. # # … Does this look like what you need? If so, read on.

Installing Beautiful Soup¶ If you’re using a recent version of Debian or Ubuntu Linux, you can install Beautiful Soup with the system package manager: $ apt – get install python3 – bs4 Beautiful Soup 4 is published through PyPi, so if you can’t install it with the system packager, you can install it with easy_install or pip . The package name is beautifulsoup4 . Make sure you use the right version of pip or easy_install for your Python version (these may be named pip3 and easy_install3 respectively). $ easy_install beautifulsoup4 $ pip install beautifulsoup4 (The BeautifulSoup package is not what you want. That’s the previous major release, Beautiful Soup 3. Lots of software uses BS3, so it’s still available, but if you’re writing new code you should install beautifulsoup4 .) If you don’t have easy_install or pip installed, you can download the Beautiful Soup 4 source tarball and install it with setup.py . $ python setup.py install If all else fails, the license for Beautiful Soup allows you to package the entire library with your application. You can download the tarball, copy its bs4 directory into your application’s codebase, and use Beautiful Soup without installing it at all. I use Python 3.8 to develop Beautiful Soup, but it should work with other recent versions. Installing a parser¶ Beautiful Soup supports the HTML parser included in Python’s standard library, but it also supports a number of third-party Python parsers. One is the lxml parser. Depending on your setup, you might install lxml with one of these commands: $ apt – get install python – lxml $ easy_install lxml $ pip install lxml Another alternative is the pure-Python html5lib parser, which parses HTML the way a web browser does. Depending on your setup, you might install html5lib with one of these commands: $ apt – get install python – html5lib $ easy_install html5lib $ pip install html5lib This table summarizes the advantages and disadvantages of each parser library: Parser Typical usage Advantages Disadvantages Python’s html.parser BeautifulSoup(markup, “html.parser”) Batteries included

Decent speed

Lenient (As of Python 3.2) Not as fast as lxml, less lenient than html5lib. lxml’s HTML parser BeautifulSoup(markup, “lxml”) Very fast

Lenient External C dependency lxml’s XML parser BeautifulSoup(markup, “lxml-xml”) BeautifulSoup(markup, “xml”) Very fast

The only currently supported XML parser External C dependency html5lib BeautifulSoup(markup, “html5lib”) Extremely lenient

Parses pages the same way a web browser does

Creates valid HTML5 Very slow

External Python dependency If you can, I recommend you install and use lxml for speed. If you’re using a very old version of Python – earlier than 3.2.2 – it’s essential that you install lxml or html5lib. Python’s built-in HTML parser is just not very good in those old versions. Note that if a document is invalid, different parsers will generate different Beautiful Soup trees for it. See Differences between parsers for details.

Making the soup¶ To parse a document, pass it into the BeautifulSoup constructor. You can pass in a string or an open filehandle: from bs4 import BeautifulSoup with open ( “index.html” ) as fp : soup = BeautifulSoup ( fp , ‘html.parser’ ) soup = BeautifulSoup ( “a web page” , ‘html.parser’ ) First, the document is converted to Unicode, and HTML entities are converted to Unicode characters: print ( BeautifulSoup ( “Sacré bleu!” , “html.parser” )) # Sacré bleu! Beautiful Soup then parses the document using the best available parser. It will use an HTML parser unless you specifically tell it to use an XML parser. (See Parsing XML.)

Kinds of objects¶ Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. But you’ll only ever have to deal with about four kinds of objects: Tag , NavigableString , BeautifulSoup , and Comment . Tag ¶ A Tag object corresponds to an XML or HTML tag in the original document: soup = BeautifulSoup ( ‘Extremely bold‘ , ‘html.parser’ ) tag = soup . b type ( tag ) # Tags have a lot of attributes and methods, and I’ll cover most of them in Navigating the tree and Searching the tree. For now, the most important features of a tag are its name and attributes. Name¶ Every tag has a name, accessible as .name : tag . name # ‘b’ If you change a tag’s name, the change will be reflected in any HTML markup generated by Beautiful Soup: tag . name = “blockquote” tag #

Extremely bold

Attributes¶ A tag may have any number of attributes. The tag has an attribute “id” whose value is “boldest”. You can access a tag’s attributes by treating the tag like a dictionary: tag = BeautifulSoup ( ‘bold‘ , ‘html.parser’ ) . b tag [ ‘id’ ] # ‘boldest’ You can access that dictionary directly as .attrs : tag . attrs # {‘id’: ‘boldest’} You can add, remove, and modify a tag’s attributes. Again, this is done by treating the tag as a dictionary: tag [ ‘id’ ] = ‘verybold’ tag [ ‘another-attribute’ ] = 1 tag # del tag [ ‘id’ ] del tag [ ‘another-attribute’ ] tag # bold tag [ ‘id’ ] # KeyError: ‘id’ tag . get ( ‘id’ ) # None Multi-valued attributes¶ HTML 4 defines a few attributes that can have multiple values. HTML 5 removes a couple of them, but defines a few more. The most common multi-valued attribute is class (that is, a tag can have more than one CSS class). Others include rel , rev , accept-charset , headers , and accesskey . Beautiful Soup presents the value(s) of a multi-valued attribute as a list: css_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) css_soup . p [ ‘class’ ] # [‘body’] css_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) css_soup . p [ ‘class’ ] # [‘body’, ‘strikeout’] If an attribute looks like it has more than one value, but it’s not a multi-valued attribute as defined by any version of the HTML standard, Beautiful Soup will leave the attribute alone: id_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) id_soup . p [ ‘id’ ] # ‘my id’ When you turn a tag back into a string, multiple attribute values are consolidated: rel_soup = BeautifulSoup ( ‘

Back to the homepage

‘ , ‘html.parser’ ) rel_soup . a [ ‘rel’ ] # [‘index’] rel_soup . a [ ‘rel’ ] = [ ‘index’ , ‘contents’ ] print ( rel_soup . p ) #

Back to the homepage

You can disable this by passing multi_valued_attributes=None as a keyword argument into the BeautifulSoup constructor: no_list_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ , multi_valued_attributes = None ) no_list_soup . p [ ‘class’ ] # ‘body strikeout’ You can use get_attribute_list to get a value that’s always a list, whether or not it’s a multi-valued atribute: id_soup . p . get_attribute_list ( ‘id’ ) # [“my id”] If you parse a document as XML, there are no multi-valued attributes: xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ ) xml_soup . p [ ‘class’ ] # ‘body strikeout’ Again, you can configure this using the multi_valued_attributes argument: class_is_multi = { ‘*’ : ‘class’ } xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ , multi_valued_attributes = class_is_multi ) xml_soup . p [ ‘class’ ] # [‘body’, ‘strikeout’] You probably won’t need to do this, but if you do, use the defaults as a guide. They implement the rules described in the HTML specification: from bs4.builder import builder_registry builder_registry . lookup ( ‘html’ ) . DEFAULT_CDATA_LIST_ATTRIBUTES NavigableString ¶ A string corresponds to a bit of text within a tag. Beautiful Soup uses the NavigableString class to contain these bits of text: soup = BeautifulSoup ( ‘Extremely bold‘ , ‘html.parser’ ) tag = soup . b tag . string # ‘Extremely bold’ type ( tag . string ) # A NavigableString is just like a Python Unicode string, except that it also supports some of the features described in Navigating the tree and Searching the tree. You can convert a NavigableString to a Unicode string with str : unicode_string = str ( tag . string ) unicode_string # ‘Extremely bold’ type ( unicode_string ) # You can’t edit a string in place, but you can replace one string with another, using replace_with(): tag . string . replace_with ( “No longer bold” ) tag # No longer bold NavigableString supports most of the features described in Navigating the tree and Searching the tree, but not all of them. In particular, since a string can’t contain anything (the way a tag may contain a string or another tag), strings don’t support the .contents or .string attributes, or the find() method. If you want to use a NavigableString outside of Beautiful Soup, you should call unicode() on it to turn it into a normal Python Unicode string. If you don’t, your string will carry around a reference to the entire Beautiful Soup parse tree, even when you’re done using Beautiful Soup. This is a big waste of memory. BeautifulSoup ¶ The BeautifulSoup object represents the parsed document as a whole. For most purposes, you can treat it as a Tag object. This means it supports most of the methods described in Navigating the tree and Searching the tree. You can also pass a BeautifulSoup object into one of the methods defined in Modifying the tree, just as you would a Tag. This lets you do things like combine two parsed documents: doc = BeautifulSoup ( “INSERT FOOTER HEREHere’s the footer

” , “xml” ) doc . find ( text = “INSERT FOOTER HERE” ) . replace_with ( footer ) # ‘INSERT FOOTER HERE’ print ( doc ) # #

Since the BeautifulSoup object doesn’t correspond to an actual HTML or XML tag, it has no name and no attributes. But sometimes it’s useful to look at its .name , so it’s been given the special .name “[document]”: soup . name # ‘[document]’

Navigating the tree¶ Here’s the “Three sisters” HTML document again: html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) I’ll use this as an example to show you how to move from one part of a document to another. Going down¶ Tags may contain strings and other tags. These elements are the tag’s children . Beautiful Soup provides a lot of different attributes for navigating and iterating over a tag’s children. Note that Beautiful Soup strings don’t support any of these attributes, because a string can’t have children. Navigating using tag names¶ The simplest way to navigate the parse tree is to say the name of the tag you want. If you want the tag, just say soup.head : soup . head # soup . title # You can do use this trick again and again to zoom in on a certain part of the parse tree. This code gets the first tag beneath the tag: soup . body . b # The Dormouse’s story Using a tag name as an attribute will give you only the first tag by that name: soup . a # Elsie If you need to get all the tags, or anything more complicated than the first tag with a certain name, you’ll need to use one of the methods described in Searching the tree, such as find_all() : soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] .contents and .children ¶ A tag’s children are available in a list called .contents : head_tag = soup . head head_tag # head_tag . contents # [] title_tag = head_tag . contents [ 0 ] title_tag # title_tag . contents # [‘The Dormouse’s story’] The BeautifulSoup object itself has children. In this case, the tag is the child of the BeautifulSoup object.: len ( soup . contents ) # 1 soup . contents [ 0 ] . name # ‘html’ A string does not have .contents , because it can’t contain anything: text = title_tag . contents [ 0 ] text . contents # AttributeError: ‘NavigableString’ object has no attribute ‘contents’ Instead of getting them as a list, you can iterate over a tag’s children using the .children generator: for child in title_tag . children : print ( child ) # The Dormouse’s story If you want to modify a tag’s children, use the methods described in Modifying the tree. Don’t modify the the .contents list directly: that can lead to problems that are subtle and difficult to spot. .descendants ¶ The .contents and .children attributes only consider a tag’s direct children. For instance, the tag has a single direct child–the ] But the # The Dormouse’s story The tag has only one child, but it has two descendants: the ] head_tag . string # ‘The Dormouse’s story’ If a tag contains more than one thing, then it’s not clear what .string should refer to, so .string is defined to be None : print ( soup . html . string ) # None .strings and stripped_strings ¶ If there’s more than one thing inside a tag, you can still look at just the strings. Use the .strings generator: for string in soup . strings : print ( repr ( string )) ‘

‘ # “The Dormouse’s story” # ‘

‘ # ‘

‘ # “The Dormouse’s story” # ‘

‘ # ‘Once upon a time there were three little sisters; and their names were

‘ # ‘Elsie’ # ‘,

‘ # ‘Lacie’ # ‘ and

‘ # ‘Tillie’ # ‘;

and they lived at the bottom of a well.’ # ‘

‘ # ‘…’ # ‘

‘ These strings tend to have a lot of extra whitespace, which you can remove by using the .stripped_strings generator instead: for string in soup . stripped_strings : print ( repr ( string )) # “The Dormouse’s story” # “The Dormouse’s story” # ‘Once upon a time there were three little sisters; and their names were’ # ‘Elsie’ # ‘,’ # ‘Lacie’ # ‘and’ # ‘Tillie’ # ‘;

and they lived at the bottom of a well.’ # ‘…’ Here, strings consisting entirely of whitespace are ignored, and whitespace at the beginning and end of strings is removed. Going up¶ Continuing the “family tree” analogy, every tag and every string has a parent : the tag that contains it. .parent ¶ You can access an element’s parent with the .parent attribute. In the example “three sisters” document, the tag is the parent of the title_tag . parent # The title string itself has a parent: the The parent of a top-level tag like is the BeautifulSoup object itself: html_tag = soup . html type ( html_tag . parent ) # And the .parent of a BeautifulSoup object is defined as None: print ( soup . parent ) # None .parents ¶ You can iterate over all of an element’s parents with .parents . This example uses .parents to travel from an tag buried deep within the document, to the very top of the document: link = soup . a link # Elsie for parent in link . parents : print ( parent . name ) # p # body # html # [document] Going sideways¶ Consider a simple document like this: sibling_soup = BeautifulSoup ( “text1text2” , ‘html.parser’ ) print ( sibling_soup . prettify ()) # # # text1 # # # text2 # # The tag and the tag are at the same level: they’re both direct children of the same tag. We call them siblings . When a document is pretty-printed, siblings show up at the same indentation level. You can also use this relationship in the code you write. .next_sibling and .previous_sibling ¶ You can use .next_sibling and .previous_sibling to navigate between page elements that are on the same level of the parse tree: sibling_soup . b . next_sibling # text2 sibling_soup . c . previous_sibling # text1 The tag has a .next_sibling , but no .previous_sibling , because there’s nothing before the tag on the same level of the tree . For the same reason, the tag has a .previous_sibling but no .next_sibling : print ( sibling_soup . b . previous_sibling ) # None print ( sibling_soup . c . next_sibling ) # None The strings “text1” and “text2” are not siblings, because they don’t have the same parent: sibling_soup . b . string # ‘text1’ print ( sibling_soup . b . string . next_sibling ) # None In real documents, the .next_sibling or .previous_sibling of a tag will usually be a string containing whitespace. Going back to the “three sisters” document: # Elsie # Lacie # Tillie You might think that the .next_sibling of the first tag would be the second tag. But actually, it’s a string: the comma and newline that separate the first tag from the second: link = soup . a link # Elsie link . next_sibling # ‘,

‘ The second tag is actually the .next_sibling of the comma: link . next_sibling . next_sibling # Lacie .next_siblings and .previous_siblings ¶ You can iterate over a tag’s siblings with .next_siblings or .previous_siblings : for sibling in soup . a . next_siblings : print ( repr ( sibling )) # ‘,

‘ # Lacie # ‘ and

‘ # Tillie # ‘; and they lived at the bottom of a well.’ for sibling in soup . find ( id = “link3” ) . previous_siblings : print ( repr ( sibling )) # ‘ and

‘ # Lacie # ‘,

‘ # Elsie # ‘Once upon a time there were three little sisters; and their names were

‘ Going back and forth¶ Take a look at the beginning of the “three sisters” document: # #

The Dormouse’s story

An HTML parser takes this string of characters and turns it into a series of events: “open an tag”, “open a tag”, “open a

The Dormouse's story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

...

""" print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_a_tags ) . prettify ()) # # Elsie # # # Lacie # # # Tillie # print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_tags_with_id_link2 ) . prettify ()) # # Lacie # print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_short_strings ) . prettify ()) # Elsie # , # Lacie # and # Tillie # ... # You can also pass a SoupStrainer into any of the methods covered in Searching the tree. This probably isn’t terribly useful, but I thought I’d mention it: soup = BeautifulSoup ( html_doc , 'html.parser' ) soup . find_all ( only_short_strings ) # ['

', '

', 'Elsie', ',

', 'Lacie', ' and

', 'Tillie', # '

', '...', '

'] Customizing multi-valued attributes¶ In an HTML document, an attribute like class is given a list of values, and an attribute like id is given a single value, because the HTML specification treats those attributes differently: markup = '' soup = BeautifulSoup ( markup , 'html.parser' ) soup . a [ 'class' ] # ['cls1', 'cls2'] soup . a [ 'id' ] # 'id1 id2' You can turn this off by passing in multi_valued_attributes=None . Than all attributes will be given a single value: soup = BeautifulSoup ( markup , 'html.parser' , multi_valued_attributes = None ) soup . a [ 'class' ] # 'cls1 cls2' soup . a [ 'id' ] # 'id1 id2' You can customize this behavior quite a bit by passing in a dictionary for multi_valued_attributes . If you need this, look at HTMLTreeBuilder.DEFAULT_CDATA_LIST_ATTRIBUTES to see the configuration Beautiful Soup uses by default, which is based on the HTML specification. (This is a new feature in Beautiful Soup 4.8.0.) Handling duplicate attributes¶ When using the html.parser parser, you can use the on_duplicate_attribute constructor argument to customize what Beautiful Soup does when it encounters a tag that defines the same attribute more than once: markup = '' The default behavior is to use the last value found for the tag: soup = BeautifulSoup ( markup , 'html.parser' ) soup . a [ 'href' ] # http://url2/ soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = 'replace' ) soup . a [ 'href' ] # http://url2/ With on_duplicate_attribute='ignore' you can tell Beautiful Soup to use the first value found and ignore the rest: soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = 'ignore' ) soup . a [ 'href' ] # http://url1/ (lxml and html5lib always do it this way; their behavior can’t be configured from within Beautiful Soup.) If you need more, you can pass in a function that’s called on each duplicate value: def accumulate ( attributes_so_far , key , value ): if not isinstance ( attributes_so_far [ key ], list ): attributes_so_far [ key ] = [ attributes_so_far [ key ]] attributes_so_far [ key ] . append ( value ) soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = accumulate ) soup . a [ 'href' ] # ["http://url1/", "http://url2/"] (This is a new feature in Beautiful Soup 4.9.1.) Instantiating custom subclasses¶ When a parser tells Beautiful Soup about a tag or a string, Beautiful Soup will instantiate a Tag or NavigableString object to contain that information. Instead of that default behavior, you can tell Beautiful Soup to instantiate subclasses of Tag or NavigableString , subclasses you define with custom behavior: from bs4 import Tag , NavigableString class MyTag ( Tag ): pass class MyString ( NavigableString ): pass markup = "

some text

" soup = BeautifulSoup ( markup , 'html.parser' ) isinstance ( soup . div , MyTag ) # False isinstance ( soup . div . string , MyString ) # False my_classes = { Tag : MyTag , NavigableString : MyString } soup = BeautifulSoup ( markup , 'html.parser' , element_classes = my_classes ) isinstance ( soup . div , MyTag ) # True isinstance ( soup . div . string , MyString ) # True This can be useful when incorporating Beautiful Soup into a test framework. (This is a new feature in Beautiful Soup 4.8.1.)

Troubleshooting¶ diagnose() ¶ If you’re having trouble understanding what Beautiful Soup does to a document, pass the document into the diagnose() function. (New in Beautiful Soup 4.2.0.) Beautiful Soup will print out a report showing you how different parsers handle the document, and tell you if you’re missing a parser that Beautiful Soup could be using: from bs4.diagnose import diagnose with open ( "bad.html" ) as fp : data = fp . read () diagnose ( data ) # Diagnostic running on Beautiful Soup 4.2.0 # Python version 2.7.3 (default, Aug 1 2012, 05:16:07) # I noticed that html5lib is not installed. Installing it may help. # Found lxml version 2.3.2.0 # # Trying to parse your data with html.parser # Here's what html.parser did with the document: # ... Just looking at the output of diagnose() may show you how to solve the problem. Even if not, you can paste the output of diagnose() when asking for help. Errors when parsing a document¶ There are two different kinds of parse errors. There are crashes, where you feed a document to Beautiful Soup and it raises an exception, usually an HTMLParser.HTMLParseError . And there is unexpected behavior, where a Beautiful Soup parse tree looks a lot different than the document used to create it. Almost none of these problems turn out to be problems with Beautiful Soup. This is not because Beautiful Soup is an amazingly well-written piece of software. It’s because Beautiful Soup doesn’t include any parsing code. Instead, it relies on external parsers. If one parser isn’t working on a certain document, the best solution is to try a different parser. See Installing a parser for details and a parser comparison. The most common parse errors are HTMLParser.HTMLParseError: malformed start tag and HTMLParser.HTMLParseError: bad end tag . These are both generated by Python’s built-in HTML parser library, and the solution is to install lxml or html5lib. The most common type of unexpected behavior is that you can’t find a tag that you know is in the document. You saw it going in, but find_all() returns [] or find() returns None . This is another common problem with Python’s built-in HTML parser, which sometimes skips tags it doesn’t understand. Again, the best solution is to install lxml or html5lib. Version mismatch problems¶ SyntaxError: Invalid syntax (on the line ROOT_TAG_NAME = '[document]' ): Caused by running an old Python 2 version of Beautiful Soup under Python 3, without converting the code.

ImportError: No module named HTMLParser - Caused by running an old Python 2 version of Beautiful Soup under Python 3.

ImportError: No module named html.parser - Caused by running the Python 3 version of Beautiful Soup under Python 2.

ImportError: No module named BeautifulSoup - Caused by running Beautiful Soup 3 code on a system that doesn’t have BS3 installed. Or, by writing Beautiful Soup 4 code without knowing that the package name has changed to bs4 .

ImportError: No module named bs4 - Caused by running Beautiful Soup 4 code on a system that doesn’t have BS4 installed. Parsing XML¶ By default, Beautiful Soup parses documents as HTML. To parse a document as XML, pass in “xml” as the second argument to the BeautifulSoup constructor: soup = BeautifulSoup ( markup , "xml" ) You’ll need to have lxml installed. Other parser problems¶ If your script works on one computer but not another, or in one virtual environment but not another, or outside the virtual environment but not inside, it’s probably because the two environments have different parser libraries available. For example, you may have developed the script on a computer that has lxml installed, and then tried to run it on a computer that only has html5lib installed. See Differences between parsers for why this matters, and fix the problem by mentioning a specific parser library in the BeautifulSoup constructor.

Because HTML tags and attributes are case-insensitive, all three HTML parsers convert tag and attribute names to lowercase. That is, the markup is converted to . If you want to preserve mixed-case or uppercase tags and attributes, you’ll need to parse the document as XML. Miscellaneous¶ UnicodeEncodeError: 'charmap' codec can't encode character '\xfoo' in position bar (or just about any other UnicodeEncodeError ) - This problem shows up in two main situations. First, when you try to print a Unicode character that your console doesn’t know how to display. (See this page on the Python wiki for help.) Second, when you’re writing to a file and you pass in a Unicode character that’s not supported by your default encoding. In this case, the simplest solution is to explicitly encode the Unicode string into UTF-8 with u.encode("utf8") .

KeyError: [attr] - Caused by accessing tag['attr'] when the tag in question doesn’t define the attr attribute. The most common errors are KeyError: 'href' and KeyError: 'class' . Use tag.get('attr') if you’re not sure attr is defined, just as you would with a Python dictionary.

AttributeError: 'ResultSet' object has no attribute 'foo' - This usually happens because you expected find_all() to return a single tag or string. But find_all() returns a _list_ of tags and strings–a ResultSet object. You need to iterate over the list and look at the .foo of each one. Or, if you really only want one result, you need to use find() instead of find_all() .

AttributeError: 'NoneType' object has no attribute 'foo' - This usually happens because you called find() and then tried to access the .foo` attribute of the result. But in your case, find() didn’t find anything, so it returned None , instead of returning a tag or a string. You need to figure out why your find() call isn’t returning anything.

AttributeError: 'NavigableString' object has no attribute 'foo' - This usually happens because you’re treating a string as though it were a tag. You may be iterating over a list, expecting that it contains nothing but tags, when it actually contains both tags and strings. Improving Performance¶ Beautiful Soup will never be as fast as the parsers it sits on top of. If response time is critical, if you’re paying for computer time by the hour, or if there’s any other reason why computer time is more valuable than programmer time, you should forget about Beautiful Soup and work directly atop lxml. That said, there are things you can do to speed up Beautiful Soup. If you’re not using lxml as the underlying parser, my advice is to start. Beautiful Soup parses documents significantly faster using lxml than using html.parser or html5lib. You can speed up encoding detection significantly by installing the cchardet library. Parsing only part of a document won’t save you much time parsing the document, but it can save a lot of memory, and it’ll make searching the document much faster.

Translating this documentation¶ New translations of the Beautiful Soup documentation are greatly appreciated. Translations should be licensed under the MIT license, just like Beautiful Soup and its English documentation are. There are two ways of getting your translation into the main code base and onto the Beautiful Soup website: Create a branch of the Beautiful Soup repository, add your translation, and propose a merge with the main branch, the same as you would do with a proposed change to the source code. Send a message to the Beautiful Soup discussion group with a link to your translation, or attach your translation to the message. Use the Chinese or Brazilian Portuguese translations as your model. In particular, please translate the source file doc/source/index.rst , rather than the HTML version of the documentation. This makes it possible to publish the documentation in a variety of formats, not just HTML.