.

Now we know the unique location of our data with the help of class tags.

Jump into the Code

Now that we know where our data is, we can start coding our web scraper. Open your text editor now!

First, we need to import all the libraries that we are going to use.

# import libraries import urllib2 from bs4 import BeautifulSoup

Next, declare a variable for the url of the page.

# specify the url quote_page = ‘http://www.bloomberg.com/quote/SPX:IND’

Then, make use of the Python urllib2 to get the HTML page of the url declared.

# query the website and return the html to the variable ‘page’ page = urllib2.urlopen(quote_page)

Finally, parse the page into BeautifulSoup format so we can use BeautifulSoup to work on it.

# parse the html using beautiful soup and store in variable `soup` soup = BeautifulSoup(page, ‘html.parser’)

Now we have a variable, soup , containing the HTML of the page. Here’s where we can start coding the part that extracts the data.

Remember the unique layers of our data? BeautifulSoup can help us get into these layers and extract the content with find() . In this case, since the HTML class name is unique on this page, we can simply query

.

# Take out the

of name and get its value name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})

After we have the tag, we can get the data by getting its text .

name = name_box.text.strip() # strip() is used to remove starting and trailing print name

Similarly, we can get the price too.

# get the index price price_box = soup.find(‘div’, attrs={‘class’:’price’}) price = price_box.text print price

When you run the program, you should be able to see that it prints out the current price of the S&P 500 Index.

Export to Excel CSV

Now that we have the data, it is time to save it. The Excel Comma Separated Format is a nice choice. It can be opened in Excel so you can see the data and process it easily.

But first, we have to import the Python csv module and the datetime module to get the record date. Insert these lines to your code in the import section.

import csv from datetime import datetime

At the bottom of your code, add the code for writing data to a csv file.

# open a csv file with append, so old data will not be erased with open(‘index.csv’, ‘a’) as csv_file: writer = csv.writer(csv_file) writer.writerow([name, price, datetime.now()])

Now if you run your program, you should able to export an index.csv file, which you can then open with Excel, where you should see a line of data.

So if you run this program everyday, you will be able to easily get the S&P 500 Index price without rummaging through the website!

Going Further (Advanced uses)

Multiple Indices

So scraping one index is not enough for you, right? We can try to extract multiple indices at the same time.

First, modify the quote_page into an array of URLs.

quote_page = [‘http://www.bloomberg.com/quote/SPX:IND’, ‘http://www.bloomberg.com/quote/CCMP:IND’]

Then we change the data extraction code into a for loop, which will process the URLs one by one and store all the data into a variable data in tuples.

# for loop data = [] for pg in quote_page: # query the website and return the html to the variable ‘page’ page = urllib2.urlopen(pg) # parse the html using beautiful soap and store in variable `soup` soup = BeautifulSoup(page, ‘html.parser’) # Take out the

of name and get its value name_box = soup.find(‘h1’, attrs={‘class’: ‘name’}) name = name_box.text.strip() # strip() is used to remove starting and trailing # get the index price price_box = soup.find(‘div’, attrs={‘class’:’price’}) price = price_box.text # save the data in tuple data.append((name, price))

Also, modify the saving section to save data row by row.

# open a csv file with append, so old data will not be erased with open(‘index.csv’, ‘a’) as csv_file: writer = csv.writer(csv_file) # The for loop for name, price in data: writer.writerow([name, price, datetime.now()])

Rerun the program and you should be able to extract two indices at the same time!

Advanced Scraping Techniques

BeautifulSoup is simple and great for small-scale web scraping. But if you are interested in scraping data at a larger scale, you should consider using these other alternatives:

Scrapy, a powerful python scraping framework Try to integrate your code with some public APIs. The efficiency of data retrieval is much higher than scraping webpages. For example, take a look at Facebook Graph API, which can help you get hidden data which is not shown on Facebook webpages. Consider using a database backend like MySQL to store your data when it gets too large.

Adopt the DRY Method

DRY stands for “Don’t Repeat Yourself”, try to automate your everyday tasks like this person. Some other fun projects to consider might be keeping track of your Facebook friends’ active time (with their consent of course), or grabbing a list of topics in a forum and trying out natural language processing (which is a hot topic for Artificial Intelligence right now)!

If you have any questions, please feel free to leave a comment below.

References

http://www.gregreda.com/2013/03/03/web-scraping-101-with-python/

Beginner’s guide to Web Scraping in Python using BeautifulSoup

This article was originally published on Altitude Labs’ blog and was written by our software engineer, Leonard Mok. Altitude Labs is a software agency that specializes in personalized, mobile-first React apps.

What is the difference beautifulsoup and bs4

I’m new to python and I tried to parse some XML files in order to add some new tags and store that new XML file.

python-beautifulsoup seams to be the right package for that. Searching around the web for tutorials, how to add an new tag to XML parsed by BeautifulSoup, i found out, that the package python-bs4 is used.

Looking at the package description, both packages have the same title:

python-bs4 – error-tolerant HTML parser for Python python-beautifulsoup – error-tolerant HTML parser for Python

So my question: what is the difference?

Scrapy VS Beautiful Soup: A Comparison Of Web Crawling Tools

One of the most critical assets for data-driven organisations is the kind of tools used by their data science professionals. Web crawler and other such web scraping tools are few of those tools that are used to gain meaningful insights. Web scraping allows efficient extraction of data from several web services and helps in converting raw and unstructured data into a structured whole.

There are several tools available for web scraping, such as lxml, BeautifulSoup, MechanicalSoup, Scrapy, Python Requests and others. Among these, Scrapy and Beautiful Soup are popular among developers.

THE BELAMY Sign up for your weekly dose of what’s up in emerging technology. Email Sign up

In this article, we will compare these two web scraping tools, and try to understand the differences between them. Before diving deep into the tools, let us first understand what these tools are.

Scrapy

Scrapy is an open-source and collaborative framework for extracting the data you need from websites in a fast and simple manner. This tool can be used for extracting data using APIs. It can also be used as a general-purpose web crawler. Thus, Scrapy is an application framework, which can be used for writing web spiders that crawl websites and extract data from them.

The framework provides a built-in mechanism for extracting data – known as selectors – and can be used for data mining, automated testing, etc. Scrapy is supported under Python 3.5+ under CPython and PyPy starting with PyPy 5.9.

Features of Scrapy:

Scrapy provides built-in support for selecting and extracting data from HTML/XML sources using extended CSS selectors and XPath expressions

An interactive shell console for trying out the CSS and XPath expressions to scrape data

Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem)

Scraping With Scrapy

Using pip

If you just want to install scrapy globally in your system, you can install scrapy library using the python package ‘pip’. Open your terminal or command prompt and type the following command.

pip install scrapy

Using Conda

If you want scrapy to be in your conda environment just type in and execute the following command in your terminal

conda install -c conda-forge scrapy

The scrapy shell: It allows to scrape web pages interactively using the command line.

To open scrapy shell type scrapy shell .

Scraping with Scrapy Shell

Follow the steps below to start scraping :

1. Open the html file in a web browser and copy the url.

2. Now in the scrapy shell type and execute the following command:

fetch(“url–”)

Replace url– with the url of the html file or any webpage and the fetch command will download the page locally to your system.

You will get a similar message in your console

[scrapy.core.engine] DEBUG: Crawled (200)

3. Viewing the response

The fetch object will store whatever page or information it fetched into a response object. To view the response object simply type in and enter the following command.

view(response)

The console will return a True and the webpage that was downloaded with fetch() will open up in your default browser.

4. Now that all the data you need is available locally. You just need to know what data you need.

5. Scraping the data: Coming back to the console, all the elements need to be printed behind the webpage that was fetched earlier. Enter the following command:

print(response.text)

Click here to get the detailed web scraping.

Beautiful Soup

Beautiful Soup is one of the most popular Python libraries which helps in parsing HTML or XML documents into a tree structure to find and extract data. This tool features a simple, Pythonic interface and automatic encoding conversion to make it easy to work with website data.

This library provides simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree, and automatically converts incoming documents to Unicode and outgoing documents to UTF-8.

Features of Beautiful Soup:

This Python library provides a few simple methods, as well as Pythonic idioms for navigating, searching, and modifying a parse tree

The library automatically converts incoming and outgoing documents to Unicode and UTF-8, respectively

This library sits on top of popular Python parsers like lxml and html5lib, allowing you to try out different parsing strategies or trade speed for flexibility

Scraping With Beautifulsoup

Installing Beautiful Soup 4

Beautiful Soup library can be installed using PIP with a very simple command. It is available on almost all platforms. Here is a way to install it using Jupyter Notebook.

This library can be imported with the following code and assign it to an object.

Getting Started

We will be using this basic, and default, HTML doc to parse the data using Beautiful Soup.

The following code will expand HTML into its hierarchy:

Exploring The Parse Tree

To navigate through the tree, we can use the following commands:

Beautiful Soup has many attributes which can be accessed and edited. This extracted parsed data can be saved onto a text file.

Click here to know more about web scraping with BeautifulSoup.

Scrapy VS Beautiful Soup

Structure

Scrapy is an open-source framework, whereas Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. A framework inverts the control of the program and informs the developer what they need. Whereas in the case of a library, the developer calls the library where and when they need it.

Performance

Due to the built-in support for generating feed exports in multiple formats, as well as selecting and extracting data from various sources, the performance of Scrapy can be said to be faster than Beautiful Soup. Working with Beautiful Soup can speed up with the help of Multithreading process.

Extensibility

Beautiful Soup works best when working on smaller projects. On the other hand, Scrapy may be the better choice for larger projects with more complexities, as this framework can add custom functionalities and can develop pipelines with flexibility and speed.

Beginner-Friendly

For a beginner who is trying hands-on web scraping for the first time, Beautiful Soup is the best choice to start with. Scrapy can be used for scraping, but it is comparatively more complex than the former.

Community

The developer’s community of Scrapy is stronger and vast compared to that of Beautiful Soup. Also, developers can use Beautiful Soup for parsing HTML responses in Scrapy callbacks by feeding the response’s body into a BeautifulSoup object and extracting whatever data they need from it.

Beautiful Soup Documentation — Beautiful Soup 4.4.0 documentation

Quick Start¶ Here’s an HTML document I’ll be using as an example throughout this document. It’s part of a story from Alice in Wonderland : html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” Running the “three sisters” document through Beautiful Soup gives us a BeautifulSoup object, which represents the document as a nested data structure: from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) print ( soup . prettify ()) # # # # # #

# # The Dormouse’s story # #

#

# Once upon a time there were three little sisters; and their names were # # Elsie # # , # # Lacie # # and # # Tillie # # ; and they lived at the bottom of a well. #

#

# … #

# # Here are some simple ways to navigate that data structure: soup . title # soup . title . name # u’title’ soup . title . string # u’The Dormouse’s story’ soup . title . parent . name # u’head’ soup . p #

The Dormouse’s story

soup . p [ ‘class’ ] # u’title’ soup . a # Elsie soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] soup . find ( id = “link3” ) # Tillie One common task is extracting all the URLs found within a page’s tags: for link in soup . find_all ( ‘a’ ): print ( link . get ( ‘href’ )) # http://example.com/elsie # http://example.com/lacie # http://example.com/tillie Another common task is extracting all the text from a page: print ( soup . get_text ()) # The Dormouse’s story # # The Dormouse’s story # # Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well. # # … Does this look like what you need? If so, read on.

Installing Beautiful Soup¶ If you’re using a recent version of Debian or Ubuntu Linux, you can install Beautiful Soup with the system package manager: $ apt-get install python-bs4 (for Python 2) $ apt-get install python3-bs4 (for Python 3) Beautiful Soup 4 is published through PyPi, so if you can’t install it with the system packager, you can install it with easy_install or pip . The package name is beautifulsoup4 , and the same package works on Python 2 and Python 3. Make sure you use the right version of pip or easy_install for your Python version (these may be named pip3 and easy_install3 respectively if you’re using Python 3). $ easy_install beautifulsoup4 $ pip install beautifulsoup4 (The BeautifulSoup package is probably not what you want. That’s the previous major release, Beautiful Soup 3. Lots of software uses BS3, so it’s still available, but if you’re writing new code you should install beautifulsoup4 .) If you don’t have easy_install or pip installed, you can download the Beautiful Soup 4 source tarball and install it with setup.py . $ python setup.py install If all else fails, the license for Beautiful Soup allows you to package the entire library with your application. You can download the tarball, copy its bs4 directory into your application’s codebase, and use Beautiful Soup without installing it at all. I use Python 2.7 and Python 3.2 to develop Beautiful Soup, but it should work with other recent versions. Problems after installation¶ Beautiful Soup is packaged as Python 2 code. When you install it for use with Python 3, it’s automatically converted to Python 3 code. If you don’t install the package, the code won’t be converted. There have also been reports on Windows machines of the wrong version being installed. If you get the ImportError “No module named HTMLParser”, your problem is that you’re running the Python 2 version of the code under Python 3. If you get the ImportError “No module named html.parser”, your problem is that you’re running the Python 3 version of the code under Python 2. In both cases, your best bet is to completely remove the Beautiful Soup installation from your system (including any directory created when you unzipped the tarball) and try the installation again. If you get the SyntaxError “Invalid syntax” on the line ROOT_TAG_NAME = u'[document]’ , you need to convert the Python 2 code to Python 3. You can do this either by installing the package: $ python3 setup.py install or by manually running Python’s 2to3 conversion script on the bs4 directory: $ 2to3-3.2 -w bs4 Installing a parser¶ Beautiful Soup supports the HTML parser included in Python’s standard library, but it also supports a number of third-party Python parsers. One is the lxml parser. Depending on your setup, you might install lxml with one of these commands: $ apt-get install python-lxml $ easy_install lxml $ pip install lxml Another alternative is the pure-Python html5lib parser, which parses HTML the way a web browser does. Depending on your setup, you might install html5lib with one of these commands: $ apt-get install python-html5lib $ easy_install html5lib $ pip install html5lib This table summarizes the advantages and disadvantages of each parser library: Parser Typical usage Advantages Disadvantages Python’s html.parser BeautifulSoup(markup, “html.parser”) Batteries included

Decent speed

Lenient (As of Python 2.7.3 and 3.2.) Not as fast as lxml, less lenient than html5lib. lxml’s HTML parser BeautifulSoup(markup, “lxml”) Very fast

Lenient External C dependency lxml’s XML parser BeautifulSoup(markup, “lxml-xml”) BeautifulSoup(markup, “xml”) Very fast

The only currently supported XML parser External C dependency html5lib BeautifulSoup(markup, “html5lib”) Extremely lenient

Parses pages the same way a web browser does

Creates valid HTML5 Very slow

External Python dependency If you can, I recommend you install and use lxml for speed. If you’re using a version of Python 2 earlier than 2.7.3, or a version of Python 3 earlier than 3.2.2, it’s essential that you install lxml or html5lib–Python’s built-in HTML parser is just not very good in older versions. Note that if a document is invalid, different parsers will generate different Beautiful Soup trees for it. See Differences between parsers for details.

Making the soup¶ To parse a document, pass it into the BeautifulSoup constructor. You can pass in a string or an open filehandle: from bs4 import BeautifulSoup with open ( “index.html” ) as fp : soup = BeautifulSoup ( fp ) soup = BeautifulSoup ( “data” ) First, the document is converted to Unicode, and HTML entities are converted to Unicode characters: BeautifulSoup(“Sacré bleu!”) Sacré bleu! Beautiful Soup then parses the document using the best available parser. It will use an HTML parser unless you specifically tell it to use an XML parser. (See Parsing XML.)

Kinds of objects¶ Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. But you’ll only ever have to deal with about four kinds of objects: Tag , NavigableString , BeautifulSoup , and Comment . Tag ¶ A Tag object corresponds to an XML or HTML tag in the original document: soup = BeautifulSoup ( ‘Extremely bold‘ ) tag = soup . b type ( tag ) # Tags have a lot of attributes and methods, and I’ll cover most of them in Navigating the tree and Searching the tree. For now, the most important features of a tag are its name and attributes. Name¶ Every tag has a name, accessible as .name : tag . name # u’b’ If you change a tag’s name, the change will be reflected in any HTML markup generated by Beautiful Soup: tag . name = “blockquote” tag #

Extremely bold

Attributes¶ A tag may have any number of attributes. The tag has an attribute “id” whose value is “boldest”. You can access a tag’s attributes by treating the tag like a dictionary: tag [ ‘id’ ] # u’boldest’ You can access that dictionary directly as .attrs : tag . attrs # {u’id’: ‘boldest’} You can add, remove, and modify a tag’s attributes. Again, this is done by treating the tag as a dictionary: tag [ ‘id’ ] = ‘verybold’ tag [ ‘another-attribute’ ] = 1 tag # del tag [ ‘id’ ] del tag [ ‘another-attribute’ ] tag # tag [ ‘id’ ] # KeyError: ‘id’ print ( tag . get ( ‘id’ )) # None Multi-valued attributes¶ HTML 4 defines a few attributes that can have multiple values. HTML 5 removes a couple of them, but defines a few more. The most common multi-valued attribute is class (that is, a tag can have more than one CSS class). Others include rel , rev , accept-charset , headers , and accesskey . Beautiful Soup presents the value(s) of a multi-valued attribute as a list: css_soup = BeautifulSoup ( ‘

‘ ) css_soup . p [ ‘class’ ] # [“body”] css_soup = BeautifulSoup ( ‘

‘ ) css_soup . p [ ‘class’ ] # [“body”, “strikeout”] If an attribute looks like it has more than one value, but it’s not a multi-valued attribute as defined by any version of the HTML standard, Beautiful Soup will leave the attribute alone: id_soup = BeautifulSoup ( ‘

‘ ) id_soup . p [ ‘id’ ] # ‘my id’ When you turn a tag back into a string, multiple attribute values are consolidated: rel_soup = BeautifulSoup ( ‘

Back to the homepage

‘ ) rel_soup . a [ ‘rel’ ] # [‘index’] rel_soup . a [ ‘rel’ ] = [ ‘index’ , ‘contents’ ] print ( rel_soup . p ) #

Back to the homepage

You can disable this by passing multi_valued_attributes=None as a keyword argument into the BeautifulSoup constructor: no_list_soup = BeautifulSoup ( ‘

‘ , ‘html’ , multi_valued_attributes = None ) no_list_soup . p [ ‘class’ ] # u’body strikeout’ You can use `get_attribute_list to get a value that’s always a list, whether or not it’s a multi-valued atribute: id_soup . p . get_attribute_list ( ‘id’ ) # [“my id”] If you parse a document as XML, there are no multi-valued attributes: xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ ) xml_soup . p [ ‘class’ ] # u’body strikeout’ Again, you can configure this using the multi_valued_attributes argument: class_is_multi = { ‘*’ : ‘class’ } xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ , multi_valued_attributes = class_is_multi ) xml_soup . p [ ‘class’ ] # [u’body’, u’strikeout’] You probably won’t need to do this, but if you do, use the defaults as a guide. They implement the rules described in the HTML specification: from bs4.builder import builder_registry builder_registry . lookup ( ‘html’ ) . DEFAULT_CDATA_LIST_ATTRIBUTES NavigableString ¶ A string corresponds to a bit of text within a tag. Beautiful Soup uses the NavigableString class to contain these bits of text: tag . string # u’Extremely bold’ type ( tag . string ) # A NavigableString is just like a Python Unicode string, except that it also supports some of the features described in Navigating the tree and Searching the tree. You can convert a NavigableString to a Unicode string with unicode() : unicode_string = unicode ( tag . string ) unicode_string # u’Extremely bold’ type ( unicode_string ) # You can’t edit a string in place, but you can replace one string with another, using replace_with(): tag . string . replace_with ( “No longer bold” ) tag #

No longer bold

NavigableString supports most of the features described in Navigating the tree and Searching the tree, but not all of them. In particular, since a string can’t contain anything (the way a tag may contain a string or another tag), strings don’t support the .contents or .string attributes, or the find() method. If you want to use a NavigableString outside of Beautiful Soup, you should call unicode() on it to turn it into a normal Python Unicode string. If you don’t, your string will carry around a reference to the entire Beautiful Soup parse tree, even when you’re done using Beautiful Soup. This is a big waste of memory. BeautifulSoup ¶ The BeautifulSoup object represents the parsed document as a whole. For most purposes, you can treat it as a Tag object. This means it supports most of the methods described in Navigating the tree and Searching the tree. You can also pass a BeautifulSoup object into one of the methods defined in Modifying the tree, just as you would a Tag. This lets you do things like combine two parsed documents: doc = BeautifulSoup ( “INSERT FOOTER HEREHere’s the footer

” , “xml” ) doc . find ( text = “INSERT FOOTER HERE” ) . replace_with ( footer ) # u’INSERT FOOTER HERE’ print ( doc ) # #

Since the BeautifulSoup object doesn’t correspond to an actual HTML or XML tag, it has no name and no attributes. But sometimes it’s useful to look at its .name , so it’s been given the special .name “[document]”: soup . name # u'[document]’

Navigating the tree¶ Here’s the “Three sisters” HTML document again: html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) I’ll use this as an example to show you how to move from one part of a document to another. Going down¶ Tags may contain strings and other tags. These elements are the tag’s children . Beautiful Soup provides a lot of different attributes for navigating and iterating over a tag’s children. Note that Beautiful Soup strings don’t support any of these attributes, because a string can’t have children. Navigating using tag names¶ The simplest way to navigate the parse tree is to say the name of the tag you want. If you want the tag, just say soup.head : soup . head # soup . title # You can do use this trick again and again to zoom in on a certain part of the parse tree. This code gets the first tag beneath the tag: soup . body . b # The Dormouse’s story Using a tag name as an attribute will give you only the first tag by that name: soup . a # Elsie If you need to get all the tags, or anything more complicated than the first tag with a certain name, you’ll need to use one of the methods described in Searching the tree, such as find_all() : soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] .contents and .children ¶ A tag’s children are available in a list called .contents : head_tag = soup . head head_tag # head_tag . contents [ < title > The Dormouse ‘s story] title_tag = head_tag . contents [ 0 ] title_tag # title_tag . contents # [u’The Dormouse’s story’] The BeautifulSoup object itself has children. In this case, the tag is the child of the BeautifulSoup object.: len ( soup . contents ) # 1 soup . contents [ 0 ] . name # u’html’ A string does not have .contents , because it can’t contain anything: text = title_tag . contents [ 0 ] text . contents # AttributeError: ‘NavigableString’ object has no attribute ‘contents’ Instead of getting them as a list, you can iterate over a tag’s children using the .children generator: for child in title_tag . children : print ( child ) # The Dormouse’s story .descendants ¶ The .contents and .children attributes only consider a tag’s direct children. For instance, the tag has a single direct child–the ] But the # The Dormouse’s story The tag has only one child, but it has two descendants: the ] head_tag . string # u’The Dormouse’s story’ If a tag contains more than one thing, then it’s not clear what .string should refer to, so .string is defined to be None : print ( soup . html . string ) # None .strings and stripped_strings ¶ If there’s more than one thing inside a tag, you can still look at just the strings. Use the .strings generator: for string in soup . strings : print ( repr ( string )) # u”The Dormouse’s story” # u’

‘ # u”The Dormouse’s story” # u’

‘ # u’Once upon a time there were three little sisters; and their names were

‘ # u’Elsie’ # u’,

‘ # u’Lacie’ # u’ and

‘ # u’Tillie’ # u’;

and they lived at the bottom of a well.’ # u’

‘ # u’…’ # u’

‘ These strings tend to have a lot of extra whitespace, which you can remove by using the .stripped_strings generator instead: for string in soup . stripped_strings : print ( repr ( string )) # u”The Dormouse’s story” # u”The Dormouse’s story” # u’Once upon a time there were three little sisters; and their names were’ # u’Elsie’ # u’,’ # u’Lacie’ # u’and’ # u’Tillie’ # u’;

and they lived at the bottom of a well.’ # u’…’ Here, strings consisting entirely of whitespace are ignored, and whitespace at the beginning and end of strings is removed. Going up¶ Continuing the “family tree” analogy, every tag and every string has a parent : the tag that contains it. .parent ¶ You can access an element’s parent with the .parent attribute. In the example “three sisters” document, the tag is the parent of the title_tag . parent # The title string itself has a parent: the The parent of a top-level tag like is the BeautifulSoup object itself: html_tag = soup . html type ( html_tag . parent ) # And the .parent of a BeautifulSoup object is defined as None: print ( soup . parent ) # None .parents ¶ You can iterate over all of an element’s parents with .parents . This example uses .parents to travel from an tag buried deep within the document, to the very top of the document: link = soup . a link # Elsie for parent in link . parents : if parent is None : print ( parent ) else : print ( parent . name ) # p # body # html # [document] # None Going sideways¶ Consider a simple document like this: sibling_soup = BeautifulSoup ( “text1text2” ) print ( sibling_soup . prettify ()) # # # # # text1 # # # text2 # # # # The tag and the tag are at the same level: they’re both direct children of the same tag. We call them siblings . When a document is pretty-printed, siblings show up at the same indentation level. You can also use this relationship in the code you write. .next_sibling and .previous_sibling ¶ You can use .next_sibling and .previous_sibling to navigate between page elements that are on the same level of the parse tree: sibling_soup . b . next_sibling # text2 sibling_soup . c . previous_sibling # text1 The tag has a .next_sibling , but no .previous_sibling , because there’s nothing before the tag on the same level of the tree . For the same reason, the tag has a .previous_sibling but no .next_sibling : print ( sibling_soup . b . previous_sibling ) # None print ( sibling_soup . c . next_sibling ) # None The strings “text1” and “text2” are not siblings, because they don’t have the same parent: sibling_soup . b . string # u’text1′ print ( sibling_soup . b . string . next_sibling ) # None In real documents, the .next_sibling or .previous_sibling of a tag will usually be a string containing whitespace. Going back to the “three sisters” document: < a href = "http://example.com/elsie" class = "sister" id = "link1" > Elsie < a href = "http://example.com/lacie" class = "sister" id = "link2" > Lacie < a href = "http://example.com/tillie" class = "sister" id = "link3" > Tillie You might think that the .next_sibling of the first tag would be the second tag. But actually, it’s a string: the comma and newline that separate the first tag from the second: link = soup . a link # Elsie link . next_sibling # u’,

‘ The second tag is actually the .next_sibling of the comma: link . next_sibling . next_sibling # Lacie .next_siblings and .previous_siblings ¶ You can iterate over a tag’s siblings with .next_siblings or .previous_siblings : for sibling in soup . a . next_siblings : print ( repr ( sibling )) # u’,

‘ # Lacie # u’ and

‘ # Tillie # u’; and they lived at the bottom of a well.’ # None for sibling in soup . find ( id = “link3” ) . previous_siblings : print ( repr ( sibling )) # ‘ and

‘ # Lacie # u’,

‘ # Elsie # u’Once upon a time there were three little sisters; and their names were

‘ # None Going back and forth¶ Take a look at the beginning of the “three sisters” document: < html >< head >< title > The Dormouse ‘s story < p class = "title" >< b > The Dormouse ‘s story

An HTML parser takes this string of characters and turns it into a series of events: “open an tag”, “open a tag”, “open a

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” print ( BeautifulSoup ( html_doc , “html.parser” , parse_only = only_a_tags ) . prettify ()) # # Elsie # # # Lacie # # # Tillie # print ( BeautifulSoup ( html_doc , “html.parser” , parse_only = only_tags_with_id_link2 ) . prettify ()) # # Lacie # print ( BeautifulSoup ( html_doc , “html.parser” , parse_only = only_short_strings ) . prettify ()) # Elsie # , # Lacie # and # Tillie # … # You can also pass a SoupStrainer into any of the methods covered in Searching the tree. This probably isn’t terribly useful, but I thought I’d mention it: soup = BeautifulSoup ( html_doc ) soup . find_all ( only_short_strings ) # [u’

‘, u’

‘, u’Elsie’, u’,

‘, u’Lacie’, u’ and

‘, u’Tillie’, # u’

‘, u’…’, u’

‘]

Troubleshooting¶ diagnose() ¶ If you’re having trouble understanding what Beautiful Soup does to a document, pass the document into the diagnose() function. (New in Beautiful Soup 4.2.0.) Beautiful Soup will print out a report showing you how different parsers handle the document, and tell you if you’re missing a parser that Beautiful Soup could be using: from bs4.diagnose import diagnose with open ( “bad.html” ) as fp : data = fp . read () diagnose ( data ) # Diagnostic running on Beautiful Soup 4.2.0 # Python version 2.7.3 (default, Aug 1 2012, 05:16:07) # I noticed that html5lib is not installed. Installing it may help. # Found lxml version 2.3.2.0 # # Trying to parse your data with html.parser # Here’s what html.parser did with the document: # … Just looking at the output of diagnose() may show you how to solve the problem. Even if not, you can paste the output of diagnose() when asking for help. Errors when parsing a document¶ There are two different kinds of parse errors. There are crashes, where you feed a document to Beautiful Soup and it raises an exception, usually an HTMLParser.HTMLParseError . And there is unexpected behavior, where a Beautiful Soup parse tree looks a lot different than the document used to create it. Almost none of these problems turn out to be problems with Beautiful Soup. This is not because Beautiful Soup is an amazingly well-written piece of software. It’s because Beautiful Soup doesn’t include any parsing code. Instead, it relies on external parsers. If one parser isn’t working on a certain document, the best solution is to try a different parser. See Installing a parser for details and a parser comparison. The most common parse errors are HTMLParser.HTMLParseError: malformed start tag and HTMLParser.HTMLParseError: bad end tag . These are both generated by Python’s built-in HTML parser library, and the solution is to install lxml or html5lib. The most common type of unexpected behavior is that you can’t find a tag that you know is in the document. You saw it going in, but find_all() returns [] or find() returns None . This is another common problem with Python’s built-in HTML parser, which sometimes skips tags it doesn’t understand. Again, the solution is to install lxml or html5lib. Version mismatch problems¶ SyntaxError: Invalid syntax (on the line ROOT_TAG_NAME = u'[document]’ ): Caused by running the Python 2 version of Beautiful Soup under Python 3, without converting the code.

(on the line ): Caused by running the Python 2 version of Beautiful Soup under Python 3, without converting the code. ImportError: No module named HTMLParser – Caused by running the Python 2 version of Beautiful Soup under Python 3.

– Caused by running the Python 2 version of Beautiful Soup under Python 3. ImportError: No module named html.parser – Caused by running the Python 3 version of Beautiful Soup under Python 2.

– Caused by running the Python 3 version of Beautiful Soup under Python 2. ImportError: No module named BeautifulSoup – Caused by running Beautiful Soup 3 code on a system that doesn’t have BS3 installed. Or, by writing Beautiful Soup 4 code without knowing that the package name has changed to bs4 .

– Caused by running Beautiful Soup 3 code on a system that doesn’t have BS3 installed. Or, by writing Beautiful Soup 4 code without knowing that the package name has changed to . ImportError: No module named bs4 – Caused by running Beautiful Soup 4 code on a system that doesn’t have BS4 installed. Parsing XML¶ By default, Beautiful Soup parses documents as HTML. To parse a document as XML, pass in “xml” as the second argument to the BeautifulSoup constructor: soup = BeautifulSoup ( markup , “xml” ) You’ll need to have lxml installed. Other parser problems¶ If your script works on one computer but not another, or in one virtual environment but not another, or outside the virtual environment but not inside, it’s probably because the two environments have different parser libraries available. For example, you may have developed the script on a computer that has lxml installed, and then tried to run it on a computer that only has html5lib installed. See Differences between parsers for why this matters, and fix the problem by mentioning a specific parser library in the BeautifulSoup constructor.

constructor. Because HTML tags and attributes are case-insensitive, all three HTML parsers convert tag and attribute names to lowercase. That is, the markup is converted to . If you want to preserve mixed-case or uppercase tags and attributes, you’ll need to parse the document as XML. Miscellaneous¶ UnicodeEncodeError: ‘charmap’ codec can’t encode character u’\xfoo’ in position bar (or just about any other UnicodeEncodeError ) – This is not a problem with Beautiful Soup. This problem shows up in two main situations. First, when you try to print a Unicode character that your console doesn’t know how to display. (See this page on the Python wiki for help.) Second, when you’re writing to a file and you pass in a Unicode character that’s not supported by your default encoding. In this case, the simplest solution is to explicitly encode the Unicode string into UTF-8 with u.encode(“utf8”) .

(or just about any other ) – This is not a problem with Beautiful Soup. This problem shows up in two main situations. First, when you try to print a Unicode character that your console doesn’t know how to display. (See this page on the Python wiki for help.) Second, when you’re writing to a file and you pass in a Unicode character that’s not supported by your default encoding. In this case, the simplest solution is to explicitly encode the Unicode string into UTF-8 with . KeyError: [attr] – Caused by accessing tag[‘attr’] when the tag in question doesn’t define the attr attribute. The most common errors are KeyError: ‘href’ and KeyError: ‘class’ . Use tag.get(‘attr’) if you’re not sure attr is defined, just as you would with a Python dictionary.

– Caused by accessing when the tag in question doesn’t define the attribute. The most common errors are and . Use if you’re not sure is defined, just as you would with a Python dictionary. AttributeError: ‘ResultSet’ object has no attribute ‘foo’ – This usually happens because you expected find_all() to return a single tag or string. But find_all() returns a _list_ of tags and strings–a ResultSet object. You need to iterate over the list and look at the .foo of each one. Or, if you really only want one result, you need to use find() instead of find_all() .

– This usually happens because you expected to return a single tag or string. But returns a _list_ of tags and strings–a object. You need to iterate over the list and look at the of each one. Or, if you really only want one result, you need to use instead of . AttributeError: ‘NoneType’ object has no attribute ‘foo’ – This usually happens because you called find() and then tried to access the .foo` attribute of the result. But in your case, find() didn’t find anything, so it returned None , instead of returning a tag or a string. You need to figure out why your find() call isn’t returning anything. Improving Performance¶ Beautiful Soup will never be as fast as the parsers it sits on top of. If response time is critical, if you’re paying for computer time by the hour, or if there’s any other reason why computer time is more valuable than programmer time, you should forget about Beautiful Soup and work directly atop lxml. That said, there are things you can do to speed up Beautiful Soup. If you’re not using lxml as the underlying parser, my advice is to start. Beautiful Soup parses documents significantly faster using lxml than using html.parser or html5lib. You can speed up encoding detection significantly by installing the cchardet library. Parsing only part of a document won’t save you much time parsing the document, but it can save a lot of memory, and it’ll make searching the document much faster.

Translating this documentation¶ New translations of the Beautiful Soup documentation are greatly appreciated. Translations should be licensed under the MIT license, just like Beautiful Soup and its English documentation are. There are two ways of getting your translation into the main code base and onto the Beautiful Soup website: Create a branch of the Beautiful Soup repository, add your translation, and propose a merge with the main branch, the same as you would do with a proposed change to the source code. Send a message to the Beautiful Soup discussion group with a link to your translation, or attach your translation to the message. Use the Chinese or Brazilian Portuguese translations as your model. In particular, please translate the source file doc/source/index.rst , rather than the HTML version of the documentation. This makes it possible to publish the documentation in a variety of formats, not just HTML.

Beautiful Soup Documentation — Beautiful Soup 4.9.0 documentation

Quick Start¶ Here’s an HTML document I’ll be using as an example throughout this document. It’s part of a story from Alice in Wonderland : html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” Running the “three sisters” document through Beautiful Soup gives us a BeautifulSoup object, which represents the document as a nested data structure: from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) print ( soup . prettify ()) # # # # # #

# # The Dormouse’s story # #

#

# Once upon a time there were three little sisters; and their names were # # Elsie # # , # # Lacie # # and # # Tillie # # ; and they lived at the bottom of a well. #

#

# … #

# # Here are some simple ways to navigate that data structure: soup . title # soup . title . name # u’title’ soup . title . string # u’The Dormouse’s story’ soup . title . parent . name # u’head’ soup . p #

The Dormouse’s story

soup . p [ ‘class’ ] # u’title’ soup . a # Elsie soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] soup . find ( id = “link3” ) # Tillie One common task is extracting all the URLs found within a page’s tags: for link in soup . find_all ( ‘a’ ): print ( link . get ( ‘href’ )) # http://example.com/elsie # http://example.com/lacie # http://example.com/tillie Another common task is extracting all the text from a page: print ( soup . get_text ()) # The Dormouse’s story # # The Dormouse’s story # # Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well. # # … Does this look like what you need? If so, read on.

Installing Beautiful Soup¶ If you’re using a recent version of Debian or Ubuntu Linux, you can install Beautiful Soup with the system package manager: $ apt – get install python3 – bs4 Beautiful Soup 4 is published through PyPi, so if you can’t install it with the system packager, you can install it with easy_install or pip . The package name is beautifulsoup4 . Make sure you use the right version of pip or easy_install for your Python version (these may be named pip3 and easy_install3 respectively). $ easy_install beautifulsoup4 $ pip install beautifulsoup4 (The BeautifulSoup package is not what you want. That’s the previous major release, Beautiful Soup 3. Lots of software uses BS3, so it’s still available, but if you’re writing new code you should install beautifulsoup4 .) If you don’t have easy_install or pip installed, you can download the Beautiful Soup 4 source tarball and install it with setup.py . $ python setup.py install If all else fails, the license for Beautiful Soup allows you to package the entire library with your application. You can download the tarball, copy its bs4 directory into your application’s codebase, and use Beautiful Soup without installing it at all. I use Python 3.8 to develop Beautiful Soup, but it should work with other recent versions. Installing a parser¶ Beautiful Soup supports the HTML parser included in Python’s standard library, but it also supports a number of third-party Python parsers. One is the lxml parser. Depending on your setup, you might install lxml with one of these commands: $ apt – get install python – lxml $ easy_install lxml $ pip install lxml Another alternative is the pure-Python html5lib parser, which parses HTML the way a web browser does. Depending on your setup, you might install html5lib with one of these commands: $ apt – get install python – html5lib $ easy_install html5lib $ pip install html5lib This table summarizes the advantages and disadvantages of each parser library: Parser Typical usage Advantages Disadvantages Python’s html.parser BeautifulSoup(markup, “html.parser”) Batteries included

Decent speed

Lenient (As of Python 3.2) Not as fast as lxml, less lenient than html5lib. lxml’s HTML parser BeautifulSoup(markup, “lxml”) Very fast

Lenient External C dependency lxml’s XML parser BeautifulSoup(markup, “lxml-xml”) BeautifulSoup(markup, “xml”) Very fast

The only currently supported XML parser External C dependency html5lib BeautifulSoup(markup, “html5lib”) Extremely lenient

Parses pages the same way a web browser does

Creates valid HTML5 Very slow

External Python dependency If you can, I recommend you install and use lxml for speed. If you’re using a very old version of Python – earlier than 3.2.2 – it’s essential that you install lxml or html5lib. Python’s built-in HTML parser is just not very good in those old versions. Note that if a document is invalid, different parsers will generate different Beautiful Soup trees for it. See Differences between parsers for details.

Making the soup¶ To parse a document, pass it into the BeautifulSoup constructor. You can pass in a string or an open filehandle: from bs4 import BeautifulSoup with open ( “index.html” ) as fp : soup = BeautifulSoup ( fp , ‘html.parser’ ) soup = BeautifulSoup ( “a web page” , ‘html.parser’ ) First, the document is converted to Unicode, and HTML entities are converted to Unicode characters: print ( BeautifulSoup ( “Sacré bleu!” , “html.parser” )) # Sacré bleu! Beautiful Soup then parses the document using the best available parser. It will use an HTML parser unless you specifically tell it to use an XML parser. (See Parsing XML.)

Kinds of objects¶ Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. But you’ll only ever have to deal with about four kinds of objects: Tag , NavigableString , BeautifulSoup , and Comment . Tag ¶ A Tag object corresponds to an XML or HTML tag in the original document: soup = BeautifulSoup ( ‘Extremely bold‘ , ‘html.parser’ ) tag = soup . b type ( tag ) # Tags have a lot of attributes and methods, and I’ll cover most of them in Navigating the tree and Searching the tree. For now, the most important features of a tag are its name and attributes. Name¶ Every tag has a name, accessible as .name : tag . name # ‘b’ If you change a tag’s name, the change will be reflected in any HTML markup generated by Beautiful Soup: tag . name = “blockquote” tag #

Extremely bold

Attributes¶ A tag may have any number of attributes. The tag has an attribute “id” whose value is “boldest”. You can access a tag’s attributes by treating the tag like a dictionary: tag = BeautifulSoup ( ‘bold‘ , ‘html.parser’ ) . b tag [ ‘id’ ] # ‘boldest’ You can access that dictionary directly as .attrs : tag . attrs # {‘id’: ‘boldest’} You can add, remove, and modify a tag’s attributes. Again, this is done by treating the tag as a dictionary: tag [ ‘id’ ] = ‘verybold’ tag [ ‘another-attribute’ ] = 1 tag # del tag [ ‘id’ ] del tag [ ‘another-attribute’ ] tag # bold tag [ ‘id’ ] # KeyError: ‘id’ tag . get ( ‘id’ ) # None Multi-valued attributes¶ HTML 4 defines a few attributes that can have multiple values. HTML 5 removes a couple of them, but defines a few more. The most common multi-valued attribute is class (that is, a tag can have more than one CSS class). Others include rel , rev , accept-charset , headers , and accesskey . Beautiful Soup presents the value(s) of a multi-valued attribute as a list: css_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) css_soup . p [ ‘class’ ] # [‘body’] css_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) css_soup . p [ ‘class’ ] # [‘body’, ‘strikeout’] If an attribute looks like it has more than one value, but it’s not a multi-valued attribute as defined by any version of the HTML standard, Beautiful Soup will leave the attribute alone: id_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ ) id_soup . p [ ‘id’ ] # ‘my id’ When you turn a tag back into a string, multiple attribute values are consolidated: rel_soup = BeautifulSoup ( ‘

Back to the homepage

‘ , ‘html.parser’ ) rel_soup . a [ ‘rel’ ] # [‘index’] rel_soup . a [ ‘rel’ ] = [ ‘index’ , ‘contents’ ] print ( rel_soup . p ) #

Back to the homepage

You can disable this by passing multi_valued_attributes=None as a keyword argument into the BeautifulSoup constructor: no_list_soup = BeautifulSoup ( ‘

‘ , ‘html.parser’ , multi_valued_attributes = None ) no_list_soup . p [ ‘class’ ] # ‘body strikeout’ You can use get_attribute_list to get a value that’s always a list, whether or not it’s a multi-valued atribute: id_soup . p . get_attribute_list ( ‘id’ ) # [“my id”] If you parse a document as XML, there are no multi-valued attributes: xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ ) xml_soup . p [ ‘class’ ] # ‘body strikeout’ Again, you can configure this using the multi_valued_attributes argument: class_is_multi = { ‘*’ : ‘class’ } xml_soup = BeautifulSoup ( ‘

‘ , ‘xml’ , multi_valued_attributes = class_is_multi ) xml_soup . p [ ‘class’ ] # [‘body’, ‘strikeout’] You probably won’t need to do this, but if you do, use the defaults as a guide. They implement the rules described in the HTML specification: from bs4.builder import builder_registry builder_registry . lookup ( ‘html’ ) . DEFAULT_CDATA_LIST_ATTRIBUTES NavigableString ¶ A string corresponds to a bit of text within a tag. Beautiful Soup uses the NavigableString class to contain these bits of text: soup = BeautifulSoup ( ‘Extremely bold‘ , ‘html.parser’ ) tag = soup . b tag . string # ‘Extremely bold’ type ( tag . string ) # A NavigableString is just like a Python Unicode string, except that it also supports some of the features described in Navigating the tree and Searching the tree. You can convert a NavigableString to a Unicode string with str : unicode_string = str ( tag . string ) unicode_string # ‘Extremely bold’ type ( unicode_string ) # You can’t edit a string in place, but you can replace one string with another, using replace_with(): tag . string . replace_with ( “No longer bold” ) tag # No longer bold NavigableString supports most of the features described in Navigating the tree and Searching the tree, but not all of them. In particular, since a string can’t contain anything (the way a tag may contain a string or another tag), strings don’t support the .contents or .string attributes, or the find() method. If you want to use a NavigableString outside of Beautiful Soup, you should call unicode() on it to turn it into a normal Python Unicode string. If you don’t, your string will carry around a reference to the entire Beautiful Soup parse tree, even when you’re done using Beautiful Soup. This is a big waste of memory. BeautifulSoup ¶ The BeautifulSoup object represents the parsed document as a whole. For most purposes, you can treat it as a Tag object. This means it supports most of the methods described in Navigating the tree and Searching the tree. You can also pass a BeautifulSoup object into one of the methods defined in Modifying the tree, just as you would a Tag. This lets you do things like combine two parsed documents: doc = BeautifulSoup ( “INSERT FOOTER HEREHere’s the footer

” , “xml” ) doc . find ( text = “INSERT FOOTER HERE” ) . replace_with ( footer ) # ‘INSERT FOOTER HERE’ print ( doc ) # #

Since the BeautifulSoup object doesn’t correspond to an actual HTML or XML tag, it has no name and no attributes. But sometimes it’s useful to look at its .name , so it’s been given the special .name “[document]”: soup . name # ‘[document]’

Navigating the tree¶ Here’s the “Three sisters” HTML document again: html_doc = “””

The Dormouse’s story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

…

“”” from bs4 import BeautifulSoup soup = BeautifulSoup ( html_doc , ‘html.parser’ ) I’ll use this as an example to show you how to move from one part of a document to another. Going down¶ Tags may contain strings and other tags. These elements are the tag’s children . Beautiful Soup provides a lot of different attributes for navigating and iterating over a tag’s children. Note that Beautiful Soup strings don’t support any of these attributes, because a string can’t have children. Navigating using tag names¶ The simplest way to navigate the parse tree is to say the name of the tag you want. If you want the tag, just say soup.head : soup . head # soup . title # You can do use this trick again and again to zoom in on a certain part of the parse tree. This code gets the first tag beneath the tag: soup . body . b # The Dormouse’s story Using a tag name as an attribute will give you only the first tag by that name: soup . a # Elsie If you need to get all the tags, or anything more complicated than the first tag with a certain name, you’ll need to use one of the methods described in Searching the tree, such as find_all() : soup . find_all ( ‘a’ ) # [Elsie, # Lacie, # Tillie] .contents and .children ¶ A tag’s children are available in a list called .contents : head_tag = soup . head head_tag # head_tag . contents # [] title_tag = head_tag . contents [ 0 ] title_tag # title_tag . contents # [‘The Dormouse’s story’] The BeautifulSoup object itself has children. In this case, the tag is the child of the BeautifulSoup object.: len ( soup . contents ) # 1 soup . contents [ 0 ] . name # ‘html’ A string does not have .contents , because it can’t contain anything: text = title_tag . contents [ 0 ] text . contents # AttributeError: ‘NavigableString’ object has no attribute ‘contents’ Instead of getting them as a list, you can iterate over a tag’s children using the .children generator: for child in title_tag . children : print ( child ) # The Dormouse’s story If you want to modify a tag’s children, use the methods described in Modifying the tree. Don’t modify the the .contents list directly: that can lead to problems that are subtle and difficult to spot. .descendants ¶ The .contents and .children attributes only consider a tag’s direct children. For instance, the tag has a single direct child–the ] But the # The Dormouse’s story The tag has only one child, but it has two descendants: the ] head_tag . string # ‘The Dormouse’s story’ If a tag contains more than one thing, then it’s not clear what .string should refer to, so .string is defined to be None : print ( soup . html . string ) # None .strings and stripped_strings ¶ If there’s more than one thing inside a tag, you can still look at just the strings. Use the .strings generator: for string in soup . strings : print ( repr ( string )) ‘

‘ # “The Dormouse’s story” # ‘

‘ # ‘

‘ # “The Dormouse’s story” # ‘

‘ # ‘Once upon a time there were three little sisters; and their names were

‘ # ‘Elsie’ # ‘,

‘ # ‘Lacie’ # ‘ and

‘ # ‘Tillie’ # ‘;

and they lived at the bottom of a well.’ # ‘

‘ # ‘…’ # ‘

‘ These strings tend to have a lot of extra whitespace, which you can remove by using the .stripped_strings generator instead: for string in soup . stripped_strings : print ( repr ( string )) # “The Dormouse’s story” # “The Dormouse’s story” # ‘Once upon a time there were three little sisters; and their names were’ # ‘Elsie’ # ‘,’ # ‘Lacie’ # ‘and’ # ‘Tillie’ # ‘;

and they lived at the bottom of a well.’ # ‘…’ Here, strings consisting entirely of whitespace are ignored, and whitespace at the beginning and end of strings is removed. Going up¶ Continuing the “family tree” analogy, every tag and every string has a parent : the tag that contains it. .parent ¶ You can access an element’s parent with the .parent attribute. In the example “three sisters” document, the tag is the parent of the title_tag . parent # The title string itself has a parent: the The parent of a top-level tag like is the BeautifulSoup object itself: html_tag = soup . html type ( html_tag . parent ) # And the .parent of a BeautifulSoup object is defined as None: print ( soup . parent ) # None .parents ¶ You can iterate over all of an element’s parents with .parents . This example uses .parents to travel from an tag buried deep within the document, to the very top of the document: link = soup . a link # Elsie for parent in link . parents : print ( parent . name ) # p # body # html # [document] Going sideways¶ Consider a simple document like this: sibling_soup = BeautifulSoup ( “text1text2” , ‘html.parser’ ) print ( sibling_soup . prettify ()) # # # text1 # # # text2 # # The tag and the tag are at the same level: they’re both direct children of the same tag. We call them siblings . When a document is pretty-printed, siblings show up at the same indentation level. You can also use this relationship in the code you write. .next_sibling and .previous_sibling ¶ You can use .next_sibling and .previous_sibling to navigate between page elements that are on the same level of the parse tree: sibling_soup . b . next_sibling # text2 sibling_soup . c . previous_sibling # text1 The tag has a .next_sibling , but no .previous_sibling , because there’s nothing before the tag on the same level of the tree . For the same reason, the tag has a .previous_sibling but no .next_sibling : print ( sibling_soup . b . previous_sibling ) # None print ( sibling_soup . c . next_sibling ) # None The strings “text1” and “text2” are not siblings, because they don’t have the same parent: sibling_soup . b . string # ‘text1’ print ( sibling_soup . b . string . next_sibling ) # None In real documents, the .next_sibling or .previous_sibling of a tag will usually be a string containing whitespace. Going back to the “three sisters” document: # Elsie # Lacie # Tillie You might think that the .next_sibling of the first tag would be the second tag. But actually, it’s a string: the comma and newline that separate the first tag from the second: link = soup . a link # Elsie link . next_sibling # ‘,

‘ The second tag is actually the .next_sibling of the comma: link . next_sibling . next_sibling # Lacie .next_siblings and .previous_siblings ¶ You can iterate over a tag’s siblings with .next_siblings or .previous_siblings : for sibling in soup . a . next_siblings : print ( repr ( sibling )) # ‘,

‘ # Lacie # ‘ and

‘ # Tillie # ‘; and they lived at the bottom of a well.’ for sibling in soup . find ( id = “link3” ) . previous_siblings : print ( repr ( sibling )) # ‘ and

‘ # Lacie # ‘,

‘ # Elsie # ‘Once upon a time there were three little sisters; and their names were

‘ Going back and forth¶ Take a look at the beginning of the “three sisters” document: # #

The Dormouse’s story

An HTML parser takes this string of characters and turns it into a series of events: “open an tag”, “open a tag”, “open a

The Dormouse's story

Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.

...

""" print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_a_tags ) . prettify ()) # # Elsie # # # Lacie # # # Tillie # print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_tags_with_id_link2 ) . prettify ()) # # Lacie # print ( BeautifulSoup ( html_doc , "html.parser" , parse_only = only_short_strings ) . prettify ()) # Elsie # , # Lacie # and # Tillie # ... # You can also pass a SoupStrainer into any of the methods covered in Searching the tree. This probably isn’t terribly useful, but I thought I’d mention it: soup = BeautifulSoup ( html_doc , 'html.parser' ) soup . find_all ( only_short_strings ) # ['

', '

', 'Elsie', ',

', 'Lacie', ' and

', 'Tillie', # '

', '...', '

'] Customizing multi-valued attributes¶ In an HTML document, an attribute like class is given a list of values, and an attribute like id is given a single value, because the HTML specification treats those attributes differently: markup = '' soup = BeautifulSoup ( markup , 'html.parser' ) soup . a [ 'class' ] # ['cls1', 'cls2'] soup . a [ 'id' ] # 'id1 id2' You can turn this off by passing in multi_valued_attributes=None . Than all attributes will be given a single value: soup = BeautifulSoup ( markup , 'html.parser' , multi_valued_attributes = None ) soup . a [ 'class' ] # 'cls1 cls2' soup . a [ 'id' ] # 'id1 id2' You can customize this behavior quite a bit by passing in a dictionary for multi_valued_attributes . If you need this, look at HTMLTreeBuilder.DEFAULT_CDATA_LIST_ATTRIBUTES to see the configuration Beautiful Soup uses by default, which is based on the HTML specification. (This is a new feature in Beautiful Soup 4.8.0.) Handling duplicate attributes¶ When using the html.parser parser, you can use the on_duplicate_attribute constructor argument to customize what Beautiful Soup does when it encounters a tag that defines the same attribute more than once: markup = '' The default behavior is to use the last value found for the tag: soup = BeautifulSoup ( markup , 'html.parser' ) soup . a [ 'href' ] # http://url2/ soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = 'replace' ) soup . a [ 'href' ] # http://url2/ With on_duplicate_attribute='ignore' you can tell Beautiful Soup to use the first value found and ignore the rest: soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = 'ignore' ) soup . a [ 'href' ] # http://url1/ (lxml and html5lib always do it this way; their behavior can’t be configured from within Beautiful Soup.) If you need more, you can pass in a function that’s called on each duplicate value: def accumulate ( attributes_so_far , key , value ): if not isinstance ( attributes_so_far [ key ], list ): attributes_so_far [ key ] = [ attributes_so_far [ key ]] attributes_so_far [ key ] . append ( value ) soup = BeautifulSoup ( markup , 'html.parser' , on_duplicate_attribute = accumulate ) soup . a [ 'href' ] # ["http://url1/", "http://url2/"] (This is a new feature in Beautiful Soup 4.9.1.) Instantiating custom subclasses¶ When a parser tells Beautiful Soup about a tag or a string, Beautiful Soup will instantiate a Tag or NavigableString object to contain that information. Instead of that default behavior, you can tell Beautiful Soup to instantiate subclasses of Tag or NavigableString , subclasses you define with custom behavior: from bs4 import Tag , NavigableString class MyTag ( Tag ): pass class MyString ( NavigableString ): pass markup = "

some text

" soup = BeautifulSoup ( markup , 'html.parser' ) isinstance ( soup . div , MyTag ) # False isinstance ( soup . div . string , MyString ) # False my_classes = { Tag : MyTag , NavigableString : MyString } soup = BeautifulSoup ( markup , 'html.parser' , element_classes = my_classes ) isinstance ( soup . div , MyTag ) # True isinstance ( soup . div . string , MyString ) # True This can be useful when incorporating Beautiful Soup into a test framework. (This is a new feature in Beautiful Soup 4.8.1.)

Troubleshooting¶ diagnose() ¶ If you’re having trouble understanding what Beautiful Soup does to a document, pass the document into the diagnose() function. (New in Beautiful Soup 4.2.0.) Beautiful Soup will print out a report showing you how different parsers handle the document, and tell you if you’re missing a parser that Beautiful Soup could be using: from bs4.diagnose import diagnose with open ( "bad.html" ) as fp : data = fp . read () diagnose ( data ) # Diagnostic running on Beautiful Soup 4.2.0 # Python version 2.7.3 (default, Aug 1 2012, 05:16:07) # I noticed that html5lib is not installed. Installing it may help. # Found lxml version 2.3.2.0 # # Trying to parse your data with html.parser # Here's what html.parser did with the document: # ... Just looking at the output of diagnose() may show you how to solve the problem. Even if not, you can paste the output of diagnose() when asking for help. Errors when parsing a document¶ There are two different kinds of parse errors. There are crashes, where you feed a document to Beautiful Soup and it raises an exception, usually an HTMLParser.HTMLParseError . And there is unexpected behavior, where a Beautiful Soup parse tree looks a lot different than the document used to create it. Almost none of these problems turn out to be problems with Beautiful Soup. This is not because Beautiful Soup is an amazingly well-written piece of software. It’s because Beautiful Soup doesn’t include any parsing code. Instead, it relies on external parsers. If one parser isn’t working on a certain document, the best solution is to try a different parser. See Installing a parser for details and a parser comparison. The most common parse errors are HTMLParser.HTMLParseError: malformed start tag and HTMLParser.HTMLParseError: bad end tag . These are both generated by Python’s built-in HTML parser library, and the solution is to install lxml or html5lib. The most common type of unexpected behavior is that you can’t find a tag that you know is in the document. You saw it going in, but find_all() returns [] or find() returns None . This is another common problem with Python’s built-in HTML parser, which sometimes skips tags it doesn’t understand. Again, the best solution is to install lxml or html5lib. Version mismatch problems¶ SyntaxError: Invalid syntax (on the line ROOT_TAG_NAME = '[document]' ): Caused by running an old Python 2 version of Beautiful Soup under Python 3, without converting the code.

ImportError: No module named HTMLParser - Caused by running an old Python 2 version of Beautiful Soup under Python 3.

ImportError: No module named html.parser - Caused by running the Python 3 version of Beautiful Soup under Python 2.

ImportError: No module named BeautifulSoup - Caused by running Beautiful Soup 3 code on a system that doesn’t have BS3 installed. Or, by writing Beautiful Soup 4 code without knowing that the package name has changed to bs4 .

ImportError: No module named bs4 - Caused by running Beautiful Soup 4 code on a system that doesn’t have BS4 installed. Parsing XML¶ By default, Beautiful Soup parses documents as HTML. To parse a document as XML, pass in “xml” as the second argument to the BeautifulSoup constructor: soup = BeautifulSoup ( markup , "xml" ) You’ll need to have lxml installed. Other parser problems¶ If your script works on one computer but not another, or in one virtual environment but not another, or outside the virtual environment but not inside, it’s probably because the two environments have different parser libraries available. For example, you may have developed the script on a computer that has lxml installed, and then tried to run it on a computer that only has html5lib installed. See Differences between parsers for why this matters, and fix the problem by mentioning a specific parser library in the BeautifulSoup constructor.

Because HTML tags and attributes are case-insensitive, all three HTML parsers convert tag and attribute names to lowercase. That is, the markup is converted to . If you want to preserve mixed-case or uppercase tags and attributes, you’ll need to parse the document as XML. Miscellaneous¶ UnicodeEncodeError: 'charmap' codec can't encode character '\xfoo' in position bar (or just about any other UnicodeEncodeError ) - This problem shows up in two main situations. First, when you try to print a Unicode character that your console doesn’t know how to display. (See this page on the Python wiki for help.) Second, when you’re writing to a file and you pass in a Unicode character that’s not supported by your default encoding. In this case, the simplest solution is to explicitly encode the Unicode string into UTF-8 with u.encode("utf8") .

KeyError: [attr] - Caused by accessing tag['attr'] when the tag in question doesn’t define the attr attribute. The most common errors are KeyError: 'href' and KeyError: 'class' . Use tag.get('attr') if you’re not sure attr is defined, just as you would with a Python dictionary.

AttributeError: 'ResultSet' object has no attribute 'foo' - This usually happens because you expected find_all() to return a single tag or string. But find_all() returns a _list_ of tags and strings–a ResultSet object. You need to iterate over the list and look at the .foo of each one. Or, if you really only want one result, you need to use find() instead of find_all() .

AttributeError: 'NoneType' object has no attribute 'foo' - This usually happens because you called find() and then tried to access the .foo` attribute of the result. But in your case, find() didn’t find anything, so it returned None , instead of returning a tag or a string. You need to figure out why your find() call isn’t returning anything.

AttributeError: 'NavigableString' object has no attribute 'foo' - This usually happens because you’re treating a string as though it were a tag. You may be iterating over a list, expecting that it contains nothing but tags, when it actually contains both tags and strings. Improving Performance¶ Beautiful Soup will never be as fast as the parsers it sits on top of. If response time is critical, if you’re paying for computer time by the hour, or if there’s any other reason why computer time is more valuable than programmer time, you should forget about Beautiful Soup and work directly atop lxml. That said, there are things you can do to speed up Beautiful Soup. If you’re not using lxml as the underlying parser, my advice is to start. Beautiful Soup parses documents significantly faster using lxml than using html.parser or html5lib. You can speed up encoding detection significantly by installing the cchardet library. Parsing only part of a document won’t save you much time parsing the document, but it can save a lot of memory, and it’ll make searching the document much faster.

Translating this documentation¶ New translations of the Beautiful Soup documentation are greatly appreciated. Translations should be licensed under the MIT license, just like Beautiful Soup and its English documentation are. There are two ways of getting your translation into the main code base and onto the Beautiful Soup website: Create a branch of the Beautiful Soup repository, add your translation, and propose a merge with the main branch, the same as you would do with a proposed change to the source code. Send a message to the Beautiful Soup discussion group with a link to your translation, or attach your translation to the message. Use the Chinese or Brazilian Portuguese translations as your model. In particular, please translate the source file doc/source/index.rst , rather than the HTML version of the documentation. This makes it possible to publish the documentation in a variety of formats, not just HTML.

PyMOTM: Beautiful Soup 4 (Part I)

Beautiful Soup 4

Mục đích: Parse HTML, XML và Website scraping

Bạn muốn parse HTML, XML hay đơn giản là muốn scraping dữ liệu từ một website nào đó với ngôn ngữ Python? Trên Viblo cũng có một số bài của Anh Tranngoc về Scraping và Crawling dữ liệu từ một website nào đó với module Scrapy như: Kỹ thuật scraping và crawling Web nâng cao với Scrapy và SQLAlchemy hay Scraping và crawling Web với Scrapy và SQLAlchemy. Nay mình cũng xin phép giới thiệu một module cũng có thể scraping dữ liệu giống Scrapy nhưng nó nhẹ và đơn giản hơn cho những yêu cầu không đòi hỏi sự phức tạp như Scrapy, đó là Beautiful Soup. Chúng ta cùng đi tìm hiểu nó nhé!

Cài đặt

Trước tiên, chúng ta có thể kiểm tra xem trên máy của mình đã cài module này chưa hoặc nếu đã cài rồi, chúng ta xem mình đã ở phiên bản mới nhất chưa bằng 1 đoạn code ngắn trên terminal như sau:

python - c "import bs4; print(bs4.__version__);"

Sau khi thử đoạn code trên, nếu máy bạn đã cài rồi thì nó sẽ xuất hiện phiên bản của module BeautifulSoup. Còn không, bạn sẽ nhận được một lỗi như sau:

Traceback ( most recent call last ) : File "" , line 1 , in < module > ImportError : No module named bs4

Để cài Beautiful Soup 4, bạn có bốn cách là thông qua APT, PIP, EasyInstall hoặc thông qua source. Mình sẽ đưa ra cả bốn cách cài để bạn có thể lựa chọn cho phù hợp với sở thích của mình nhé

Qua APT: sudo apt-get install python-bs4

Qua PIP: sudo pip install beautifulsoup4

Qua EasyInstall: sudo easy_install beautifulsoup4

Qua source: Vào trang download của Beautiful Soup tại đây rồi download phiên bản mà bạn muốn sử dụng. Xả nén source bằng lệnh: tar -vfx Thay đổi thư mục hiện thời bằng lệnh: cd / Cài đặt: python setup.py install

Nếu tất cả cách trên đều không thể cài đặt được, thì bạn vẫn có thể sử dụng nó bằng cách copy thư mục bs4 ở phần cài đặt bằng source vào thư mục source code của bạn rồi sử dụng như bình thường !

Cài đặt parser

Beautiful Soup hỗ trợ thư viện HTML parser mặc định của Python và một số thư viện của bên thứ ba. Ví dụ như lxml parse hoặc html5lib parser. Mình sẽ hướng dẫn các bạn cài đặt cả 2 thư viện này nhé !

HTML5Lib

Để cài đặt HTML5Lib, bạn có thể làm theo 1 trong 3 cách:

Thông qua APT: sudo apt-get install python-html5lib

Thông qua PIP: pip install html5lib

Thông qua EasyInstall easy_install html5lib

LXML

Để có thể cài đặt được LXML parser, bạn cần phải có 3 package sau: libxml2-dev , libxslt1-dev và python-dev .

Cài đặt các required packages: sudo apt-get install libxml2-dev libxslt1-dev python-dev

Cài đặt LXML parser: Thông qua APT: sudo apt-get install python-lxml Thông qua PIP: pip install lxml Thông qua EasyInstall: easy_install lxml

Cách sử dụng các parser

Python html.parser BeautifulSoup(markup, "html.parser")

LXML HTML parser BeautifulSoup(markup, "lxml")

LXML XLM parser BeautifulSoup(markup, "lxml-xml") hoặc BeautifulSoup(markup, "xml")

HTML5Lib BeautifulSoup(markup, "html5lib")

Chuẩn bị nguyên liệu

Để thử Beautiful Soup, chúng ta sẽ tạo một file HTML với tên bs4.html trong thư mục tmp với nội dung như sau:

< html lang = " en " > < head > < meta charset = " UTF-8 " > < title > Document < link rel = " stylesheet " href = " css.css " type = " text/css " > < body > < div class = " items-list " > < div class = " item pull-right " > < p class = " title " > Item 001 < p class = " price " > Price: 01$ < p > < a href = " # " > Buy < div class = " item pull-right " > < p class = " title " > Item 002 < p class = " price " > Price: 02$ < p > < a href = " # " > Buy < div class = " item pull-right " > < p class = " title " > Item 003 < p class = " price " > Price: 03$ < p > < a href = " # " > Buy < div class = " item pull-right " > < p class = " title " > Item 004 < p class = " price " > Price: 04$ < p > < a href = " # " > Buy < div class = " item pull-right " > < p class = " title " > Item 005 < p class = " price " > Price: 05$ < p > < a href = " # " > Buy

Making the soup

Sau khi đã có đầy đủ nguyên liệu và dụng cụ rồi, chúng ta cùng nhau đi chế biến món soup nhé ). Để parser một tài liệu, bạn chỉ cần import BeautifulSoup từ thư viện bs4 rồi truyền file handle hoặc một chuỗi HTML (XML) vào constructor của BeautifulSoup là có thể sử dụng được ngay. Xem ra món này nấu khá đơn giản đấy nhỉ ?

from bs4 import BeautifulSoup from_file_handle = BeautifulSoup ( open ( '/tmp/bs4.html' ) ) from_string = BeautifulSoup ( 'data' )

Ở BeautifulSoup constructor, nếu bạn không truyền tên của parser mà bạn muốn sử dụng ở tham số thứ 2 thì mặc định nó sẽ sử dụng parser tốt nhất đang có sẵn trên hệ thống của bạn.

OK, bây giờ chúng ta sẽ đi vào chi tiết 1 chút về Beautiful Soup nhé. Trước khi đi vào chi tiết, chúng ta sẽ tạo 1 file code python đơn giản là đọc dữ liệu từ file bs4.html mà chúng ta đã chuẩn bị từ trước để sử dụng nhé. Trong file code này, mình có sử dụng module pdb (Python Debugger) để tiện cho việc debug code của chúng ta trên terminal. Nếu có thể, mình sẽ giới thiệu module pdb này trong bài viết tiếp theo của series PyMOTM này ! Còn việc sử dụng parser nào thì tùy bạn quyết định nhé. Mình sẽ dùng HTML5Lib !

from bs4 import BeautifulSoup import pdb html_dom = BeautifulSoup ( open ( '/tmp/bs4.html' ) , 'html5lib' ) pdb . set_trace ( ) ;

Let's go (gogo)!

Các loại object của Beautiful Soup

Beautiful Soup sẽ chuyển đổi tài liệu HTML sang Python tree object. Nhưng bạn chỉ cần quan tâm 4 loại object là: Tag , NavigableString , BeautifulSoup và Comment thôi nhé !

Tag

Là một HTML (XML) tag

( Pdb ) title_tag = html_dom . find ( 'title' ) ( Pdb ) type ( title_tag ) < class 'bs4.element.Tag' > ( Pdb )

Name

Tên của tag. Bạn có thể lấy ra bằng cách sử dụng .name

( Pdb ) title_tag . name u 'title'

Bạn cũng có thể thay đổi tên của tag bằng cách gán cho nó 1 cái tên

( Pdb ) title_tag . name = 'blockquote' ( Pdb ) title_tag < blockquote > Document < / blockquote >

Attributes

Một tag có thể có một hoặc nhiều attribute. Bạn có thể lấy giá trị của một attribute như sau:

( Pdb ) link_tag = html_dom . find ( 'link' ) ( Pdb ) link_tag [ 'rel' ] [ u 'stylesheet' ]

Hoặc bạn cũng có thể lấy ra tất cả các attribute của 1 tag bằng .attrs . Dữ liệu trả ra là 1 dictionary:

(Pdb) link_tag.attrs { u 'href' : u 'css.css' , u 'type' : u 'text/css' , u 'rel' : [u 'stylesheet' ] }

Bạn cũng có thể thêm/sửa/xóa một attribute của tag:

(Pdb) link_tag['href'] = 'lorem.css' (Pdb) link_tag < link href = " lorem.css " rel = " stylesheet " type = " text/css " /> (Pdb) del link_tag['rel'] (Pdb) link_tag < link href = " lorem.css " type = " text/css " /> (Pdb) link_tag['media'] = 'all' (Pdb) link_tag < link href = " lorem.css " media = " all " type = " text/css " />

Multi-valued attributes

Ở HTML4 nó định nghĩa khá nhiều attribute được phép nhiều hơn một giá trị (các giá trị cách nhau 1 space). Nhưng sang HTML5 thì nó đã được định nghĩa lại chỉ còn một vài attribute được phép nhiều hơn 1 giá trị là class , rel , rev , accept-charset , headers và accesskey . Kết quả trả về là một danh sách các giá trị của attribute đó (nếu nó thuộc các attribute được liệt kê ở trên), hoặc là một string (nếu attribute đó không thuộc danh sách được liệt kê ở trên):

( Pdb ) items = html_dom . select ( '.items-list .item' ) ( Pdb ) items [ 0 ] [ 'class' ] [ u'item' , u'pull-right' ]

Bạn cũng có thể thêm/sửa/xóa một giá trị trong danh sách các giá trị của attribute đó:

(Pdb) del items[0]['class'][0] (Pdb) items[0]['class'] [u'pull-right'] (Pdb) items[0]['class'] = ['lorem', 'lipsum'] (Pdb) items[0]['class'] ['lorem', 'lipsum'] (Pdb) items[0]['class'][0] = 'lorem_updated' (Pdb) items[0]['class'] ['lorem_updated', 'lipsum']

NavigableString

Là nội dung text của tag đó. NavigableString giống như một Python Unicode string, ngoại trừ việc nó hỗ trợ bạn một số tính năng như duyệt và tìm kiếm (sẽ được giới thiệu ở bài tiếp theo). Bạn cũng có thể convert nó sang kiểu Unicode string bằng function unicode()

( Pdb ) title_tag . string u 'Document' ( Pdb ) type ( title_tag . string ) < class 'bs4.element.NavigableString' > ( Pdb ) unicode_title = unicode ( title_tag . string ) ( Pdb ) type ( unicode_title ) < type 'unicode' >

Bạn không thể sửa nội dung văn bản của 1 tag bằng cách trực tiếp, nhưng có thể thay thế nó thông qua function replace_with() :

( Pdb ) title_tag . string u 'Document' ( Pdb ) title_tag . string . replace_with ( 'PyMOTM: BeautifulSoup4' ) u 'Document' ( Pdb ) title_tag < title > PyMOTM: BeautifulSoup4 < / title >

Bài viết cũng đã khá dài rồi mà chúng ta mới chỉ đi chưa được một nửa của Beautiful Soup, nên mình xin phép tạm dừng tại đây nhé. Phần 2 mình sẽ giới thiệu về việc duyệt và tìm kiếm nhé !

Kỹ thuật Scraping Trang web trong Python bằng Beautiful Soup: Cơ bản

Subscribe below and we’ll send you a weekly email summary of all new Code tutorials. Never miss out on learning about the next big thing.

Monty Shokeen Freelancer, India

I am a full-stack developer who also loves to write tutorials. After trying out a bunch of things till second year of college, I decided to work on my web development skills. Starting with just HTML and CSS, I kept moving forward and gained experience in PHP, JavaScript and Python. I usually spend my free time either working on some side projects or traveling around.

Beautiful Soup - Installation

Beautiful Soup - Installation

Advertisements

As BeautifulSoup is not a standard python library, we need to install it first. We are going to install the BeautifulSoup 4 library (also known as BS4), which is the latest one.

To isolate our working environment so as not to disturb the existing setup, let us first create a virtual environment.

Creating a virtual environment (optional)

A virtual environment allows us to create an isolated working copy of python for a specific project without affecting the outside setup.

Best way to install any python package machine is using pip, however, if pip is not installed already (you can check it using – “pip –version” in your command or shell prompt), you can install by giving below command −

Linux environment

$sudo apt-get install python-pip

Windows environment

To install pip in windows, do the following −

Download the get-pip.py from https://bootstrap.pypa.io/get-pip.py or from the github to your computer.

Open the command prompt and navigate to the folder containing get-pip.py file.

Run the following command −

>python get-pip.py

That’s it, pip is now installed in your windows machine.

You can verify your pip installed by running below command −

>pip --version pip 19.2.3 from c:\users\yadur\appdata\local\programs\python\python37\lib\site-packages\pip (python 3.7)

Installing virtual environment

Run the below command in your command prompt −

>pip install virtualenv

After running, you will see the below screenshot −

Below command will create a virtual environment (“myEnv”) in your current directory −

>virtualenv myEnv

Screenshot

To activate your virtual environment, run the following command −

>myEnv\Scripts\activate

In the above screenshot, you can see we have “myEnv” as prefix which tells us that we are under virtual environment “myEnv”.

To come out of virtual environment, run deactivate.

(myEnv) C:\Users\yadur>deactivate C:\Users\yadur>

As our virtual environment is ready, now let us install beautifulsoup.

Installing BeautifulSoup

As BeautifulSoup is not a standard library, we need to install it. We are going to use the BeautifulSoup 4 package (known as bs4).

Linux Machine

To install bs4 on Debian or Ubuntu linux using system package manager, run the below command −

$sudo apt-get install python-bs4 (for python 2.x) $sudo apt-get install python3-bs4 (for python 3.x)

You can install bs4 using easy_install or pip (in case you find problem in installing using system packager).

$easy_install beautifulsoup4 $pip install beautifulsoup4

(You may need to use easy_install3 or pip3 respectively if you’re using python3)

Windows Machine

To install beautifulsoup4 in windows is very simple, especially if you have pip already installed.

>pip install beautifulsoup4

So now beautifulsoup4 is installed in our machine. Let us talk about some problems encountered after installation.

Problems after installation

On windows machine you might encounter, wrong version being installed error mainly through −

error: ImportError “No module named HTMLParser” , then you must be running python 2 version of the code under Python 3.

error: ImportError “No module named html.parser” error, then you must be running Python 3 version of the code under Python 2.

Best way to get out of above two situations is to re-install the BeautifulSoup again, completely removing existing installation.

If you get the SyntaxError “Invalid syntax” on the line ROOT_TAG_NAME = u’[document]’, then you need to convert the python 2 code to python 3, just by either installing the package −

$ python3 setup.py install

or by manually running python’s 2 to 3 conversion script on the bs4 directory −

$ 2to3-3.2 -w bs4

Installing a Parser

By default, Beautiful Soup supports the HTML parser included in Python’s standard library, however it also supports many external third party python parsers like lxml parser or html5lib parser.

To install lxml or html5lib parser, use the command −

Linux Machine

$apt-get install python-lxml $apt-get insall python-html5lib

Windows Machine

$pip install lxml $pip install html5lib

Generally, users use lxml for speed and it is recommended to use lxml or html5lib parser if you are using older version of python 2 (before 2.7.3 version) or python 3 (before 3.2.2) as python’s built-in HTML parser is not very good in handling older version.

Running Beautiful Soup

It is time to test our Beautiful Soup package in one of the html pages (taking web page – https://www.tutorialspoint.com/index.htm, you can choose any-other web page you want) and extract some information from it.

In the below code, we are trying to extract the title from the webpage −

from bs4 import BeautifulSoup import requests url = "https://www.tutorialspoint.com/index.htm" req = requests.get(url) soup = BeautifulSoup(req.text, "html.parser") print(soup.title)

Output

One common task is to extract all the URLs within a webpage. For that we just need to add the below line of code −

for link in soup.find_all('a'): print(link.get('href'))

Output

https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/about/about_careers.htm https://www.tutorialspoint.com/questions/index.php https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/codingground.htm https://www.tutorialspoint.com/current_affairs.htm https://www.tutorialspoint.com/upsc_ias_exams.htm https://www.tutorialspoint.com/tutor_connect/index.php https://www.tutorialspoint.com/whiteboard.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/tutorialslibrary.htm https://www.tutorialspoint.com/videotutorials/index.php https://store.tutorialspoint.com https://www.tutorialspoint.com/gate_exams_tutorials.htm https://www.tutorialspoint.com/html_online_training/index.asp https://www.tutorialspoint.com/css_online_training/index.asp https://www.tutorialspoint.com/3d_animation_online_training/index.asp https://www.tutorialspoint.com/swift_4_online_training/index.asp https://www.tutorialspoint.com/blockchain_online_training/index.asp https://www.tutorialspoint.com/reactjs_online_training/index.asp https://www.tutorix.com https://www.tutorialspoint.com/videotutorials/top-courses.php https://www.tutorialspoint.com/the_full_stack_web_development/index.asp …. …. https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/free_web_graphics.htm https://www.tutorialspoint.com/online_file_conversion.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/free_online_whiteboard.htm http://www.tutorialspoint.com https://www.facebook.com/tutorialspointindia https://plus.google.com/u/0/+tutorialspoint http://www.twitter.com/tutorialspoint http://www.linkedin.com/company/tutorialspoint https://www.youtube.com/channel/UCVLbzhxVTiTLiVKeGV7WEBg https://www.tutorialspoint.com/index.htm /about/about_privacy.htm#cookies /about/faq.htm /about/about_helping.htm /about/contact_us.htm

Similarly, we can extract useful information using beautifulsoup4.

Now let us understand more about “soup” in above example.

コメントセクションでBeautiful Soup 4 Tutorial #1 - Web Scraping With Pythonに関連する詳細情報を参照するか、トピックに関連するその他の記事を参照してくださいbeautifulsoup4 python.

キーワードに関する情報 beautifulsoup4 python

以下はの検索結果です beautifulsoup4 python Bingサイトから. 必要に応じてもっと読むことができます.

投稿 beautifulsoup4 python - Beautiful Soup 4 Tutorial #1 - Web Scraping With Python インターネット上のさまざまな情報源から編集しました。この記事がお役に立てば幸いです。より多くの人に見てもらえるように共有して応援してください！ありがとうございました！

記事のキーワード Beautiful Soup 4 Tutorial #1 - Web Scraping With Python

• tech with tim
• beautiful soup 4
• web scraping
• web scraping with python
• python web scraping
• web scraping python
• web scraping python beautifulsoup
• python
• html
• html files
• tag name
• parsing website html
• locating text
• tree structure
• beautiful soup tree structure
• beautiful
• soup
• beautiful soup 4 python tutorial
• beautiful soup module in python
• beautiful soup 4 python
• html parsing python
• python web scraping tutorial
• web scraping using python
• python tree structure code

Beautiful #Soup #4 #Tutorial ##1 #- #Web #Scraping #With #Python

---

Youtubeでトピックbeautifulsoup4 pythonに関するビデオをもっと見る

---

また、最新のニュースレターでキーワードbeautifulsoup4 pythonに関連するニュースをさらに見ることができます。.

トピック beautifulsoup4 python - Beautiful Soup 4 Tutorial #1 - Web Scraping With Python に関する記事の表示が終了しました。この記事の情報が役に立った場合は、共有してください。どうもありがとうございます。

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Name *

Email *

Website

Comment *

Save my name, email, and website in this browser for the next time I comment.

fapjunk