Difference between revisions of "Web Scraping with BeautifulSoup"
From EBA_Documentation
Brad Holland (Talk | contribs) |
Brad Holland (Talk | contribs) (→Easier Scrapes) |
||
| Line 27: | Line 27: | ||
===== Easier Scrapes ===== | ===== Easier Scrapes ===== | ||
| − | [https://onedrive.live.com/redir?resid=18C32DCEEE22CD57! | + | [https://onedrive.live.com/redir?resid=18C32DCEEE22CD57!58391&authkey=!AIT7d2U1Ea7qsKs&ithint=file%2czip Scrape tabular data from Northeastern University's Women Writers Project] |
HTML included from [http://www.wwp.northeastern.edu/wwo/texts/titleURLs_TR.html this website] | HTML included from [http://www.wwp.northeastern.edu/wwo/texts/titleURLs_TR.html this website] | ||
| + | |||
===== Difficult Scrapes ===== | ===== Difficult Scrapes ===== | ||
[https://onedrive.live.com/redir?resid=18C32DCEEE22CD57!57838&authkey=!AM2aYhRjgYldJd8&ithint=file%2czip Scrape and extract inner-document names and their hidden biographies from Mitford Letters] | [https://onedrive.live.com/redir?resid=18C32DCEEE22CD57!57838&authkey=!AM2aYhRjgYldJd8&ithint=file%2czip Scrape and extract inner-document names and their hidden biographies from Mitford Letters] | ||
HTML included from [http://digitalmitford.org/1825-06-29-Talfourd.html this website] | HTML included from [http://digitalmitford.org/1825-06-29-Talfourd.html this website] | ||
Latest revision as of 11:22, 4 February 2016
Beautiful Soup Homepage We use version 4
Beautiful Soup
From the homepage:
"Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. These features make it powerful:
- Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. It doesn't take much code to write an application
- Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't detect one. Then you just have to specify the original encoding.
- Beautiful Soup sits on top of popular Python parsers like lxml and html5lib, allowing you to try out different parsing strategies or trade speed for flexibility.
- Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. You can tell it "Find all the links", or "Find all the links of class externalLink", or "Find all the links whose urls match "foo.com", or "Find the table heading that's got bold text, then give me that text."
Valuable data that was once locked up in poorly-designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup."
Resources
- Installing Beautiful Soup (and Python) by Brad
- Official Beautiful Soup documentation
- Great step-by-step scraping exercise by The Programming Historian
Examples
Note: in order to run these programs you must have both Python and Beautiful Soup installed. See above links for installation instructions.
- Unzip the files into a folder on your computer. Open up the Python IDLE program and click File -> Open...
- Browse to the files you extracted and choose the .py file
- Once the Python program opens, press F5 or click Run -> Run Module to start the program
- To see the data output, open up the included .csv file
Easier Scrapes
Scrape tabular data from Northeastern University's Women Writers Project
HTML included from this website
Difficult Scrapes
Scrape and extract inner-document names and their hidden biographies from Mitford Letters
HTML included from this website
