In the first of this 2-part mini-series on scraping Amazon for data, I described how to write a python code to auto-login to your Amazon.com account. Here I’ll detail another code that automates the process of searching Amazon for a particular item, filtered by its brand name, star rating, and price range. While logging in […]
Scraping Amazon with selenium (P.2): Price lookup Read More »
In the previous post, I showed an example of scraping dynamic websites with python’s requests-html package. While requests-html is a great light-weight tool to harvest real-time data, modern websites can easily recognize it as a bot script from the user-agent string (looks something like python-requests/2.28.2), and may refuse entry after a while. The selenium package
For a data scientist, web scraping, or harvesting information from websites (when done legally) is often the only way to get data. There are excellent python libraries available for scraping, such as beautifulsoup, requests-html, selenium, and scrapy. The choice usually comes down to whether the website serves static or dynamic content, interaction with the server,
Those like me who use Calibre to manage their ereader library are familiar with the hassle of syncing between the two. Standard choices are to either 1) physically hook up your ereader to the computer where Calibre is installed, or 2) access Calibre server or a similar service over your local network. I am not
