In the first of this 2-part mini-series on scraping Amazon for data, I described how to write a python code to auto-login to your Amazon.com account. Here I’ll detail another code that automates the process of searching Amazon for a particular item, filtered by its brand name, star rating, and price range. While logging in […]
Scraping Amazon with selenium (P.2): Price lookup Read More »
In the previous post, I showed an example of scraping dynamic websites with python’s requests-html package. While requests-html is a great light-weight tool to harvest real-time data, modern websites can easily recognize it as a bot script from the user-agent string (looks something like python-requests/2.28.2), and may refuse entry after a while. The selenium package
For a data scientist, web scraping, or harvesting information from websites (when done legally) is often the only way to get data. There are excellent python libraries available for scraping, such as beautifulsoup, requests-html, selenium, and scrapy. The choice usually comes down to whether the website serves static or dynamic content, interaction with the server,
WordPress is the most popular CMS (content management system) platform available today, powering 43% of all websites on the Internet, including this one. To install WordPress locally on a computer requires first creating a “localhost” consisting of the Apache web server, MySQL (or its community fork MariaDB) database management system, and optionally phpMyAdmin for administering
Those like me who use Calibre to manage their ereader library are familiar with the hassle of syncing between the two. Standard choices are to either 1) physically hook up your ereader to the computer where Calibre is installed, or 2) access Calibre server or a similar service over your local network. I am not
Implementing a kriging algorithm for modeling spatial data is not difficult, unless you choose to code every step yourself. Once the data is prepared with correct spatial attributes, the legwork is done by software packages available in all major programming languages (here is an example with Baltimore crime data implemented in R). Sometimes it helps
In the previous post I analyzed a 12-yr long crime data from 2011 to 2022 in the city of Baltimore, and described the yearly trends and spatial patterns observed in the crime activity. Here I talk about how to model this data to predict crimes anywhere in the city. I chose year 2022 data to
Availability of high-quality data on the time and location of major crimes in the city of Baltimore makes studying the characteristics of such crimes a data scientist’s delight. I have used a 12-year long dataset spanning from Jan-2011 to Dec-2022 (updated from the time of posting) to explore trends and patterns of violent crimes within
