Scraping In The Name Of!

Extracting The Data

We will be using the Client URL Library (cURL) to do our web scraping. You can view the documentation at http://php.net/manual/en/book.curl.php. This library allows us to communicate with different types of servers using multiple different protocols. We are interested in connecting to weather.com's Apache server via HTTP so we are golden because cURL supports both!

Now that we've found our method of extraction, let's begin writing the code to do the work. Let's begin by writing the code necessary to extract information for current conditions of our first location, Silverthorne:

Code:

<?php
    $url = "http://www.weather.com/weather/right-now/Silverthorne+CO+USCO0357";
    $crl = curl_init();
 
    curl_setopt ($crl, CURLOPT_URL, $url);
    curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, 5);
 
    $content = curl_exec ($crl);
    curl_close ($crl);
?>

If you print out the variable $content, you will see the HTML from the URL you loaded. Since we will have to run this same code for all the pages from which we want to extract information, we should put it in a function, passing the URL as a parameter:

Code:

<?php
    function scrapeWebsite ($url)
    {
        $crl = curl_init ();
 
        curl_setopt ($crl, CURLOPT_URL, $url);
        curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, 5);
 
        $content = curl_exec ($crl);
        curl_close ($crl);
 
        return $content;
    }
?>

Now we have a function ready to scrape any pages we want. Next, we need to pass each of the URLs to our function and store the scraped data so we can interpret and manipulate it. We could build an array to store the data and pass in each URL as follows:

Code:

<?php 
    function scrapeWebsite ($url)
    {
        $crl = curl_init ();
 
        curl_setopt ($crl, CURLOPT_URL, $url);
        curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, 5);
 
        $content = curl_exec ($crl);
        curl_close ($crl);
 
        return $content;
    }
 
    $weather = array ();
    $weather['silverthorne'] = array (
        'current' => array (
            'url' => 'http://www.weather.com/weather/right-now/Silverthorne+CO+USCO0357'
            'content' => ""
        ),
        'forecast' => array (
            'url' => 'http://www.weather.com/weather/today/Silverthorne+CO+USCO0357:1:US'
            'content' => ""
        )
    );
 
    $weather['silverthorne']['current']['content'] = scrapeWebsite($weather['silverthorne']['current']['url']);
    $weather['silverthorne']['forecast']['content'] = scrapeWebsite($weather['silverthorne']['forecast']['url']);
?>

This would work and we could add as many cities as we’d like, but would also mean some manual work every time we wanted to add or delete a city. Instead, let’s make the software do the work for us! Let’s modify our function so all we need to do is pass it a URL and our storage array and it will do the rest! Warning, fancy thinking ahead!

If you look at our URLs, each city has a code associated with it. For example, Silverthorne’s code is USCO0357. Weather.com accepts this code alone in its URL (as opposed to the entire “Silverthorne+CO+USCO0357:1:US” string), making it possible for us to create a standard pattern for our URLs. Let’s split up the URL for current conditions in Silverthorne to recreate an array similar to how we planned our original one:

So our array will have an entry for each location code, and each location code will have an entry for each page from which we want data. We will use regular expressions to parse each URL for the $page and $code. You can learn more about regular expressions in PHP at http://php.net/manual/en/book.pcre.php.

Please note we MUST escape slashes ‘/’ with backslashes ‘\’ in our pattern when using preg_match() since slash ‘/’ normally denotes the beginning and end of the pattern. Code:

<?php
    function scrapeWebsite ($url, &$weather)
    {
        // Parse the URL to retrieve the city name and page
        $result = preg_match ("/^.*\/weather\/(?P<page>[^\/]+)\/(?P<code>[^\/]+)$/", $url, $matches);
 
        // If the result from preg_match is not 1, the pattern was not found so return nothing
        if ($result !== 1)
        {
            return false;
        }
        else
        {
            $page = $matches['page'];
            $code = $matches['code'];
        }
 
        // If the code has not yet been added to the container, create it
        if (!isset($weather[$code]))
        {
            $weather[$code] = array ();
        }
 
        // Initialize a new session and return a cURL handle
        $crl = curl_init ();
 
        // Set options for cURL
        curl_setopt ($crl, CURLOPT_URL, $url); // The URL to fetch
        curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1); // Return the transfer as a string
        curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, 5); // Allow 5 seconds for connecting
 
        // Execute the given cURL session
        $content = curl_exec ($crl);
 
        // Store the content in the container using $page as the key
        $weather[$code][$page] = $content;
 
        // Close the cURL session
        curl_close ($crl);
    }
 
    $weather = array ();
    $urls = array ('http://www.weather.com/weather/right-now/USCO0357', 'http://www.weather.com/weather/today/USCO0357', 'http://www.weather.com/weather/right-now/USCO0105', 'http://www.weather.com/weather/today/USCO0105');
 
    foreach ($urls as $url)
    {
        scrapeWebsite ($url, $weather);
    }
?>

If you print out the contents of the $weather array now, it will show the HTML for each of the four pages we supplied.

Please note, your URLs MUST be in the pattern http://www.weather.com/weather/$page/$code for this to work. Result:

Array (
        [USCO0357] => Array (
                [right-now] => (HTML)
                [today] => (HTML)
        )
        [USCO0105] => Array (
                [right-now] => (HTML)
                [today] => (HTML)
        )
)

Success! The beauty of doing it this way is we can add as many locations as we’d like as well as include any pages (5-day forecast, 10-day forecast, etc) too. As we have done in the past, let’s look at the regular expression in greater detail.

Regex:

This regex is pretty simple but let’s translate it to English to be absolutely clear what is happening.

Regex	English
^.*/weather/	Starting at the beginning of the string, match any characters until reaching a slash ‘/’ followed by the string “weather” and another slash ‘/’
(?P<page>[^/]+)/	Match any characters that are NOT a slash ‘/’ until reaching a slash ‘/’ and capture as a group named ‘page’
(?P<code>[^/]+)$	Match any characters that are NOT a slash ‘/’ until reaching the end of the string and capture as a group named ‘code’

Now let’s do something useful with that data!