Long PHP process with HTML requests results in Error 502

html
php

#1

Hi everybody,

I’m currently working on a project where I get a list of games on Steam, and collect videos from those games to put together. At the moment I’ve got the first part working where I parse the games from the Steam Store page. At the moment that means loading around 33 pages. Everything is a-okay when I limit the loading loop to 30 pages, but 33 is apparently too much too handle, resulting in a Error 502 page from C9.

Error:

<html>
  <head>
    <meta charset='utf-8'> 
    <title>Error 502 - Bad Gateway</title>
    <link rel="stylesheet" type="text/css" href="https://cdn.c9.io/errors/style.css" />
    <style type="text/css">
    .error_content {
        background: rgba(255, 255, 255, 0.23);
        padding: 10px;
        width: 641px;
        margin: 25px 0;
        display: none;
    }
    
    #error-msg {
        display: block;
    }
    </style>
  </head>
  <body class="errorUnknown light">
    <div id="wrapper">
      <h1>Error 502 - Bad Gateway</h1>
      <div class="error_content" id="error-msg">
          <p>Please click <a href="javascript:location.reload(true)">here</a> to try again, if the issue persists please contact <a href="https://c9.io/support">support</a></p>
      </div>
      
      <a href="http://status.c9.io">Status Page</a> |
      <a href="https://c9.io/support">Support</a> |
      <a href="https://c9.io/dashboard.html">Dashboard</a> |
      <a href="https://c9.io">Home</a>
    </div>
  </body>
</html>

The loop loading the pages:

//Iterate though all the pages
    for ($i = 1; $i < $gameAmount; $i++) {
        //Wait a second or two as to not overload something
        sleep(2);
        //Get content of current page
        $gamePage = file_get_html($steamURL . "&page=" . $i);
        //Loop through all the result rows
        foreach($gamePage->find('.search_result_row') as $game) {
            //Get the game title from the span element
            $games[] = $game->find('span.title', 0)->plaintext;
        }
    }