Download URL with PHP Curl emulating firefox browser

There exist many ways to download a recourse using PHP, It is possible across FTP or http. However we need to know choose the better way to doing if we want obtain always success result.

Let's see how easy it can be using the function: file_get_contents

$content = file_get_contents(“http://www.moises-soft.net”);

However we must take in account that some servers prevents the use of download automatic tools.

The way to know if the server must answers or not a request:

1. In the first place by the (IP) from where the request, but this article is not intended to teach as to circumvent a restriction on IP , only disclose that it is can a reason that you have not access to a recourse . For example , Cuba have not access to somes google services as Google ADsence.


2. Second by the headers sending in the HTTP request. Normally each browser (IExplorer, Firefox, etc) sends data in their headers with useful information. A way to evader a programmer download by a software is to compare the headers to know if the request is make by a browser and therefore by a person.

How to send requests as firefox browser?


function getURL($url)
{
$curl = curl_init();

// headers send by firefox

$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";

// browsers keep this blank.

$referers = array("google.com", "yahoo.com", "msn.com", "ask.com", "live.com");
$choice = array_rand($referers);
$referer = "http://" . $referers[$choice] . "";

$browsers = array("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Firefox/3.0.3", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)");

$choice2 = array_rand($browsers);

$browser = $browsers[$choice2];

curl_setopt($curl, CURLOPT_URL, $url);

curl_setopt($curl, CURLOPT_USERAGENT, $browser);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_REFERER, $referer);
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 30);
curl_setopt($curl, CURLOPT_MAXREDIRS, 7);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

// ejecuta el comando

$data = curl_exec($curl);

if ($data === false) {

$data = curl_error($curl);

}

// cierra la conexión

curl_close($curl);

return $data;

}

Keyword : download, URL, PHP, browser, CURL, emulate, download url php, curl url

Descargar URL con PHP (curl) emulando navegador FIREFOX

Existen en PHP formas varias formas de descargar un recurso , ya sea de a través de ftp o de http sin embargo hay que saber elegir la mejor forma de hacerlo si queremos lograr siempre resultados satisfactorios.
Veamos que sencillo puede ser usando la función: file_get_contents

Tan simple como:
$contenido = file_get_contents(“http://www.moises-soft.net”);

Sin embargo hay que tener en cuenta que algunos servidores previenen el uso de herramientas que automáticamente descargan recursos, ya sea con buenas o malas intenciones.
Las formas que tienen de saber los servidores si responden o no a una petición:

En primer lugar por el IP de donde viene la solicitud, sin embargo este artículo no tiene como objetivo ensenar como burlar una restricción de IP , solo dar a conocer que este puede ser el motivo de que usted no tenga acceso algún recurso. Por ejemplo Cuba no tiene acceso a algunas páginas de Google como es Google ADsence.

En segundo lugar por las cabeceras enviadas en la petición HTTP. Normalmente cada navegador (IExplorer, Firefox, etc) envía datos en sus cabeceras con mucha información útil. Un forma de burlar una descaga programada por cualquier programa es comprara las cabeceras enviadas para saber si no es petición de un navegador y por consiguiente de una persona.



¿Cómo enviar peticiones como si provinieran de Firefox?

function getURL($url)
{
$curl = curl_init();

// cabeceras enviadas por firefox
$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";



// browsers keep this blank.
$referers = array("google.com", "yahoo.com", "msn.com", "ask.com", "live.com");
$choice = array_rand($referers);
$referer = "http://" . $referers[$choice] . "";

$browsers = array("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Firefox/3.0.3", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20060918 Firefox/2.0", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)");
$choice2 = array_rand($browsers);
$browser = $browsers[$choice2];

curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_USERAGENT, $browser);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_REFERER, $referer);
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 30);
curl_setopt($curl, CURLOPT_MAXREDIRS, 7);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

// ejecuta el comando
$data = curl_exec($curl);

if ($data === false) {
$data = curl_error($curl);
}

// cierra la conexión
curl_close($curl);


return $data;

}