Linux curl via proxy

How to use curl command with proxy username/password on Linux/ Unix

Unix and Linux curl command with proxy syntax

Another option is to pass the -x option to the curl command. To use the specified proxy:

Linux use curl command with proxy

First set the http_proxy:

In this example, I’m downloading a pdf file:
$ export http_proxy=»vivek:myPasswordHere@10.12.249.194:3128/»
$ curl -v -O http://dl.cyberciti.biz/pdfdownloads/b8bf71be9da19d3feeee27a0a6960cb3/569b7f08/cms/631.pdf
OR use the -x option:
curl -x ‘http://vivek:myPasswordHere@10.12.249.194:3128’ -v -O https://dl.cyberciti.biz/pdfdownloads/b8bf71be9da19d3feeee27a0a6960cb3/569b7f08/cms/631.pdf
Sample outputs:

Fig.01: curl in action (click to enlarge)

How to use the specified proxy server with curl on Unix

$ curl -x http://prox_server_vpn:3128/ -I https://www.cyberciti.biz/faq/howto-nginx-customizing-404-403-error-page/

How to use socks protocol?

The syntax is same:

How do I configure and setup curl to permanently use a proxy connection?

/.curlrc file using a text editor such as vim:
$ vi

/.curlrc
Append the following:

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Save and close the file. Another option is create a bash shell alias in your

Remember, the proxy string can be specified with a protocol:// prefix to specify alternative proxy protocols. Use socks4://, socks4a://, socks5:// or socks5h:// to request the specific SOCKS version to be used. No protocol specified, http:// and all others will be treated as HTTP proxies. If the port number is not specified in the proxy string, it is assumed to be 1080. The -x option overrides existing environment variables that set the proxy to use. If there’s an environment variable setting a proxy, you can set proxy to “” to override it. See curl command man page here for more info.

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Category	List of Unix and Linux commands
Documentation	help • mandb • man • pinfo
Disk space analyzers	df • duf • ncdu • pydf
File Management	cat • cp • less • mkdir • more • tree
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Linux Desktop Apps	Skype • Spotify • VLC 3
Modern utilities	bat • exa
Network Utilities	NetHogs • dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtop
Searching	ag • grep • whereis • which
Shell builtins	compgen • echo • printf
Text processing	cut • rev
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04

Comments on this entry are closed.

I want to learn linux + networking for example iptables etc.
What is the best place to start?

I need a help to curl comand to download a large files from google drive, nothing works for me, sorry about my poor english.

thx for the post. this is exactly what i needed.

Источник

Использование HTTP proxy и SOCKS в Linux

В Linux существует много полезных консольных команд, которые при необходимости хотелось бы запустить через proxy. Некоторые приложения имеют встроенную поддержку proxy, а некоторые нет. Далее описано как пользоваться востребованными утилитами через proxy, даже теми, которые этой поддержки не имеют.

curl: передача данных через proxy

curl имеет полноценную поддержку как HTTP proxy так и SOCKS.

Для тестирования возможно использовать proxy сервера из бесплатных списков (socks — sockslist.net, и HTTP proxy — proxyhttp.net). Проверка IP адреса будет производиться с помощью ресурса check-host.net

Часть параметров curl можно записать в файл

С помощью time и curl также можно замерить время отклика сервера:

Результат будет выглядеть так:

wget: закачка файлов через proxy

wget имеет встроенную поддержку proxy. Недостаток лишь в том, что только поддержку HTTP proxy. Для использования совместно с SOCKS рекомендуется использовать соксификатор dante.

Чтобы все время не указывать —proxy-user и —proxy-password можно их прописать в файл

ssh: доступ к серверам

Для доступа к серверам через ssh и proxy также лучше использовать соксификатор dante.

Соксификатор dante

С помощью socksify можно направить через proxy почти любое приложение, не только консольное.

Чтобы все время не вводить данные о proxy можно создать файл /etc/socks.conf
Пример для SOCKS:

Пример для HTTP proxy с авторизацией:

А также экспортировать переменные SOCKS_USERNAME и SOCKS_PASSWORD, если для SOCKS или HTTP proxy требуется авторизация:

DNS запросы через proxy

Часто требуется чтобы и преобразование имен происходило через proxy. Если использовать dante, то запрос на преобразование имен идет и через proxy, и через именной сервер указанный в /etc/resolv.conf . Понять почему же идет два одинаковых запроса вместо одного не удалось. Поэтому можно предложить два варианта:
1) Закомментировать именные сервера в файле /etc/resolv.conf, чтобы преобразование имен шло только через proxy. Это отразится на всей системе.
2) Изменить /etc/resolv.conf и выставить именные сервера необходимой страны, или просто отличные от серверов провайдера. Например установить сервера Google:

Чтобы данные не были перезаписаны именными серверами провайдера (при переподключении), можно запретить обновление списка именных серверов сетевому менеджеру (NetworkManager/wicd) или DHCP-клиенту (спасибо ergil за корректировку).

Или воспользоваться «грубым» методом — запрещением изменения файла /etc/resolv.conf:

Если есть какие-то дополнения, пожалуйста, напишите, это будет полезно узнать и применить.

Источник

curl through proxy syntax

I don’t understand how to read this general syntax. I want to request a URL through our proxy and it requires a specific host, port, username, and password. I don’t know how to figure out what the protocol is. I want to do curl through a proxy. Based on the helpfile below, I would guess the line should be:

curl -x [whatever-my-protocol-is://]my-host-which-i-know[:my-port-which-i-know] -U my-username-which-i-know[:my-pass-which-i-know] http://www.google.com

Is this right? How do I figure out the protocol?

Relevant info from man:

2 Answers 2

Ignore «[» when reading general syntax. Also, windows likes double vs single quotes.

So the command is:

If you use domain

Not the answer you’re looking for? Browse other questions tagged curl proxy or ask your own question.

Linked

Related

Hot Network Questions

Subscribe to RSS

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

site design / logo © 2021 Stack Exchange Inc; user contributions licensed under cc by-sa. rev 2021.10.8.40416

By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

Источник

How to Use cURL With Proxy?

This step-by-step guide will explain how to use cURL or simply, curl, with proxy servers. It covers all the aspects, beginning from installation to explaining various options to set the proxy.

We did not target any specific proxy service. Therefore this tutorial should work with all proxy servers. All you need to know is the server details and credentials.

This is a fairly technical tutorial and expects readers to have a basic understanding of what a proxy is. It would be especially interesting and useful for those starting with web scraping.

Navigation:

What is cURL?

cURL is a command line tool for sending and receiving data using the url. Let’s look at the simplest example of using curl. Open your terminal or command prompt and type in this command and press Enter:

This will get the HTML of the page and print it on the console.

This will print the document information.

The question “what is cURL?” is also answered in one of our previous articles. We recommend reading it if you want to learn how it became such a universal asset.

Installation

cURL is provided with many Linux distributions and with MacOS. Now it is provided with Windows 10 as well.

If your Linux distribution is not provided with it, you can install it by running the install command. For example, on Ubuntu, open Terminal and run this command:

If you are running an older version of Windows, or if you want to install an alternate version, you can download curl from the official download page.

What you need to connect to a proxy

Irrespective of which proxy service you use, you will need the following information to use a:

• proxy server address
• port
• protocol
• username (if authentication is required)
• password (if authentication is required)

In this tutorial, we are going to assume that the proxy server is 127.0.0.1, the port is 1234, the user name is user, and the password is pwd. We will look into multiple examples covering various protocols.

NOTE. If you are on a network that uses NTLM authentication, you can use the switch –proxy-ntlm while running curl. Similarly, –proxy-digest can be used for digest authentication. You can look at all the available options by running curl –help. This tutorial will have examples for the scenario when a username and password has to be specified.

The next section will cover the first curl proxy scenario, which happens to be the most common one – HTTP and HTTPS proxy with curl.

Using cURL with HTTP/HTTPS proxy

If you recall, we looked at using curl without proxy like this:

This particular website is especially useful for testing out proxies as the output of this page is the origin IP address. If you are using a proxy correctly, the page will return an IP address that is different from your machine’s, that is, the proxy’s IP address.

There are multiple ways to run curl with proxy command. The next section will cover sending proxy details as a command line argument.

NOTE. All the command line options, or switches, are case sensitive. For example, -f instructs curl to fail silently, while -F denotes a form to be submitted.

Command line argument to set proxy in cURL

Open terminal and type the following command, and press Enter:

The output is going to be a huge list of options. One of them is going to look like this:

Note that x is small, and it is case-sensitive. The proxy details can be supplied using -x or –proxy switch. Both mean the same thing. Bot of the curl with proxy commands are same:

NOTE. If there are SSL certificate errors, add -k (note the small k) to the curl command. This will allow insecure server connections when using SSL.

You may have noticed that both the proxy url and target url are surrounded in double quotes. This is a recommended practice to handle special characters in the url.

Another interesting thing to note here is that the default proxy protocol is http. Thus, following two commands will do exactly the same:

Using environment variables

Another way to use proxy with curl is to set the environment variables http_proxy and https_proxy.

Note that setting proxy using environment variables works only with MacOS and Linux. For Windows, see the next section which explains how to use _curlrc file.

If you look at the first part of these variable names, it clearly shows the protocol for which these proxies will be used. It has nothing to do with the protocol used for the proxy server itself.

• http_proxy – the proxy will be used to access addresses that use http protocol

• https_proxy – the proxy will be used to access addresses that use https protocol

Simply set the variables http_proxy to http proxy address and https_proxy to set https proxy address. Open terminal and run these two commands.

After running these two commands, run curl normally.

If you see SSL Certificate errors, add -k to ignore these errors.

Another thing to note here is that these variables apply system wide. If this behavior is not desired, turn off the global proxy by unsetting these two variables:

See the next section to set default proxy only for curl and not system wide.

Configure cURL to always use proxy

If you want a proxy for curl but not for other programs, this can be achieved by creating a curl config file.

For Linux and MacOS, open terminal and navigate to your home directory. If there is already a .curlrc file, open it. If there is none, create a new file. Here are the set of commands that can be run:

In this file, add this line:

Save the file. Now curl with proxy is ready to be used. Simply run curl normally and it will read the proxy from .curlrc file.

On Windows, the file is named _curlrc. This file can be placed in the %APPDATA% directory.

To find the exact path of %APPDATA%, open command prompt and run the following command:

This directory will be something like C:\Users\ \AppData\Roaming. Now go to this directory, and create a new file _curlrc, and set the proxy by adding this line:

This works exactly the same way in Linux, MacOS, and Windows.

Ignore or override proxy for one request

If the proxy is set globally, or by modifying the .curlrc file, this can still be overridden to set another proxy or even bypass it.

To override proxy for one request, set the new proxy using -x or –proxy switch as usual:

If you want to bypass proxy altogether for a request, you can pass –noproxy followed by “*”. This instructs curl to not use proxy for all URLs.

If you have many curl requests to execute without a proxy, but not change system wide proxy settings, the following section will show you exactly how to do that.

Bonus tip – turning proxies off and on quickly

This tip is dedicated only for advanced users. If you do not know what a .bashrc file is, you may skip this section.

You can create an alias in your .bashrc file to set proxies and unset proxies. For example, open .bashrc file using any editor and add these lines:

After adding these lines, save the .bashrc and update the shell to read this .bashrc. To do this, run this this command in the terminal:

Now, whenever you need to turn on the proxy, you can quickly turn on the proxy, run one or more curl commands and then turn off the proxies like this:

cURL socks proxy

If the proxy server is using socks protocol, the syntax remains the same:

Similarly, socks4://, socks4a://, socks5:// or socks5h:// can be used depending on the socks version.

Alternatively, curl socks proxy can also be set using the switch –socks5 instead of -x. You can follow the same command, but use the different switch: username and password can be sent using the –proxy-user switch.

Again, –socks4, –socks4a or –socks5 can be used, depending on the version.

Summary

cURL is a very powerful tool for automation and is arguably the best command line interface in terms of proxy support. Lastly, as libcurl works very well with php, many web applications use it for web scraping projects, making it a must-have for any web scraper.

You can learn more on web scraping using Selenium and some other useful libraries like Beautiful Soup or lxml tutorial in our blog.

About Iveta Vistorskyte

Iveta Vistorskyte is a Content Manager at Oxylabs. Growing up as a writer and a challenge seeker, she decided to welcome herself to the tech-side, and instantly became interested in this field. When she is not at work, you’ll probably find her just chillin’ while listening to her favorite music or playing board games with friends.

All information on Oxylabs Blog is provided on an «as is» basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.

Источник