Following the previous article is to create a port scanner (port scanner) in python. In this article, I will guide you to familiarize yourself with the requests library in python, and how to apply that library in network security.
Requests . Library
Dealing with HTTP requests is not an easy task in any programming language. If when talking about Python, it comes with two built-in libraries urllib and urllib2, to handle HTTP related operations. Both of these included libraries have a different set of functions (functions) and often need to be used together. Main limitations of use urllib is that it is confusing (some methods are available in both urllib, urllib2), the documentation is not clear and we need to write a lot of code to make a simple HTTP request.
To make the work simpler, an easy-to-use third-party library was born, called requests, is available and most developers prefer to use it to replace urllib/urllib2. It is an Apache2 HTTP library and is provided by urllib3 and httplib.
Setting
Since in this article using python3, I will use pip3 to install third-party libraries.
Enter the following command into Terminal:
sudo pip3 install requests
Comeinand pip that will install the library request on your system.
Basics of requests
To start using the library request of python, you need to import it first. The following are the commands that will be used to create a program that collects data from a website or API. But first, you need to read this article to understand how HTTP works.
request.get(url): This command will make a GET request to the website.request.post(url, data): This command will make a POST request to the URL and the data of the article content can be converted to a dictionary.
Note: Other request methods will be implemented in the same way as request.head, request.put, etc.
To make it easy to understand, let’s take an example of the request object:
>>>import requests
>>>r = requests.get("https://google.com")
r.status_code: This command will return the response code received from the request like 200, 404, 301, etc.r.text: This command will return the data you received from a web page.r.json: This command will get the response data from the web in the form of a dictionary.
Arguments for request methods:
timeout: This command is used to set the timeout for a request.allow_redirects: This command is used to specify whether or not a redirect may or may not be allowedallow_redirects = Truewill allow redirect requests.r.encoding: This command will display the encrypted form of the received data.cookie: This command will pass the cookie to the session request.headers: This command will be used to provide the header for the session request.
Exercise 1: Lookup IP
Applying the above knowledge, we will make a simple request to the IP lookup API to collect information for our target.
Note: This will be a very basic request, but it should help you understand how to crawl in Python anyway. I am using this API to get general information about IP address.
import requests
import json # Thư viện dùng để định dạng dữ liệu nhận được
def iplookup(public_ip):
r = requests.get("http://ip-api.com/json/"+public_ip)
if r.status_code == 200: # Nếu thành công
data = json.loads(r.text) # Chuyển dữ liệu nhận được vào json
for key, value in (data).items(): # Định dạng dữ liệu json sang dictionary
print("{}:{}".format(key, value))
else: # If error occurs
print("Error Occured while making request")
if __name__ == "__main__":
try:
ip = input("Enter IP: ")
iplookup(ip)
except:
print("Error Occured!")
How it works
On the first and second lines, we import two libraries requests and json. Then we create a function named iplookup, contains the parameter public_ip. Next, we send the request to the API and see if it succeeds through r.status_code. Convert the received data to json and dictionary format and then print it to Terminal. I added try...except to handle any possible errors.
Exercise 2: Block folder
Now, let’s use a more complicated, but very useful tool, a directory blocker. But before I continue, I will explain to you how to read and write files in Python.
File handling: Read and write
We need to read files to do directory blocking using dictionary attack. In python we use the function open is a built-in function that returns a file object and can be used to open a file in various ways like reading, writing, and appending.
Eg:
#!/usr/bin/python3
f = open("new.txt", "r") # Mở file ở chế độ đọc
print(f.read()) # Đọc nội dung file
new = open("new1.txt","w") # Mở file ở chế độ đọc ghi
data = f.read()
new.write(data) # Ghi dữ liệu vào file new1.txt
new.close() # Đóng file
We use the function open There are 2 parameters: the path of the file and the file opening mode. In the example above, file.txt is the file path, and r is read mode and then open file new1.txt to write data.
Different file modes:
r: Reading mode.w: Record mode.a: Concatenation mode (When opening the file, the cursor position will always be at the end of the file).r+: Read and write mode.
Note: Adding
bentering a mode will open the file in binary operation mode i.e all the contents of the file will be treated as byte objects likef = open ("new.txt", "rb")will read the file in binary.
import requests
def dirb(url, dict):
try:
wordlist = open(dict,"rb")
for path in wordlist.readlines():
path = path.strip().decode("utf-8")
urlpath = url+"/"+path
r = requests.get(urlpath)
if r.status_code != 404:
print("{} -> {}".format(r.status_code, urlpath))
except: # Catching exceptions
print("Error Occured!")
if __name__ == "__main__":
dirb("http://10.0.0.210", "all.txt")
How it works
We have created a function named dirb with parameters url and dict, this will be the file containing the directory listing to brute force on the site. Next, I also don’t forget to use try...except. Then I open the file dict using readlines and read the list of words contained in the file, then split the string and decode, append the data to your specified urls, then add the urls and paths along with “/”. Finally, do a GET request to the generated url and print the output of the request as long as the response code is not 404 (meaning “Not Found”).
I will use this script to hack a server on TryHackMe MrRobotCTF and this is the result:
[email protected]:~$ python3 temp.py 200 -> http://10.0.0.210/admin 403 -> http://10.0.0.210/.htaccess 200 -> http://10.0.0.210/readme.html 200 -> http://10.0.0.210/image --snip--
You can also deploy MrRobot server and use this script to get the directories of the website.
summary
This article will give you enough ideas and knowledge to create your own tools using Python3. But in the second project, you can see that the program runs a bit long. To solve that problem, we can use multithreading. I haven’t added that feature yet as it will require some additional libraries and different knowledge. In addition, you can also see more python articles here.