In Python, what are the differences between the urllib, urllib2, urllib3 and requests modules?

In Python, the modules urllib, urllib2, urllib3, and requests are used for handling HTTP requests, but they differ in functionality, ease of use, and scope. Below is a detailed breakdown of their differences, along with examples:

1. urllib (Standard Library)

• Scope: A collection of modules for URL handling in Python’s standard library.
• Components:
• urllib.request: Open and read URLs.
• urllib.parse: Parse URLs.
• urllib.error: Handle exceptions.
• urllib.robotparser: Parse robots.txt files.
• Features:
• Low-level HTTP client.
• Requires manual handling of headers, encoding, and cookies.
• No connection pooling or thread safety.
• Python 2 vs. 3:
• In Python 2, urllib and urllib2 were separate.
• In Python 3, urllib2 was merged into urllib.

Example (Python 3):

from urllib.request import urlopen, Request
from urllib.parse import urlencode

# GET request
response = urlopen("https://api.example.com/data")
print(response.read())

# POST request with data
data = urlencode({"key": "value"}).encode("utf-8")
req = Request("https://api.example.com/post", data=data, method="POST")
response = urlopen(req)
print(response.read())

2. urllib2 (Python 2 Only)

• Scope: Extended urllib in Python 2 with features like authentication and cookies.
• Features:
• Added support for HTTP authentication, cookies, and headers.
• Deprecated in Python 3 (merged into urllib).

Example (Python 2):

import urllib2

# GET request with headers
req = urllib2.Request("https://api.example.com")
req.add_header("User-Agent", "MyApp")
response = urllib2.urlopen(req)
print(response.read())

3. urllib3 (Third-Party Library)

• Scope: A high-level, third-party library for advanced HTTP features.
• Features:
• Connection pooling, retries, and thread safety.
• Supports file uploads, TLS/SSL, and proxies.
• Used as a dependency by requests.
• Requires installation: pip install urllib3.

Example:

import urllib3

# Create a PoolManager for connection reuse
http = urllib3.PoolManager()

# GET request
response = http.request("GET", "https://api.example.com/data")
print(response.data)

# POST request with JSON
response = http.request(
    "POST",
    "https://api.example.com/post",
    body='{"key": "value"}',
    headers={"Content-Type": "application/json"}
)
print(response.data)

4. requests (Third-Party Library)

• Scope: A user-friendly, third-party library built on urllib3.
• Features:
• Simplified API for GET/POST requests, sessions, and cookies.
• Automatic JSON parsing, timeout handling, and redirects.
• Requires installation: pip install requests.

Example:

import requests

# GET request with parameters
response = requests.get(
    "https://api.example.com/data",
    params={"key": "value"},
    headers={"User-Agent": "MyApp"}
)
print(response.json())  # Auto-parse JSON

# POST request with JSON data
response = requests.post(
    "https://api.example.com/post",
    json={"key": "value"}
)
print(response.status_code)

Key Differences

Feature	urllib (Python 3)	urllib2 (Python 2)	urllib3	requests
Standard Library	✅	✅ (Python 2)	❌ (Third-party)	❌ (Third-party)
Ease of Use	Low (manual handling)	Low	Medium	High (simplified API)
Connection Pooling	❌	❌	✅	✅ (via urllib3)
JSON Handling	Manual parsing	Manual parsing	Manual parsing	✅ (auto-parsing)
Cookies	Manual handling	Manual handling	Manual handling	✅ (session objects)
Error Handling	Basic exceptions	Basic exceptions	Retries/timeouts	Detailed exceptions

When to Use Which

1. urllib:

• For simple tasks within the standard library (no external dependencies).
• Parsing URLs or handling basic requests in Python 3.

1. urllib3:

• Advanced use cases requiring connection pooling or retries.
• Building higher-level libraries (e.g., requests).

1. requests:

• Most common use cases (APIs, web scraping).
• Prioritizing readability and simplicity.

Summary

• urllib: Low-level, standard library tools for basic HTTP operations.
• urllib3: Robust third-party library for performance and reliability.
• requests: The go-to library for user-friendly HTTP interactions.

Use requests for simplicity and urllib3/urllib for specialized needs or minimal dependencies.