In the previous tutorial, we learned about async/await and asyncio. Now let’s learn about HTTP requests and APIs — how to call REST APIs, send data, handle responses, and build a reusable API client.

Almost every Python application talks to an API at some point. Weather data, payment processing, social media, databases — they all use HTTP. By the end of this tutorial, you will know how to make HTTP requests, handle errors, and build a clean API client.

What is a REST API?

A REST API is a way for programs to talk to each other over HTTP. When you open a website, your browser sends HTTP requests. An API works the same way, but instead of HTML, it sends and receives JSON data.

HTTP Methods:

MethodPurposeExample
GETRead dataGet a list of users
POSTCreate dataCreate a new user
PUTUpdate data (full replace)Update a user’s profile
PATCHUpdate data (partial)Change just the email
DELETERemove dataDelete a user

Status Codes:

CodeMeaning
200OK — request succeeded
201Created — new resource created
204No Content — success, nothing to return
400Bad Request — invalid data
401Unauthorized — missing or bad auth
403Forbidden — no permission
404Not Found — resource does not exist
500Internal Server Error — server broke

httpx — The Modern Python HTTP Library

We use httpx throughout this tutorial. It supports both sync and async requests, has a clean API, and is the recommended HTTP library for modern Python.

Install it:

pip install httpx

Your First GET Request

import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/todos/1")

print(response.status_code)  # 200
print(response.json())
# {'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

response.json() parses the JSON body into a Python dictionary. response.status_code tells you if it worked.

Checking for Errors

Always check the status code:

import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/todos/9999")

if response.status_code == 200:
    data = response.json()
    print(data)
elif response.status_code == 404:
    print("Not found!")
else:
    print(f"Error: {response.status_code}")

Or use raise_for_status() to throw an exception on error:

import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/todos/1")
response.raise_for_status()  # Raises httpx.HTTPStatusError on 4xx/5xx
data = response.json()

GET Requests — Reading Data

Query Parameters

Use the params argument to add query parameters:

import httpx

# This calls: https://jsonplaceholder.typicode.com/posts?userId=1
response = httpx.get(
    "https://jsonplaceholder.typicode.com/posts",
    params={"userId": 1},
)
posts = response.json()
print(f"User 1 has {len(posts)} posts")

Custom Headers

Add headers for authentication, content type, or custom values:

import httpx

response = httpx.get(
    "https://api.example.com/data",
    headers={
        "Authorization": "Bearer your-api-key",
        "Accept": "application/json",
    },
)

Timeouts

Always set a timeout. Without one, a stuck server will hang your program forever:

import httpx

# Timeout after 10 seconds
response = httpx.get(
    "https://jsonplaceholder.typicode.com/todos/1",
    timeout=10.0,
)

The default timeout in httpx is 5 seconds. In production, always set an explicit timeout.

POST Requests — Creating Data

Send JSON data with a POST request:

import httpx

new_post = {
    "title": "My New Post",
    "body": "This is the content.",
    "userId": 1,
}

response = httpx.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=new_post,  # Automatically sets Content-Type: application/json
)

print(response.status_code)  # 201 Created
print(response.json())       # Returns the created post with an id

The json= parameter automatically converts your dictionary to JSON and sets the correct Content-Type header.

PUT and DELETE — Updating and Removing Data

PUT (Full Update)

import httpx

updated_post = {
    "id": 1,
    "title": "Updated Title",
    "body": "Updated content.",
    "userId": 1,
}

response = httpx.put(
    "https://jsonplaceholder.typicode.com/posts/1",
    json=updated_post,
)
print(response.status_code)  # 200

DELETE

import httpx

response = httpx.delete("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code)  # 200

Using httpx.Client — Connection Pooling

For multiple requests, use httpx.Client(). It reuses connections (faster) and lets you set shared headers and base URL:

import httpx

with httpx.Client(
    base_url="https://jsonplaceholder.typicode.com",
    headers={"Accept": "application/json"},
    timeout=10.0,
) as client:
    # All requests share the base URL, headers, and timeout
    todos = client.get("/todos", params={"userId": 1}).json()
    posts = client.get("/posts", params={"userId": 1}).json()

    print(f"User 1: {len(todos)} todos, {len(posts)} posts")

Always use a Client when making multiple requests. It is faster and cleaner.

Async HTTP with httpx

In Tutorial #18, we learned async/await. Here is how to use it with httpx:

import asyncio
import httpx

async def fetch_todos_and_posts():
    """Fetch todos and posts concurrently."""
    async with httpx.AsyncClient(
        base_url="https://jsonplaceholder.typicode.com",
        timeout=10.0,
    ) as client:
        # Fetch both at the same time
        todos_response, posts_response = await asyncio.gather(
            client.get("/todos", params={"userId": 1}),
            client.get("/posts", params={"userId": 1}),
        )

        todos = todos_response.json()
        posts = posts_response.json()
        print(f"Todos: {len(todos)}, Posts: {len(posts)}")

asyncio.run(fetch_todos_and_posts())

The async version runs both requests concurrently. If each takes 200ms, the total time is ~200ms instead of ~400ms.

JSON Handling

Parsing JSON Safely

Always handle parse errors:

import json

def parse_json_safe(text: str) -> dict | None:
    """Safely parse JSON. Returns None on error."""
    try:
        return json.loads(text)
    except (json.JSONDecodeError, TypeError):
        return None

print(parse_json_safe('{"name": "Alex"}'))  # {'name': 'Alex'}
print(parse_json_safe("not json"))           # None

Pretty-Printing JSON

import json

data = {"name": "Alex", "scores": [95, 87, 92]}
print(json.dumps(data, indent=2, sort_keys=True))

Output:

{
  "name": "Alex",
  "scores": [95, 87, 92]
}

Authentication

API Key in Headers

import httpx

response = httpx.get(
    "https://api.example.com/data",
    headers={"X-API-Key": "your-key-here"},
)

Bearer Token (JWT)

import httpx

token = "eyJhbGciOiJIUzI1NiIs..."

response = httpx.get(
    "https://api.example.com/profile",
    headers={"Authorization": f"Bearer {token}"},
)

Basic Authentication

import httpx

response = httpx.get(
    "https://api.example.com/data",
    auth=("username", "password"),
)

Never hardcode API keys in your code. Use environment variables:

import os
import httpx

api_key = os.environ["API_KEY"]  # Set in .env or system environment
response = httpx.get(
    "https://api.example.com/data",
    headers={"Authorization": f"Bearer {api_key}"},
)

Error Handling for HTTP Requests

Here is a robust pattern for handling HTTP errors:

import httpx

def fetch_user(user_id: int) -> dict | None:
    """Fetch a user from the API with proper error handling."""
    try:
        response = httpx.get(
            f"https://jsonplaceholder.typicode.com/users/{user_id}",
            timeout=10.0,
        )
        response.raise_for_status()
        return response.json()

    except httpx.TimeoutException:
        print(f"Request timed out for user {user_id}")
        return None

    except httpx.HTTPStatusError as e:
        if e.response.status_code == 404:
            print(f"User {user_id} not found")
        else:
            print(f"HTTP error: {e.response.status_code}")
        return None

    except httpx.RequestError as e:
        print(f"Network error: {e}")
        return None

Error hierarchy in httpx:

  • httpx.RequestError — any network/connection error
  • httpx.TimeoutException — request took too long
  • httpx.HTTPStatusError — raised by raise_for_status() for 4xx/5xx codes

Rate Limiting

When calling APIs, respect their rate limits. Here is a simple rate limiter:

import time

class RateLimiter:
    """Simple rate limiter that tracks request timestamps."""

    def __init__(self, max_requests: int, window_seconds: float) -> None:
        self.max_requests = max_requests
        self.window_seconds = window_seconds
        self._timestamps: list[float] = []

    def try_request(self) -> bool:
        """Try to make a request. Returns True if allowed."""
        now = time.monotonic()
        # Remove old timestamps
        self._timestamps = [
            t for t in self._timestamps if now - t < self.window_seconds
        ]
        if len(self._timestamps) < self.max_requests:
            self._timestamps.append(now)
            return True
        return False

# Allow 5 requests per second
limiter = RateLimiter(max_requests=5, window_seconds=1.0)

for i in range(10):
    if limiter.try_request():
        print(f"Request {i + 1}: allowed")
    else:
        print(f"Request {i + 1}: blocked — rate limit reached")

Retry Logic

Network requests can fail. Add retry logic with exponential backoff:

import time
import httpx

def fetch_with_retry(
    url: str,
    max_retries: int = 3,
    retry_delay: float = 1.0,
) -> httpx.Response:
    """Fetch a URL with retry on failure."""
    for attempt in range(max_retries):
        try:
            response = httpx.get(url, timeout=10.0)
            if response.status_code < 500:
                return response  # Success or client error — don't retry
            # Server error — retry
            print(f"Server error {response.status_code}, retrying...")
        except httpx.RequestError as e:
            print(f"Network error: {e}, retrying...")

        # Exponential backoff: 1s, 2s, 4s
        time.sleep(retry_delay * (2 ** attempt))

    raise RuntimeError(f"Failed after {max_retries} retries")

Key idea: Retry on server errors (5xx) and network errors. Do not retry on client errors (4xx) — those mean you sent bad data.

Building an API Client

For production code, wrap API calls in a class:

import httpx

class UserClient:
    """Client for the Users API."""

    def __init__(self, base_url: str, token: str) -> None:
        self.client = httpx.Client(
            base_url=base_url,
            headers={"Authorization": f"Bearer {token}"},
            timeout=10.0,
        )

    def list_users(self) -> list[dict]:
        """Get all users."""
        response = self.client.get("/users")
        response.raise_for_status()
        return response.json()

    def get_user(self, user_id: int) -> dict:
        """Get a single user."""
        response = self.client.get(f"/users/{user_id}")
        response.raise_for_status()
        return response.json()

    def create_user(self, name: str, email: str) -> dict:
        """Create a new user."""
        response = self.client.post(
            "/users",
            json={"name": name, "email": email},
        )
        response.raise_for_status()
        return response.json()

    def delete_user(self, user_id: int) -> None:
        """Delete a user."""
        response = self.client.delete(f"/users/{user_id}")
        response.raise_for_status()

    def close(self) -> None:
        """Close the HTTP client."""
        self.client.close()

Usage:

client = UserClient("https://api.example.com", token="your-token")
try:
    users = client.list_users()
    new_user = client.create_user("Alex", "alex@example.com")
    print(new_user)
finally:
    client.close()

This pattern gives you:

  • Shared configuration (base URL, auth, timeout)
  • Connection reuse (faster)
  • Typed methods with clear names
  • One place to add error handling, retry, logging

Pagination

Most APIs return data in pages. Here is how to fetch all pages:

import httpx

def fetch_all_pages(base_url: str, endpoint: str) -> list[dict]:
    """Fetch all pages from a paginated API."""
    all_items = []
    page = 1

    with httpx.Client(base_url=base_url, timeout=10.0) as client:
        while True:
            response = client.get(endpoint, params={"page": page, "per_page": 20})
            response.raise_for_status()
            items = response.json()

            if not items:  # Empty page = no more data
                break

            all_items.extend(items)
            page += 1

    return all_items

Some APIs use a next URL instead of page numbers. Check the API documentation for the pagination format.

Common Mistakes

1. No Timeout

# BAD — hangs forever if server is down
response = httpx.get("https://api.example.com/data")

# GOOD — fails after 10 seconds
response = httpx.get("https://api.example.com/data", timeout=10.0)

An infinite timeout is a production bug. Always set a timeout.

2. Not Checking Status Codes

# BAD — assumes success
response = httpx.get("https://api.example.com/users/999")
data = response.json()  # Might be an error response!

# GOOD — check before using
response = httpx.get("https://api.example.com/users/999")
response.raise_for_status()
data = response.json()

3. Hardcoding API Keys

# BAD — secret is in your code (and git history)
headers = {"Authorization": "Bearer sk-abc123secret"}

# GOOD — use environment variables
import os
headers = {"Authorization": f"Bearer {os.environ['API_KEY']}"}

Source Code

You can find all the code from this tutorial on GitHub:

GitHub: python-tutorial/tutorial-19-http

What’s Next?

In the next tutorial, we will learn about databases — SQLite with Python’s built-in sqlite3 module and SQLAlchemy ORM for building data-driven applications.