Python has topped the TIOBE index since 2021, and it is the default language of the AI boom — PyTorch, LangChain, and every major LLM framework are Python-first. It is also genuinely easy to start: no build step, no compiler, a text editor and a terminal are enough.

This guide takes you from print("Hello, World!") to a working project: BookTrail, a reading-list tracker with a shared data model, a colored CLI, a web scraper that imports real book data, and a tested REST API. Every section adds a concept. The final part connects them into one project — nothing here is a disconnected snippet.

What you will have by the end: a solid grip on Python’s type system, error handling, and OOP, comfort with the tools that separate scripts from real software (generators, decorators, context managers, async), and a real multi-part Python project — a CLI, a scraper, and a REST API sharing one model, tested with pytest.

Two things before you start:

  • No prior Python experience needed. Some programming background helps. Every example targets Python 3.12+ — install it first if you have not (see Setup below).
  • Full source code for the capstone project is on GitHub: github.com/kemalcodes/python-tutorial.

Need a quick syntax lookup instead of a full tutorial? See the Python Cheat Sheet.

Part 1: Foundations

Why Python

Python was created by Guido van Rossum in 1991. Its defining trait is readability — code reads close to English, with no semicolons and no class wrapper required for a script:

print("Hello, World!")

That one line is a complete program. Compare it to Java or C++, and the difference in ceremony is the whole pitch. Python is not just easy to start with, though — it runs some of the largest systems in the world. Instagram (Django) serves over two billion users. Google, Netflix, Spotify, and NASA all run Python in production.

PythonJavaScriptJava/KotlinRust
TypingDynamicDynamicStaticStatic
SpeedSlowerFaster (V8)FasterVery fast
Main useAI, data, scriptingWeb frontendsAndroid, enterpriseSystems, performance
Package managerpip, uvnpmGradle/MavenCargo

“Python is slow” is true for CPU-heavy work, but mostly irrelevant: most Python programs spend their time waiting on the network, a database, or disk — not computing. For the rare case where raw speed matters, NumPy and pandas call C under the hood, and pyo3 lets you drop into Rust. Python 2 reached end of life in January 2020 — this guide, like everything you should read in 2026, means Python 3.

The ecosystem is what actually makes “Python is the default choice” true in practice — PyPI hosts hundreds of thousands of packages, and there’s a well-established default for nearly every common task: FastAPI or Django for web backends, pandas and Polars for data manipulation, PyTorch for machine learning, requests/httpx for HTTP, pytest for testing, ruff for linting. Knowing these names matters as much as knowing the syntax — picking the standard tool for a job is usually the right call over building it yourself.

Setup

Check what you already have:

python3 --version

If it prints 3.12 or newer, skip ahead. Otherwise:

# macOS
brew install python

# Ubuntu/Debian
sudo apt install python3 python3-pip python3-venv

# Windows: download from python.org — CHECK "Add Python to PATH" during install

The REPL (Read-Eval-Print Loop) is Python’s interactive mode — type python3 in a terminal and try expressions one at a time (2 + 3, "Hello".upper()). Exit with exit() or Ctrl+D. For real files, save a .py file and run python3 filename.py.

Get user input with input() — it always returns a string, so convert it yourself: age = int(input("Age: ")). Comments start with #. Python uses indentation (4 spaces, never tabs) instead of curly braces to mark code blocks — this is not a style preference, it is syntax.

Variables and Types

Assign a variable with = — no type declaration needed. Python infers the type from the value (dynamic typing), and a variable can be reassigned to a different type entirely:

x = 42          # int
x = "hello"     # now a string — no error
x = [1, 2, 3]   # now a list

The five basic types: int (whole numbers, no size limit), float (decimals), str (text, single/double/triple quotes), bool (True/False, capitalized), and None (absence of a value — Python’s null, always compared with is, never ==).

Type hints document intent without changing runtime behavior — Python ignores them at execution, but your IDE and tools like mypy use them:

username: str = "Sam"
nickname: str | None = None   # can be str or None

f-strings are the standard way to format text — put a variable or expression directly in {}:

name, age = "Alex", 25
print(f"{name} is {age} years old")     # Alex is 25 years old
print(f"{3.14159:.2f}")                  # 3.14 — 2 decimal places
print(f"{1_000_000:,}")                  # 1,000,000 — comma separator
print(f"{x=}")                            # x=42 — prints name AND value, great for debugging

Arithmetic has one gotcha worth memorizing: / always returns a float, even for even division (10 / 2 is 5.0); use // for integer (floor) division. Converting float to int truncates, it does not round — int(3.9) is 3, use round() if you want 4. Python integers have no size limit — there’s no overflow to think about the way there is in a fixed-width language. Truthy/falsy values let you write if name: instead of if name != "": 0, 0.0, "", None, and empty collections ([], {}, set()) are all falsy; everything else is truthy.

Use type(x) to see a value’s exact type and isinstance(x, int) to check it — prefer isinstance in real code, because it also accounts for inheritance (isinstance(True, int) is True, since bool is a subclass of int in Python) where type(x) == int would miss subclasses entirely. isinstance also accepts a tuple of types for an either/or check: isinstance(value, (int, float)).

Control Flow

if/elif/else branches on conditions; combine with and, or, not. For simple cases, a ternary expression fits on one line: status = "adult" if age >= 18 else "minor".

for iterates over any sequence. range(5) gives 0..4 (the end is exclusive — the most common beginner off-by-one). enumerate() gives you index and value together; zip() walks two sequences in parallel and stops at the shorter one:

for i, fruit in enumerate(["apple", "banana"]):
    print(f"{i}: {fruit}")

for name, score in zip(["Alex", "Sam"], [95, 87]):
    print(f"{name}: {score}")

while loops until its condition is false — always make sure something inside the loop changes that condition, or you get an infinite loop. break exits a loop early; continue skips to the next iteration. The walrus operator := assigns and returns a value in the same expression, useful in a while condition or a comprehension:

while (line := input("> ")) != "quit":
    print(f"You said: {line}")

match/case (Python 3.10+) is a more powerful switch — it matches values, ranges via guards, and even destructures tuples and lists:

match point:
    case (0, 0): print("origin")
    case (x, 0): print(f"on x-axis at x={x}")
    case (x, y): print(f"({x}, {y})")

match also destructures lists, which makes it a clean fit for parsing simple command strings — split on whitespace and match the resulting list’s shape directly:

match command.split():
    case ["quit"]: ...
    case ["move", direction, distance]: ...
    case _: print("unknown command")

Add an if guard to a case for conditions the pattern alone can’t express (case _ if 400 <= code < 500: return "Client Error"); _ is the wildcard, matching anything, like default in other languages’ switch.

One rarely-used but interview-favorite feature: for and while support an else clause that runs only if the loop finished without hitting a break — useful for a “search and report not-found” pattern without a separate found-flag variable:

for i in range(2, n):
    if n % i == 0:
        break
else:
    print(f"{n} is prime")   # only runs if the loop never broke

Functions

def defines a function; type hints on parameters and the -> ReturnType are optional but standard practice:

def greet(name: str, greeting: str = "Hello") -> str:
    """Return a greeting message."""
    return f"{greeting}, {name}!"

Default parameters (greeting: str = "Hello") and named arguments (greet(name="Alex")) cover most of what other languages need method overloading for. *args collects extra positional arguments into a tuple; **kwargs collects extra keyword arguments into a dict; a * on its own forces everything after it to be passed by name:

def create_user(name: str, *, email: str, active: bool = True) -> dict:
    return {"name": name, "email": email, "active": active}

create_user("Alex", email="alex@example.com")   # OK
# create_user("Alex", "alex@example.com")       # TypeError — email is keyword-only

A lambda is a small anonymous function, limited to one expression — mostly useful as an argument to sorted(), map(), or filter(): sorted(users, key=lambda u: u["age"]). For anything longer, use a named def.

A closure is a function that remembers a variable from its enclosing scope, even after that scope has returned:

def make_multiplier(factor: int):
    def multiplier(n: int) -> int:
        return n * factor
    return multiplier

triple = make_multiplier(3)
print(triple(5))   # 15

Closures are the foundation decorators are built on (Part 2). The single most-repeated Python gotcha: never use a mutable object as a default argument.

# WRONG — the same list is shared across every call
def append_to(item, lst=[]):
    lst.append(item)
    return lst

# RIGHT
def append_to(item, lst=None):
    if lst is None:
        lst = []
    lst.append(item)
    return lst

The default is created once, when the function is defined — not on every call. This bites everyone exactly once.

Variable lookup follows the LEGB rule — Local, Enclosing, Global, Built-in, checked in that order. A name inside a function resolves to the innermost matching scope first; a function can read a global variable without any special syntax, but assigning to it requires an explicit global keyword (rarely a good idea — prefer passing values in and returning them out over mutating global state). A triple-quoted docstring as the first line of a function documents parameters, return value, and exceptions, and shows up when someone calls help(your_function):

def calculate_bmi(weight_kg: float, height_m: float) -> float:
    """Calculate Body Mass Index.

    Args:
        weight_kg: Weight in kilograms.
        height_m: Height in meters.

    Raises:
        ValueError: If height_m is zero or negative.
    """

Data Structures

List — ordered, mutable, allows duplicates. listOf-style creation is just [1, 2, 3]; .append(), .insert(i, x), .remove(x), .pop() mutate in place. Slicing (list[start:end:step]) is the tool you will use constantly — numbers[::-1] reverses, numbers[:3] takes the first three, the end index is always exclusive.

Tuple — ordered, immutable, allows duplicates. Use it for fixed data (coordinates), dict keys, and returning multiple values from a function. Tuple unpacking is one of Python’s nicest features — a, b = b, a swaps two variables with no temp variable, and first, *middle, last = [1,2,3,4,5] captures the rest into a list.

Dict — key-value pairs, insertion-ordered since 3.7. Use .get(key, default) instead of dict[key] when a key might be missing — [] raises KeyError. Iterate with .items() to get both key and value:

for name, score in scores.items():
    print(f"{name}: {score}")

Set — unordered, unique elements, supports math operations (| union, & intersection, - difference). Membership testing (x in collection) is O(1) on a set versus O(n) on a list — convert to a set first if you check membership repeatedly on a large collection.

Comprehensions build any of these in one line and are more idiomatic than a manual loop:

squares = [x**2 for x in range(6)]                    # list
lengths = {word: len(word) for word in words}          # dict
unique = {x for x in numbers}                            # set

collections.Counter counts occurrences (Counter(text.split()).most_common(3)); collections.defaultdict(list) removes the “check if key exists first” boilerplate when grouping data:

groups = defaultdict(list)
for word in words:
    groups[len(word)].append(word)   # no "if key not in groups" needed

sorted(items, key=...) returns a new list and works on anything iterable (tuples, dict items, custom objects); .sort() exists only on lists and sorts in place, returning None — a common bug is writing items = items.sort(), which throws away the list. Both take reverse=True and a key function for sorting by something other than natural order:

sorted(students, key=lambda s: s.score, reverse=True)   # highest score first

Comprehensions accept a trailing if to filter while building, and nest naturally for real-world, not-flat data — a common shape is a list of dicts where you need one field’s aggregate:

evens = [x for x in range(20) if x % 2 == 0]
avg_by_user = {u["name"]: sum(u["scores"]) / len(u["scores"]) for u in users}

For genuinely nested data (a list of dicts, each with its own list), a normal for loop is usually more readable than forcing everything into one comprehension — reach for the simpler tool once the comprehension needs more than one for or if.

Strings

Beyond the f-string basics from earlier: .strip()/.lower()/.upper() for cleaning, .split(sep)/sep.join(list) for the split-transform-rejoin pattern you will use constantly, .replace(old, new), and in for substring checks.

Raw strings (r"...") treat backslashes literally — essential for Windows paths and regex patterns, where you would otherwise double-escape everything:

path = r"C:\Users\Alex\data"
pattern = r"\d+\.\d+"

For pattern matching beyond simple substring checks, use re:

import re

match = re.search(r"[\w.+-]+@[\w-]+\.[\w.-]+", text)
if match:
    print(match.group())

numbers = [int(n) for n in re.findall(r"\d+", text)]
clean = re.sub(r"\s+", " ", messy_text)   # normalize whitespace

Always check re.search() and re.match() for None before calling .group() — an unmatched pattern returns None, not an exception, and calling .group() on it crashes. Reach for regex only when a plain in, .startswith(), or .split() will not do — a common (and true) programmer joke: “some people, faced with a problem, think ‘I’ll use regex.’ Now they have two problems.”

f-string format specs go well beyond decimal places: f"{255:b}" gives binary (11111111), f"{255:x}" gives hex (ff), f"{42:08d}" zero-pads to 8 digits, and f"{value:.0%}" formats a fraction as a percentage. Alignment specs build simple text tables without a library:

print(f"{'Name':<15}{'Score':>8}")   # < left-align, > right-align, ^ center, all take a width

A slug generator is a compact, realistic combination of the string tools above — lowercase, strip special characters, collapse whitespace into hyphens:

def slugify(text: str) -> str:
    text = text.lower().strip()
    text = re.sub(r"[^\w\s-]", "", text)   # drop punctuation
    text = re.sub(r"[\s_]+", "-", text)     # spaces/underscores -> hyphens
    return text.strip("-")

slugify("What's New in 2026?")   # "whats-new-in-2026"

Modules and Packages

Every .py file is a module; import it with import module_name and call module_name.function(). Prefer this over from module import *, which pollutes your namespace and hides where names came from.

The if __name__ == "__main__": guard is one of the most important Python idioms — code inside it only runs when the file is executed directly, not when it is imported:

def add(a: int, b: int) -> int:
    return a + b

if __name__ == "__main__":
    print(add(3, 4))   # only prints when you run this file directly

A package is a directory with an __init__.py file, letting you organize modules into a tree and control the public API via what __init__.py re-exports — from .math_utils import add inside __init__.py lets callers write from my_project import add instead of reaching into the submodule directly. Inside a package, from .sibling import helper (one dot, same package) and from ..other_package import thing (two dots, parent package) are relative imports — they only work inside a package, not in a standalone script.

Three import styles cover almost everything: import json then json.dumps(...) is the clearest about where a name came from; from datetime import datetime, timedelta is worth it for names you use constantly; from module import * should be avoided entirely — it pollutes your namespace and makes it impossible to tell where a given name was defined just by reading the file.

The standard library is large enough that “is there a module for this” is usually “yes”: pathlib.Path for filesystem paths (preferred over the older os.path), datetime for dates and times, json for encoding/decoding, and collections for Counter/defaultdict. A conventional src layout for anything beyond a single script keeps source, tests, and config cleanly separated:

my-project/
    src/my_project/__init__.py
    tests/test_main.py
    pyproject.toml

Virtual environments isolate one project’s dependencies from another’s — without one, two projects needing different versions of the same library will conflict:

python3 -m venv .venv
source .venv/bin/activate      # .venv\Scripts\activate on Windows
pip install requests pydantic

pyproject.toml is the modern standard for declaring dependencies (replacing requirements.txt, setup.py, and setup.cfg with one file). uv is a newer package manager, 10-100x faster than pip and largely a drop-in replacement (uv pip install instead of pip install) — it is becoming the default choice for new projects in 2026. Never install packages globally; always activate a virtual environment first.

Part 2: Writing Idiomatic Python

OOP: Classes, Inheritance, Magic Methods

A class bundles data and behavior. __init__ is the constructor; self refers to the current instance and is always the first parameter of every method:

class Dog:
    species = "Canis familiaris"          # class variable — shared by all instances

    def __init__(self, name: str, age: int) -> None:
        self.name = name                   # instance variable — unique per object
        self.age = age

    def bark(self) -> str:
        return f"{self.name} says: Woof!"

@property turns a method into an attribute-like accessor, letting you add validation without changing the calling code:

class Circle:
    def __init__(self, radius: float) -> None:
        self._radius = radius

    @property
    def radius(self) -> float:
        return self._radius

    @radius.setter
    def radius(self, value: float) -> None:
        if value <= 0:
            raise ValueError("Radius must be positive.")
        self._radius = value

Magic methods (double-underscore, “dunder”) wire your class into Python’s built-in operators: __repr__ (developer-facing string, shown in the REPL), __str__ (user-facing string, used by print()), __eq__ (==), __add__ (+), __len__ (len()). If you implement only one, implement __repr__ — Python falls back to it.

Inheritance shares behavior: class Cat(Animal) inherits everything, and super().__init__(...) calls the parent constructor. isinstance() checks membership including parent classes; issubclass() checks the class hierarchy itself. Inheritance chains as many levels as you like (class Kitten(Cat) inherits everything Cat has, which itself inherited from Animal) but keep it to two or three levels in practice — beyond that, favor composition: store the collaborator as an attribute (self.engine = Engine(...)) instead of extending a class you don’t fully control, which keeps each class’s behavior easy to reason about in isolation.

@classmethod (receives cls, not self) is the standard pattern for alternative constructors — Temperature.from_fahrenheit(212) and Temperature.from_kelvin(373.15) both build the same class from different starting units, which is cleaner than cramming unit-conversion logic into __init__ — and @staticmethod is a plain function that happens to live inside a class for organizational purposes, taking neither self nor cls.

Abstract base classes (from abc import ABC, abstractmethod) define a contract: a class inheriting from an ABC cannot be instantiated until it implements every @abstractmethod, and Python enforces this with a TypeError at instantiation time rather than a runtime AttributeError the first time the missing method is called — catching the mistake immediately instead of somewhere downstream:

class Shape(ABC):
    @abstractmethod
    def area(self) -> float: ...

    def describe(self) -> str:              # concrete — shared by every subclass
        return f"{type(self).__name__}: area={self.area():.2f}"

Beyond __init__ and __repr__/__str__, the magic methods worth knowing: __eq__ for ==, __lt__ for < (needed if you want a class to work with sorted()), __len__ for len(obj), __contains__ for in, and __getitem__ for obj[key] — implement whichever ones make your object behave like the built-in type it most resembles, and let Python’s own syntax operate on it directly instead of exposing .get_length()-style methods.

The rule of thumb for when to reach for a class at all: use one when data and behavior belong together and change over time; use a plain function when you are just transforming data; use a dataclass (next) when you mainly need to store data.

Class variables are shared across every instance — usually what you want for a constant like species above, but a genuine trap when the class variable is mutable. Declaring members = [] at class level gives every instance the same list:

# WRONG — one list shared by every Team instance
class Team:
    members = []

t1, t2 = Team(), Team()
t1.members.append("Alex")
print(t2.members)   # ["Alex"] — surprise, they share state

# RIGHT — each instance gets its own list, set in __init__
class Team:
    def __init__(self):
        self.members = []

This is the class-level twin of the mutable-default-argument bug from Part 1 — same root cause, a mutable object created once and shared, not per instance. A leading underscore (self._radius) is Python’s convention for “internal, don’t touch directly” — nothing enforces it at runtime, the @property pattern above is what actually provides the safe public interface.

Dataclasses and Pydantic

A regular class needs 10+ lines of boilerplate (__init__, __repr__, __eq__) just to hold three fields. @dataclass generates all of it from type-annotated fields:

from dataclasses import dataclass, field

@dataclass
class User:
    name: str
    email: str
    active: bool = True
    tags: list[str] = field(default_factory=list)   # never use tags: list = [] — same mutable-default bug as functions

field() gives you per-field control beyond a plain default: default_factory for mutable defaults (shown above), init=False to exclude a field from the generated __init__ (useful for a value computed in __post_init__), and repr=False to hide a field from the auto-generated __repr__. __post_init__ runs right after __init__, the natural place for derived fields or validation:

@dataclass
class Rectangle:
    width: float
    height: float
    area: float = field(init=False)

    def __post_init__(self) -> None:
        self.area = self.width * self.height   # computed, not passed in

@dataclass(frozen=True) makes instances immutable and, as a direct consequence, hashable — usable as dict keys or set members, something a normal mutable dataclass cannot do because Python needs a guarantee the object won’t change after being hashed. @dataclass(slots=True) trades the ability to add attributes later for lower memory use and faster attribute access, worth it when you create thousands of instances. @dataclass(order=True) auto-generates </>/<=/>= by comparing fields in declaration order — if you want to sort by a specific field rather than field order, add a leading sort_index field set in __post_init__ to control it.

Pydantic goes further: it validates data at creation time, not just stores it, and is the standard for anything crossing an API boundary (request bodies, config files):

from pydantic import BaseModel, Field, field_validator

class UserProfile(BaseModel):
    name: str
    email: str
    age: int = Field(ge=0, le=150)

    @field_validator("email")
    @classmethod
    def email_has_at(cls, v: str) -> str:
        if "@" not in v:
            raise ValueError("Email must contain @.")
        return v.lower()

UserProfile(name="Alex", email="ALEX@x.com", age=25)   # OK, email lowered
UserProfile(name="Alex", email="bad", age=200)          # raises ValidationError — two problems at once

Dataclasses inherit normally — a child dataclass gets every field from the parent, with its own fields appended after:

@dataclass
class Person:
    name: str
    age: int

@dataclass
class Employee(Person):
    company: str
    role: str = "Developer"   # ok — has a default

Employee("Alex", 30, "Acme")

The same rule that applies to function parameters applies here: once a field has a default, every field after it (including inherited ones checked in declaration order) must also have one, or the generated __init__ can’t be built consistently.

Pydantic models nest naturally and validate the nesting for free — passing a plain dict for a nested field is enough, Pydantic builds the inner model and validates it against the same rules as if you’d constructed it directly:

class Address(BaseModel):
    city: str
    country: str = "Germany"

class Employee(BaseModel):
    name: str
    address: Address

Employee.model_validate({"name": "Sam", "address": {"city": "Berlin"}})

model_dump() converts a model back to a plain dict (exclude={"internal_field"} or exclude_none=True to trim what comes out); model_validate() is the reverse, building a model from a dict — the pair you’ll use constantly at the boundary between your database layer and your API layer.

Rule of thumb: dataclass for internal data structures with no external validation need; Pydantic wherever data enters your program from outside (an HTTP request, a config file, user input).

Error Handling

try/except catches errors; always catch specific exception types, never a bare except: — that also swallows Ctrl+C and system exits.

def parse_int(value: str) -> int | None:
    try:
        return int(value)
    except ValueError:
        return None

The full form has four parts: try (code that might fail), except (handle it), else (runs only on success — keeps success logic out of the try block so its own errors are not accidentally caught), finally (always runs, for cleanup).

Build your own exception hierarchy by inheriting from Exception. A base AppError plus specific subclasses lets callers catch broadly or narrowly:

class AppError(Exception):
    pass

class NotFoundError(AppError):
    def __init__(self, resource: str, resource_id: str) -> None:
        super().__init__(f"{resource} '{resource_id}' not found.")

raise NewError(...) from original_error chains exceptions so the traceback shows the real root cause instead of a confusing “during handling of the above exception.” A base exception plus specific subclasses is the standard shape for a library or application’s error hierarchy — callers can catch broadly (except AppError) or narrowly (except NotFoundError) depending on how much detail they need:

try:
    user = get_user("99")
except NotFoundError as e:
    print(f"Not found: {e.resource} {e.resource_id}")   # specific — has extra fields
except AppError as e:
    log_error(e)   # broad — catches anything else from your app

Python 3.11 added ExceptionGroup for the case where you want to collect every problem instead of stopping at the first — form validation is the canonical example, since a user fixing one typo shouldn’t have to resubmit five times to discover the other four errors one at a time:

errors = []
if not name.strip(): errors.append(ValidationError("name", "required"))
if "@" not in email: errors.append(ValidationError("email", "invalid"))
if errors:
    raise ExceptionGroup("Form validation failed", errors)

# caught with:
except ExceptionGroup as eg:
    for error in eg.exceptions:
        print(f"  - {error.field}: {error.message}")

For the simple case of “ignore this one specific exception,” contextlib.suppress is cleaner than an empty except block:

with contextlib.suppress(FileNotFoundError):
    Path("maybe-missing.txt").unlink()

Python’s preferred style is EAFP — Easier to Ask Forgiveness than Permission — try the operation and catch the failure, rather than checking preconditions first (LBYL). It is often faster (no wasted check on the happy path) and avoids race conditions where the state changes between the check and the use:

# EAFP — Pythonic
try:
    return data[key]
except KeyError:
    return default

File I/O

Always open files with a with block — it guarantees the file closes even if an error is raised inside:

with open(path, "r", encoding="utf-8") as f:
    content = f.read()

Always pass encoding="utf-8" explicitly — without it, Python uses the OS default, which is inconsistent across platforms. For line-by-line processing of large files, iterate the file object directly instead of .read()ing it all into memory:

with open("huge.log", encoding="utf-8") as f:
    for line in f:
        process(line.strip())

pathlib.Path is the modern way to handle paths — build them with /, and get .name, .stem, .suffix, .parent for free:

from pathlib import Path

path = Path("data") / "users" / "config.json"
path.parent.mkdir(parents=True, exist_ok=True)
for py_file in Path("src").rglob("*.py"):   # recursive glob
    ...

json.dump(data, f, indent=2, ensure_ascii=False) writes readable, non-ASCII-safe JSON (ensure_ascii=False keeps accented characters as-is instead of escaping them to \uXXXX); json.load(f) reads it back. csv.DictReader/csv.DictWriter handle CSV as dicts keyed by header row instead of positional lists — note that every CSV value comes back as a string, converting "25" to 25 is on you:

with open("out.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerows(rows)   # rows is a list of dicts with matching keys

The newline="" argument to open() for CSV specifically prevents doubled blank lines on Windows — easy to forget, annoying to debug. For genuinely binary data (images, any non-text file), open with "rb"/"wb" — opening a binary file in text mode corrupts it. tempfile.TemporaryDirectory() is a context manager that creates a directory and deletes it (and everything in it) when the block ends — ideal for tests, since nothing it creates outlives the test.

Generators and Iterators

A function with yield instead of return becomes a generator — calling it does not run the body, it returns a lazy iterator that computes one value per next() call:

def fibonacci(limit: int):
    a, b = 0, 1
    while a <= limit:
        yield a
        a, b = b, a + b

for num in fibonacci(100):
    print(num, end=" ")   # 0 1 1 2 3 5 8 13 21 34 55 89

A generator expression is the lazy counterpart to a list comprehension — same syntax, parentheses instead of brackets — and uses a few hundred bytes regardless of how many values it will eventually produce, versus a list comprehension that allocates everything upfront:

total = sum(x * x for x in range(1_000_000))   # never builds a list of a million squares

Chain generators into a pipeline and each item flows through every stage before the next item starts — no intermediate lists anywhere:

def read_values(data): yield from (v.strip() for v in data)
def parse_ints(values):
    for v in values:
        try: yield int(v)
        except ValueError: continue

result = list(parse_ints(read_values(["10", "abc", "20"])))   # [10, 20]

Use a generator when you process something once and do not need random access or len(); use a list when you need either of those, or need to iterate more than once (a generator is exhausted after one pass) — calling list() on an already-consumed generator returns [], silently, not an error.

The memory difference is the entire reason this feature exists: sys.getsizeof on a list comprehension over a million items reports roughly 8MB; the equivalent generator expression reports around 200 bytes, because it stores only the formula and its current position, never the full sequence.

itertools is the standard-library toolbox for working with iterables lazily, without writing the generator functions yourself. itertools.chain(a, b, c) walks several iterables as one, without concatenating them into a new list first; itertools.islice(gen, 5) takes a slice from any iterable, including an infinite generator; itertools.groupby groups consecutive matching elements — it only works correctly on data already sorted by the grouping key, a common source of “why didn’t this group everything together” bugs; itertools.batched(range(10), 3) (3.12+) splits an iterable into fixed-size chunks, handy for batching database inserts.

A custom class becomes iterable by implementing __iter__ and __next__, raising StopIteration when exhausted — but a class written this way can only be iterated once; making __iter__ itself a generator function (using yield) instead produces a fresh iterator on every call, so the same object can be iterated repeatedly:

class Countdown:
    def __init__(self, start): self.start = start
    def __iter__(self):
        current = self.start
        while current > 0:
            yield current
            current -= 1

c = Countdown(3)
list(c)   # [3, 2, 1]
list(c)   # [3, 2, 1] again — works, because __iter__ builds a new generator each time

Decorators

Because functions are first-class objects in Python, a decorator can wrap one function in another that adds behavior — logging, timing, caching, retries — without touching the original code:

import functools

def timer(func):
    @functools.wraps(func)   # preserves func.__name__ and __doc__ — always include this
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.perf_counter() - start:.4f}s")
        return result
    return wrapper

@timer
def process_data(items: list) -> int:
    return len(items)

@timer above def process_data is exactly process_data = timer(process_data) — nothing magic, just a function replacing another function. A decorator that itself takes arguments needs one more layer of nesting (a “decorator factory”):

def retry(max_attempts: int = 3):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    last = e
            raise last
        return wrapper
    return decorator

@retry(max_attempts=3)
def fetch_data(url: str) -> str: ...

The standard library already ships production-quality decorators — reach for functools.lru_cache instead of hand-writing a memoization cache. Stacked decorators apply bottom-to-top: @a @b def f() is f = a(b(f)).

Decorators aren’t limited to functions — a class decorator receives the class itself and can return a modified version of it, which is how a singleton pattern is implemented without touching the class body:

def singleton(cls):
    instances = {}
    def get_instance(*args, **kwargs):
        if cls not in instances:
            instances[cls] = cls(*args, **kwargs)
        return instances[cls]
    return get_instance

@singleton
class DatabaseConnection: ...

You’ve already used class decorators — @dataclass is one. functools.wraps sets a __wrapped__ attribute on the wrapper pointing back to the original function, which lets you bypass the decorator entirely when needed (greet.__wrapped__("Alex") skips any logging/caching the decorator added) — useful for testing and debugging a decorated function in isolation. You’ll recognize this whole pattern everywhere once you know it: @app.get("/users") in FastAPI, @pytest.mark.parametrize in pytest, @property and @classmethod in plain Python — all the same mechanism, just registering or wrapping in a different way.

Context Managers

You have used with since file I/O — it is powered by any object implementing __enter__ and __exit__. __enter__ runs on entry (its return value is what as name receives); __exit__ always runs on exit, even if the block raised an exception, which is exactly the guarantee try/finally gives you but cleaner:

class Timer:
    def __enter__(self):
        self._start = time.perf_counter()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.elapsed = time.perf_counter() - self._start
        return False   # False = let exceptions propagate; True would silently swallow them

For simple cases, @contextmanager from contextlib turns a generator into a context manager — code before yield is the setup, code in finally after it is the cleanup:

from contextlib import contextmanager

@contextmanager
def temp_directory():
    tmp = Path(tempfile.mkdtemp())
    try:
        yield tmp
    finally:
        shutil.rmtree(tmp)

Use a context manager for anything needing guaranteed setup/cleanup: files, database connections, locks, timers, temp resources. Multiple context managers can share one with statement (Python 3.10+), entered in order and exited in reverse — cleaner than nesting them:

with (
    temp_directory() as tmp,
    open(tmp / "out.txt", "w") as f,
):
    f.write("data")

A transaction pattern — roll back changes if the block raises — is a natural fit for @contextmanager, backing up state before yield and restoring it in the except:

@contextmanager
def transaction(data: dict):
    backup = dict(data)
    try:
        yield data
    except Exception:
        data.clear()
        data.update(backup)   # restore on failure
        raise                  # still propagate the error to the caller

async with is the same pattern with __aenter__/__aexit__ for resources that need to await during setup or teardown — covered next.

Type Hints Deep Dive

Beyond the str/int basics: list[int], dict[str, int], and tuple[int, ...] are the modern (3.9+) collection hints, no typing import needed. str | None (3.10+) replaces Optional[str]; int | str replaces Union[int, str]. Literal["low", "medium", "high"] restricts a parameter to specific values — your IDE autocompletes them and mypy flags anything else. Callable[[int], bool] types a function-taking-a-function parameter: argument types in [], return type after.

Naming a complex type pays off once it appears more than once — Python 3.12’s type statement is the cleanest way (no import needed): type UserMap = dict[int, str] documents intent far better than dict[int, str] repeated at every call site. Annotated[int, "metadata"] attaches extra information to a type that Python itself ignores but frameworks read — FastAPI uses it to attach validation constraints directly to a parameter’s type: page: Annotated[int, Query(ge=1)] = 1. For the rare case of two modules needing each other’s types only for annotations (a circular import that would otherwise fail at runtime), if TYPE_CHECKING: guards an import so it only happens during static analysis, never when the code actually runs.

mypy checks all of this statically, without running your code:

mypy src/

Type hints are purely developer tooling — Python ignores them at runtime, so they add zero performance cost and catch zero bugs on their own. If you need runtime enforcement, that is what Pydantic (above) is for. Add hints gradually: start with public function signatures — the ones other code depends on — before bothering with local variables mypy can usually infer anyway. Turning on strict = true in pyproject.toml’s [tool.mypy] section from day one on an existing, untyped codebase produces an overwhelming wall of errors; start with check_untyped_defs = true and tighten the settings incrementally as coverage grows, which is how every large Python codebase actually got typed rather than doing it all in one pass.

Part 3: Async, Testing, and the Web

Testing with pytest

Install pytest; write functions named test_* in files named test_*.py, and use plain assert:

def test_add():
    assert add(3, 4) == 7

Failures show the actual and expected values automatically — no self.assertEqual boilerplate needed. pytest.raises checks that an exception was raised:

def test_divide_by_zero():
    with pytest.raises(ValueError, match="Cannot divide by zero"):
        divide(10, 0)

Fixtures provide reusable setup, injected as function parameters — pytest calls the fixture fresh for every test that requests it, so tests stay isolated:

@pytest.fixture
def empty_cart():
    return ShoppingCart()

def test_empty_cart(empty_cart):
    assert empty_cart.total() == 0.0

Put shared fixtures in conftest.py to make them available to every test file in the directory without an import. @pytest.mark.parametrize runs one test function against many input/expected pairs — much better than asserting five things in one test, because you see exactly which input failed:

@pytest.mark.parametrize("n, expected", [(3, "Fizz"), (5, "Buzz"), (15, "FizzBuzz"), (7, "7")])
def test_fizzbuzz(n, expected):
    assert fizzbuzz(n) == expected

For external dependencies (a database, an API), use unittest.mock.MagicMock so your tests stay fast and do not depend on the network. Pass spec=WeatherService and the mock only allows attributes that actually exist on WeatherService — calling a typo’d method raises AttributeError immediately instead of silently returning another mock, which is what makes spec= worth adding every time:

mock_service = MagicMock(spec=WeatherService)
mock_service.get_temperature.return_value = 22.0
result = format_weather_report(mock_service, "Berlin")
mock_service.get_temperature.assert_called_once_with("Berlin")   # verify, not just stub

pytest.approx handles floating-point comparisons that == gets wrong due to binary rounding — 0.1 + 0.2 == 0.3 is False in every language that uses IEEE floats, Python included, so write 0.1 + 0.2 == pytest.approx(0.3) for any assertion involving computed floats rather than exact literals.

Put a mock behind its own @pytest.fixture when several tests need the same one, and override the return_value per test as needed — the fixture builds a fresh mock every time, so tests never leak mock state into each other. @pytest.mark.slow (or any custom name) tags tests for selective runs — pytest -m "not slow" skips them for a fast local loop, pytest -m slow runs only them in CI; register custom markers in pyproject.toml to avoid warnings. The TDD cycle — write a failing test, write the minimum code to pass it, then refactor with the safety net in place — is worth knowing even if you don’t follow it strictly everywhere; it’s most valuable exactly where BookTrail’s capstone applies it later, on the validation and not-found paths where the expected behavior is easy to state before the code exists.

Async/Await

Three independent 2-second API calls take 6 seconds run sequentially. Async runs them concurrently — 2 seconds total — because while one task is waiting (for network, disk, a database), the event loop lets another task run:

import asyncio

async def fetch_data(name: str) -> str:
    await asyncio.sleep(2)      # suspends here — does NOT block the thread
    return f"Data from {name}"

async def main():
    results = await asyncio.gather(
        fetch_data("API-1"), fetch_data("API-2"), fetch_data("API-3"),
    )

asyncio.run(main())   # the one entry point from sync code — call it once, at the top level

async def creates a coroutine; you can only await one inside another async def — using await in a regular function is a SyntaxError. asyncio.gather() runs several coroutines concurrently and returns their results in input order; pass return_exceptions=True to collect failures instead of raising on the first one. async for and async with are the async equivalents of for and with, for iterators and context managers that need to await during their protocol methods (__anext__, __aenter__).

The most common mistake: calling time.sleep() inside async code, which blocks the entire event loop, not just one task — always use await asyncio.sleep() instead. Decision rule: async for I/O-bound work with async-compatible libraries (httpx, asyncpg); threading for I/O-bound work with sync-only libraries; multiprocessing for CPU-bound work that needs multiple cores.

asyncio.create_task() starts a coroutine immediately without waiting for it — useful when you want to kick off work and do something else before collecting the result, where gather() assumes you want to launch and immediately await everything as a batch. asyncio.wait_for(coro, timeout=2.0) bounds a single call that might hang, raising TimeoutError if it runs too long — always wrap network calls with a timeout, since a stuck connection with none hangs your whole program. asyncio.Semaphore(n) caps concurrency inside a batch of tasks, the async equivalent of a worker pool — async with semaphore: around the actual request body ensures only n requests are in flight at once, which is how you avoid hammering an API that rate-limits you:

semaphore = asyncio.Semaphore(3)   # max 3 concurrent requests

async def fetch(url):
    async with semaphore:
        return await client.get(url)

For streaming data rather than a single value, async for consumes an async iterator — a class implementing __aiter__/__anext__ (the await-capable counterparts of __iter__/__next__), the shape you’ll see behind database cursors and paginated API clients that fetch the next page lazily as you iterate.

HTTP and APIs

httpx is the modern HTTP client — same API for sync and async code:

import httpx

response = httpx.get("https://api.example.com/todos/1", timeout=10.0)
response.raise_for_status()   # raises on 4xx/5xx instead of silently returning bad data
data = response.json()

Always set an explicit timeout — the default exists but is easy to forget, and a hung server without one hangs your program forever. For repeated calls, use httpx.Client() (or AsyncClient()) as a context manager — it pools connections and lets you share a base_url and headers across every call:

with httpx.Client(base_url="https://api.example.com", timeout=10.0) as client:
    todos = client.get("/todos", params={"userId": 1}).json()
    posts = client.get("/posts", params={"userId": 1}).json()

POST with json=data (auto-sets the content type and serializes it). Never hardcode API keys — read them from environment variables (os.environ["API_KEY"]) and pass them as an Authorization: Bearer ... header.

Network calls fail transiently — retry with exponential backoff rather than hammering a struggling server at a fixed interval, and only retry server errors (5xx) or connection failures, never a 4xx, since a client error means you sent something the server will reject every time:

for attempt in range(max_retries):
    try:
        response = httpx.get(url, timeout=10.0)
        if response.status_code < 500:
            return response
    except httpx.RequestError:
        pass
    time.sleep(delay * (2 ** attempt))   # 1s, 2s, 4s...

Most real APIs paginate list endpoints — loop until a page comes back empty, or follow whatever cursor/next-URL field the API returns:

def fetch_all_pages(client, endpoint):
    page, items = 1, []
    while True:
        batch = client.get(endpoint, params={"page": page}).json()
        if not batch:
            break
        items.extend(batch)
        page += 1
    return items

For a service you call repeatedly, wrap the calls in a small client class — shared base_url, headers, and timeout in one place, one method per endpoint, and a single spot to add retry or logging later instead of duplicating it at every call site.

Databases

Python’s built-in sqlite3 needs no installation — good for small apps and prototypes. Always use ? placeholders, never f-strings, for values in a query — string interpolation into SQL is how SQL injection happens:

import sqlite3

conn = sqlite3.connect("tasks.db")
conn.row_factory = sqlite3.Row   # access columns by name: row["title"]
conn.execute("INSERT INTO tasks (title) VALUES (?)", (title,))   # safe
conn.commit()

One sqlite3 surprise worth flagging: using a connection as a with block (with sqlite3.connect(...) as conn:) only wraps the transaction — it commits on success and rolls back on an exception inside the block — it does not close the connection the way a file’s with block closes the file. Call conn.close() yourself afterward if you need the connection actually released.

For anything beyond a handful of queries, SQLAlchemy (the standard Python ORM) lets you model tables as classes instead of hand-writing SQL:

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

class Base(DeclarativeBase): pass

class Book(Base):
    __tablename__ = "books"
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str]

session.add(obj) + session.commit() inserts; session.get(Book, id) reads by primary key; session.query(Book).filter(...) for anything more complex.

relationship() connects two tables — a User with posts: Mapped[list["Post"]] = relationship(back_populates="author") and a matching Post.author gives you navigation in both directions: alex.posts lists a user’s posts, post.author.name walks back to the owner, without writing the join yourself. cascade="all, delete-orphan" on the relationship means deleting a User also deletes their Posts automatically — without it, deleting a user with existing posts fails on the foreign key constraint instead.

Watch for the N+1 query problem — accessing user.posts inside a loop over users issues one query per user instead of one query total:

# BAD — one extra query per user
for user in session.query(User).all():
    print(len(user.posts))

# GOOD — eager-load with one JOIN
users = session.query(User).options(joinedload(User.posts)).all()

Wrap multi-step writes in a transaction so they succeed or fail together — session.commit() only if every step reaches it; on an exception, call session.rollback() (or wrap the block in with session.begin():, which commits on success and rolls back automatically on exception) so a half-finished transfer never leaves the database in an inconsistent state.

Logging and Debugging

print() debugging does not scale: no levels, no timestamps, cannot be turned off in production. logging fixes all of it:

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logging.info("Application started")
logging.error("Failed to connect to database")

Five levels, ascending severity: DEBUG, INFO, WARNING, ERROR, CRITICAL. Setting the logger to WARNING silently drops DEBUG and INFO calls — use DEBUG in development, WARNING/INFO in production. Use %-style placeholders (logger.info("Order %d", order_id)), not f-strings, in log calls — the string is only formatted if the message will actually be emitted, so an f-string wastes work on messages that get filtered out. logger.exception(...) inside an except block automatically attaches the full traceback.

For interactive debugging, breakpoint() drops you into pdb at that exact line — n (next line), s (step into), p variable (print), c (continue). VS Code’s built-in debugger does the same thing with a GUI: click left of a line number to set a breakpoint, press F5.

A handler decides where a log record goes, and a logger can have several at once — StreamHandler for the console, FileHandler for a file, each with its own level and Formatter, so you can show INFO on screen while a file captures everything down to DEBUG:

console = logging.StreamHandler()
console.setLevel(logging.INFO)

file_handler = logging.FileHandler("app.log")
file_handler.setLevel(logging.DEBUG)   # file gets everything, console only INFO+

logger.addHandler(console)
logger.addHandler(file_handler)

A TimedContext context manager (from Part 2’s context-manager pattern) logs how long a block took without a manual start = time.perf_counter() / elapsed = ... pair scattered through the codebase:

class TimedContext:
    def __enter__(self):
        self._start = time.perf_counter()
        return self
    def __exit__(self, *exc):
        self.elapsed = time.perf_counter() - self._start
        logger.info("completed in %.3fs", self.elapsed)

Automation Scripts

A large share of real-world Python is small automation scripts, not services — and the patterns above compose into one directly. A file organizer that sorts a Downloads folder by extension is a representative example: a category lookup, a dry-run mode you always build first, and a conflict resolver that never silently overwrites an existing file:

FILE_CATEGORIES = {"images": [".jpg", ".png"], "documents": [".pdf", ".txt"]}

def get_category(path: Path) -> str:
    for category, extensions in FILE_CATEGORIES.items():
        if path.suffix.lower() in extensions:
            return category
    return "other"

def resolve_conflict(target: Path) -> Path:
    if not target.exists():
        return target
    stem, suffix, counter = target.stem, target.suffix, 1
    while target.exists():
        target = target.parent / f"{stem}_{counter}{suffix}"
        counter += 1
    return target

Always build the dry_run=True path first and default to it — log what would move without calling shutil.move(), verify the output looks right, then flip the flag. Read the source and target directories from environment variables (os.environ.get("SOURCE_DIR", str(Path.home() / "Downloads"))) rather than hardcoding a path, so the same script works on any machine it’s copied to. For catching a script running unattended, try/except OSError around the actual move — shutil.move can fail mid-run if a file disappears or permissions are wrong, and a script that crashes on the first bad file instead of logging it and continuing defeats the point of automating something you’d otherwise babysit by hand.

Part 4: Build BookTrail — a CLI, a Scraper, and a REST API Sharing One Model

Everything above is enough to build something real. BookTrail is a reading-list tracker: one shared Book model backs a CLI you run locally, a scraper that imports real book data, and a REST API — showing how a dataclass, error handling, and file I/O designed in isolation hold up once three different front ends depend on them.

The Shared Model

from dataclasses import dataclass, asdict, field
from datetime import datetime
from pathlib import Path
import json

@dataclass
class Book:
    id: int
    title: str
    author: str = ""
    status: str = "to_read"     # to_read, reading, finished
    rating: int | None = None
    added_at: str = field(default_factory=lambda: datetime.now().strftime("%Y-%m-%d"))

    def to_dict(self) -> dict:
        return asdict(self)

    @classmethod
    def from_dict(cls, data: dict) -> "Book":
        return cls(**data)


class BookShelf:
    """Core logic — no CLI or web framework dependencies, easy to test."""

    def __init__(self, filepath: Path) -> None:
        self.filepath = filepath
        self.books: list[Book] = self._load()

    def _load(self) -> list[Book]:
        if not self.filepath.exists():
            return []
        try:
            data = json.loads(self.filepath.read_text(encoding="utf-8"))
            return [Book.from_dict(item) for item in data]
        except (json.JSONDecodeError, TypeError):
            return []

    def _save(self) -> None:
        self.filepath.parent.mkdir(parents=True, exist_ok=True)
        self.filepath.write_text(
            json.dumps([b.to_dict() for b in self.books], indent=2), encoding="utf-8"
        )

    def add(self, title: str, author: str = "") -> Book:
        if not title.strip():
            raise ValueError("Title cannot be empty")
        next_id = max((b.id for b in self.books), default=0) + 1
        book = Book(id=next_id, title=title.strip(), author=author)
        self.books.append(book)
        self._save()
        return book

    def mark_finished(self, book_id: int, rating: int | None = None) -> Book:
        book = self._find(book_id)
        book.status = "finished"
        book.rating = rating
        self._save()
        return book

    def by_status(self, status: str) -> list[Book]:
        return [b for b in self.books if b.status == status]

    def _find(self, book_id: int) -> Book:
        for book in self.books:
            if book.id == book_id:
                return book
        raise KeyError(f"Book #{book_id} not found")

This is Part 2 in miniature: a dataclass for the shape, ValueError/KeyError for the two ways add and _find can fail, pathlib + json for persistence, and a list comprehension for filtering. Keeping BookShelf free of CLI or web framework imports is what makes the next three sections possible without duplicating logic.

CLI: Click + Rich

import click
from rich.console import Console
from rich.table import Table

console = Console()
SHELF_FILE = Path.home() / ".booktrail" / "shelf.json"

def get_shelf() -> BookShelf:
    return BookShelf(SHELF_FILE)

@click.group()
def cli():
    """BookTrail — track what you read."""

@cli.command()
@click.argument("title")
@click.option("--author", default="")
def add(title: str, author: str):
    shelf = get_shelf()
    book = shelf.add(title, author)
    console.print(f"[green]Added #{book.id}:[/green] {book.title}")

@cli.command("list")
@click.option("--status", default=None, type=click.Choice(["to_read", "reading", "finished"]))
def list_books(status: str | None):
    shelf = get_shelf()
    books = shelf.by_status(status) if status else shelf.books
    table = Table(title="BookTrail")
    table.add_column("ID"); table.add_column("Title"); table.add_column("Status")
    for b in books:
        table.add_row(str(b.id), b.title, b.status)
    console.print(table)

if __name__ == "__main__":
    cli()

click.group() plus @cli.command() gives you a git-style CLI (booktrail add "...", booktrail list --status reading) with zero manual argument parsing. click.option(..., is_flag=True) gives you a boolean switch (--pending) with no value needed; click.Choice([...]) restricts an option to a fixed set of values and Click validates it before your function even runs, rejecting --status unknown with a clear error instead of silently passing the bad string through. Rich’s Table turns the plain list into aligned, colored terminal output, and markup tags (console.print("[red]text[/red]")) color output inline without manual ANSI escape codes — useful for coloring the status column by value (red for overdue, green for finished) the way BookTrail’s real CLI does.

Scraper: Import Real Books

books.toscrape.com is a public sandbox built for scraping practice — use it (or your own data source) rather than a live site you have not checked the terms of service for. BeautifulSoup plus CSS selectors extract structured data from the HTML:

import httpx
from bs4 import BeautifulSoup

def scrape_books(page: int = 1) -> list[dict]:
    url = f"https://books.toscrape.com/catalogue/page-{page}.html"
    response = httpx.get(url, timeout=10.0, headers={"User-Agent": "BookTrail/1.0"})
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "lxml")

    return [
        {"title": a["title"]}
        for a in soup.select("article.product_pod h3 a")
    ]

def import_scraped(shelf: BookShelf, page: int = 1) -> int:
    books = scrape_books(page)
    for data in books:
        shelf.add(data["title"])
    return len(books)

import_scraped reuses BookShelf.add — the same validation and persistence the CLI uses, with a different source of titles. Always set a User-Agent and a timeout, and add a delay between requests if scraping multiple pages (see Part 3’s httpx section for a rate limiter pattern). CSS selectors via soup.select(...) are usually easier to write than .find() chains if you already know CSS: article.product_pod h3 a reads directly as “an <a> inside an <h3> inside an element with class product_pod.” Before scraping any site that isn’t a sandbox, check https://<site>/robots.txt for disallowed paths, and respect them — a Disallow entry is the site telling you explicitly what it doesn’t want crawled.

REST API: FastAPI

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI(title="BookTrail API")
shelf = BookShelf(SHELF_FILE)

class BookCreate(BaseModel):
    title: str
    author: str = ""

@app.post("/books", status_code=201)
def create_book(data: BookCreate):
    try:
        return shelf.add(data.title, data.author).to_dict()
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))

@app.get("/books")
def list_books(status: str | None = None):
    books = shelf.by_status(status) if status else shelf.books
    return [b.to_dict() for b in books]

@app.post("/books/{book_id}/finish")
def finish_book(book_id: int, rating: int | None = None):
    try:
        return shelf.mark_finished(book_id, rating).to_dict()
    except KeyError:
        raise HTTPException(status_code=404, detail="Book not found")

The ValueError/KeyError pair from BookShelf maps directly onto HTTP 400/404 — the same two error paths, three times, once per front end, with no logic duplicated. uvicorn main:app --reload runs it; visit /docs for automatic Swagger documentation, generated from the Pydantic model and route signatures with zero extra code.

For a shelf backed by a real database instead of the JSON file, FastAPI’s Depends() is the idiomatic way to supply and clean up a resource per request — a generator function that yields the resource and closes it in finally after the response is sent:

def get_db():
    db = SessionFactory()
    try:
        yield db
    finally:
        db.close()

@app.get("/books")
def list_books(db: Session = Depends(get_db)):
    return db.query(BookDB).all()

Every route that declares db: Session = Depends(get_db) gets a fresh, automatically-closed session — no manual connection management scattered across handlers. Query(ge=1, le=100) on a query parameter adds the same kind of validation Field() adds to a Pydantic model, directly in the function signature: page: int = Query(default=1, ge=1) rejects a negative or absurdly large page number before your handler code ever runs, and shows up in the generated /docs automatically.

Testing All Three

import pytest

@pytest.fixture
def shelf(tmp_path):
    return BookShelf(tmp_path / "shelf.json")

def test_add_and_persist(tmp_path):
    path = tmp_path / "shelf.json"
    BookShelf(path).add("Dune")
    assert len(BookShelf(path).books) == 1   # reloaded from disk — persistence works

def test_add_empty_title_raises(shelf):
    with pytest.raises(ValueError):
        shelf.add("")

def test_finish_unknown_raises(shelf):
    with pytest.raises(KeyError):
        shelf.mark_finished(999)

from fastapi.testclient import TestClient

def test_api_create_and_list(tmp_path, monkeypatch):
    monkeypatch.setattr("booktrail.api.shelf", BookShelf(tmp_path / "shelf.json"))
    client = TestClient(app)
    response = client.post("/books", json={"title": "Dune"})
    assert response.status_code == 201
    assert client.get("/books").json()[0]["title"] == "Dune"

tmp_path (a built-in pytest fixture) gives every test an isolated, throwaway directory — no test ever touches your real ~/.booktrail/shelf.json, and tests never interfere with each other.

Project Layout

booktrail/
    src/booktrail/
        model.py       # Book, BookShelf — the shared core
        cli.py          # Click + Rich
        scraper.py      # httpx + BeautifulSoup
        api.py           # FastAPI
    tests/
        test_model.py
        test_api.py
    pyproject.toml

One BookShelf class, three ways in. That is the actual payoff of Part 1 through 3: dataclasses and clean error handling made the model hard to misuse, and keeping it free of CLI/web imports meant the CLI, scraper, and API could all reuse it verbatim instead of re-implementing validation three times. Natural next steps: swap the JSON file for the SQLAlchemy model from Part 3, add async scraping with httpx.AsyncClient for multiple pages at once, or add the automation-script pattern (from the original series) to export a weekly reading report.

Where to Go From Here

The complete, working code for the capstone project (BookTrail) is on GitHub: github.com/kemalcodes/python-tutorial.