How to Use This Handbook

This series is designed for readers who can already write functions, classes, comprehensions, and small applications. It takes you from “comfortable Python programmer” to “engineer who can reason about a Python system.”

Examples with a Run control execute in Pyodide's WebAssembly build of Python, inside a disposable browser worker with a five-second execution limit. Use them for language experiments, not for conclusions about native CPython object sizes, threading, processes, allocator behavior, or benchmark timings.

Every chapter follows a repeatable rhythm:

1. Mental model — the idea in plain language.
2. Mechanics — what Python actually does.
3. Worked example — a small, focused implementation.
4. Engineering judgment — when the technique helps and when it does not.
5. Exercises — deliberate practice.
6. Review checklist — prompts you can reuse on real projects.

All examples target modern Python 3.12+ unless stated otherwise. Most work on Python 3.10+. Commands use a Unix-like shell, but the concepts are platform-independent.

Prerequisites

You should be able to:

• create and activate a virtual environment;
• use lists, dictionaries, sets, functions, exceptions, and classes;
• import modules and install packages;
• read a traceback;
• use Git for basic version control;
• write a small test with pytest or unittest.

The engineering loop

Use this loop throughout the course:

Observe → form a hypothesis → make the smallest change → measure → explain the result.

That loop matters more than memorizing implementation trivia. It prevents speculative optimization, accidental redesign, and cargo-cult security.

Curriculum Map

Suggested learning sequence

| Phase | Weeks | Topics | Primary outcome | |---|---:|---|---| | I. Runtime foundations | 1–3 | Object model, execution, memory, protocols | Explain Python behavior from first principles | | II. Quality and design | 4–6 | Refactoring, design, typing, architecture | Change code safely and make boundaries explicit | | III. Reliability | 7–9 | Testing, debugging, tooling, security | Find defects systematically and reduce risk | | IV. Scale and speed | 10–12 | Profiling, algorithms, concurrency, async | Improve measured bottlenecks and throughput | | V. Delivery | 13–14 | Packaging, releases, observability | Ship maintainable libraries and services | | VI. AI engineering | 15–16 | LLM APIs, tools, agents, evaluation, safety | Build bounded, testable agent workflows | | VII. Capstones | 17–20+ | Integrated projects | Demonstrate senior-level engineering judgment |

Three pacing options

• Focused, 8 weeks: two chapters per week, one exercise per chapter, one capstone.
• Thorough, 16 weeks: one chapter per week, all core exercises, two capstones.
• Deep study, 24 weeks: add source-code reading, benchmarks, write-ups, and all capstones.

Repository shape for course work

advanced-python/
├── pyproject.toml
├── README.md
├── src/
│   └── course_labs/
├── tests/
├── benchmarks/
├── docs/
│   └── decisions/
└── capstones/

Keep an engineering journal in docs/decisions/. For each substantial exercise, record the problem, evidence, decision, alternatives, and result.

Part I — Python’s Runtime Model

Chapter 1 — Names, Objects, Identity, and Mutability

Mental model

Python variables are names bound to objects. Assignment normally changes a binding; it does not copy an object. Objects have an identity, a type, and a value. Mutability determines whether the value can change while identity remains stable.

original = {"tags": ["python"]}
alias = original
shallow = original.copy()

alias["tags"].append("runtime")

assert original is alias
assert shallow is not original
assert shallow["tags"] is original["tags"]

print("[check] original is alias:", original is alias)
print("[check] nested list is shared:", shallow["tags"] is original["tags"])
print("[state] original tags after alias mutation:", original["tags"])

The dictionary copy is new, but its nested list is shared. This is why “I copied it” is incomplete: ask which layer was copied and which references remain shared.

Equality versus identity

Use == for value equality and is for identity. The common production use of is is comparing with a singleton:

MISSING = object()

def get_timeout(config: dict[str, object]) -> float | None:
    value = config.get("timeout", MISSING)
    if value is MISSING:
        return 30.0
    if value is None:
        return None
    return float(value)

Here, missing and explicitly disabled are different states.

Function arguments and defaults

Arguments bind local names to objects. Mutating a passed mutable object can affect the caller; rebinding the local name cannot.

def add_label(labels: list[str], label: str) -> None:
    labels.append(label)          # visible to caller

def replace_labels(labels: list[str]) -> None:
    labels = ["replacement"]     # local rebinding only

Default argument expressions run once, when the function is defined:

def append_event(event: str, events: list[str] | None = None) -> list[str]:
    if events is None:
        events = []
    events.append(event)
    return events

Engineering judgment

Prefer immutable value objects at boundaries. Copy only where ownership changes, and document whether an API borrows, consumes, or returns shared mutable state.

Exercises

1. Predict the identities and values of nested objects after shallow and deep copies; verify with id().
2. Create a bug caused by a mutable default, write a failing test, and repair it.
3. Implement a freeze() function that recursively converts lists to tuples and dictionaries to immutable key/value tuples. Define how cycles are handled.
4. Model “missing,” “present with None,” and “present with a value” without confusing the three states.

Review checklist

• Are mutable objects shared intentionally?
• Is is limited to singleton/sentinel checks?
• Do function defaults avoid mutable instances and time-dependent calls?
• Is ownership clear when objects cross boundaries?

Chapter 2 — Bytecode, Frames, Scope, and the Import System

Python source is compiled to bytecode, then evaluated by the interpreter. The exact bytecode is an implementation detail and changes between versions, but inspecting it builds intuition.

import dis

def total_with_tax(amount: float, rate: float) -> float:
    return amount * (1 + rate)

print("[state] bytecode disassembly for total_with_tax:")
dis.dis(total_with_tax)
print("[result] total for amount=100 and tax rate=20%:", total_with_tax(100, 0.20))

A running function has a frame containing instruction state, local bindings, globals, and a link to the previous frame. Tracebacks are chains of frames. This explains both their diagnostic power and why retaining tracebacks can retain many objects.

LEGB name lookup

Python resolves ordinary names through Local, Enclosing, Global, and Builtins scopes.

def make_counter() -> callable:
    count = 0

    def increment() -> int:
        nonlocal count
        count += 1
        return count

    return increment

Late binding in closures is a frequent trap:

# Wrong: every function looks up the final value of i.
functions = [lambda: i for i in range(3)]

# Deliberate snapshot through a default argument.
functions = [lambda i=i: i for i in range(3)]

Imports are execution plus caching

On first import, Python finds a module, creates its module object, executes its top-level code, and caches it in sys.modules. Circular imports expose partially initialized modules.

Avoid heavy work at import time. Keep dependency direction acyclic, and move shared concepts to a lower-level module rather than hiding circularity with scattered local imports.

# Good module boundary: definitions at import time, effects on request.
def build_client(settings: "Settings") -> "Client":
    return Client(endpoint=settings.endpoint)

Exercises

1. Use dis to compare a loop, a list comprehension, and sum().
2. Reproduce a closure late-binding bug in a callback registry and fix it two ways.
3. Build two modules with a circular import, diagram the dependency, then extract the shared abstraction.
4. Inspect sys.modules before and after importing a package.

Review checklist

• Do modules perform network, filesystem, or expensive work during import?
• Are closures capturing values or names intentionally?
• Does the import graph point from policy toward stable abstractions?
• Are tracebacks logged without retaining sensitive local values unnecessarily?

Chapter 3 — The Data Model and Python Protocols

Python’s “dunder” methods are protocol hooks. Iteration, context management, containment, formatting, arithmetic, and attribute access are all behaviors expressed through protocols.

Build a value object

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True, slots=True)
class Money:
    amount: Decimal
    currency: str

    def __post_init__(self) -> None:
        if not self.currency or len(self.currency) != 3:
            raise ValueError("currency must be a three-letter code")

    def __add__(self, other: object) -> "Money":
        if not isinstance(other, Money):
            return NotImplemented
        if self.currency != other.currency:
            raise ValueError("cannot add different currencies")
        return Money(self.amount + other.amount, self.currency)

subtotal = Money(Decimal("19.95"), "GBP")
tax = Money(Decimal("4.00"), "GBP")
print("[result] GBP subtotal plus tax:", subtotal + tax)

Returning NotImplemented allows Python to try reflected operations or raise the correct error. It is not the same as raising NotImplementedError.

Iterators and generators

An iterable can produce an iterator. An iterator maintains traversal state. A generator is a concise iterator whose frame pauses at yield.

from collections.abc import Iterable, Iterator

def batched(items: Iterable[str], size: int) -> Iterator[list[str]]:
    if size < 1:
        raise ValueError("size must be positive")
    batch: list[str] = []
    for item in items:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

Generators support streaming, but they are usually single-use and defer errors until iteration.

Context managers express lifetime

from contextlib import contextmanager
from collections.abc import Iterator

@contextmanager
def transaction(connection) -> Iterator[object]:
    try:
        yield connection
    except Exception:
        connection.rollback()
        raise
    else:
        connection.commit()

The context manager owns a lifecycle. It does not silently swallow errors.

Descriptors and attribute access

Functions stored on classes are descriptors: accessing one through an instance binds the instance as self. property, classmethod, and many ORM fields build on descriptors.

Use custom descriptors sparingly. They are valuable for reusable attribute behavior, but a plain property or explicit method is often easier to understand.

Exercises

1. Implement a replayable iterable and a single-use iterator over the same data.
2. Add multiplication and readable formatting to Money while preserving invariants.
3. Write synchronous and asynchronous timing context managers.
4. Implement a validating descriptor, then compare it with a dataclass plus explicit validation.

Review checklist

• Does a custom class behave predictably under equality, hashing, and representation?
• Are iterators clearly single-pass or replayable?
• Do resource owners implement context management?
• Is protocol magic improving the API rather than concealing side effects?

Chapter 4 — Memory, Garbage Collection, and Object Layout

CPython primarily uses reference counting, supplemented by a cyclic garbage collector. An object is usually reclaimed when its reference count reaches zero, but cycles require separate detection.

import gc
import weakref

class Node:
    def __init__(self, name: str) -> None:
        self.name = name
        self.neighbor: Node | None = None

a = Node("a")
b = Node("b")
a.neighbor = b
b.neighbor = a

watch = weakref.ref(a)
del a, b
collected = gc.collect()
assert watch() is None

print("[check] reference cycle was reclaimed:", watch() is None)
print("[result] unreachable objects found by gc.collect():", collected)

Measure retained memory

sys.getsizeof() reports shallow size. For allocations over time, use tracemalloc:

import tracemalloc

tracemalloc.start()
before = tracemalloc.take_snapshot()

data = [{"id": i, "payload": "x" * 100} for i in range(20_000)]

after = tracemalloc.take_snapshot()
for stat in after.compare_to(before, "lineno")[:5]:
    print("[result] allocation growth by source line:", stat)

__slots__ can reduce per-instance overhead for large numbers of simple objects, but it constrains dynamic attributes and can complicate inheritance. Measure before adopting it.

Common retention causes

• unbounded caches;
• global registries;
• queues whose consumers cannot keep up;
• callbacks that close over large objects;
• long-lived task references;
• tracebacks and exception locals;
• native extensions with their own allocation behavior.

Exercises

1. Compare memory for 500,000 regular dataclass and slotted dataclass instances.
2. Create an unbounded cache leak, capture snapshots, and replace it with a bounded policy.
3. Use weak references for an observer registry; document the lifetime tradeoff.
4. Diagnose why a list “cleared” in one scope remains reachable elsewhere.

Review checklist

• Is memory growth a leak, a cache, fragmentation, or expected workload state?
• Are caches bounded by size, time, or both?
• Are queues bounded and overload policies explicit?
• Is object-layout optimization justified by measured instance counts?

Part II — Refactoring, Design, Typing, and Architecture

Chapter 5 — Refactoring as Controlled Change

Refactoring changes structure without intentionally changing observable behavior. Safe refactoring depends on fast feedback, small steps, and an explicit definition of behavior.

From conditional tangle to policy objects

Before:

def shipping_cost(order, customer_type, country):
    if customer_type == "premium":
        if country == "GB":
            return 0
        return 5
    if order.total > 100:
        return 0
    if country == "GB":
        return 8
    return 15

After:

from dataclasses import dataclass
from decimal import Decimal
from typing import Protocol

@dataclass(frozen=True)
class Order:
    total: Decimal
    destination: str

class ShippingPolicy(Protocol):
    def price(self, order: Order) -> Decimal: ...

class StandardShipping:
    def price(self, order: Order) -> Decimal:
        if order.total > Decimal("100"):
            return Decimal("0")
        return Decimal("8") if order.destination == "GB" else Decimal("15")

class PremiumShipping:
    def price(self, order: Order) -> Decimal:
        return Decimal("0") if order.destination == "GB" else Decimal("5")

The key improvement is not “more classes.” It is separating a varying business rule behind a small contract. If the rule will never vary independently, extracted functions may be better.

A refactoring sequence

1. Characterize current behavior with tests.
2. Name the responsibility being separated.
3. Extract a pure function or value object.
4. Introduce an interface only if multiple implementations or test seams justify it.
5. Move one caller at a time.
6. Remove obsolete paths.
7. Run tests, type checks, and representative benchmarks.

Code smells as questions

• Long function: does it mix levels of abstraction?
• Primitive obsession: is an important concept missing a type?
• Feature envy: is behavior located away from the data/invariants it needs?
• Shotgun surgery: is one business change scattered across modules?
• Boolean parameter: does the function secretly have two responsibilities?
• Inheritance tangle: would composition make capabilities clearer?

Exercises

1. Refactor a 60-line invoice function into calculation, policy, and rendering stages.
2. Replace three related primitive arguments with a validated value object.
3. Remove an unnecessary abstract base class and compare the resulting design.
4. Write a characterization test for intentionally strange legacy behavior.

Review checklist

• Is each commit behavior-preserving or explicitly behavior-changing?
• Does every abstraction name a real concept?
• Are pure decisions separated from effects?
• Has duplication of knowledge—not merely similar syntax—been reduced?

Chapter 6 — Composition, Dependency Inversion, and Useful Patterns

Patterns are vocabulary for recurring forces, not decorations to apply everywhere.

Functional core, imperative shell

Keep business decisions pure and push I/O to the edges:

from dataclasses import dataclass

@dataclass(frozen=True)
class Account:
    balance_cents: int
    frozen: bool = False

def can_withdraw(account: Account, amount_cents: int) -> bool:
    return not account.frozen and 0 < amount_cents <= account.balance_cents

def withdraw(account_id: str, amount_cents: int, repository, gateway) -> None:
    account = repository.get(account_id)
    if not can_withdraw(account, amount_cents):
        raise ValueError("withdrawal rejected")
    gateway.dispense(amount_cents)
    repository.save(Account(account.balance_cents - amount_cents, account.frozen))

Dependency injection without a framework

Pass collaborators through constructors or functions. This exposes dependencies and makes tests cheap.

class ReportService:
    def __init__(self, repository: "ReportRepository", clock: "Clock") -> None:
        self.repository = repository
        self.clock = clock

Useful patterns in Python include:

• Strategy: interchangeable policy behind a small protocol.
• Adapter: translate an external interface into an internal one.
• Repository: collection-like access to persisted domain objects.
• Command: represent a requested action as data.
• Decorator: add behavior around a callable or object.
• State machine: make valid transitions explicit.

Avoid “service,” “manager,” and “utils” modules that become unbounded responsibility buckets.

Exercises

1. Wrap two payment SDKs behind one internal protocol.
2. Model an order lifecycle as explicit states and legal transitions.
3. Replace monkeypatch-heavy tests with constructor injection.
4. Identify a pattern that makes a sample system worse; remove it and explain why.

Review checklist

• Are dependencies visible at construction time?
• Do interfaces belong near the code that consumes them?
• Is composition favored when inheritance would couple unrelated change axes?
• Can domain policy be tested without databases, networks, or clocks?

Chapter 7 — Type-Driven Design

Type hints are executable design documentation for tools and humans. They are most useful at boundaries, public APIs, and complicated transformations.

from dataclasses import dataclass
from typing import NewType, Protocol, TypeVar

UserId = NewType("UserId", str)

@dataclass(frozen=True)
class User:
    id: UserId
    email: str

class UserReader(Protocol):
    def get(self, user_id: UserId) -> User | None: ...

T = TypeVar("T")

def first(items: list[T]) -> T | None:
    return items[0] if items else None

Make invalid states difficult to represent

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class Pending:
    status: Literal["pending"] = "pending"

@dataclass(frozen=True)
class Completed:
    receipt_id: str
    status: Literal["completed"] = "completed"

Payment = Pending | Completed

def receipt(payment: Payment) -> str | None:
    match payment:
        case Completed(receipt_id=value):
            return value
        case Pending():
            return None

This is stronger than one object with status: str and an optional receipt whose validity depends on comments.

Static types do not validate untrusted input

Type checkers analyze code; they do not make JSON trustworthy. Parse and validate input at runtime, then convert it to trusted domain types.

Advanced tools

• Protocol for structural interfaces;
• TypeVar and generics for type-preserving containers and functions;
• ParamSpec for decorators that preserve callable signatures;
• TypedDict for dictionary-shaped external structures;
• overloads for APIs whose return type truly depends on input types;
• Never and exhaustive matching for closed state models.

Do not chase a type-checker victory by obscuring the runtime design. A targeted boundary cast with a reason is better than a forest of clever generics.

Exercises

1. Add strict typing to an untyped service boundary without typing the entire application.
2. Replace a dictionary carrying domain state with a discriminated union.
3. Write a typed retry decorator using ParamSpec and TypeVar.
4. Introduce NewType identifiers to prevent mixing user, order, and account IDs.

Review checklist

• Are public interfaces fully typed?
• Does Any enter only through explicit boundaries and get narrowed promptly?
• Are None, missing, error, and empty distinct where the domain requires it?
• Do runtime validators protect untrusted data?

Chapter 8 — Architecture That Preserves Changeability

Architecture is the set of boundaries that makes some changes easy and others costly. Start with change drivers, not a folder template.

A pragmatic layered shape

src/shop/
├── domain/          # entities, value objects, pure policy
├── application/     # use cases and ports
├── adapters/        # database, HTTP, queues, external SDKs
└── entrypoints/     # CLI, web routes, worker handlers

Dependencies point inward. Domain code does not import the web framework or ORM. The application layer coordinates domain behavior and describes ports. Adapters implement those ports.

from typing import Protocol

class UnitOfWork(Protocol):
    orders: "OrderRepository"
    def commit(self) -> None: ...
    def __enter__(self) -> "UnitOfWork": ...
    def __exit__(self, *args: object) -> None: ...

def confirm_order(order_id: str, uow: UnitOfWork) -> None:
    with uow:
        order = uow.orders.get(order_id)
        if order is None:
            raise LookupError(order_id)
        order.confirm()
        uow.commit()

Architectural decision record

Use a short ADR when a decision changes system shape:

# ADR-007: Use an outbox for order events

Status: Accepted

Context: Database updates and message publication can fail independently.
Decision: Persist events with the transaction and relay asynchronously.
Consequences: At-least-once delivery; consumers must be idempotent.
Alternatives: Distributed transaction, best-effort publish.

Modular monolith before distributed system

A well-bounded monolith often provides most organizational benefits of services with fewer failure modes. Split deployment units when there is evidence: independent scaling, security isolation, ownership, or release cadence—not because folders feel large.

Exercises

1. Draw the dependency graph of an existing application and mark cycles.
2. Extract one use case from a framework handler into an application service.
3. Write an ADR comparing a modular monolith with two services.
4. Design an idempotency strategy for a command that may be delivered twice.

Review checklist

• Can core rules run without infrastructure?
• Are transaction boundaries aligned with business invariants?
• Do external models get translated at adapters?
• Is every distributed boundary justified by a concrete force?

Part III — Testing, Debugging, Tooling, and Security

Chapter 9 — A Testing Strategy That Produces Confidence

A test suite is a risk-control system. Optimize for confidence, diagnostic quality, and feedback speed—not raw test count.

Test behavior through stable seams

from dataclasses import dataclass
from datetime import datetime
from typing import Protocol

class Clock(Protocol):
    def now(self) -> datetime: ...

@dataclass
class FixedClock:
    instant: datetime
    def now(self) -> datetime:
        return self.instant

def create_trial(clock: Clock) -> dict[str, datetime]:
    return {"started_at": clock.now()}

Injecting time removes sleeps, global patches, and flaky assertions.

The test portfolio

• Unit tests: pure rules and small collaborations.
• Integration tests: databases, queues, filesystems, and SDK adapters.
• Contract tests: assumptions between services or internal ports/adapters.
• End-to-end tests: a few critical user journeys.
• Property-based tests: invariants over generated examples.
• Mutation testing: whether tests detect plausible code defects.

Properties beat anecdotes

For a money transfer, useful properties include conservation of total value, rejection of negative amounts, idempotency under a repeated key, and no partial update on failure.

def test_transfer_conserves_money(accounts):
    before = sum(a.balance for a in accounts)
    transfer(accounts[0], accounts[1], 10)
    assert sum(a.balance for a in accounts) == before

Test doubles

Prefer small fakes and stubs over interaction-heavy mocks. Mock at boundaries you own. Do not reproduce a third-party client’s entire behavior in mocks; integration-test the adapter against a sandbox or recorded contract where appropriate.

Exercises

1. Turn a flaky time-based test into a deterministic one.
2. Write properties for a parser, serializer, or state machine.
3. Build an in-memory repository fake that honors the production port.
4. Delete a brittle implementation-coupled test and replace it with a behavior test.

Review checklist

• Does each test state the risk it controls?
• Are failures deterministic and messages diagnostic?
• Are critical boundaries integration-tested?
• Can implementation change without rewriting unrelated tests?

Chapter 10 — Debugging and Observability

Debugging is hypothesis testing under uncertainty. Reproduce first; reduce second; instrument third; change code only after evidence identifies a cause.

Read the whole traceback

The last line names the exception; preceding frames explain the path. Find the first frame in code you control, inspect inputs and invariants, then work outward.

import logging

logger = logging.getLogger(__name__)

def process(job: "Job") -> None:
    try:
        run(job)
    except TemporaryFailure:
        logger.exception("job failed temporarily", extra={"job_id": job.id})
        raise

Never log secrets, tokens, full authorization headers, or raw personal data. Prefer stable identifiers and structured fields.

Debugging toolkit

• breakpoint() and an interactive debugger for state inspection;
• focused logging with correlation IDs;
• minimal reproducible examples;
• git bisect for finding a regression commit;
• faulthandler for crashes and hangs;
• stack dumps for blocked threads;
• tracing for cross-service latency;
• metrics for rates, errors, durations, and saturation.

Observability questions

Logs explain individual events. Metrics show aggregate health. Traces show causal paths through a distributed request. All three should share correlation attributes while respecting privacy.

Exercises

1. Diagnose a deliberately nested exception from only its traceback.
2. Add correlation IDs across a CLI-to-service-to-adapter call path.
3. Use binary search over commits or feature flags to isolate a regression.
4. Design a redaction filter and test it against representative secrets.

Incident checklist

• What changed?
• What is the blast radius?
• Can the failure be reproduced safely?
• Which observation would falsify the leading hypothesis?
• Is mitigation safer than diagnosis in production?
• What evidence must be preserved for the retrospective?

Chapter 11 — Tooling and Automated Quality Gates

A good toolchain makes the correct path cheap. Keep configuration centralized in pyproject.toml where supported.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "advanced-python-labs"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = []

[project.optional-dependencies]
dev = ["pytest", "pytest-cov", "mypy", "ruff"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra --strict-markers"

[tool.mypy]
python_version = "3.12"
strict = true

[tool.ruff]
line-length = 100

A useful local/CI sequence

1. formatting and linting;
2. static type checking;
3. fast unit tests;
4. integration tests;
5. package build and metadata check;
6. security and dependency checks;
7. smoke test the built artifact.

Pin dependencies for applications to make deployment reproducible; libraries usually declare compatible ranges and test a version matrix. Automated updates are useful only when tests can meaningfully evaluate them.

Exercises

1. Create a pyproject.toml for the course repository.
2. Introduce one quality gate at a time and repair the codebase.
3. Build an sdist and wheel, install the wheel into a clean environment, and run a smoke test.
4. Configure coverage around critical modules without treating 100% as the goal.

Review checklist

• Can a new contributor run the same checks as CI with one command?
• Does CI test the built package rather than only the source tree?
• Are tool suppressions narrow, explained, and reviewable?
• Are dependency and interpreter versions intentional?

Chapter 12 — Secure Python Code Review

Security review begins with trust boundaries and assets, not a scanner. Ask: what input is attacker-controlled, what authority does the process hold, and what would happen if a check failed?

Injection: preserve structure, bind data

# Vulnerable
query = f"SELECT id FROM users WHERE email = '{email}'"

# Correct shape: parameterized query
cursor.execute("SELECT id FROM users WHERE email = %s", (email,))

Parameterization applies to data values, not arbitrary identifiers. For table or column selection, use a strict allowlist and library-supported identifier composition.

For processes, avoid a shell when arguments can be passed directly:

import subprocess

subprocess.run(
    ["convert", "--", input_path, output_path],
    check=True,
    timeout=30,
)

An argument list prevents shell metacharacter interpretation, but it does not make the called program safe. Validate file types, paths, sizes, and program-specific flags.

Paths and uploads

from pathlib import Path

def safe_destination(root: Path, user_name: str) -> Path:
    if Path(user_name).name != user_name:
        raise ValueError("nested paths are not allowed")
    candidate = (root / user_name).resolve()
    candidate.relative_to(root.resolve())
    return candidate

root = Path("/srv/uploads")
print("[result] accepted upload destination:", safe_destination(root, "report.csv"))
try:
    safe_destination(root, "../secret.txt")
except ValueError as error:
    print("[error] rejected upload path '../secret.txt':", error)

Also consider symlink races, archive traversal, device files, decompression bombs, name collisions, and storage quotas. A check followed by a privileged use can be vulnerable if the path changes between the two operations.

Dangerous deserialization and dynamic execution

Treat pickle as code execution for untrusted data. Treat eval, exec, dynamic imports, unsafe YAML loaders, and template execution as high-risk. Prefer small data formats with explicit schemas.

Authentication and authorization

Authentication establishes identity; authorization decides whether that identity may perform this action on this object. Review authorization at every entry point and object access, not only UI navigation.

def get_invoice(actor: User, invoice_id: str, repository) -> Invoice:
    invoice = repository.get(invoice_id)
    if invoice is None:
        raise NotFound()
    if invoice.account_id not in actor.account_ids:
        raise NotFound()  # avoid revealing cross-tenant existence
    return invoice

Secrets and cryptography

• never commit secrets;
• source them from an approved secret store;
• rotate after exposure;
• avoid logging them;
• use established password hashing and cryptographic libraries;
• use secrets, not random, for security tokens;
• compare sensitive tokens with constant-time library helpers where applicable;
• give credentials the least privilege and shortest useful lifetime.

Denial of service and resource controls

Bound input size, recursion, decompression, concurrency, request duration, queue depth, retries, and response size. “Valid input” can still be operationally hostile.

Secure review workflow

1. Map entry points, assets, identities, and trust boundaries.
2. Trace untrusted data to interpreters, storage, logs, and outbound requests.
3. Check authentication, object-level authorization, and tenant isolation.
4. Review file/process/network behavior and resource bounds.
5. Inspect secret handling and dependency provenance.
6. Write abuse-case tests.
7. Rank findings by exploitability and impact; state assumptions.

Security review checklist

• Are database operations parameterized?
• Can user data reach a shell, template engine, query language, or evaluator?
• Are outbound URLs restricted against server-side request forgery?
• Are redirects, DNS resolution, private address ranges, and response sizes considered?
• Are uploads stored outside executable/static paths and assigned server-generated names?
• Is every object access authorized for the current principal and tenant?
• Are cookies and tokens configured for transport and lifecycle safety?
• Are errors useful to operators without leaking internals to clients?
• Are rate, time, memory, and concurrency limits present?
• Are dependencies maintained, pinned appropriately, and sourced reliably?

Exercises

1. Review a sample upload endpoint and produce findings with severity, evidence, impact, and remediation.
2. Replace shell and SQL injection flaws and add regression tests.
3. Threat-model a webhook receiver, including replay, spoofing, parsing, and retry abuse.
4. Design tenant-isolation tests for a multi-tenant API.

Part IV — Performance, Concurrency, and Async

Chapter 13 — Performance Engineering by Measurement

Performance is a requirement with a workload and a budget. “Make it faster” becomes actionable only after defining latency percentiles, throughput, memory, cost, dataset size, and concurrency.

The measurement ladder

1. Reproduce the representative workload.
2. Record an end-to-end baseline.
3. Profile to locate expensive regions.
4. Form a hypothesis about the cause.
5. Benchmark the smallest meaningful unit.
6. Change one thing.
7. Re-measure the end-to-end result.
8. Preserve a regression benchmark if the risk warrants it.

Choose the right profiler

• wall-clock sampling for where real time is spent;
• deterministic function profiling for call counts and cumulative time;
• line profiling for a hot function;
• tracemalloc for Python allocation growth;
• database query plans for query cost;
• system metrics for CPU, I/O, paging, and contention.

Benchmark carefully

from timeit import repeat

samples = repeat(
    "sum(x * x for x in range(10_000))",
    repeat=7,
    number=100,
)
print("[result] fastest of 7 benchmark samples in seconds:", min(samples))

Warm-up, data construction, caches, network variance, and garbage collection can dominate small benchmarks. A microbenchmark can show mechanism, but not automatically user impact.

Optimization order

1. remove unnecessary work;
2. improve the algorithm or data structure;
3. reduce I/O round trips and serialization;
4. batch work;
5. cache with a clear invalidation policy;
6. use optimized built-ins or vectorized/native libraries;
7. parallelize only when the workload and runtime permit it.

Exercises

1. Compare list membership with set membership across dataset sizes.
2. Find and repair an N+1 database access pattern in a fake repository.
3. Add a bounded cache and define freshness, eviction, and failure behavior.
4. Write a short performance report containing workload, baseline, profile evidence, change, result, and limitations.

Performance checklist

• Is the workload representative?
• Is the metric tied to a user or system objective?
• Was the bottleneck measured rather than guessed?
• Does the optimization preserve correctness and readability?
• Are memory, tail latency, and cost considered alongside averages?

Chapter 14 — Algorithms, Data Structures, and Data-Oriented Choices

Python-level constant factors matter, but asymptotic behavior dominates at scale.

# Quadratic lookup pattern
def common_slow(left: list[str], right: list[str]) -> list[str]:
    return [item for item in left if item in right]

# Build once, then average constant-time membership
def common_fast(left: list[str], right: list[str]) -> list[str]:
    right_set = set(right)
    return [item for item in left if item in right_set]

left = ["python", "rust", "go", "python"]
right = ["python", "typescript"]
print("[result] common items using list membership:", common_slow(left, right))
print("[result] common items using set membership:", common_fast(left, right))

The second version trades memory and setup cost for faster repeated membership. For tiny inputs the difference may be irrelevant.

Practical structures

• dict: mapping, grouping, deduplication with associated values;
• set: membership and set algebra;
• deque: efficient operations at both ends;
• heapq: repeated access to a smallest item or top-k processing;
• bisect: maintain/search sorted sequences when insertion cost is acceptable;
• generator pipelines: streaming transformations;
• arrays/dataframe/native libraries: dense numeric work.

Batching and locality

Batching reduces fixed overhead in database, network, and serialization calls. But large batches increase memory and tail latency. Choose batch size experimentally and support backpressure.

Exercises

1. Implement top-k items using full sorting and a heap; benchmark different n and k.
2. Replace a list queue with deque and explain the complexity difference.
3. Build a streaming log aggregator that never loads the full file.
4. Compare per-record database writes with bounded batches under simulated latency.

Chapter 15 — Threads, Processes, and the GIL

Concurrency is multiple tasks making progress; parallelism is simultaneous execution. They solve related but distinct problems.

Selection guide

| Workload | Typical first choice | Reason | |---|---|---| | Many blocking network calls | threads or async | overlap waiting | | CPU-heavy pure Python | processes | separate interpreters/cores | | Native code releasing the GIL | threads may help | native work can run in parallel | | Mixed pipeline | bounded staged design | isolate CPU and I/O concerns |

The Global Interpreter Lock in standard CPython builds means one thread executes Python bytecode at a time per interpreter. It does not make compound operations logically atomic, and it does not remove the need for locks around shared invariants.

from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_all(urls: list[str], fetch) -> list[bytes]:
    results: list[bytes] = []
    with ThreadPoolExecutor(max_workers=8) as pool:
        futures = [pool.submit(fetch, url) for url in urls]
        for future in as_completed(futures):
            results.append(future.result())
    return results

Production questions: What is the timeout? How is cancellation propagated? Is concurrency bounded? Does result order matter? What happens after partial failure?

Processes cost serialization

Process pools avoid the GIL for CPU-bound Python but require serializable arguments/results and incur startup and communication overhead. Large shared datasets may erase expected speedups.

Exercises

1. Benchmark sequential, threaded, and process execution for simulated I/O and CPU work.
2. Introduce a race around a multi-step invariant, reproduce it, and protect it.
3. Implement ordered and completion-order result collection.
4. Design graceful shutdown for a bounded worker queue.

Review checklist

• Is the workload CPU-bound, I/O-bound, or mixed?
• Is concurrency bounded by a capacity estimate?
• Are shared invariants synchronized at the right granularity?
• Are cancellation, timeout, retry, and partial failure explicit?

Chapter 16 — Asyncio and Structured Concurrency

Asyncio uses cooperative scheduling. A task runs until it awaits something that is not ready. Blocking the event-loop thread blocks every other task on that loop.

import asyncio

async def fetch_one(client, url: str, semaphore: asyncio.Semaphore) -> bytes:
    async with semaphore:
        async with asyncio.timeout(5):
            return await client.get_bytes(url)

async def fetch_many(client, urls: list[str]) -> list[bytes]:
    semaphore = asyncio.Semaphore(20)
    async with asyncio.TaskGroup() as group:
        tasks = [
            group.create_task(fetch_one(client, url, semaphore))
            for url in urls
        ]
    return [task.result() for task in tasks]

TaskGroup ties child task lifetime to a lexical scope. If one fails, sibling cancellation and error aggregation are handled coherently.

Cancellation is normal control flow

Use try/finally for cleanup. Do not broadly catch and suppress cancellation. Give external operations timeouts, but distinguish an operation timeout from an overall request deadline.

Backpressure with queues

async def producer(queue: asyncio.Queue[str], source) -> None:
    async for item in source:
        await queue.put(item)  # waits when bounded queue is full

async def consumer(queue: asyncio.Queue[str], handle) -> None:
    while True:
        item = await queue.get()
        try:
            await handle(item)
        finally:
            queue.task_done()

A bounded queue turns overload into waiting. The full production design must also define shutdown sentinels, cancellation, retry, poison items, and dead-letter behavior.

Exercises

1. Find a blocking call inside an async service and move it to an appropriate boundary.
2. Add per-request and overall timeouts to a fan-out operation.
3. Build a bounded producer/consumer pipeline with graceful shutdown.
4. Test cancellation cleanup without relying on long sleeps.

Async checklist

• Does every awaited external operation have an appropriate deadline?
• Is fan-out bounded?
• Are tasks owned by a scope rather than leaked in the background?
• Does cleanup run when cancellation occurs?
• Are synchronous libraries kept off the event loop?

Part V — Packaging, Delivery, and Operations

Chapter 17 — Packaging Python Correctly

A distributable project separates import packages, metadata, tests, and build artifacts. Use the src layout to reduce accidental imports from the repository root.

example-project/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/example_project/
│   ├── __init__.py
│   └── cli.py
└── tests/

Versioning and compatibility

Treat public behavior as more than function signatures. Exceptions, import paths, CLI output, configuration keys, serialized data, timing assumptions, and side effects can all be compatibility surfaces.

For libraries:

• keep runtime dependencies minimal;
• use environment markers only when necessary;
• publish wheels and source distributions;
• test minimum and current supported dependency versions where risk justifies it;
• document deprecations before removals.

For applications:

• produce reproducible deployments;
• lock exact transitive versions;
• scan and update continuously;
• separate build and runtime stages;
• attach provenance where your delivery environment supports it.

CLI entry point

[project.scripts]
course-labs = "course_labs.cli:main"

Keep main() thin: parse input, call application behavior, translate known failures to exit codes, and let the core remain reusable.

Exercises

1. Package a small library with a src layout and typed public API.
2. Build both distribution formats and inspect their contents.
3. Define a deprecation path for a renamed function.
4. Install the wheel in a clean environment and test its CLI.

Release checklist

• Does metadata accurately describe Python and dependency compatibility?
• Does the built artifact include required data and exclude secrets/tests unintentionally?
• Is the changelog clear about breaking changes?
• Was the artifact tested after installation?
• Can the release be traced to source and CI execution?

Chapter 18 — Resilient Services and Operational Design

Retries, timeouts, circuit breakers, and idempotency are not generic wrappers. They encode assumptions about failures.

Retry only safe, transient failures

import random
import time
from collections.abc import Callable
from typing import TypeVar

T = TypeVar("T")

def retry(operation: Callable[[], T], attempts: int = 3) -> T:
    for attempt in range(attempts):
        try:
            return operation()
        except TemporaryError:
            if attempt == attempts - 1:
                raise
            delay = min(2 ** attempt, 8) + random.random()
            time.sleep(delay)
    raise AssertionError("unreachable")

In real systems, accept a deadline/clock, expose retry metrics, and honor server guidance. Never automatically retry a non-idempotent action unless an idempotency mechanism makes repetition safe.

Capacity and overload

Every resource should have a bound: connection pools, task pools, queues, request bodies, caches, and in-flight downstream calls. Decide which work waits, which fails fast, and which is shed during overload.

Delivery semantics

“Exactly once” is usually an end-to-end property built from idempotency and deduplication, not a magical transport setting. Record message IDs or business idempotency keys inside the same transaction as the effect when possible.

Exercises

1. Add deadline-aware retry to an HTTP adapter.
2. Design an outbox relay and idempotent consumer.
3. Define service-level indicators and objectives for a sample API.
4. Conduct a failure-mode review for database exhaustion and downstream timeout.

Operations checklist

• Are health checks meaningful and inexpensive?
• Can the service shut down without abandoning accepted work?
• Do retry budgets amplify or absorb an outage?
• Are dashboards organized around user-visible symptoms?
• Do runbooks state safe mitigation steps and owners?

Part VI — Working with AI Models and Agents

Chapter 19 — LLM Applications as Typed, Untrusted Integrations

An LLM is a probabilistic external dependency. Treat its output as untrusted input, its context as limited, and its availability/cost as operational concerns.

Separate model transport from application policy

from dataclasses import dataclass
from typing import Protocol

@dataclass(frozen=True)
class ModelRequest:
    system: str
    user: str

@dataclass(frozen=True)
class ModelResponse:
    text: str
    input_tokens: int
    output_tokens: int

class LanguageModel(Protocol):
    def complete(self, request: ModelRequest) -> ModelResponse: ...

The adapter handles provider-specific authentication, timeouts, retries, and response translation. Application code owns prompts, validation, budgets, and fallback behavior.

Structured output pipeline

1. Define a small output schema.
2. Ask the model for that structure using provider-supported structured output where available.
3. Parse and validate at runtime.
4. Reject unknown actions and invalid ranges.
5. Apply domain authorization independently of model output.
6. Log safe metadata and evaluation signals.

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class TicketDecision:
    category: Literal["billing", "technical", "other"]
    priority: int

def validate_decision(raw: dict[str, object]) -> TicketDecision:
    category = raw.get("category")
    priority = raw.get("priority")
    if category not in {"billing", "technical", "other"}:
        raise ValueError("invalid category")
    if not isinstance(priority, int) or not 1 <= priority <= 5:
        raise ValueError("invalid priority")
    return TicketDecision(category, priority)

Retrieval-augmented generation

A retrieval pipeline has distinct quality stages: document ingestion, chunking, indexing, retrieval, reranking, context assembly, generation, and citation verification. Evaluate retrieval separately from answer generation; a fluent model cannot recover facts it never received.

Evaluation before optimization

Build a versioned evaluation set containing ordinary cases, edge cases, adversarial inputs, and previously observed failures. Measure task success, schema validity, citation support, latency, token usage, cost, and safety violations. Human review remains important for subjective quality.

Exercises

1. Wrap a model client behind a provider-neutral protocol and build a deterministic fake.
2. Create a 30-case evaluation set for a classification task.
3. Add runtime validation and a safe fallback for malformed output.
4. Evaluate retrieval recall independently of generated answer quality.

Review checklist

• Are model inputs and outputs treated as untrusted?
• Is the prompt versioned alongside evaluation results?
• Are token, latency, and monetary budgets explicit?
• Can the system operate safely when the provider is slow, unavailable, or wrong?

Chapter 20 — Tool-Using Agents and Bounded Autonomy

An agent is a control loop in which a model proposes actions, tools affect or inspect an environment, and observations inform subsequent steps. Reliability comes from the surrounding system, not from asking the model to “be careful.”

A minimal agent loop

from dataclasses import dataclass
from typing import Any, Literal

@dataclass(frozen=True)
class Action:
    kind: Literal["tool", "finish"]
    name: str | None = None
    arguments: dict[str, Any] | None = None
    answer: str | None = None

def run_agent(model, tools, task: str, max_steps: int = 8) -> str:
    history: list[dict[str, object]] = [{"role": "user", "content": task}]
    for _ in range(max_steps):
        action: Action = model.next_action(history, tools.schemas())
        if action.kind == "finish":
            return action.answer or ""
        result = tools.call_checked(action.name, action.arguments or {})
        history.append({"role": "tool", "name": action.name, "content": result})
    raise RuntimeError("agent exceeded step budget")

The simple loop is incomplete until it has authentication, authorization, validation, deadlines, cancellation, observability, output limits, and controls for consequential actions.

Tool design principles

• give tools narrow capabilities and precise names;
• use strict schemas with small enums and explicit required fields;
• return compact, structured results;
• separate read-only and mutating operations;
• make repeat calls idempotent where possible;
• keep authorization in trusted code;
• require approval for irreversible or high-impact operations;
• defend against prompt injection in retrieved/tool content.

Prompt injection is a trust-boundary problem

Webpages, documents, emails, and tool results may contain instructions aimed at the model. They are data, not authority. The orchestrator must distinguish system policy, user intent, and untrusted content. A model should not gain permission merely because a document asks it to.

Human oversight

Approval is meaningful only if the person sees the proposed action, exact target, relevant consequences, and enough context to decide. Avoid vague confirmation such as “continue?” for a destructive operation.

Agent evaluations

Test more than final-answer quality:

• correct tool selection;
• argument validity;
• refusal of unauthorized actions;
• resistance to injected instructions;
• recovery from tool errors;
• step and cost efficiency;
• idempotency under retries;
• correct escalation to a human.

Exercises

1. Implement a read-only research agent with a six-step budget and cited evidence.
2. Add a fake malicious document and test that its embedded instructions are ignored.
3. Design approval records for a file-deletion tool without implementing deletion.
4. Create traces for a failing agent run and classify the failure: model, prompt, tool, retrieval, orchestration, or policy.

Agent security checklist

• Can the agent distinguish instructions from untrusted content?
• Does every tool enforce authorization outside the model?
• Are mutating and destructive actions gated appropriately?
• Are targets resolved and displayed before approval?
• Are loops bounded by steps, time, tokens, and money?
• Are secrets kept out of prompts and traces?
• Can actions be audited and, where practical, reversed?

Part VII — Advanced Project Briefs

Capstone 1 — Profile-Guided Data Pipeline

Mission

Build a command-line pipeline that ingests newline-delimited JSON, validates records, enriches them from a local reference dataset, aggregates results, and writes a report. It must process inputs larger than memory.

Required capabilities

• streaming ingestion and bounded memory;
• typed domain records and explicit validation failures;
• pluggable input/output adapters;
• deterministic unit and integration tests;
• structured error reporting with line numbers;
• baseline and optimized performance report;
• distributable wheel with CLI entry point.

Performance experiment

Generate datasets at 10k, 100k, and 1m records. Measure throughput, peak memory, and invalid-record handling. Profile before changing code. Compare at least two aggregation structures and two batch sizes.

Stretch goals

• compressed input;
• parallel parsing with ordered output;
• checkpoint/resume;
• property-based parser tests;
• reproducible benchmark command.

Acceptance checklist

• [ ] The pipeline never requires the entire input in memory.
• [ ] Invalid records do not corrupt aggregate state.
• [ ] Benchmarks describe hardware, Python version, data, and method.
• [ ] Optimizations are supported by profile evidence.
• [ ] The installed wheel works outside the repository.

Capstone 2 — Secure Multi-Tenant Task API

Mission

Build an API where users create, assign, and complete tasks inside organizations. The engineering focus is authorization, tenant isolation, auditability, and maintainable architecture.

Threat model highlights

• cross-tenant object access;
• ID guessing and enumeration;
• mass assignment;
• malicious filenames on attachments;
• injection in search/filter inputs;
• forged or replayed webhooks;
• denial of service through large bodies and expensive queries;
• sensitive fields in logs.

Required capabilities

• inward-pointing domain/application/adapter boundaries;
• object-level authorization tests for every endpoint;
• parameterized storage queries;
• safe attachment policy;
• idempotency keys for create operations;
• security headers and safe error responses;
• dependency, secret, and audit-log strategy;
• structured threat model and security review report.

Acceptance checklist

• [ ] Tenant A can never read or modify Tenant B resources.
• [ ] Authorization is enforced in trusted server-side code.
• [ ] Repeated requests with one idempotency key produce one effect.
• [ ] Logs contain useful identifiers but no credentials or sensitive payloads.
• [ ] Abuse cases are automated regression tests.

Capstone 3 — Resilient Async Crawler

Mission

Build a polite, bounded crawler for a controlled set of domains. Extract titles and links, obey configured limits, and produce a crawl report.

Required capabilities

• structured concurrency;
• per-host and global concurrency limits;
• connect/read/overall deadlines;
• URL canonicalization and deduplication;
• response-size and content-type limits;
• bounded queue and graceful shutdown;
• retry only for eligible failures;
• metrics for queue depth, success, failure, and latency;
• deterministic tests using a local fake server.

Security concerns

Treat URLs as hostile. Restrict schemes, credentials, ports, redirects, and address ranges according to your deployment. Revalidate redirects and consider DNS rebinding. Never use this project to crawl systems without permission.

Acceptance checklist

• [ ] Concurrency cannot grow without bound.
• [ ] Cancellation closes clients and releases workers.
• [ ] A slow or huge response cannot monopolize the crawler.
• [ ] Duplicate canonical URLs are processed once.
• [ ] Tests cover redirect loops, timeout, oversized response, and shutdown.

Capstone 4 — Auditable Support-Triage Agent

Mission

Build an agent that reads synthetic support tickets, retrieves relevant policy passages, proposes a category and response, and optionally creates a draft through a tool. It must never send messages automatically.

Required capabilities

• provider-neutral model adapter and deterministic fake;
• typed structured outputs;
• retrieval with source identifiers;
• evidence-backed response drafts;
• read-only tools separated from draft-creation tools;
• human approval before any external mutation;
• prompt-injection defenses;
• trace redaction and cost/latency budgets;
• versioned evaluation suite of at least 75 cases.

Evaluation dimensions

• correct category and priority;
• relevant policy retrieval;
• claims supported by cited passages;
• no invented customer/account facts;
• correct refusal or escalation;
• resistance to injected ticket text;
• valid tool arguments;
• median and tail latency, tokens, and cost.

Acceptance checklist

• [ ] Malicious ticket content cannot expand tool permissions.
• [ ] The agent cannot send a reply, only create a draft.
• [ ] Every factual policy claim maps to retrieved evidence.
• [ ] Invalid model output fails closed.
• [ ] Evaluation results are reproducible and compared across prompt versions.

Part VIII — Practice System

Weekly study template

Session A: learn and reproduce

Read one chapter, type the examples rather than pasting them, and predict results before execution. Record surprises.

Session B: vary and break

Change inputs, introduce a failure, and observe the resulting traceback, profile, type error, race, or security impact.

Session C: apply

Use the idea in a small existing codebase. Keep the change narrow and document the tradeoff.

Session D: explain

Write 300–500 words answering:

• What problem does this technique solve?
• What mechanism makes it work?
• What is the simplest viable alternative?
• When would I reject this technique?
• What evidence would change my decision?

Code-reading prompts

When studying a mature Python project, ask:

1. Where are side effects initiated?
2. How are dependencies constructed?
3. Which types carry domain meaning?
4. What owns resource lifetime?
5. How do errors cross boundaries?
6. Which operations are bounded?
7. What can be tested without infrastructure?
8. Where does untrusted data become trusted?
9. What compatibility promises exist?
10. How would an operator diagnose failure?

Pull-request review template

Correctness

• What invariant is changed or preserved?
• Are edge cases and failure paths covered?
• Are errors translated at the correct boundary?

Design

• Is responsibility located with the knowledge it needs?
• Are dependencies and side effects visible?
• Is this abstraction paid for by real variation or complexity?

Security

• What new input or authority is introduced?
• Is authorization checked on the target object?
• Can resource use be forced beyond a safe bound?
• Could logs or traces expose sensitive data?

Performance

• Does complexity change with realistic input size?
• Does this add queries, network calls, serialization, or copies?
• Is a benchmark needed to support the claim?

Operability

• Can failure be detected and diagnosed?
• Are timeouts, retries, and idempotency appropriate?
• Can the change be rolled back or disabled safely?

Secure code-review finding template

### [Severity] Short finding title

Location: module/function/line
Asset at risk: ...
Trust boundary: ...

Evidence:
Describe the data/control flow and the unsafe operation.

Impact:
State what an attacker could achieve and under which assumptions.

Recommendation:
Give the smallest robust remediation, including validation or authorization.

Regression test:
Describe an abuse case that should fail safely.

Performance report template

# Performance Investigation: <name>

## Objective and budget
Metric, target, workload, constraints.

## Baseline
Environment, data, repetitions, distribution of results.

## Profile evidence
Where time/memory is spent and how it was measured.

## Hypothesis
The suspected mechanism and predicted change.

## Intervention
The smallest code/design change made.

## Result
Before/after numbers, correctness checks, and tradeoffs.

## Limitations and follow-up
What this experiment does not establish.

Final Competency Checklist

Runtime and language

• [ ] I can explain binding, identity, equality, mutability, and copying.
• [ ] I can reason about scope, closures, frames, imports, and bytecode at a useful level.
• [ ] I can implement and evaluate Python protocols without overusing magic.
• [ ] I can investigate memory retention with reachability and allocation evidence.

Design and architecture

• [ ] I can refactor in behavior-preserving steps with characterization tests.
• [ ] I can separate pure policy from side effects.
• [ ] I can use protocols and dependency injection without a framework.
• [ ] I can design boundaries around change drivers and document decisions.

Reliability and security

• [ ] I can choose unit, integration, contract, property, and end-to-end tests by risk.
• [ ] I can debug from evidence and add privacy-aware observability.
• [ ] I can trace untrusted data to sensitive operations.
• [ ] I can review authorization, injection, deserialization, files, SSRF, secrets, and resource bounds.

Scale and delivery

• [ ] I can define a workload, profile it, and validate an optimization.
• [ ] I can choose appropriate data structures and streaming/batching strategies.
• [ ] I can choose threads, processes, or async based on workload mechanics.
• [ ] I can package, build, install, test, version, and release a Python project.

AI engineering

• [ ] I treat model output and retrieved content as untrusted.
• [ ] I can validate structured output and evaluate model behavior systematically.
• [ ] I can build narrow tools with trusted authorization and meaningful approvals.
• [ ] I can bound an agent by capability, steps, time, tokens, cost, and human oversight.

Where to Go Next

Advanced Python is not a finish line. Continue in three directions:

1. Read implementation code. Study selected standard-library modules, an interpreter implementation, and mature open-source packages. Trace one behavior end to end.
2. Run experiments. Replace assumptions with minimal programs, profiles, failure injection, and benchmarks.
3. Operate what you build. Real engineering judgment grows when software meets changing data, partial failure, security boundaries, upgrades, and users.

The central habit of this curriculum is simple: make important behavior explicit. Explicit types clarify states. Explicit boundaries clarify authority. Explicit measurements clarify performance. Explicit lifetimes clarify concurrency. Explicit evaluation clarifies agent quality. And explicit tradeoffs make Python systems easier to change safely.

Glossary

Adapter: Code translating between an external interface and an internal contract.
Backpressure: A mechanism that slows producers when consumers or resources are saturated.
Characterization test: A test capturing current behavior, often before legacy refactoring.
Closure: A function that refers to names from an enclosing scope.
Contract test: A test checking assumptions at a boundary between components or services.
Descriptor: An object controlling attribute access through protocol methods.
Idempotency: The property that repeating an operation has the same intended effect as performing it once.
Invariant: A condition that must remain true for a model or operation to be valid.
Port: An interface describing how application code communicates across a boundary.
Percentile latency: A latency threshold below which a percentage of observations fall, such as p95.
Prompt injection: Untrusted content attempting to influence a model as though it were authorized instruction.
Protocol: A behavioral contract; in typing, structural requirements an object can satisfy without inheritance.
Structured concurrency: Managing concurrent task lifetimes within explicit scopes.
Trust boundary: A place where data or control crosses between different levels of confidence or authority.
Value object: An object defined by its value and invariants rather than an independent identity.