Calling a Python string "an array of characters" is a useful first approximation and a dangerous final model. What is a character? How many bytes does it occupy? Does slicing share storage? Why can appending one symbol change the size of an entire string?

Python's str answers one part precisely: it is an immutable sequence of Unicode code points. CPython answers the storage question separately. Since PEP 393, it chooses among several internal element widths according to the largest code point in each string. The same public type can therefore use materially different storage strategies.

That separation matters in real systems. It explains surprising memory cliffs, prevents byte offsets from leaking into text logic, and clarifies where network protocols, files, databases, and cryptographic APIs require an encoding decision.

Version note. Examples target Python 3.10 through 3.14 and were verified on 64-bit CPython 3.14.7. Unicode semantics and immutability are Python guarantees. Object headers, compact layouts, interning choices, allocation shortcuts, and exact byte counts are CPython 3.14 details.

Code points are not encoded bytes

Begin with three strings that look equally short in Python:

Pyodide / WebAssembly
samples = ["A", "\u00e9", "\U0001f40d"]

for text in samples:
    print("[result] text, code points, UTF-8 bytes, hex:", ascii(text), len(text), len(text.encode("utf-8")), text.encode("utf-8").hex())

'A' 1 1 41
'\xe9' 1 2 c3a9
'\U0001f40d' 1 4 f09f908d

Each str contains one code point, so each has length one. UTF-8 represents those values with one, two, and four bytes. len(text) never reports the eventual UTF-8 payload size; len(text.encode("utf-8")) does.

Python indexing follows the same rule:

Pyodide / WebAssembly
text = "A\u00e9\U0001f40d"

print("[result] code-point length:", len(text))
print("[result] code points:", [f"U+{ord(char):04X}" for char in text])
print("[check] third code point is snake:", text[2] == "\U0001f40d")

3
['U+0041', 'U+00E9', 'U+1F40D']
True

This behavior is independent of how CPython stores the string and independent of how you later encode it. It also means that a byte offset from an external UTF-8 protocol is not a valid str index. Convert offsets deliberately or keep the operation in bytes.

Python guarantee. A string is a sequence of Unicode code points, with indexing, slicing, and length defined over those code points. Python does not expose UTF-8 code units as string elements.

Code points are not user-perceived characters

Unicode can represent what a user sees as one character with multiple code points. A letter followed by a combining mark is the standard small example:

Pyodide / WebAssembly
import unicodedata


composed = "\u00e9"
decomposed = "e\u0301"

print("[state] composed and decomposed lengths:", len(composed), len(decomposed))
print("[check] equal before normalization:", composed == decomposed)
print("[check] equal after NFC normalization:", unicodedata.normalize("NFC", decomposed) == composed)

1 2
False
True

The two strings may render identically, but they are different code-point sequences and compare unequal until normalized. More elaborate emoji can contain several code points joined into one grapheme cluster. Python's standard len, indexing, and slicing do not segment grapheme clusters.

Choose the unit that belongs to the requirement:

  • code points for Python's ordinary text operations;
  • grapheme clusters for cursor movement or user-visible character limits, using a Unicode-aware segmentation library;
  • encoded bytes for protocol limits, storage quotas, and wire offsets;
  • display columns for terminal layout, which is another distinct problem.

Normalization is not a universal cleanup step. Apply a named normalization form at a documented boundary only when the domain wants canonically equivalent sequences treated alike. Passwords, signed payloads, source code, and identifiers governed by another specification may need different policies.

PEP 393: width follows the largest code point

Older CPython builds were configured around either two-byte or four-byte Unicode elements. That made memory usage platform-dependent and could expose surrogate pairs on narrow builds. PEP 393 replaced this with a flexible representation.

For its canonical representation, a modern CPython string selects the smallest element width capable of holding every code point:

maximum code point       CPython storage kind
U+0000 .. U+00FF         1 byte per element
U+0100 .. U+FFFF         2 bytes per element
U+10000 .. U+10FFFF      4 bytes per element

ASCII-only strings are a particularly compact subset of the one-byte form. The object records properties such as length and storage kind, and compact strings keep character data in the same allocation as the object header. Legacy and C-API states add nuance, but ordinary Python-created strings normally use compact ready representations.

We can observe the consequence without reading private memory:

import sys


samples = {
    "ascii": "a" * 1000,
    "latin1": "\u00e9" * 1000,
    "bmp": "\u0100" * 1000,
    "astral": "\U0001f40d" * 1000,
}

for name, text in samples.items():
    print(name, len(text), sys.getsizeof(text))

On our 64-bit CPython 3.14.7 build:

ascii 1000 1041
latin1 1000 1057
bmp 1000 2058
astral 1000 4060

The fixed overhead differs for some forms, and the payload trend is the important evidence: roughly one, one, two, or four bytes per code point. sys.getsizeof() is shallow and build-specific. It does not report a portable string formula.

The chosen width belongs to the whole string, not each character. One high code point can widen all elements:

import sys


ascii_text = "a" * 1000
wide_text = ascii_text + "\U0001f40d"

print(sys.getsizeof(ascii_text))
print(sys.getsizeof(wide_text))
print(len(wide_text), len(wide_text.encode("utf-8")))

1041
4064
1001 1004

The UTF-8 payload grows by four bytes, but CPython's canonical in-memory representation grows by about three kilobytes because all 1001 code points now need the four-byte kind. This is an implementation tradeoff: constant-time indexing by code point and simple contiguous storage, rather than storing every string internally as variable-width UTF-8.

Do not turn this into a reason to strip non-ASCII text. It is a reason to measure representative multilingual data and to avoid retaining giant transformed strings unnecessarily.

Immutability means transformations allocate values

No string operation changes an existing string's sequence of code points. Methods return strings, and slicing produces a string value:

text = "configuration"

print(text.upper(), text)
print(text[:] is text)
print(text[0:5] is text)
print(text[0:5])

On CPython 3.14 this prints:

CONFIGURATION configuration
True
False
confi

Immutability is guaranteed; these identities are not. CPython can return the original object for a full slice because the value cannot be changed. A proper substring ordinarily gets its own string allocation. Python does not promise slice identity or storage sharing.

Independent substring storage avoids a classic retention trap: a ten-character slice does not keep a one-gigabyte source buffer alive merely by referring into it. The cost is copying the selected code points. Repeatedly slicing progressively smaller strings can therefore copy much more data than the final result suggests. Prefer indexes or one-pass parsing when working through a large buffer.

Some methods may also return the original object when no change is required:

text = "already clean"

print(text.removeprefix("missing") is text)
print(text.replace("x", "y") is text)
print(text.upper() is text)

The observed CPython 3.14 output is True, True, and False. Treat those as allocation optimizations, not API contracts. Write code around values, not id() or is, except for documented singletons such as None.

Concatenation and the quadratic trap

Because strings are immutable, left + right conceptually creates the combined value. Repeating that operation can copy the growing prefix again and again:

from timeit import timeit


def with_plus(parts):
    result = ""
    for part in parts:
        result = result + part
    return result


def with_join(parts):
    return "".join(parts)


parts = ["abcdefghij"] * 20_000
print(with_plus(parts) == with_join(parts))
print(round(timeit(lambda: with_plus(parts), number=20), 3))
print(round(timeit(lambda: with_join(parts), number=20), 3))

Our run printed equal results and approximately 0.011 versus 0.001 seconds. Timing varies, and CPython can optimize some local-variable concatenation by resizing when the left string has no other references. That shortcut makes simple loops less disastrous than the abstract model predicts, but it is not a Python guarantee and does not apply reliably when aliases or more complex expressions intervene.

str.join() is the deliberate bulk-construction operation. CPython can inspect the parts, calculate the final length and maximum storage kind, allocate once, and copy into the result. Prefer it when assembling an iterable of many pieces. For a handful of known pieces, + and f-strings remain clear and appropriate. Do not replace readable f"{name}: {value}" with ritual joins.

For streaming output, avoid constructing the giant string at all. Write chunks to a text stream, yield them from a generator, or use io.StringIO when an API ultimately requires one string.

Interning is an optimization, not value semantics

CPython interns selected strings so equal values can share one object. Identifier-like constants are common candidates, and sys.intern() lets an application request an interned canonical object:

import sys


left = "customer_identifier"
right = "".join(["customer", "_identifier"])

print(left == right, left is right)
right = sys.intern(right)
left = sys.intern(left)
print(left is right)

On the tested build this prints:

True False
True

The first identity can change with compilation context, implementation, or optimizer decisions. The equality cannot. Always compare strings with ==, never is.

Interning can reduce duplicate storage and accelerate equality checks in workloads with a bounded vocabulary that is compared repeatedly, such as parsers or symbol tables. It is not a general-purpose deduplication switch. Interning unbounded user input can lengthen object lifetimes and turn a memory optimization into a leak-shaped cache. Measure the whole workload before adding explicit interning.

Encoding is a boundary, not string storage

str and bytes answer different questions. A string models Unicode text. Bytes model integer octets. Encoding maps text to bytes under a named codec; decoding maps bytes to text:

Pyodide / WebAssembly
text = "caf\u00e9"
payload = text.encode("utf-8")

print("[state] text and payload types:", type(text).__name__, type(payload).__name__)
print("[check] UTF-8 payload and round trip:", payload, payload.decode("utf-8") == text)

try:
    payload.decode("ascii")
except UnicodeDecodeError as error:
    print("[error] ASCII decode type and byte offset:", type(error).__name__, error.start)

str bytes
b'caf\xc3\xa9' True
UnicodeDecodeError 3

The codec is part of the data contract. Saying "this is text" is insufficient at a byte boundary; the producer and consumer must agree on UTF-8, UTF-16, Latin-1, or another specified encoding and error policy. UTF-8 is usually the interoperable default, not an inference Python can safely make from arbitrary bytes.

Keep boundaries visible:

  • decode bytes once when entering text-oriented application logic;
  • perform parsing, normalization, validation, and formatting as str;
  • encode once when writing to a byte-oriented protocol;
  • keep opaque compressed, encrypted, hashed, or binary payloads as bytes;
  • pass encoding= explicitly when opening text files unless the format intentionally uses the locale default.

Error handlers are policy, not plumbing. errors="ignore" silently discards information and can merge distinct inputs. errors="replace" is useful for display but rarely for identifiers or round trips. surrogateescape supports lossless handling of undecodable filesystem-style bytes in specific interfaces; it is not ordinary Unicode text to send blindly elsewhere.

Measuring the representation you actually own

sys.getsizeof(text) counts the shallow CPython string allocation. len(text.encode("utf-8")) counts a newly created UTF-8 payload but not its object header. Neither is "the memory used by text" in every architecture.

A service may simultaneously retain an input bytes, decoded str, normalized str, parsed fields, and encoded output. Peak memory can be several representations at once. Use tracemalloc around the complete pipeline when that peak matters, and decide which layer owns each copy.

Representation also changes performance. CPython's one-, two-, and four-byte forms make indexing constant-time, while UTF-8 is compact for mostly ASCII text but needs variable-length decoding to find a code point. Databases and runtimes may make different tradeoffs. Never estimate process memory by multiplying character count by UTF-8 file size.

Practical decisions

Use str when the value is text and operations have text semantics. Use bytes when byte identity, byte offsets, or a binary protocol matters. Then make these decisions explicit:

  • Define whether limits count code points, grapheme clusters, encoded bytes, or display width.
  • Normalize only where the domain specification requires it, and state the form.
  • Use join() for many retained pieces; use streaming when no complete string is needed.
  • Avoid chains of large slices when indexes can describe the same region.
  • Assume transformations allocate unless documentation guarantees otherwise.
  • Treat interning and identity reuse as optional CPython optimizations.
  • Measure multilingual production-shaped samples, not only ASCII fixtures.
  • Include encoding and error handling in every external text contract.

Most string bugs are not failures of Unicode trivia. They are unit errors: applying a byte limit to code points, applying code-point slicing to graphemes, or assuming an in-memory representation is the wire format.

Exercises: test the model

  1. Measure sys.getsizeof() for 10,000-character strings whose maximum code point falls in each PEP 393 storage kind. Explain which conclusions are CPython-specific.
  2. Build composed and decomposed versions of several accented words. Compare equality before and after NFC and NFD normalization.
  3. Given a UTF-8 limit of 20 bytes, write validation that never truncates in the middle of an encoded sequence. Decide how it should report an oversized grapheme cluster.
  4. Compare repeated +, join(), and io.StringIO for representative fragment sizes. Verify results before interpreting timings.
  5. Parse a large string using shrinking slices, then rewrite the parser using source indexes. Compare peak allocations with tracemalloc.
  6. Find a protocol boundary in an application and document its codec, error policy, normalization policy, and unit for length limits.

Keep this model

A Python string is an immutable sequence of Unicode code points. It is not UTF-8, not a sequence of grapheme clusters, and not a promise that every element consumes the same number of bytes across implementations.

CPython 3.14 uses PEP 393's flexible representation: each string chooses a one-, two-, or four-byte canonical element width from its largest code point, with an extra-compact ASCII form. That design explains memory cliffs and fast indexing, but it remains an implementation strategy.

When string behavior surprises you, ask:

  1. Is this operation defined in code points, grapheme clusters, bytes, or display columns?
  2. Which encoded representation exists at the system boundary?
  3. Is the observed allocation or identity behavior guaranteed by Python or optimized by CPython?

Correct units and explicit boundaries matter more than memorizing header sizes.

Primary sources