13. Input, Output, and Serialization

Every chapter so far kept its data inside one running process, where objects are reachable by reference and vanish when the JVM exits. Real programs read configuration from disk, stream gigabytes they cannot hold in memory, and turn live object graphs into bytes that survive a restart or cross a network. This chapter covers the stream abstractions that move bytes and characters, the NIO.2 file API that replaced the flawed java.io.File, and the built-in serialisation mechanism together with the substantial reasons to distrust it.

<a id="131-streams-and-file-access"></a>

13.1 Streams and File Access

This subchapter covers the four stream families that every Java I/O operation is built from, and the modern API for naming, inspecting, and manipulating the files themselves.

Streams, Readers, and Writers

A program's own heap is fast, structured, and gone the moment the process ends. Everything else — a file, the keyboard, a socket, another process, a compression codec — lives outside it. Those sources differ in every physical respect, yet they share one property: data arrives or departs as an ordered sequence, a piece at a time, in one direction. Java names that shared property a stream, and builds all of its I/O on it.

An array and a stream are opposites in a useful way. An array is a warehouse shelf: everything is present at once and you can reach for item 500 directly. A stream is a conveyor belt: items pass you one at a time, you cannot skip ahead cheaply, and once an item has gone by it is gone. That restriction is what allows the same four interfaces to describe a 2 KB file and an endless network socket.

One naming collision must be cleared up immediately, because it confuses everyone.

Name	Package	What it is
InputStream / OutputStream	java.io	A byte pipe to an external resource
Stream<T>	java.util.stream	A pipeline of in-memory values (Chapter 8)

They are unrelated types with unrelated purposes. The rest of this topic means the java.io kind.

Why there are two families, not one. A byte stream moves raw 8-bit values, which is exactly right for a JPEG, a ZIP archive, or a protocol frame, where every byte is significant and no interpretation is wanted. Text is different: a character is not a byte. Turning 'è' into bytes requires a charset, and different charsets produce different byte counts for the same character.

   FILE ON DISK  (UTF-8 bytes)          IN MEMORY  (Java chars, UTF-16)
   48 65 6C 6C 6F          decode
   C3 A8                   ------->     'H' 'e' 'l' 'l' 'o' 'è'
   21                      charset
                                         6 chars from 8 bytes
   ^^^^^                                 (C3 A8 is ONE character)

Reading that file with a byte stream and casting each byte to char produces mojibake, and splitting the byte array in the middle of a multi-byte sequence corrupts a character. Reader and Writer exist so that the decoding happens in one correct place.

That gives four abstract root classes, one per combination of direction and unit.

	Bytes (binary data)	Characters (text)
Input	InputStream	Reader
Output	OutputStream	Writer

Their core contracts are deliberately tiny, because every concrete subclass must implement them over a different physical medium.

Method	Declared on	Returns
int read()	InputStream	Next byte as 0..255, or -1 at end
int read(byte[] b, int off, int len)	InputStream	Count actually read, or -1 at end
int read(char[] c)	Reader	Count of characters, or -1 at end
void write(int b)	OutputStream	Writes the low 8 bits
void flush()	Output side	Pushes buffered data onward
void close()	All four	Flushes, then releases the OS handle

Two details in that table cause real bugs. read() returns an int, not a byte, precisely so that -1 can mean end-of-stream without colliding with the legitimate byte value 0xFF. And the bulk read returns how many elements it filled, which is often fewer than the array length even when the stream is not finished, because it returns as soon as some data is available.

The decorator pattern is the organising idea. Rather than offering a BufferedEncryptedFileReader, Java gives you thin wrappers that each add one capability and delegate the rest, exactly the structural pattern from Chapter 9.4. You compose the stack you need.

   new BufferedReader( new InputStreamReader( new FileInputStream(f), UTF_8 ) )
        |                      |                       |
        |                      |                       +-- SOURCE: raw bytes off disk
        |                      +-- BRIDGE: bytes -> chars using a charset
        +-- BUFFER: batches syscalls, and adds readLine()

   Each layer IS-A Reader, so callers never care how deep the stack goes.

InputStreamReader and OutputStreamWriter are the only bridges between the two families, and they are where a charset belongs.

Why buffering is not optional. An unbuffered read() can mean one system call per byte. Copying a one-megabyte file that way issues over a million transitions into the kernel. An 8 KB buffer turns the same work into 128 of them.

   Unbuffered: app -> kernel -> disk, once per byte      1,048,576 crossings
   Buffered:   app -> buffer (in-process) ... refill     128 crossings
               |
               +-- roughly three orders of magnitude fewer boundary crossings

Here is the canonical binary copy. Note the handling of the returned count.

try (InputStream in = new BufferedInputStream(new FileInputStream("source.bin"));
     OutputStream out = new BufferedOutputStream(new FileOutputStream("dest.bin"))) {

    byte[] buffer = new byte[8192];          // one page-sized chunk
    int n;
    while ((n = in.read(buffer)) != -1) {    // -1 means end; 0 is possible and legal
        out.write(buffer, 0, n);             // write exactly n bytes, never buffer.length
    }
}                                            // close() flushes, then releases handles

Writing out.write(buffer) instead of out.write(buffer, 0, n) is a classic corruption bug: the final partial chunk pads the output with stale bytes from the previous iteration.

Text input follows the same shape, with the charset stated explicitly.

try (BufferedReader reader = new BufferedReader(
         new InputStreamReader(new FileInputStream("data.csv"),
                               StandardCharsets.UTF_8))) {

    String line;
    while ((line = reader.readLine()) != null) {   // null ends; "" is a blank line
        process(line);
    }
}

readLine() strips the terminator and accepts \n, \r, or \r\n interchangeably, so it silently normalises line endings and cannot tell you which convention the file used.

On the output side, PrintWriter is the convenient choice and the one with a trap.

try (PrintWriter out = new PrintWriter(
         new BufferedWriter(new OutputStreamWriter(
             new FileOutputStream("report.txt"), StandardCharsets.UTF_8)))) {

    out.printf("total=%d%n", 42);     // %n is the platform line separator
    if (out.checkError()) {           // PrintWriter SWALLOWS IOException
        throw new IOException("write failed");
    }
}

PrintWriter and PrintStream never throw IOException from their write methods; they set an internal error flag instead. System.out behaves the same way, which is why a failed console write is silent.

Charset defaults have changed, so state the charset anyway. FileReader and FileWriter had no charset parameter at all until Java 11, and before Java 18 the default charset came from the operating system locale, so the same program produced different bytes on a developer laptop and a server. Java 18 fixed the default to UTF-8. Code that must run correctly on older releases, or that must read a file in a stated encoding, passes StandardCharsets.UTF_8 explicitly every time.

Unflushed data is lost data. Buffered output sits in memory until the buffer fills, flush() is called, or close() runs. A program that exits without closing leaves the tail of its output unwritten. Try-with-resources, covered in Chapter 5.3, is the mechanism that makes this reliable; nothing in this topic should be written without it.

The standard streams are ordinary members of these families.

Field	Type	Notes
System.in	InputStream	Raw bytes; wrap in a Reader for text
System.out	PrintStream	Buffered, flushed on newline by default
System.err	PrintStream	Unbuffered, so errors are not lost
System.console()	Console	null when not attached to a terminal; offers readPassword()

Beyond the basics, the decorator catalogue covers most needs.

Decorator	Adds
BufferedInputStream / BufferedReader	Batched syscalls, readLine()
DataInputStream / DataOutputStream	Portable big-endian primitives and UTF-8 strings
ObjectInputStream / ObjectOutputStream	Whole object graphs (see 13.2)
GZIPInputStream / GZIPOutputStream	Transparent compression
ByteArrayInputStream / StringReader	An in-memory source, ideal for tests
PushbackInputStream	unread(), for hand-written parsers
SequenceInputStream	Concatenates several streams into one

Finally, modern JDKs offer shortcuts that remove the loop entirely when the data is small enough to hold in memory.

String text  = Files.readString(Path.of("small.txt"));     // whole file, UTF-8
byte[] bytes = in.readAllBytes();                          // Java 9+
in.transferTo(out);                                        // Java 9+, the whole copy

Use them for configuration and fixtures. Use the explicit loop when the input might be larger than the heap.

Key Takeaways

• A java.io stream is a one-directional, sequential pipe to something outside the heap, and is unrelated to java.util.stream.Stream.
• Byte streams (InputStream, OutputStream) carry raw data; character streams (Reader, Writer) carry decoded text.
• Text is not bytes: converting between them requires a charset, and InputStreamReader and OutputStreamWriter are the bridges.
• read() returns an int so that -1 can signal end-of-stream unambiguously.
• Bulk read returns the count actually filled, so always write back exactly that many elements.
• Streams are composed with the decorator pattern, one capability per wrapper.
• Buffering replaces one syscall per byte with one per block, and is a thousandfold difference in practice.
• PrintWriter and PrintStream never throw on write; they set an error flag you must check.
• Always pass the charset explicitly, because the default was platform-dependent before Java 18.
• Buffered output is lost unless the stream is flushed or closed, so use try-with-resources.

🧪 Practice

1. Write a file copier that reads with FileInputStream one byte at a time, then a second that uses an 8 KB buffer, and time both on a 50 MB file.
2. Write the string "café" to a file as UTF-8, read it back as ISO-8859-1, and print what you get. Explain the result byte by byte.
3. Build a stack that writes GZIP-compressed UTF-8 text, then read it back with the mirror-image stack, using try-with-resources on both sides.
4. Take a program that prints to System.out inside a loop and make it write to a BufferedWriter instead, then deliberately omit the close and observe what reaches the file.
5. Interview: Why does Java provide both InputStream and Reader rather than one hierarchy with a charset parameter? (Hint: what is the correct charset for a JPEG, and what happens if a buffer boundary falls inside a multi-byte sequence?)

File Handling with the NIO.2 API

Streams move the contents of a file. Something else must name files, create and delete them, list a directory, and ask how large or how old a file is. Java's original answer was java.io.File, and it was weak enough that Java 7 replaced it wholesale with the java.nio.file package, universally called NIO.2.

The complaints against java.io.File are worth knowing, because they explain every design decision in its replacement.

Problem with java.io.File	Consequence
delete() returns boolean	You learn it failed, never why: missing, locked, or denied
listFiles() returns null on error	A NullPointerException far from the cause
No copy method at all	Everyone hand-rolled a stream loop
renameTo() is platform-dependent	Works in testing, fails across filesystems
No symbolic-link awareness	Traversals follow links into infinite loops
One syscall per attribute query	Slow, and open to time-of-check races
No access to POSIX permissions or owners	Fell back to shelling out

NIO.2 fixes these by splitting one overloaded class into two collaborators with a clean division of labour.

• Path is a name. It is pure syntax, like an address written on an envelope: it can be manipulated, joined, and compared without any file existing anywhere.
• Files is a toolbox of static methods that act on the filesystem, throwing a specific IOException subtype when something goes wrong.

   Path  ------------- names ------------->  a location
    |                                            |
    | (pure text manipulation, no disk access)   |
    v                                            v
   Files ----------- operates on -----------> the actual file
         (readString, copy, delete, walk, ... each throws on failure)

A third piece, FileSystems, makes the whole API pluggable: the same Path and Files code can address entries inside a ZIP archive, because a ZIP is exposed as a filesystem provider.

Working with Path. Every method here is string surgery; none of it touches the disk.

Path base = Path.of("/srv/app");
Path cfg  = base.resolve("conf/app.properties");   // /srv/app/conf/app.properties
Path rel  = base.relativize(cfg);                  // conf/app.properties
Path sib  = cfg.resolveSibling("app.bak");         // /srv/app/conf/app.bak

Path messy = Path.of("/srv/app/../app/./logs");
Path clean = messy.normalize();                    // /srv/app/logs  (text only)

cfg.getFileName();      // app.properties
cfg.getParent();        // /srv/app/conf
cfg.getNameCount();     // 3
cfg.isAbsolute();       // true

Two subtleties matter. resolve on an absolute argument discards the base entirely, so base.resolve("/etc/passwd") is /etc/passwd. And normalize is purely textual: if app is a symbolic link, removing ../ can change which file you mean. Files.toRealPath resolves links properly, but it touches the disk and fails if the file does not exist.

Path.equals is likewise syntactic. Two different paths can name one file through a link or a case-insensitive filesystem, and only Files.isSameFile answers that question honestly.

That syntactic nature gives the standard defence against path traversal, where user input like ../../etc/passwd escapes an intended directory.

Path root = Path.of("/var/www").toAbsolutePath().normalize();
Path target = root.resolve(userSuppliedName).normalize();   // normalize AFTER resolve
if (!target.startsWith(root)) {                             // syntactic containment check
    throw new SecurityException("path escapes the served root");
}

Working with Files. The static methods group into five jobs.

Job	Methods
Read	readString, readAllLines, readAllBytes, lines, newBufferedReader, newInputStream
Write	writeString, write, newBufferedWriter, newOutputStream
Create	createFile, createDirectory, createDirectories, createTempFile, createSymbolicLink
Manipulate	copy, move, delete, deleteIfExists
Inspect	exists, size, isDirectory, isReadable, readAttributes, probeContentType
Traverse	list, walk, find, newDirectoryStream, walkFileTree

Whole-file reads and writes collapse a stream loop into one line, at the cost of holding the content in memory.

String text = Files.readString(cfg, StandardCharsets.UTF_8);
List<String> lines = Files.readAllLines(cfg, StandardCharsets.UTF_8);

Files.writeString(out, text, StandardCharsets.UTF_8,
                  StandardOpenOption.CREATE,             // make it if absent
                  StandardOpenOption.TRUNCATE_EXISTING); // otherwise empty it first

// For a file larger than the heap, stream it lazily instead:
try (Stream<String> lazy = Files.lines(cfg, StandardCharsets.UTF_8)) {   // IS a resource
    long errors = lazy.filter(l -> l.contains("ERROR")).count();
}

The try-with-resources on Files.lines is mandatory, not stylistic: unlike most streams, the ones returned by lines, list, walk, and find hold an open file handle, as Chapter 5.3 explained.

Copy, move, and delete take option flags that state intent precisely.

Option	Effect
REPLACE_EXISTING	Overwrite the target instead of throwing FileAlreadyExistsException
COPY_ATTRIBUTES	Carry timestamps and permissions across
ATOMIC_MOVE	The move either fully happens or does not; same filesystem only
NOFOLLOW_LINKS	Act on the symbolic link itself, not its target

ATOMIC_MOVE enables the single most useful pattern in file handling: never let a reader see a half-written file.

Path target = Path.of("/srv/app/conf/app.properties");
Path tmp = Files.createTempFile(target.getParent(), "app", ".tmp");  // same filesystem

Files.writeString(tmp, newContent, StandardCharsets.UTF_8);
Files.move(tmp, target,                                   // the swap is instantaneous
           StandardCopyOption.REPLACE_EXISTING,
           StandardCopyOption.ATOMIC_MOVE);

Traversing directories. Four methods differ in depth and in what they hand back.

   /srv/app
   |-- conf
   |   |-- app.properties
   |   `-- log4j2.xml
   |-- logs
   |   `-- app.log
   `-- lib
       `-- core.jar

   Files.newDirectoryStream(dir, "*.jar") -> (nothing here; one level, glob filter)
   Files.list(dir)                        -> conf, logs, lib          (one level)
   Files.walk(dir)                        -> every path, depth-first  (recursive)
   Files.find(dir, 5, (p, a) -> a.size() > 1024)  -> walk plus an attribute test
   Files.walkFileTree(dir, visitor)       -> callbacks, including error handling

// Every .java file under src, sorted, with the handle released afterwards
try (Stream<Path> paths = Files.walk(Path.of("src"))) {
    paths.filter(Files::isRegularFile)
         .filter(p -> p.toString().endsWith(".java"))
         .sorted()
         .forEach(System.out::println);
}

walk does not follow symbolic links unless you pass FOLLOW_LINKS, which is the sane default: following them invites cycles. Deleting a tree needs walkFileTree or a reverse-sorted walk, because a directory cannot be removed until it is empty.

Reading metadata. Asking one question at a time costs one syscall each and opens a window in which the file changes between questions.

// Three separate trips to the filesystem, and three chances to race
long size = Files.size(p);
FileTime when = Files.getLastModifiedTime(p);
boolean dir = Files.isDirectory(p);

// One trip, one consistent snapshot
BasicFileAttributes a = Files.readAttributes(p, BasicFileAttributes.class);
a.size(); a.lastModifiedTime(); a.isDirectory(); a.creationTime();

Files.exists deserves suspicion. It is three-valued in disguise: exists and notExists can both return false when the answer is unknown, for instance when a parent directory denies traversal. Worse, anything you do after checking is a race. Prefer attempting the operation and catching NoSuchFileException or FileAlreadyExistsException.

Finally, the mapping from old code to new.

java.io.File	NIO.2 equivalent
new File(s)	Path.of(s)
f.exists()	Files.exists(p), or just attempt the operation
f.delete()	Files.delete(p) (throws) or deleteIfExists
f.renameTo(g)	Files.move(p, q, REPLACE_EXISTING)
f.listFiles()	Files.list(p) or newDirectoryStream(p)
f.length()	Files.size(p)
f.mkdirs()	Files.createDirectories(p)
(no equivalent)	Files.copy, ATOMIC_MOVE, walk, WatchService, POSIX permissions

Interoperation is easy in both directions with file.toPath() and path.toFile(), so legacy APIs are never a reason to avoid NIO.2.

For reacting to changes rather than polling, WatchService registers a directory with the operating system's notification mechanism.

try (WatchService watcher = FileSystems.getDefault().newWatchService()) {
    Path.of("/srv/app/conf").register(watcher,
            StandardWatchEventKinds.ENTRY_CREATE,
            StandardWatchEventKinds.ENTRY_MODIFY);

    WatchKey key = watcher.take();            // blocks until something happens
    for (WatchEvent<?> event : key.pollEvents()) {
        System.out.println(event.kind() + " on " + event.context());
    }
    key.reset();                              // REQUIRED, or no further events arrive
}

Key Takeaways

• NIO.2 splits naming (Path) from acting (Files), which java.io.File confused.
• Path operations are pure text and never touch the disk, so a Path can name a file that does not exist.
• Files methods throw specific IOException subtypes instead of returning an unexplained boolean.
• resolve against an absolute path discards the base, and normalize is textual, so symbolic links can defeat it.
• Guard against path traversal by resolving, normalising, then checking startsWith the intended root.
• Streams from list, walk, find, and lines hold file handles and need try-with-resources.
• Write-to-temp then ATOMIC_MOVE is the standard way to publish a file without exposing a partial state.
• readAttributes gets a consistent metadata snapshot in one syscall.
• Files.exists can be false when the truth is unknown, and any check is a race; prefer attempting the operation.
• toPath() and toFile() bridge to legacy APIs, so there is no reason to keep writing java.io.File.

🧪 Practice

1. Print the absolute, normalised, and real paths of ./../<your project> and explain any differences between the last two.
2. Write a method that counts the lines of every .java file under a directory, using Files.walk and Files.lines, with both handles closed.
3. Implement a safe file server helper that maps a user-supplied name to a path under /var/www and rejects any input that escapes it. Test it with ../../etc/passwd and with an absolute path.
4. Implement saveConfig so that a reader never observes a partial file, and explain why the temporary file must live in the target's own directory.
5. Write a recursive delete using walkFileTree, then explain why the naive Files.walk(...).forEach(Files::delete) version fails.
6. Interview: Why did java.nio.file replace java.io.File rather than extending it? (Hint: what does delete() returning false tell the caller, and what would a subclass have to do about that?)

<a id="132-object-serialization"></a>

13.2 Object Serialization

This subchapter restates the Serializable marker from Chapter 4.4 and then goes beneath it, covering how the object graph becomes bytes, how to control and version that format, and why most projects should choose something else entirely.

The Serializable Marker and a File Round Trip

Chapter 4.4 introduced Serializable alongside the other core Java interfaces. This topic restates the essentials so the rest of the subchapter stands on its own, and builds the round-trip helper that the later examples use.

The problem is easy to state. An object's fields hold references, and a reference is an address that means something only inside one running JVM. To store an object on disk or send it to another machine, that structure has to become a flat sequence of bytes and then be rebuilt at the other end. Java automates the whole job for any class that declares itself willing.

Declaring willingness is all Serializable does:

public interface Serializable { }    // no methods, no constants, nothing

This is a marker interface, the idiom covered in Chapter 4.4 alongside Cloneable and RandomAccess. You are not supplying an implementation; you are granting permission. The JVM supplies the reading and writing machinery itself, reflectively, by walking your fields.

public class Session implements Serializable {
    private static final long serialVersionUID = 1L;   // version stamp, see below

    private String user;
    private int requestCount;
    private transient String authToken;         // deliberately NOT written
    private static String appName = "demo";     // static: belongs to the class
}

Four rules account for almost everything the mechanism does.

Element	Written to the stream?
Instance field	Yes, if its type is also serialisable
transient field	No; restored as null, 0, or false
static field	No; it belongs to the class, not to any instance
Referenced object	Yes, recursively, the entire reachable graph

The two streams that drive it are decorators in exactly the sense of 13.1: they add object semantics on top of any byte stream, so the destination can be a file, a socket, or an array in memory.

Path file = Path.of("session.bin");
Session session = new Session();

// WRITE: flatten the object and everything it can reach
try (ObjectOutputStream out = new ObjectOutputStream(
         new BufferedOutputStream(Files.newOutputStream(file)))) {
    out.writeObject(session);
}

// READ: rebuild it
try (ObjectInputStream in = new ObjectInputStream(
         new BufferedInputStream(Files.newInputStream(file)))) {
    Session restored = (Session) in.readObject();   // the stream is untyped
}

Two details in that example matter later. readObject returns Object, so the cast is yours to write, and the security topic explains why it comes far too late to protect you. And it declares ClassNotFoundException, because the stream names its classes as text that the reader must be able to load.

Pointing the same pair at a byte array instead of a file gives the helper used by the examples in the rest of this subchapter.

static <T extends Serializable> T roundTrip(T original)
        throws IOException, ClassNotFoundException {

    ByteArrayOutputStream bytes = new ByteArrayOutputStream();
    try (ObjectOutputStream out = new ObjectOutputStream(bytes)) {
        out.writeObject(original);                  // flatten into memory
    }
    try (ObjectInputStream in = new ObjectInputStream(
             new ByteArrayInputStream(bytes.toByteArray()))) {
        @SuppressWarnings("unchecked")
        T copy = (T) in.readObject();               // rebuild a detached graph
        return copy;
    }
}

When something goes wrong, the exception tells you which half of the process failed.

Exception	Raised at	Usual cause
NotSerializableException	Write time	Some reachable object's class is not Serializable
InvalidClassException	Read time	serialVersionUID mismatch, or no valid constructor
ClassNotFoundException	Read time	The class named in the stream is not on the classpath
StreamCorruptedException	Read time	The bytes are not a valid object stream
OptionalDataException	Read time	Primitive data found where an object was expected

NotSerializableException is the one beginners meet first, and its message names the offending class rather than the object you asked to write, because the failure is usually several references deep in the graph.

Key Takeaways

• Serializable is a marker interface with no members; implementing it grants permission rather than supplying behaviour.
• The JVM provides the read and write machinery reflectively by walking the declared fields.
• Instance fields are written, transient and static fields are not, and referenced objects are written recursively.
• ObjectOutputStream and ObjectInputStream are decorators, so the same code serialises to a file, a socket, or a byte array.
• readObject returns Object and declares ClassNotFoundException, because the stream carries class names as text.
• NotSerializableException fires at write time and names the deep class that caused it, not the object you passed in.

🧪 Practice

1. Write a small class, serialise an instance to a file, read it back in a second program run, and print every field.
2. Add a field of a non-serialisable type, observe the exception and its message, then fix it two ways: with transient, and by making that type serialisable.
3. Implement the generic roundTrip helper above and use it to deep-copy an object holding a nested list. Prove the copy is independent by mutating it.
4. Interview: Serializable declares no methods, so what does implementing it actually change? (Hint: who writes the code that reads your private fields, and what is it checking before it runs?)

Object Serialization Mechanics

An object in memory is a node in a graph. Its fields point at other objects, those point at more, and the whole structure is held together by addresses that mean nothing once the process exits. Serialisation is the act of flattening that graph into a linear byte sequence; deserialisation rebuilds an equivalent graph elsewhere. Java ships a mechanism that does this automatically for any class marked Serializable.

The crucial thing to understand before the syntax is that this mechanism gives your class a second public API that you never wrote. The byte format is derived from your private field names and types, it is visible to anyone holding the bytes, and it constrains how you may refactor the class forever after.

The graph problem, and how handles solve it. Naively writing each referenced object recursively fails on two structures: a cycle loops forever, and a shared object is written twice, so reading it back produces two distinct copies where the original had one.

Java's format solves both with a handle table. The first time an object is written it receives a sequential handle and its contents follow. Every later appearance of the same object writes only a back-reference to that handle.

   IN MEMORY                                 IN THE STREAM
                                             +----------------------------+
   order ------> customer <----+             | #1 Order    (fields...)    |
     |                         |             | #2 Customer "Ada"          |
     +--> items[0] ------------+             | #3 Item   owner -> ref #2  |
     |                         |             | #4 Item   owner -> ref #2  |
     +--> items[1] ------------+             +----------------------------+

   one Customer, referenced three times      written ONCE, referenced twice more

The consequence is that sharing and cycles survive the round trip, while the result is a deep copy disconnected from the originals.

Customer ada = new Customer("Ada");
Order original = new Order(ada, List.of(new Item(ada), new Item(ada)));

Order copy = roundTrip(original);            // the helper from the topic above

copy.items().get(0).owner() == copy.items().get(1).owner();  // true: sharing preserved
copy.customer() == ada;                                      // false: it is a deep copy

That property makes serialisation a convenient deep-clone tool, and also explains a common memory surprise: writing one small object can drag its entire reachable graph into the stream, including a collection you did not think was part of it.

The wire format, briefly. Every stream begins with a magic number and a version, followed by typed records.

   AC ED 00 05   73   72   00 07  53 65 73 73 69 6F 6E  ...
   ^^^^^ ^^^^^   ^^   ^^   ^^^^^  ^^^^^^^^^^^^^^^^^^^^
   magic  ver     |    |   length  "Session" (the class NAME, as text)
                  |    +-- TC_CLASSDESC: a class description follows
                  +-- TC_OBJECT: an object follows

A class descriptor carries the class name, its serialVersionUID, flags, and the name and type of every non-transient, non-static field. Field values are then written superclass-first. Two facts follow directly from this layout: the stream names classes as strings, which the receiver will resolve and load, and your private field names are written in plain text.

What actually happens on read. The sequence is unusual enough to be worth memorising, because several surprising behaviours fall out of it.

   1. Read the class descriptor; resolve the name to a Class
                                   -> ClassNotFoundException if absent
   2. Compare serialVersionUID with the local class
                                   -> InvalidClassException on mismatch
   3. Allocate the instance WITHOUT running any of its constructors
   4. Run the no-arg constructor of the nearest NON-serialisable superclass
   5. Fill fields class by class, calling each private readObject hook
   6. If readResolve exists, call it and use whatever it returns

Step 3 is the origin of the constructor-bypass problem from Chapter 4.4: field validation written in a constructor simply never runs. Step 4 is a rule people trip over: if the first non-serialisable superclass lacks an accessible no-arg constructor, the class cannot be deserialised at all, and you get InvalidClassException: no valid constructor.

The hook methods. Serialisation is customised through five specially named members that the mechanism finds reflectively.

Member	Signature	Purpose
writeObject	private void writeObject(ObjectOutputStream)	Control what is written
readObject	private void readObject(ObjectInputStream)	Control what is read, and validate
readObjectNoData	private void readObjectNoData()	Initialise when the stream lacks this class
writeReplace	Object writeReplace()	Substitute a stand-in object before writing
readResolve	Object readResolve()	Substitute a different object after reading

writeObject and readObject are private on purpose. They are not overrides and are never inherited; the mechanism invokes the one declared on each class in the hierarchy separately, so making them protected or public would only invite accidental overriding.

The idiomatic pair calls the default machinery first, then adds whatever the default machinery could not handle.

private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();              // all non-transient, non-static fields
    out.writeInt(entries.size());          // then the transient state, by hand
    for (Entry e : entries) {
        out.writeObject(e.key());
        out.writeObject(e.value());
    }
}

private void readObject(ObjectInputStream in)
        throws IOException, ClassNotFoundException {
    in.defaultReadObject();                // restore the ordinary fields
    int n = in.readInt();                  // read back in EXACTLY the same order
    entries = new ArrayList<>(n);
    for (int i = 0; i < n; i++) {
        entries.add(new Entry(in.readObject(), in.readObject()));
    }
    if (n < 0) throw new InvalidObjectException("negative size");   // validate here
}

This is exactly how HashMap serialises itself. Its bucket array is transient, because writing a hash table's internal layout would be both wasteful and wrong: hash codes can differ between JVMs, so the table is rebuilt by re-inserting the entries on read.

writeReplace and readResolve substitute objects. The most famous use is protecting a singleton, which plain serialisation otherwise breaks by manufacturing a second instance.

public class Registry implements Serializable {
    public static final Registry INSTANCE = new Registry();
    private Registry() { }

    private Object readResolve() {
        return INSTANCE;      // discard the freshly built copy, return the real one
    }
}

Enums need no such care: they are serialised by name alone and resolved back through valueOf, which is one of the reasons Chapter 4.5 recommends an enum for singletons.

Externalizable hands you the whole job. It replaces the automatic mechanism rather than adjusting it.

Aspect	Serializable	Externalizable
Fields written	Automatically, by reflection	Only what you write
Superclass state	Handled for you	Your responsibility
Constructor on read	Bypassed	Public no-arg constructor IS called
Control and speed	Limited, slower	Total, usually faster
Risk	Forgetting transient	Forgetting a field, silently losing data

public class Point implements Externalizable {
    private int x, y;

    public Point() { }                              // REQUIRED, public, no-arg

    @Override public void writeExternal(ObjectOutput out) throws IOException {
        out.writeInt(x);
        out.writeInt(y);
    }

    @Override public void readExternal(ObjectInput in) throws IOException {
        x = in.readInt();                           // same order, or garbage results
        y = in.readInt();
    }
}

Records change the picture for the better. A record's serialised form is defined by its state components, and deserialisation goes through the canonical constructor rather than bypassing it. Validation in a compact constructor therefore runs on every read, and the stream cannot forge an invalid instance.

public record Money(String currency, long cents) implements Serializable {
    public Money {                                  // compact constructor
        if (cents < 0) throw new IllegalArgumentException("negative");
    }
}
// Deserialising a stream that claims cents = -1 throws. A normal class would not.

This is the mechanism serialisation should always have had, and it is the direction the platform is moving.

Key Takeaways

• Serialisation flattens the whole reachable object graph, not just the one object you named.
• A handle table gives each object a number, so cycles terminate and shared references stay shared after the round trip.
• The result is a deep copy: identity inside the graph is preserved, identity with the originals is not.
• The stream names classes as text and writes private field names, making the format a de facto public API of your class.
• Reading allocates the object without running its constructors, then runs the no-arg constructor of the nearest non-serialisable superclass.
• writeObject and readObject are private, are not inherited, and must read back in exactly the order they wrote.
• readResolve substitutes an object after reading and is how a singleton survives; enums get this for free.
• Externalizable gives total control including superclass state, calls a public no-arg constructor, and loses data silently if you forget a field.
• Records deserialise through their canonical constructor, so their invariants actually hold.

🧪 Practice

1. Serialise an object whose two fields point at the same List, read it back, and prove with == that the sharing survived.
2. Create a class with a cyclic reference (a.peer = b; b.peer = a;), serialise it, and explain why the write terminates.
3. Write a class with a transient collection and restore it in readObject, then verify the round trip. Compare your approach with the JDK source of HashMap.writeObject.
4. Make a singleton that breaks under serialisation, prove it with ==, then fix it with readResolve.
5. Convert a small serialisable class into a record with validation, and show that a hand-edited stream of an invalid value is now rejected.
6. Interview: A serialisable class extends a non-serialisable one. What must that superclass provide, and what happens to its fields? (Hint: walk through steps 3 and 4 of the read sequence.)

transient Fields and serialVersionUID

Two keywords control the two dimensions in which a serialised class goes wrong: transient decides what crosses the boundary in space, and serialVersionUID decides what is compatible across time. Both are easy to ignore and expensive to get wrong.

transient marks a field as not part of the object's persistent state. Four distinct situations call for it.

Reason	Example field	What happens without transient
The value is derived	A cached hash, a computed total	Wasted bytes, and stale data on read
The type cannot be written	Socket, Thread, Connection, a lambda	NotSerializableException at write time
The value is a secret	A password, a session token, a key	The secret is written in the clear
The value is huge	A memoisation table, a loaded image	A tiny object drags megabytes with it

A transient field is not merely skipped on write; on read it is left at the JVM's default value, null for references and zero or false for primitives. Nothing warns you. If an invariant depended on that field being non-null, the object comes back broken.

public class SearchIndex implements Serializable {
    private static final long serialVersionUID = 1L;

    private final List<String> documents;        // the real state, written
    private transient Map<String, List<Integer>> index;   // derived, not written
    private transient volatile int hash;          // cached, recomputed lazily

    private void readObject(ObjectInputStream in)
            throws IOException, ClassNotFoundException {
        in.defaultReadObject();                   // documents is restored, index is null
        this.index = buildIndex(documents);       // rebuild what was deliberately skipped
    }
}

static fields are excluded too, for a different reason: they belong to the class, not to any instance, so there is no instance state to write. A common beginner bug is expecting a static counter to travel with an object.

One combination is a genuine trap: transient final. A final field cannot be reassigned inside readObject, so there is no legal way to restore it. Escape routes are reflection (ugly), dropping final, or the serialisation proxy pattern covered in the next topic.

serialVersionUID is the compatibility stamp. Every serialisable class has one. If you do not declare it, the JVM computes it at runtime as a hash over the class name, its modifiers, its implemented interfaces, and the signatures of its fields, constructors, and methods.

That last inclusion is what makes the implicit value unusable. Adding a private helper method changes the hash. So does changing a field from private to package-private, or compiling with a different compiler that orders members differently. Any of those makes yesterday's files unreadable.

private static final long serialVersionUID = 1L;

The declaration must be exactly static final long; private is not required but is strongly recommended, because a non-private one is inherited by subclasses, which then wrongly share their parent's identity. Once declared, the value changes only when you decide the format has broken compatibility.

A mismatch produces an unmistakable message.

java.io.InvalidClassException: com.example.Session; local class incompatible:
    stream classdesc serialVersionUID = -4938261571854927892,
    local class serialVersionUID = 1

The JDK ships serialver for inspecting the computed value of a compiled class:

serialver -classpath build/classes com.example.Session

Keeping the UID fixed is a promise, and you must know which changes you can keep it through. The rule of thumb is that changes which only add or remove optional state are compatible; changes to the shape of existing state are not.

Change to the class	Compatible?	Effect on data written by the old version
Add a field	Yes	New field reads as null / 0 / false
Remove a field	Yes	The old value in the stream is discarded
Add or remove a method or constructor	Yes	None
Change a field from transient to normal	Yes	Old streams have no value; reads as default
Change a field from normal to transient	Yes	Value is present but ignored on read
Change a field's type	No	InvalidClassException
Rename a field	No	Treated as remove plus add; the value is lost silently
Change a field from instance to static	No	The field stops being written
Change the position of a class in the hierarchy	No	Field values attach to the wrong class
Change class to enum, interface, or a record	No	Fundamentally different serialised form
Remove implements Serializable	No	Nothing can be read

Two compatible changes still deserve care. A field added in version 2 arrives as null when reading version 1 data, so any code assuming it is populated needs a fallback, which is what readObject validation is for. And a renamed field is the worst case in the table, because nothing throws: the old value is dropped and the new field silently holds a default.

// Version 2 of the class: 'region' did not exist in version 1
private void readObject(ObjectInputStream in)
        throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    if (region == null) {
        region = "UNKNOWN";        // migrate old streams to a usable default
    }
}

The honest conclusion is that keeping the UID unchanged says the stream can still be read, not that the resulting object is meaningful. Long-lived data needs an explicit version field and migration code, at which point a format designed for versioning, covered next, is the better tool.

Key Takeaways

• transient excludes a field from the serialised form and is required for derived, unserialisable, secret, and oversized state.
• A transient field comes back as null, 0, or false with no warning, so rebuild it in readObject if an invariant depends on it.
• static fields are never serialised, because they belong to the class.
• transient final cannot be restored in readObject, which is a reason to use a serialisation proxy.
• An undeclared serialVersionUID is computed from the class structure and changes when you add a mere private method.
• Declare private static final long serialVersionUID, and change it only when you intend to break old data.
• Adding and removing fields is compatible; changing a type, renaming a field, or moving a class in the hierarchy is not.
• Renaming a field fails silently: the old value is discarded and the new field holds a default.
• A matching UID guarantees readability, never meaningfulness, so versioned data needs explicit migration.

🧪 Practice

1. Serialise an object with a transient field holding a non-null value, read it back, and print the field. Then restore it correctly in readObject.
2. Write a class without a serialVersionUID, serialise an instance to a file, add an unused private method, recompile, and read the file back. Record the exact exception message.
3. Repeat exercise 2 with an explicitly declared UID and explain the difference in one sentence.
4. Take a serialised version-1 file, add a field to the class, and write the readObject migration that gives the new field a sensible default.
5. Rename a field between versions while keeping the UID fixed, then explain what the deserialised object contains and why nothing threw.
6. Interview: Why does the implicitly computed serialVersionUID include method signatures when methods are not serialised? (Hint: what question was the designers' hash originally trying to answer about two class versions?)

Serialization Risks and Alternatives

Java serialisation is not merely old-fashioned. It is the source of an entire class of remote code execution vulnerabilities, it welds your private fields to your compatibility story, and the platform's own architects describe it as a mistake to be undone. Understanding why leads directly to what to use instead.

Risk 1: deserialising untrusted bytes can execute arbitrary code. This is the important one, and the reason it surprises people is a matter of ordering.

   attacker-controlled bytes arrive
        |
        v
   readObject()
        |-- reads class NAMES from the stream
        |-- loads those classes from YOUR classpath
        |-- allocates instances, bypassing constructors
        `-- runs their readObject / readResolve hooks   <-- code executes HERE
        |
        v
   (Session) result     <-- your cast, your validation, your first look
                            ALREADY TOO LATE

The stream decides which classes get instantiated, and their hooks run before your code sees anything. Writing (Session) in.readObject() does not protect you, because the ClassCastException is thrown after the damage.

A gadget chain exploits this. The attacker does not supply malicious code; they supply a carefully nested graph of ordinary classes that are already on your classpath, chosen so that the side effects of their readObject, hashCode, equals, or toString methods compose into something dangerous, typically ending at reflection or Runtime.exec. The 2015 Apache Commons Collections chain demonstrated this against WebLogic, JBoss, Jenkins, and others, and it did not require a single line of vulnerable code in the applications themselves. Any library with a suitable side effect is ammunition.

Risk 2: denial of service from a tiny input. A stream of a few dozen bytes can describe a structure whose reconstruction is exponential. The classic is a nest of HashSets where each level contains the previous one twice: computing hash codes on read takes time that doubles per level. Deep nesting similarly overflows the stack. The receiver has no chance to check the size of the resulting object graph before building it.

Risk 3: invariants are bypassed. As Chapter 4.4 showed and the mechanics topic explained, no constructor runs, so a hand-crafted stream can produce an object your constructors would never have allowed: a negative age, an empty required list, a Set containing duplicates.

Risk 4: the format becomes a permanent constraint. Private field names and types leak into the serialised form, so renaming a private field is a compatibility break. Encapsulation is real at compile time and fictional at serialisation time.

Risk 5: practical limitations. The format is Java-only, so no other language can read it. It is verbose and slow compared with modern codecs. And it has no schema, so you cannot inspect, validate, or evolve the data with tooling.

If you must keep using it, three defences.

The first is the simplest and the most effective: never deserialise data that crossed a trust boundary. Serialisation is acceptable between two processes you control, on an authenticated channel, ideally with the bytes signed or encrypted.

The second is filtering, added in Java 9 and backported to 8. A filter is consulted before any class is resolved, so an allowlist prevents the gadget classes from ever being instantiated.

ObjectInputFilter filter = ObjectInputFilter.Config.createFilter(
        "com.example.model.*;"     // allow our own model classes
      + "java.base/java.util.*;"   // allow the collections we actually use
      + "maxdepth=20;maxrefs=1000;maxbytes=100000;"   // and cap the work
      + "!*");                     // reject everything else

try (ObjectInputStream in = new ObjectInputStream(source)) {
    in.setObjectInputFilter(filter);        // MUST be set before readObject
    Session s = (Session) in.readObject();
}

The same syntax works process-wide through the jdk.serialFilter system property, and Java 17 added a filter factory for per-context policies. Note that the pattern ends with !*: an allowlist that forgets to reject the remainder allows everything.

The third is the serialisation proxy pattern, which is the most robust structural fix. Instead of serialising the real class, you write a small private nested class carrying the logical state, and reconstruct the real object through its ordinary public constructor.

public final class Period implements Serializable {
    private final Date start, end;

    public Period(Date start, Date end) {
        if (start.after(end)) throw new IllegalArgumentException("start after end");
        this.start = new Date(start.getTime());     // defensive copies
        this.end   = new Date(end.getTime());
    }

    private static class SerializationProxy implements Serializable {
        private static final long serialVersionUID = 1L;
        private final Date start, end;

        SerializationProxy(Period p) { this.start = p.start; this.end = p.end; }

        private Object readResolve() {
            return new Period(start, end);          // the CONSTRUCTOR runs: validation holds
        }
    }

    private Object writeReplace() {
        return new SerializationProxy(this);        // never write a real Period
    }

    private void readObject(ObjectInputStream in) throws InvalidObjectException {
        throw new InvalidObjectException("proxy required");   // block forged streams
    }
}

This restores constructor validation, allows final fields, decouples the wire format from the internal fields, and makes a forged stream useless. Its cost is extra code and object allocation, and it cannot be used for classes that are extendable or mutually referential.

The alternatives, and how to choose. Every serious system today uses an explicit, schema-described format instead.

Format	Schema	Human readable	Cross-language	Size and speed	Typical Java library
JSON	Optional (JSON Schema)	Yes	Yes	Verbose, adequate	Jackson, Gson
Protocol Buffers	Required .proto	No	Yes	Compact, fast	protobuf-java
Avro	Required, travels with data	No	Yes	Compact, fast	avro
CBOR / MessagePack	Optional	No	Yes	Compact	Jackson dataformats
XML	Optional (XSD)	Yes	Yes	Very verbose	JAXB
Java serialisation	Implicit, the class itself	No	No	Verbose, slow	Built in

The decisive advantage is not performance. It is that these formats describe data, not objects. Nothing in a JSON document names a class to instantiate, so there is no gadget chain to build, and the mapping layer decides what types may be produced.

public record Session(String user, int requestCount) { }

ObjectMapper mapper = new ObjectMapper();
String json = mapper.writeValueAsString(new Session("ada", 3));   // {"user":"ada",...}
Session back = mapper.readValue(json, Session.class);   // canonical constructor RUNS

One warning carries over: JSON libraries can be made unsafe by re-introducing the same idea. Jackson's polymorphic default typing embeds Java class names in the document and hands the attacker back the same power, so it must never be enabled for untrusted input.

A short decision guide:

Situation	Use
Data crosses a trust boundary	JSON or Protobuf, never Java serialisation
A public API or anything another language reads	JSON, or Protobuf for high volume
High-volume internal service traffic	Protobuf or Avro
Long-term storage needing schema evolution	Avro or Protobuf
A quick deep copy of a graph you fully control	Java serialisation is defensible
Two JVMs you own, on a private authenticated link	Java serialisation with a strict filter

Where the platform is going. The JDK team has stated the intent to remove the current serialisation mechanism, and the direction of travel is visible already: records deserialise through their canonical constructor, serialisation filters are standard, and Project Amber is exploring a replacement built on constructors and pattern matching rather than on reflective field injection. Write new code as if the automatic mechanism is already gone, because for new designs it should be.

Key Takeaways

• Deserialisation runs attacker-influenced code before your cast, so a cast can never be a safety check.
• A gadget chain composes ordinary classpath classes into an exploit without any vulnerable code in your application.
• A few bytes can describe a graph whose reconstruction is exponential, giving cheap denial of service.
• Constructors are bypassed, so a forged stream can produce objects your class forbids.
• Private field names and types become part of the compatibility contract, which defeats encapsulation over time.
• The first rule is simply never to deserialise data from outside a trust boundary.
• ObjectInputFilter allowlists classes and caps depth, references, and size, and must end with a reject-everything-else pattern.
• The serialisation proxy pattern routes reading through a real constructor and rejects forged streams outright.
• JSON, Protocol Buffers, and Avro describe data rather than objects, so they carry no instruction to instantiate a class.
• Jackson's polymorphic default typing reintroduces the gadget problem and must stay off for untrusted input.

🧪 Practice

1. Write a method that deserialises from a byte array, then add an ObjectInputFilter that allows only your own package and the JDK base module. Verify that a java.util.ArrayList and an arbitrary third-party class are treated differently.
2. Build the nested-HashSet structure described above to six levels, serialise it, and measure both the stream size and the time to read it back. Extend it to twenty levels and predict the result before running it.
3. Convert a class with validating constructor and final fields to the serialisation proxy pattern, then confirm that reading a stream containing the real class throws.
4. Replace Java serialisation in a small program with Jackson and records. Compare the output size and note which invariants are now enforced on read.
5. Given a service that accepts serialised objects over HTTP, list every change you would make, ordered by how much risk each removes.
6. Interview: An endpoint calls (Order) in.readObject() on a request body and the team argues the cast makes it safe. Explain precisely why it does not. (Hint: describe the order of operations, and say when the ClassCastException would be thrown relative to the hook methods.)

<a id="14-networking-in-java"></a>