Table of Contents

Sequences and TryParse

The earlier lessons read one value from bytes you already had. Real inputs are messier: a buffer may or may not hold a whole value, a file holds thousands of records in a row, and a socket delivers them while the program does other work. This lesson covers the generated members for those cases. Each one runs the reader you already know; what changes is how it is called and what it does when the bytes run out.

TryParse: failure as a value

/// <summary>Shows the forms of TryParse and what each reports.</summary>
private static void TryParseForms()
{
    byte[] bytes = [0x02, 0x00, 0x06, 0x00, 0x00, 0x00];

    // TryParse returns false instead of throwing for a read, path, or limit failure; the value is null then.
    True(Wire.TryParse(bytes, out Wire.Header? header) && header.Length == 6, "six bytes are a header");
    True(!Wire.TryParse(bytes.AsSpan(0, 3), out Wire.Header? missing) && missing is null, "three bytes are not");

    // The second form hands over the failure the throwing form would have raised, so a log line can say why.
    True(!Wire.TryParse(bytes.AsSpan(0, 3), out _, out CStructException? failure), "the same outcome");
    True(failure is CStructReadException, "a short read: the kind was read, the length needs four more bytes");
    True(failure!.Message.Contains("needed 4, available 1", StringComparison.Ordinal), failure.Message);

    // A stream is left at its origin after a failure, so the caller can try another layout at the same place.
    using var stream = new MemoryStream([0xFF, 0x02, 0x00, 0x06]) { Position = 1 };
    True(!Wire.TryParse(stream, out _), "three bytes remain");
    Equal(1L, stream.Position);

    // Cancellation is not a failure: it throws through TryParse as it does through Parse.
    using var cancelled = new CancellationTokenSource();
    cancelled.Cancel();
    Throws<OperationCanceledException>(() => Wire.TryParse(bytes, out _, null, new ReadOptions { CancellationToken = cancelled.Token }));

    // The runtime's TryReadValue<T> is the same idea for a path; TryGet and GetOrDefault are its members' twins.
    True(Wire.Layout.TryReadValue(bytes, "header.length", out uint length) && length == 6, "typed path read");
    StructValue parsed = Wire.Layout.Parse(bytes, "header");
    True(!parsed.TryGet("flags", out byte _, out CStructException? absent) && absent is CStructPathException, "no such member");
    Equal((byte)0, parsed.GetOrDefault("flags", (byte)0));
}

Parse throws when the bytes are wrong. That is right when wrong bytes are a bug, and wrong when they are expected - a truncated network read, a file of unknown format, a user's upload. Catching an exception for an expected case is slow (an exception captures a stack trace) and noisy (every caller writes the same try).

TryParse returns a bool and an out value: true and the object, or false and null. The second overload adds out CStructException? failure, the exception the throwing form would have raised, already built but never thrown. A caller that only needs yes-or-no ignores it; a caller that logs why gets the same text Parse would have shown - the field, the path, the offset.

What TryParse catches is precise: a read failure (short input, a bad value), a path failure, and a limit failure - the CStructException family. What it lets through is just as precise: an OperationCanceledException (cancelled work says nothing about the bytes) and argument errors (a null stream is a bug in the caller). A stream is left at its origin after a failure, so another layout can be tried at the same place.

Trying without an exception

For most layouts TryParse runs Parse and catches what it throws, which keeps the same text and the same behavior but costs an exception for each failure. A struct of fixed size with constraints does better: its TryParse decodes the bytes and checks the conditions itself, so a value that breaks one is returned as failure and nothing is thrown. When the generated type is a class and the conditions name only plain numeric members at fixed offsets, the conditions are checked on the bytes before the object is built, so a rejected input allocates no value. A source shorter than the struct is returned as a failure too ("Not enough bytes: needed 12, available 6"), naming the struct's size rather than the member the bytes run out in. A read limit and a cancelled token are not constraints and still go the long way (limit failures are caught, cancellation passes through). A generated view of such a struct has the matching TryValidate(out failure), and with @expect conditions on members at fixed offsets the layout has Matches:

// Which of two layouts is this? Neither question builds an object or throws.
if (ExtSuperblock.Matches(bytes)) { ... }          // decodes only the fields @expect names
else if (ReiserSuperblock.Matches(bytes)) { ... }

Matches(source) is true when the source holds the bytes the evaluated conditions read and none of them is broken. It evaluates @expect conditions only (an unsupported value is still recognized), skips a condition on a member that follows a runtime-sized array or sits in a conditional arm or a nested struct, and treats a source that is too short or a condition that cannot be evaluated as no match. A condition that names a caller variable you did not pass in variables is a mistake in the call, not a property of the bytes, so Matches throws an ArgumentException for it. A match therefore says "this looks like the format", not "a read succeeds": call TryParse next.

The Views surface has Matches too, as static methods of the layout class: recognizing a format is the question to ask before making a view of the bytes. A layout that is compiled at run time answers the same question with CStruct.Matches(declaration, bytes, variables), which decodes the same members and returns the same answers; it throws CStructLayoutException for a struct with no condition it can evaluate (the generated layouts have no Matches method then).

var layout = new CStruct("struct sb { uint32 inodes; uint16 magic @expect(magic == 0xEF53); };");
bool looksLikeExt = layout.Matches("sb", bytes);   // no value is built, nothing is thrown for short or wrong input

Every composite has the five input kinds - span, array, memory, ReadOnlySequence<byte>, stream - as TryParse<Name>, and the root has them as TryParse. The interface ICStructGenerated<TSelf> exposes the span form, so generic code can parse any generated root without knowing its class.

ReadValue<T> on the layout class

Wire.Layout.ReadValue<HeaderRecord>(bytes, "header") reads the root into a mapped class. The layout class shortens it to Wire.ReadValue<HeaderRecord>(bytes) - the same call with the root's name filled in - and Wire.TryReadValue<HeaderRecord>(bytes, out record) is its non-throwing form. They are forwarding members: a runtime read, not generated code, so use them where a mapped class is the right result type and Parse where the generated class is.

Records: one after another

private static async Task RecordSequence()
{
    // Three headers, one after another, and nothing else: Records reads them lazily, one per step of the loop.
    byte[] bytes = [1, 0, 6, 0, 0, 0, 2, 0, 7, 0, 0, 0, 3, 0, 8, 0, 0, 0];
    var kinds = new List<ushort>();
    foreach (Wire.Header header in Wire.Records(bytes))
    {
        kinds.Add(header.Kind);
    }

    Equal("1,2,3", string.Join(",", kinds));

    // The view enumerator walks the same records without allocating a single object: each step is a view over
    // the next six bytes. The record's size (Sizes.Header) is the stride.
    uint total = 0;
    foreach (Wire.HeaderView view in Wire.HeaderView.Enumerate(bytes))
    {
        total += view.Length;
    }

    Equal(21u, total);

    // The runtime's ParseMany reads the same records as StructValues, from memory or from a stream.
    Equal(3, Wire.Layout.ParseMany(bytes, "header").Count());
    var lengths = new List<uint>();
    await foreach (Wire.Header header in Wire.RecordsAsync(new MemoryStream(bytes)))
    {
        lengths.Add(header.Length);
    }

    Equal("6,7,8", string.Join(",", lengths));

    // Trailing bytes shorter than one record are not ignored: the step that meets them fails, naming the record.
    byte[] trailing = [.. bytes, 9, 9];
    int whole = 0;
    try
    {
        foreach (Wire.Header header in Wire.Records(trailing))
        {
            whole++;
        }

        True(false, "unreachable");
    }
    catch (CStructReadException failure)
    {
        Equal(3, whole);
        True(failure.Message.Contains("remaining 2 bytes are not a whole number of 6-byte elements", StringComparison.Ordinal), failure.Message);
        Equal("[3].header", failure.Path);
    }
}

A file of log entries or a message body of frames is one struct after another with nothing in between. Wire.Records(bytes) returns an IEnumerable<Header> over such a sequence, and foreach walks it.

An IEnumerable<T> is a promise to produce values one at a time; foreach asks for the next one at each step. Records is an iterator: it parses a record when the loop reaches it, not before. Ten million records cost one object at a time, and break after the third parses three. The rules it follows:

  • Stride. A composite whose size the layout fixes (Sizes.Header exists) advances by that size. A runtime-sized composite - a count-prefixed payload - advances by the bytes the previous record consumed, so its records are read in order and each one's length comes from its own contents.
  • Trailing bytes. Bytes left over that are shorter than one record are not skipped. The step that meets them throws, with the text a T v[EOF] array uses for a partial element ("The remaining 2 bytes are not a whole number of 6-byte elements") when the size is fixed, or the short read itself when it is not. A file with a trailer slices it off first.
  • The record's index. A failure names the record before the path: [3].header.length is the length of the fourth record. failure.Path carries the same text.
  • Each record is its own region. The read limits apply per record, and a stored absolute pointer address counts from the record's first byte - so a pointer's target must lie inside its own record's bytes, which is what a record-per-message format guarantees and a shared-table format does not.

The inputs are memory, a ReadOnlySequence<byte> (one segment in place, a chain through one pooled copy), and a stream: a fixed-size composite is read from any readable stream exactly one record at a time, byte for byte, and a runtime-sized one through a pooled window that refills from the start of a record it could not hold, which needs a seekable stream. RecordsAsync(stream) is the same over await foreach, with a cancellation token checked between records. Every composite has Records<Name>/Records<Name>Async; the runtime's ParseMany reads the same sequences as StructValues, with the same rules and the same failure texts.

The view enumerator

A loop that reads one field from each of a million records should not create a million objects. Wire.HeaderView.Enumerate(bytes) returns a foreach-able enumeration whose Current is a HeaderView over the next record - a view, so nothing is allocated for it either. The full surface always generates the enumerator; on the Reader, Parser and Views surfaces add Enumerators = true to the attribute (see Generate less); there Enumerate returns the enumerator itself, which is its own foreach source, so the loop reads the same.

foreach does not require IEnumerable<T>. It requires a GetEnumerator() method whose result has MoveNext() and Current, and it is happy for both to be ref structs - types that live on the stack and cannot escape the method. That is what makes the enumeration allocation-free: the enumerable, the enumerator, and every view are stack values that vanish when the loop ends. It also means they cannot be stored in a field, captured by a lambda, or handed to LINQ; for those, Records returns objects.

The enumerator exists for composites with a static size only (a view needs one), advances by that size, and fails on trailing bytes exactly as Records does. On the reference machine, walking 256 records through the enumerator costs what the hand-written offset loop costs and allocates 0 bytes; Records over the same bytes allocates one 48-byte class per record, the same as a Parse loop. The performance page has the measured table.

ParseAsync and WriteAsync on the class

/// <summary>Reads and writes through the asynchronous forms of the generated layout.</summary>
private static async Task GeneratedAsync()
{
    // ParseAsync and WriteAsync on the layout class: the same reader and writer, with the bytes moved by
    // ReadAsync and WriteAsync. A MemoryStream that exposes its buffer is read in place and the task is already
    // complete; a file or a socket pays the real asynchronous I/O.
    byte[] bytes = [0x02, 0x00, 0x06, 0x00, 0x00, 0x00];
    using var source = new MemoryStream(bytes);
    Wire.Header header = await Wire.ParseAsync(source);
    Equal(6u, header.Length);
    Equal(6L, source.Position);

    using var target = new MemoryStream();
    await Wire.WriteAsync(target, header);
    SequenceEqual(bytes, target.ToArray());

    // A cancelled token ends the read before a byte is taken and the write before a byte is written.
    using var cancelled = new CancellationTokenSource();
    cancelled.Cancel();
    try
    {
        await Wire.WriteAsync(new MemoryStream(), header, cancellationToken: cancelled.Token);
        True(false, "unreachable");
    }
    catch (OperationCanceledException)
    {
    }
}

Wire.ParseAsync(stream) reads the stream with ReadAsync into a pooled buffer - first a seekable stream up to its remaining length, any stream up to MaxTotalBytesRead plus one byte - and runs the generated reader over it, growing the buffer and running the reader again when the value needs bytes past it (padding and pointer targets are not charged to the budget): the same value, limits, and messages as Parse(stream), with the thread free while the bytes arrive. A seekable stream ends after the value and returns to its origin on a failure. Wire.WriteAsync(stream, header) serializes first and writes once, so a validation failure writes nothing. The async guide explains the rule, the position, the pointer coordinates, and cancellation in full.

Check yourself

  1. Wire.TryParse(bytes, out var header, out var failure) returns false. What is header, and what is failure?
  2. Which of these can Records do that the view enumerator cannot: read a runtime-sized composite, read from a stream, be passed to .Select(...), avoid allocating?
  3. Two bytes follow the last whole record. Which step of the foreach throws, and what does its path say?
Answers
  1. header is null; failure is the CStructReadException (or path or limit exception) that Parse would have thrown.
  2. The first three. Only the view enumerator avoids allocating.
  3. The step after the last whole record - the fourth MoveNext for three records - with the path [3].header.

Exercise

Add a uint8 flags; member to the header layout, making it seven bytes, and enumerate the same eighteen bytes. Predict what happens before you run it: the stride is now 7, so the third step fails with "The remaining 4 bytes are not a whole number of 7-byte elements" at [2].header. Then fix the bytes.

Edit this page