Table of Contents

Configure variables, options, and limits

Some layout choices are fixed when you construct CStruct. Others change for each piece of data. CStructSharp keeps those two groups separate:

  • constructor and compilation options describe the format and bound layout preparation;
  • operation variables provide integer values such as an externally known array count; and
  • read, write, or update options limit work performed on one input.

Supply a runtime variable

This layout cannot know the payload count until the caller supplies COUNT:

struct packet {
    uint8 kind;
    uint8 payload[COUNT];
};

Pass a read-only integer dictionary to every operation that needs the count:

private static void RuntimePayload()
{
    var layout = new CStruct("struct packet { uint8 kind; uint8 payload[COUNT]; };");
    var variables = new Dictionary<string, int> { ["COUNT"] = 3 };
    byte[] bytes = [0x7F, 0x10, 0x20, 0x30];
    StructValue packet = layout.Parse(bytes, "packet", variables);
    Equal((byte)0x7F, packet.Get<byte>("kind"));
    Equal(3, packet.Get<byte[]>("payload").Length);
    object? secondPayload = layout.ReadValue(bytes, "packet.payload[1]", variables);
    Equal((byte)0x20, (byte)secondPayload!);

    using var stream = new MemoryStream(bytes);
    stream.Position = 1;
    Equal(3, layout.GetArrayLength(stream, "packet.payload", variables));
    Equal(1L, stream.Position);
}

A generated reader takes the dictionary as its variables argument, which comes before the options on the unnamed root Parse as on Parse<Name>: Wire.Parse(bytes, variables), Wire.Parse(bytes, variables, options), and with limits as a value Wire.Parse(bytes, variables, in limits), which is the order of Parse(bytes, variables, options); a read with limits and no variables is Wire.Parse(bytes, in limits). The named readers and the TryParse forms follow the same order. Write Wire.Parse(bytes, null, options) (or options: options) to pass only the options. Wire.Parse(bytes, default) does not compile: default could be the variables or the limits, so name the argument (default(ReadLimits) or (IReadOnlyDictionary<string, int>?)null).

The operation copies the entries before it evaluates the layout. A caller value overrides a layout #define with the same name for that operation, but CStructSharp does not change the dictionary.

Use the same variables for related read, address, length, write, and update calls. Omitting or changing COUNT can make a later path refer to a different byte position than the initial parse.

Choose the right option type

Option type When it applies Examples of work it limits
CStructCompilationOptions Constructing CStruct Source length, layout/expression depth, expression work
ReadOptions Parse, read, debug, address, length, pointer traversal Arrays, strings, total bytes, nesting, pointers
WriteOptions Serialize and direct stream writes Arrays, strings, total output, nesting, pointer encoding
UpdateOptions Locating and replacing existing storage Separate traversal-read limits plus inherited write limits

Properties are init-only, so configure a complete object with an initializer:

var options = new ReadOptions
{
    MaxArrayElements = 10_000,
    MaxStringBytes = 1_024 * 1_024,
    DereferencePointers = false,
};

The public operation reads the supplied values at its outer entry. Reuse an initialized options object when several calls use the same policy; create another object for a different policy.

The four option types are C# records: a with expression makes a copy that changes only the members you name, and two option objects with the same members are equal. That is how a shared policy gets one variation without repeating every setting:

private static void OptionsWith()
{
    var layout = new CStruct("struct sample { uint8 count; uint16 values[count]; char name[4]; };");
    byte[] bytes = [1, 0x34, 0x12, (byte)'a', (byte)'b', 0, 0];

    // One shared policy, and a variation that changes a single member.
    var strict = new ReadOptions { MaxArrayElements = 8, MaxStringBytes = 64, };
    ReadOptions trimmed = strict with { TrimFixedText = true, };
    Equal(8, trimmed.MaxArrayElements);
    Equal("ab\0\0", layout.Parse(bytes, "sample", options: strict).Get<string>("name"));
    Equal("ab", layout.Parse(bytes, "sample", options: trimmed).Get<string>("name"));

    // Records compare by their members, so an equal policy is the same policy.
    True(strict == new ReadOptions { MaxArrayElements = 8, MaxStringBytes = 64, }, "equal members, equal options");
    True(strict != trimmed, "one member differs");
}

The Codecs and Defined members of CStructCompilationOptions hold collections, which compare by reference: two compilation options that carry different list instances are not equal even when the lists have the same contents.

Understand the defaults

Default limits are intentionally finite:

  • layout source: 128 KiB;
  • layout or expression/dependency depth: 256;
  • expression work: 100,000 steps/nodes;
  • one read or write array: 1,000,000 elements;
  • one encoded read or write string: 16 MiB;
  • total bytes read or written: 64 MiB;
  • read pointer depth: 64; and
  • read or write nesting depth: 256.

The memory-analysis API budgets its operations with MemoryAccessContext instead of these option records. It uses the same defaults for total bytes (64 MiB), pointer depth (64), and nesting depth (256); see memory budgets.

MaxTotalBytesRead counts the bytes an operation consumes, each byte once per read of it. It is not a limit on the input's file size. Two kinds of field look at bytes before they know how long they are, and neither look is charged:

  • A terminated array (uint8 a[]) is scanned for its all-zero terminator element before its elements are read. The elements and the terminator count once, when they are consumed. The scan stops where the budget ends, so an array that does not fit the budget fails with CStructReadLimitException before any element is read.
  • A terminated string (cstring, char name[], unicode_string_zero) reads ahead in chunks of up to 256 bytes while it searches for its terminator. Only the bytes through the terminator count; a stream source may still read the rest of the chunk physically, and the reader moves back over it.

For example, struct one { uint8 a[]; } over 01 02 00 09 consumes three bytes (two elements and the terminator), so a parse succeeds with a budget of 3 and fails at 2, on every input form and in the generated reader too. Debug parsing spends the same budget as a plain parse: a packed header with a uint16 and a uint32 needs a budget of 6 either way, because debug records carry byte ranges rather than copies. Layouts with unions, pointers, or selected reads may reread bytes for traversal or overlapping fields, and each reread counts, so the input size is a lower bound, not the exact cost.

UpdateOptions has separate MaxTraversal* values for bytes read while finding the destination. After the target is found, its inherited write limits apply to the replacement.

These are safety ceilings, not a promise that every value below them is appropriate for your application. For a network message expected to contain at most 100 items, set a limit near 100 rather than relying on one million.

Pass the limits as a value

ReadOptions is an object, so a program that builds one for every record it parses allocates 88 bytes each time. When the only thing that changes is a limit, the generated Parse has an overload that takes the limits as a value:

var limits = new ReadLimits { MaxArrayElements = 4096, MaxTotalBytesRead = 512 };
Header header = Wire.Parse(bytes, limits);         // no ReadOptions object is created
Entry entry = Wire.ParseEntry(bytes, in limits);   // the named reader of any declaration

ReadLimits holds the array, string, total-byte and nesting limits, TrimFixedText, InvalidText, OffsetBase, the constraint Constraints policy and the cancellation token, with the meaning and the defaults they have on ReadOptions; a default(ReadLimits) has the default limits. It does not hold the pointer policy, which keeps its defaults, so a read that follows pointers with another policy uses ReadOptions. ReadLimits.From(options) and limits.ToOptions() convert between the two. The overload exists on every generated surface that has a reader; views and the stream and sequence forms take ReadOptions.

Report offsets in the larger input

A failure reports Offset, the position where the read stopped. For a span that is a position in the span, but a program that reads a header out of an image reads it into a buffer first, and the position it wants in a report is the one in the image. ReadOptions.OffsetBase and ReadLimits.OffsetBase give the position of the buffer's first byte in the larger input, and it is added to the Offset of every failure of a span, array or memory read and to the offset in its message:

byte[] sector = ReadSector(image, 0x1BE0);
var limits = new ReadLimits { OffsetBase = 0x1BE0 };
if (!Wire.TryParse(sector, out Header? header, out CStructException? failure, in limits))
{
    Console.WriteLine(failure!.Message);   // "... (field 'magic', in 'header', offset 7136)": 0x1BE0 + 2
}

It changes nothing else: positions, pointer addresses and Origin still count from the buffer's first byte. It is at most 2^48 - 1 (ReadLimits.MaxOffsetBase), so that ReadLimits stays as small as it is; a negative or larger value is an ArgumentOutOfRangeException when it is set. The runtime layout's stream readers and the record sequences over a stream (ParseMany, Records) report absolute stream positions already and ignore it; the generated Parse of a stream counts from the stream's position at the call, so it adds it.

Choose a policy for the quiet cases

Four options change behavior rather than a limit. The first two default to the permissive choice, and the third to the strict one:

  • ReadOptions.TrimFixedText (default false): fixed-capacity text such as char name[8], wchar[N], or a bounded utf8 name[N] buffer keeps its NUL padding when read, so 61 62 00 00 is "ab\0\0". Set the option to read "ab"; only trailing NULs are removed and writing still zero-pads to the declared capacity.
  • ReadOptions.InvalidText (default Fail): text that is not valid in its encoding - malformed UTF-8, an unpaired UTF-16 surrogate, a non-ASCII byte in a cstring - fails the read at its field. InvalidTextHandling.Replace reads each invalid sequence as U+FFFD and continues; see invalid text.
  • ReadOptions.Constraints and ReadLimits.Constraints (default ConstraintPolicy.Enforce): the read evaluates the @expect and @supports conditions of its layout. ConstraintPolicy.Skip evaluates none and collects none, so a layout that states its header rules can still be read where those rules are not wanted; see read without the constraints.
  • WriteOptions.UnknownMembers (default Ignore): a supplied value may carry members the struct does not declare and they are skipped. UnknownMemberPolicy.Reject fails the write before any byte is written - 'bogus' is not a member of 'root' (WriteOptions.UnknownMembers is Reject). The layout declares: kind, tail. - for dictionaries, StructValues, and .NET objects alike, nested structs included. The members of an anonymous promoted struct or union are declared members of the struct that contains it, so they are accepted on that parent value, as a parse returns them. It also applies to UpdateOptions.

The what is not an error list explains the other quiet cases.

Handle limit failures

Exceeding a read limit throws CStructReadLimitException with code ReadLimitExceeded. Exceeding a write limit throws CStructWriteLimitException with code WriteLimitExceeded. Invalid non-positive option values are argument errors and are rejected before work begins.

Do not respond to a limit failure by raising every limit globally. Confirm the real format maximum, distinguish trusted from untrusted data, and change only the relevant policy. A count that is unexpectedly huge may indicate wrong byte order, a wrong starting position, or an incorrect runtime variable rather than a legitimate large value.

Cancel a long operation

Limits bound how much work an operation may do; they do not bound how long a caller is willing to wait. The three option records carry a CancellationToken for that. A token is a small handle that another part of the program - a timeout, a request that was abandoned, a user pressing Stop - can switch to "cancelled"; code that holds the token checks it at sensible points and stops. The library checks it where a read or write reaches a boundary: when it enters a struct or union, when it follows a pointer, between the 64 KiB blocks of a numeric array, between the elements of an array of structs, and between the 256-byte chunks of a terminated string. It never checks per primitive, so a small read costs nothing for it.

private static void Cancellation()
{
    var layout = new CStruct("struct point { uint16 x; uint16 y; }; struct path { uint8 count; point points[count]; };");
    byte[] bytes = [2, 1, 0, 2, 0, 3, 0, 4, 0];

    // A token that is already cancelled ends the read at its first boundary, before any byte is decoded.
    using var cancelled = new CancellationTokenSource();
    cancelled.Cancel();
    var options = new ReadOptions { CancellationToken = cancelled.Token, };
    Throws<OperationCanceledException>(() => layout.Parse(bytes, "path", options: options));

    // The same options with a live token read normally; a timeout is the usual source of one.
    using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(30));
    StructValue path = layout.Parse(bytes, "path", options: options with { CancellationToken = timeout.Token, });
    Equal(2, path.Get<StructValue[]>("points").Length);
}

Cancellation is not a read failure: the operation ends with the runtime's OperationCanceledException, which TryReadValue<T> lets through (it restores the stream position first) rather than turning into false. An update stages its bytes before it commits, so a cancelled update leaves the destination unchanged; a direct stream write may have written a prefix, as any late failure may. Generated classes observe the same token through their ReadOptions/WriteOptions arguments.

Continue with Handle errors and recovery, or use the exact language limits when defining an input policy.

Edit this page