Memory schemas and metadata import

A schema describes how bytes represent values: which member starts at which offset, how many bytes it occupies, and which codec turns those bytes into a number. For a file format you usually write that description yourself. For a memory image the description already exists, because a compiler made those decisions when it built the program. This guide shows how to state such decisions explicitly and how to import them from two common metadata formats.

Why a memory schema takes offsets instead of computing them

Consider this C declaration compiled for a typical 64-bit platform:

struct sample { uint8_t state; uint32_t count; };

The compiler does not place count at offset 1. Its platform's application binary interface (ABI) requires a four-byte integer to sit at an address divisible by four, so it inserts three padding bytes and places count at offset 4. The struct is eight bytes long. A different compiler, a #pragma pack directive, or a different CPU can produce a different answer, and a bit-packed member such as int flag : 3 adds compiler-specific bit placement on top. How C structs occupy memory explains these rules in depth.

An analyzer running on your machine cannot safely reproduce another platform's decisions from type names alone. MemorySchema therefore never guesses. Every composite member carries an explicit byte offset, every type carries an explicit size that includes padding, and bit-packed members carry an explicit bit offset and width. Where the Portable language computes placement from declaration order, a memory schema records placement as given.

Compile a schema once and reuse it in sessions that inspect many records or several address spaces. Validation happens at construction, so a bad description fails before any source is read.

Three ways to obtain a schema

Project an existing layout. PortableMemorySchema.Create(layout, "Root") converts a compiled CStruct with a fixed layout into descriptors. It copies the compiled offsets, pointer width, byte order, and codecs; it does not parse the declaration again. Use this when your own Portable declaration is already the authority.
Write descriptors by hand. Construct MemoryTypeDefinition and MemoryField objects when your application knows the placement, for example from a hardware manual or a debugger.
Import metadata. Feed BTF or ISF to the importers and use the returned MetadataImportResult.Schema and RootTypeId. Use this for operating-system structures whose placement was recorded by the build.

All three routes produce the same MemorySchema and support the same session operations. The Portable projection rejects conditional and runtime-sized members, because a memory schema must know every size in advance. For those layouts, open a bounded stream view and use the core API.

The building blocks of a schema

A schema is a graph of MemoryTypeDefinition nodes connected by IDs. Each definition has a Kind:

Kind	Meaning	Extra information carried
`Scalar`	An integer, float, boolean, or enum decoded by a core codec	`ScalarType` names the codec, for example `"uint32"`
`Pointer`	An unsigned stored address	`Size` is the pointer width; `ElementTypeId` names the target, or null for opaque
`Struct`	Members at explicit offsets that must not overlap	`Fields`
`Union`	Members at explicit offsets that deliberately overlap	`Fields`
`Array`	A fixed number of equally sized elements	`ElementTypeId` and `Count`; `Size` is the total byte length
`Incomplete`	A type with identity but no readable representation, such as a forward declaration	Nothing; size is zero

Every definition has an Id and a Name, and they serve different purposes. Id is the reference key used by fields and by session calls. Name is a display string. Native metadata often contains several distinct types with the same name, so the importers generate unique IDs such as btf:7 and keep the original name for display.

A MemoryField connects a containing struct or union to a member: a Name, a TypeId, a byte Offset relative to the start of the containing record, and optionally a bit slice (BitOffset, BitWidth, Signed). A field marked Promoted makes the members of an anonymous nested struct or union visible in its parent, mirroring how C lets you write record.flag when flag lives in an unnamed inner union.

The MemorySchema constructor validates the whole graph before it returns:

every referenced ID must exist, and a scalar's declared size must equal its codec's size;
every member must fit inside its container, and an array's size must equal element size times count;
struct members must not overlap (two bit slices may share one storage unit as long as their bits are disjoint);
a type may not contain itself by value, directly or through other types, because that would need infinite storage. A pointer back to the same type is fine: the pointer has a finite size.

An array's element may have size zero only when that makes the whole array zero bytes wide - count times zero can only ever equal a declared array size of zero. This is a real case, not just an allowed edge: native metadata sometimes describes an empty marker struct with no members (Linux's lock_class_key, for example, exists only so the lock validator has a stable address to key on, never to hold data) and arrays of that struct. Any other size disagreement, zero-size element or not, is still rejected.

Describe a padded record with a signed bit slice

Suppose metadata says a record is eight bytes long. A three-bit signed state occupies bits 1 through 3 of byte zero; a four-byte count starts at offset 4. Bytes 1 through 3 are padding. We describe the state's storage unit as a one-byte scalar, even though the selected value uses only three of its bits.

Byte offset	Input	Meaning
0	`8F`	State bits are `111`; the neighboring bits are also present
1–3	`AA BB CC`	Padding, preserved by an update
4–7	`2A 00 00 00`	Little-endian count 42

// Every offset and size is stated, as metadata would state it; nothing is computed from declaration order.
var schema = new MemorySchema(new[]
{
    new MemoryTypeDefinition("byte", "byte", MemoryTypeKind.Scalar, 1, scalarType: "uint8"),
    new MemoryTypeDefinition("word", "word", MemoryTypeKind.Scalar, 4, scalarType: "uint32"),
    new MemoryTypeDefinition("record", "Record", MemoryTypeKind.Struct, 8, fields: new[]
    {
        // "state" is bits 1..3 of the byte at offset 0, read as a signed two's-complement number.
        new MemoryField("state", "byte", 0, bitOffset: 1, bitWidth: 3, signed: true),
        new MemoryField("count", "word", 4),
    }),
});
var session = new MemorySession(schema);

// Byte 0 is 1000 1111: bits 1..3 are 111, which is -1 in three-bit two's complement.
var source = new ByteArrayMemorySource("record", new byte[] { 0x8f, 0xaa, 0xbb, 0xcc, 42, 0, 0, 0 });
var region = new MemoryRegion(source, 0, 8);
Require((long)session.Read(region, "record", "state")! == -1, "Signed three-bit value");
Require((uint)session.Read(region, "record", "count")! == 42, "Member at offset four");

// Writing -2 (bits 110) touches only bits 1..3; bit 0, the high nibble, and the padding survive.
session.PlanUpdate(region, "record", "state", -2L).Commit();
Require(source.ToArray().SequenceEqual(new byte[] { 0x8d, 0xaa, 0xbb, 0xcc, 42, 0, 0, 0 }), "Neighbor preservation");

Byte 8F is 1000 1111 in binary. Bits are numbered from the least significant end, so bit 0 is the rightmost digit. Bits 1 through 3 are 111. A three-bit two's-complement number covers -4 to 3:

Bits	Unsigned	Signed
`011`	3	3
`100`	4	-4
`110`	6	-2
`111`	7	-1

Reading state therefore returns -1. Updating it to -2 stores 110 in bits 1 through 3, changing 8F to 8D; bit 0, the high four bits, the padding, and the count remain unchanged. Bit offsets count from the least significant bit of the decoded storage integer. Byte order decides how a multi-byte storage integer is assembled from its bytes; it does not change what bit index zero means.

Two rules from this example are worth remembering. Reads sign-extend a signed slice and writes check the signed range, so -5 is rejected rather than wrapped. And an update reads the storage unit first, so bits that are not part of the selected slice survive untouched; the update guide builds on this.

Import a small ISF description

ISF (Intermediate Symbol Format) is the JSON metadata format used by the Volatility 3 memory-forensics framework. Its symbol tables are generated from a kernel's debug information and describe base types, user types with member offsets, enums, and symbol addresses. This importer consumes the value-type subset: it turns types into descriptors. Choosing the profile that matches a capture, and finding the address of a record, remain the application's job. An imported type says how a record is laid out, not where one is.

The following independently authored metadata places value at offset 4 in an eight-byte record. The first four bytes are outside the selected member, so their values do not affect the result, 7.

const string json = """
    { "metadata": { "format": "6.2.0" },
      "base_types": { "u32": { "kind": "int", "size": 4, "signed": false, "endian": "little" } },
      "user_types": { "counter": { "kind": "struct", "size": 8, "fields": {
        "value": { "offset": 4, "type": { "kind": "base", "name": "u32" } }
      } } }, "enums": {}, "symbols": {} }
    """;
// The importer follows "counter" and the types it references; RootTypeId is the importer's ID for it.
MetadataImportResult imported = IsfMetadata.Import(Encoding.UTF8.GetBytes(json), "counter", pointerSize: 8);
var session = new MemorySession(imported.Schema);

// Offsets 0..3 are outside the selected member, so their contents cannot influence the result.
var source = new ByteArrayMemorySource("capture", new byte[] { 0xaa, 0xbb, 0xcc, 0xdd, 7, 0, 0, 0 });
object? value = session.Read(new MemoryRegion(source, 0, 8), imported.RootTypeId, "value");
Require((uint)value! == 7, "Imported explicit offset");
Require(imported.Schema.GetField(imported.RootTypeId, "value").Offset == 4, "Metadata inspection");

Use using CStructSharp.Memory.Metadata; and using System.Text; for the importer and the UTF-8 conversion. Take RootTypeId from the result rather than guessing the importer's ID spelling. Review Diagnostics alongside Schema.Types when deciding which imported types can be read by value; a successful import can retain address-only types as pointer targets without making those targets readable.

ISF import requires format 6.2.0. It supports base integers, floats and booleans, structs, classes, unions, arrays, enums, pointers, and member bitfields. Pass the target's pointer width explicitly when it is not eight bytes. A base type's explicit endianness overrides the schema default. maxBytes, maxTypes, and cancellation bound the import; symbol evaluation, relocations, profile matching, and guesses about older formats are outside its scope.

Import BTF and split tables

BTF (BPF Type Format) is a compact binary type description emitted for the Linux kernel and its modules. A BTF blob has a small header, a table of numbered type records, and a string table. Each record has a kind (integer, pointer, struct, and so on), a size or referenced type, and a variable-length payload such as struct members with their bit offsets. Tools usually read the blob from the .BTF section of a kernel image or from /sys/kernel/btf; this API does neither. You obtain the bytes and pass them in.

First supply the blob to new BtfMetadata(bytes), which validates and indexes it. Then FindType("task") locates a unique named type and Import(typeId, pointerSize: 8) compiles that type and everything it references. FindType rejects duplicate names rather than picking one, because two kernel types can share a name while having different layouts. When names are ambiguous, use the numeric ID from the metadata producer.

The blob's magic number tells the parser its byte order (9F EB for little-endian, EB 9F for big-endian). The target's pointer width is a separate setting, because BTF describes types, not the machine word size.

Here is a complete synthetic blob with two types: a four-byte unsigned integer and an eight-byte record whose member starts at byte 4. BTF stores member offsets in bits, so the record says 32 and the importer produces byte offset 4. The comments identify each section; applications normally receive these bytes from a build tool rather than writing them by hand.

// BTF header: little-endian magic, v1, 24-byte header, 40-byte type table, 18-byte string table.
byte[] blob = Convert.FromHexString(
    "9FEB01001800000000000000280000002800000012000000" +
    // Type 1: name offset 1 (u32), INT kind, size 4, unsigned 32-bit encoding.
    "01000000000000010400000020000000" +
    // Type 2: name offset 5 (record), STRUCT kind with one member, size 8.
    "050000000100000408000000" +
    // Member: name offset 12 (value), type 1, bit offset 32 (byte offset 4).
    "0C0000000100000020000000" +
    // UTF-8 strings: empty, u32, record, value; each is zero-terminated.
    "00753332007265636F72640076616C756500");
// Parsing indexes the table; FindType demands a unique name; Import compiles the reachable graph.
var metadata = new BtfMetadata(blob);
uint rootId = metadata.FindType("record");
MetadataImportResult imported = metadata.Import(rootId, pointerSize: 8);
var session = new MemorySession(imported.Schema);
var image = new ByteArrayMemorySource("BTF record", new byte[] { 0xaa, 0xbb, 0xcc, 0xdd, 9, 0, 0, 0 });
Require((uint)session.Read(new MemoryRegion(image, 0, 8), imported.RootTypeId, "value")! == 9, "BTF member placement");
Require(imported.Schema.GetField(imported.RootTypeId, "value").Offset == 4, "BTF bit-to-byte offset");

The decoded value is 9; the first four image bytes are padding. Because the blob was written independently of the importer, the example checks that offsets are interpreted rather than inferred.

A split BTF table extends a base table; the kernel uses this so each module only ships the types it adds. Construct the base BtfMetadata first, then pass it as baseMetadata when constructing the split table. Base type IDs and string offsets keep their identity. Do not concatenate the blobs or renumber IDs yourself.

Supported representations are integers, supported floating-point sizes, arrays, pointers, structs, unions, enums and ENUM64, qualifiers such as const, typedefs, and member bit slices. Function and forward-declaration targets remain address-only. Older BTF encoded bitfields inside the integer type itself; the importer normalizes such legacy slices when they are used as members, but does not offer them as standalone scalars. Any other reachable value kind fails explicitly instead of producing a guessed layout.

BTF gives each bitfield an absolute bit offset counted from the start of the containing struct, not from the storage word it happens to live in. When two bitfields share one storage word (a compact C idiom - for example unsigned long value : 11; unsigned long tasks : 53; packed into a single eight-byte unsigned long), the importer places both at the same byte Offset - the start of that shared word - and tells them apart by BitOffset, the position within the word. A member whose bit offset merely happens to fall in the word's second byte does not get its own byte-1 storage slot; it is still part of the same word its neighbor started.

Describe one type without importing its graph

Import compiles a type and everything it references, recursively - so a single incompatible type anywhere in a large graph fails the whole call, even if the type you actually asked for is perfectly fine. A real kernel's BTF easily reaches thousands of types from one root, so this matters in practice, not just in theory.

Describe(id) answers a narrower question: what does this one type record look like? It reports the kind BTF resolves to (following typedef/const/volatile/restrict/type-tag chains transparently, the same way Import does), its size, and - for a struct or union - its direct members' names, declared type IDs, and placement. It never imports or validates anything a member merely points to; that member's own type ID is reported as-is.

This makes Describe the right tool for finding out what a specific ID actually is - typically while diagnosing why Import rejected something deep in a large graph, by walking one level at a time from the root down toward whichever type is causing trouble - rather than a replacement for Import when you actually want to read values.

// The same blob ImportBtf uses: one integer type and one struct with a single member.
byte[] blob = Convert.FromHexString(
    "9FEB01001800000000000000280000002800000012000000" +
    "01000000000000010400000020000000" +
    "050000000100000408000000" +
    "0C0000000100000020000000" +
    "00753332007265636F72640076616C756500");
var metadata = new BtfMetadata(blob);
uint rootId = metadata.FindType("record");

// Unlike Import, Describe never follows a member into its own type - it reports the declared ID as-is,
// so this succeeds even if "value"'s own type would fail to import.
BtfTypeDescription description = metadata.Describe(rootId);
Require(description.Kind == BtfKind.Struct, "Describe reports the struct kind");
Require(description.Members.Count == 1, "Describe reports one member");
BtfMemberDescription value = description.Members[0];
Require(value.Name == "value" && value.Offset == 4, "Describe reports the same placement Import would");
Require(value.BitOffset is null && value.BitWidth is null, "Describe reports a whole-value (non-bitfield) member");

Import best-effort, tolerating one locally broken type

By default, Import (whether from BTF, ISF, or a hand-built MemorySchema) is all-or-nothing: if any type reachable from the root fails validation - a member placed past its container's extent, a bitfield that does not fit its storage, and similar defects in the source metadata, not in this library - the whole import fails, even when the type you actually asked for is perfectly consistent on its own terms. For a real kernel, where one root like task_struct can reach several thousand types, that means one bad type anywhere in a huge graph blocks everything.

Passing bestEffort: true to MemorySchema's constructor, or to BtfMetadata.Import, changes this: a type that fails validation is demoted in place to a same-sized MemoryTypeKind.Opaque placeholder instead, and the substitution is recorded as a diagnostic rather than thrown. A placeholder keeps the type's real, checked size, so every other definition that embeds it - by pointer or directly by value - still places its own members correctly; only that one type's own contents become unreadable, as raw bytes rather than a decoded value. This is squarely aimed at forensic memory captures, which are often torn or partial - one subsystem's metadata can be locally corrupt while the rest of a large graph remains perfectly readable - rather than at a healthy, complete kernel, which should not need it.

var scalar = new MemoryTypeDefinition("byte", "byte", MemoryTypeKind.Scalar, 1, scalarType: "uint8");

// "inner" is intrinsically broken: its one member sits past inner's own declared one-byte extent. This is
// the shape a torn or partial forensic capture produces - not something a real, complete kernel emits.
var inner = new MemoryTypeDefinition("inner", "inner", MemoryTypeKind.Struct, 1, new[]
{
    new MemoryField("x", "byte", 4),
});

// "outer" is fine on its own terms; it just happens to embed the broken "inner" by value.
var outer = new MemoryTypeDefinition("outer", "Outer", MemoryTypeKind.Struct, 1, new[]
{
    new MemoryField("field", "inner", 0),
});

MemoryTypeDefinition[] types = { scalar, inner, outer };

// Strict validation (the default) fails the whole schema for "inner"'s sake, even though "outer" is fine.
bool strictFailed = false;
try
{
    _ = new MemorySchema(types);
}
catch (ArgumentException)
{
    strictFailed = true;
}

Require(strictFailed, "Strict import fails outright");

// With bestEffort, "inner" is demoted to a same-sized opaque placeholder instead, so "outer" still imports.
var schema = new MemorySchema(types, bestEffort: true);
Require(schema.GetType("inner").Kind == MemoryTypeKind.Opaque, "The broken member becomes opaque");
Require(schema.Diagnostics.Count == 1, "The demotion is reported as a diagnostic");

// An opaque value reads and writes as raw bytes - the size is trustworthy, its layout is not.
var session = new MemorySession(schema);
var image = new ByteArrayMemorySource("capture", new byte[] { 42 });
var read = (global::CStructSharp.Values.StructValue)session.Read(new MemoryRegion(image, 0, 1), "outer")!;
Require(((byte[])read["field"]!)[0] == 42, "Opaque member reads as its raw byte");

Semantic metadata versus compiled storage views

Schema.Types, GetType, and GetField present the imported model: real names, IDs, offsets, bit slices, and Provenance strings that say where a definition came from. Use these in an analyzer's user interface. GetField looks up immediate members; session paths additionally resolve uniquely promoted members.

To decode bytes, the schema also compiles a CompiledLayout: an ordinary Portable CStruct in which every metadata type becomes a union of byte arrays placed at the recorded offsets. This trick lets the memory APIs reuse the core's scalar and bitfield codecs without a second decoder. Its generated names such as m0 and f0 are placement labels, not metadata names. GetCompiledName(id) connects a semantic type to its view when you need to inspect the compilation. Do not show these generated names to users as if the metadata had supplied them.

Check your understanding

Metadata describes struct pair { uint8 a; uint8 b; } with b at offset 0 and a at offset 0. The schema constructor throws. What is wrong, and how would you describe a type where both really do share byte 0?
A four-byte storage unit holds a signed 5-bit slice at bit offset 12. What is the range of values the slice can store, and what does an update to 20 do?
FindType("list_head") throws ArgumentException saying the name is ambiguous. What does that tell you about the blob, and what should your program do?

Answers: struct members may not overlap; declare the type as a Union, whose members are expected to share storage, or give the fields different offsets. -16 to 15, so an update to 20 throws ArgumentOutOfRangeException before any byte is staged. The blob contains two or more types named list_head, perhaps one from the base table and one from a module; ask the metadata producer for the numeric ID you want and call Import with that ID instead of a name.

The overview links the source format specifications and summarizes their limits. Next, use the schema to follow pointers or create and update records.

Table of Contents