.NET P/Invoke

Workflow

Step 1: Choose DllImport or LibraryImport

| Aspect | DllImport | LibraryImport (.NET 7+) | |--------|-------------|---------------------------| | Mechanism | Runtime marshalling | Source generator (compile-time) | | AOT / Trim safe | No | Yes | | String marshalling | CharSet enum | StringMarshalling enum | | Error handling | SetLastError | SetLastPInvokeError | | Availability | .NET Framework 1.0+ | .NET 7+ only |

Step 2: Map Native Types to .NET Types

The most dangerous mappings — these cause the majority of bugs:

| C / Win32 Type | .NET Type | Why | |----------------|-----------|-----| | long | `CLong` | 32-bit on Windows, 64-bit on 64-bit Unix. With LibraryImport, requires [assembly: DisableRuntimeMarshalling] | | size_t | nuint / UIntPtr | Pointer-sized. Use nuint on .NET 8+ and UIntPtr on earlier .NET. Never use ulong | | BOOL (Win32) | int | Not bool — Win32 BOOL is 4 bytes | | bool (C99) | [MarshalAs(UnmanagedType.U1)] bool | Must specify 1-byte marshal | | HANDLE, HWND | SafeHandle | Prefer over raw IntPtr | | LPWSTR / wchar_t* | string | UTF-16 on Windows (lowest cost for in strings). Avoid in cross-platform code — wchar_t width is compiler-defined (typically UTF-32 on non-Windows) | | LPSTR / char* | string | Must specify encoding (ANSI or UTF-8). Always requires marshalling cost for in parameters |

For the complete type mapping table, struct layout, and blittable type rules, see references/type-mapping.md.

> ❌ NEVER use int or long for C long — it's 32-bit on Windows, 64-bit on Unix. Always use CLong. > ❌ NEVER use ulong for size_t — causes stack corruption on 32-bit. Use nuint or UIntPtr. > ❌ NEVER use bool without MarshalAs — the default marshal size is wrong.

Step 3: Write the Declaration

Given a C header:

int32_t process_records(const Record* records, size_t count, uint32_t* out_processed);

DllImport:

[DllImport("mylib")]
private static extern int ProcessRecords(
    [In] Record[] records, UIntPtr count, out uint outProcessed);

LibraryImport:

[LibraryImport("mylib")]
internal static partial int ProcessRecords(
    [In] Record[] records, nuint count, out uint outProcessed);

Calling conventions only need to be specified when targeting Windows x86 (32-bit), where Cdecl and StdCall differ. On x64, ARM, and ARM64, there is a single calling convention and the attribute is unnecessary.

Agent behavior: If you detect that Windows x86 is a target — through project properties (e.g., <PlatformTarget>x86</PlatformTarget>), runtime identifiers (e.g., win-x86), build scripts, comments, or developer instructions — flag this to the developer and recommend explicit calling conventions on all P/Invoke declarations.

// DllImport (x86 targets)
[DllImport("mylib", CallingConvention = CallingConvention.Cdecl)]

// LibraryImport (x86 targets)
[LibraryImport("mylib")]
[UnmanagedCallConv(CallConvs = [typeof(CallConvCdecl)])]

If the managed method name differs from the native export name, specify EntryPoint to avoid EntryPointNotFoundException:

// DllImport
[DllImport("mylib", EntryPoint = "process_records")]
private static extern int ProcessRecords(
    [In] Record[] records, UIntPtr count, out uint outProcessed);

// LibraryImport
[LibraryImport("mylib", EntryPoint = "process_records")]
internal static partial int ProcessRecords(
    [In] Record[] records, nuint count, out uint outProcessed);

Step 4: Handle Strings Correctly

1. Know what encoding the native function expects. There is no safe default.
2. Windows APIs: Always call the W (UTF-16) variant. The A variant needs a specific reason and explicit ANSI encoding.
3. Cross-platform C libraries: Usually expect UTF-8.
4. Specify encoding explicitly. Never rely on CharSet.Auto.
5. Never introduce `StringBuilder` for output buffers.

> ❌ NEVER rely on CharSet.Auto or omit string encoding — there is no safe default.

// DllImport — Windows API (UTF-16)
[DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)]
private static extern int GetModuleFileNameW(
    IntPtr hModule, [Out] char[] filename, int size);

// DllImport — Cross-platform C library (UTF-8)
[DllImport("mylib")]
private static extern int SetName(
    [MarshalAs(UnmanagedType.LPUTF8Str)] string name);

// LibraryImport — UTF-16
[LibraryImport("kernel32", StringMarshalling = StringMarshalling.Utf16,
    SetLastPInvokeError = true)]
internal static partial int GetModuleFileNameW(
    IntPtr hModule, [Out] char[] filename, int size);

// LibraryImport — UTF-8
[LibraryImport("mylib", StringMarshalling = StringMarshalling.Utf8)]
internal static partial int SetName(string name);

String lifetime warning: Marshalled strings are freed after the call returns. If native code stores the pointer (instead of copying), the lifetime must be manually managed. On Windows or .NET Framework, CoTaskMemAlloc/CoTaskMemFree is the first choice for cross-boundary ownership; on non-Windows targets, use NativeMemory APIs. The library may have its own allocator that must be used instead.

Step 5: Establish Memory Ownership

When memory crosses the boundary, exactly one side must own it — and both sides must agree.

> ❌ NEVER free with a mismatched allocator — Marshal.FreeHGlobal on malloc'd memory is heap corruption.

Model 1 — Caller allocates, caller frees (safest):

[LibraryImport("mylib")]
private static partial int GetName(
    Span<byte> buffer, nuint bufferSize, out nuint actualSize);

public static string GetName()
{
    Span<byte> buffer = stackalloc byte[256];
    int result = GetName(buffer, (nuint)buffer.Length, out nuint actualSize);
    if (result != 0) throw new InvalidOperationException($"Failed: {result}");
    return Encoding.UTF8.GetString(buffer[..(int)actualSize]);
}

Model 2 — Callee allocates, caller frees (common in Win32):

[LibraryImport("mylib")]
private static partial IntPtr GetVersion();
[LibraryImport("mylib")]
private static partial void FreeString(IntPtr s);

public static string GetVersion()
{
    IntPtr ptr = GetVersion();
    try { return Marshal.PtrToStringUTF8(ptr) ?? throw new InvalidOperationException(); }
    finally { FreeString(ptr); } // Must use the library's own free function
}

Critical rule: Always free with the matching allocator. Never use Marshal.FreeHGlobal or Marshal.FreeCoTaskMem on malloc'd memory.

Model 3 — Handle-based (callee allocates, callee frees): Use SafeHandle (see Step 6).

Pinning managed objects — when native code stores the pointer or runs asynchronously:

// Synchronous: use fixed
public static unsafe void ProcessSync(byte[] data)
{
    fixed (byte* ptr = data) { ProcessData(ptr, (nuint)data.Length); }
}

// Asynchronous: use GCHandle
var gcHandle = GCHandle.Alloc(data, GCHandleType.Pinned);
// Must keep pinned until native processing completes, then call gcHandle.Free()

Step 6: Use SafeHandle for Native Handles

Raw IntPtr leaks on exceptions and has no double-free protection. SafeHandle is non-negotiable.

internal sealed class MyLibHandle : SafeHandleZeroOrMinusOneIsInvalid
{
    // Required by the marshalling infrastructure to instantiate the handle.
    // Do not remove — there are no direct callers.
    private MyLibHandle() : base(ownsHandle: true) { }

    [LibraryImport("mylib", StringMarshalling = StringMarshalling.Utf8)]
    private static partial MyLibHandle CreateHandle(string config);

    [LibraryImport("mylib")]
    private static partial int UseHandle(MyLibHandle h, ReadOnlySpan<byte> data, nuint len);

    [LibraryImport("mylib")]
    private static partial void DestroyHandle(IntPtr h);

    protected override bool ReleaseHandle() { DestroyHandle(handle); return true; }

    public static MyLibHandle Create(string config)
    {
        var h = CreateHandle(config);
        if (h.IsInvalid) throw new InvalidOperationException("Failed to create handle");
        return h;
    }

    public int Use(ReadOnlySpan<byte> data) => UseHandle(this, data, (nuint)data.Length);
}

// Usage: SafeHandle is IDisposable
using var handle = MyLibHandle.Create("config=value");
int result = handle.Use(myData);

Step 7: Handle Errors

// Win32 APIs — check SetLastError
[LibraryImport("kernel32", SetLastPInvokeError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static partial bool CloseHandle(IntPtr hObject);

if (!CloseHandle(handle))
    throw new Win32Exception(Marshal.GetLastPInvokeError());

// HRESULT APIs
int hr = NativeDoWork(context);
Marshal.ThrowExceptionForHR(hr);

Step 8: Handle Callbacks (if needed)

Preferred (.NET 8+): `UnmanagedCallersOnly` — avoids delegates entirely, no GC lifetime risk:

[UnmanagedCallersOnly]
private static void LogCallback(int level, IntPtr message)
{
    string msg = Marshal.PtrToStringUTF8(message) ?? string.Empty;
    Console.WriteLine($"[{level}] {msg}");
}

[LibraryImport("mylib")]
private static unsafe partial void SetLogCallback(
    delegate* unmanaged<int, IntPtr, void> cb);

unsafe { SetLogCallback(&LogCallback); }

The method must be static, must not throw exceptions back to native code, and can only use blittable parameter types.

Fallback (older TFMs or when instance state is needed): delegate with rooting

[UnmanagedFunctionPointer(CallingConvention.Cdecl)] // Only needed on Windows x86
private delegate void LogCallbackDelegate(int level, IntPtr message);

// CRITICAL: prevent delegate from being garbage collected
private static LogCallbackDelegate? s_logCallback;

public static void EnableLogging(Action<int, string> handler)
{
    s_logCallback = (level, msgPtr) =>
    {
        string msg = Marshal.PtrToStringUTF8(msgPtr) ?? string.Empty;
        handler(level, msg);
    };
    SetLogCallback(s_logCallback);
}

If native code stores the function pointer, the delegate must stay rooted for its entire lifetime. A collected delegate means a crash.

`GC.KeepAlive` for short-lived callbacks: When converting a delegate to a function pointer with Marshal.GetFunctionPointerForDelegate, the GC does not track the relationship between the pointer and the delegate. Use GC.KeepAlive to prevent collection before the native call completes:

var callback = new LogCallbackDelegate((level, msgPtr) =>
{
    string msg = Marshal.PtrToStringUTF8(msgPtr) ?? string.Empty;
    Console.WriteLine($"[{level}] {msg}");
});

IntPtr fnPtr = Marshal.GetFunctionPointerForDelegate(callback);
NativeUsesCallback(fnPtr);
GC.KeepAlive(callback); // prevent collection — fnPtr does not root the delegate

---

Related skills