Windows process start and dynamic linking

 I was looking at Apple dyld.

And it struck me as a little wierd, er, surprising.

That they have to do work without benefit of a heap allocator.

It is true, heap allocators require initialization

and code that runs before that initialization must do without heap.

But it struck me that the structure of Windows is a bit more

elegant here, and simpler, possibly smaller, faster, easier to maintain, more general, etc.

(ok, maybe not smaller, due to an "extra" C runtime, but still, this is not much, and provides a lot of value, for example you can printf to the debugger via this code: DbgPrint).

So here is a brief description of Windows process startup and dynamic loading.

There are just a few basic aspects to the structure from which the rest follow.

 - ntdll.dll has a very special relationship with the kernel. (Yes, "dll dll", hereafter just "ntdll").

 - All usermode processes begin in ntdll. (I am ignoring Pico processes.)

   They do *not* begin in executables. No matter what flags

   the executable is built with. There is no "alternate dyld" or "ELF interpreter".

   There is no such thing as a statically linked executable on Windows.

   Well, yes, the executable can just ret or int3. It can even try to

   statically link system service stubs (their interface is sadly unstable).

   It need not have any imports. It can seem statically linked. 

   But execution still starts in ntdll no matter what the executable looks like, and such an executable cannot do much, given the unstable undocumented kernel interface (NtOpenFile, etc.)

(Prior to Windows XP, an executable with no imports would actually crash, attempting to run the address in kernel32.dll that the creater assumed would be mapped in all processes.)

 - Rather, all usermode threads begin in ntdll. There is no specific kernel to user upcall for new processes, only new threads. It suffices.

 - Process initialization occurs by virtue of thread start noticing

   the process has not been initialized. If you create a process suspended,

   and multiple threads in it suspended, and then resume them all "quickly" (NtResumeProcess?), they will race to initialize the process first.

   This is synchronized and safe.

 - ntdll contains statically linked all the system service stubs (NtOpenFile, etc.)

   They are exported from there to the rest of the usermode OS.

   Aside: win32u.dll contains the ones for win32k.sys, for use by user32.dll/gdi32.dll.

 In the past these were statically linked into user32.dll and gdi32.dll but got separated at some point. I don't know how DirectX works. Maybe via gdi32!Escape()?

 - ntdll has no imports. It probably could have some if it was careful, but this is kinda the point.

 - ntdll has thread locals, via an internal non-extensible mechanism. Not declspec(thread), not TlsAlloc.

   All of ntdll's thread locals are allocated along with all usermode threads, by the kernel. This is not really about ntdll per se. These "built in" thread locals are also very efficient to access, fixed register + offset. They should be spent wisely, at least each page of them, as every thread pays for them. This is known as the TEB, the thread environment block. It is at e.g. GS:0 on AMD64, FS:0 for x86, and dedicated registers for other processors. See NtCurrentTeb in winnt.h.

 - The special relationship between ntdll and the kernel extends, such that

   all kernel to user upcalls go through ntdll. For example exception dispatch (KiUserExceptionDispatcher), asynchronous I/O completion (KiUserApcDispatcher), win32k callbacks (KiUserCallbackDispatcher).

 - ntdll contains its own statically linked C runtime. Such as strcmp, bsearch.

   This is partially exported to the system, but is not generally reused. Usermode generally uses the universal C runtime (ucrt) or older msvcrt.dll. They are more complete and support e.g. C++ exception handling. The ntdll C runtime is msvcrt.dll, but ifdef'ed ("not all lines") and selectively built ("not all files"), e.g. to avoid kernel dependencies, though they could work.

   i.e. no fread or malloc, though malloc would be trivial. While this is somewhat wasteful, it is not terrible. This C runtime, libcntpr.lib, is also statically linked to and exported from the kernel, which is why e.g. malloc makes sense to omit (usermode heap is built on NtAllocateVirtualMemory / VirtualAlloc, kernel has those but it is not usually what kernel code wants, kernel "heap" is historically ExAllocatePool, etc.) (In truth, this C runtime was later ifdefed again to separate ntdll and kernel).

 - Other than mapping ntdll into all processes, the kernel either maps the executable and/or passes its path and/or mapped base to ntdll. Passing the path would suffice, since ntdll could map it, just as it maps .dlls.

 - ntdll process initialization then proceeds like so:

   initialize heap (process heap, i.e. GetProcessHeap())

   recursively walk executables imports,

    searching for .dlls

      mapping them (roughly: CreateFile + CreateFileMapping(SEC_IMAGE) + MapViewOfFile)

      resolving their imports

      calling DllMain

  call the executable's entry

Since ntdll has no imports, the only dependencies here, by static construction, are the system services and ntdll itself (being careful to initialize ntdll in the right order, e.g. heap first, but some other things too, like critical section support; critical sections optionally use heap).

- You can step through all this. The "magic" is asking the debugger to stop on module loads, ntdll specifically:

 cdb /xe ld:ntdll.dll foo.exe (or maybe just /xe ld).

This breaks very early in a usermode process, long before main and long before the builtin initial breakpoint.

 - It should be noted that exception dispatch is also in ntdll.

   Dynamic loading has no problem using exceptions.

   (Glossing over: ntdll is written mostly in C. It can use C exceptions. It does

   not have a C++ runtime. The C++ runtime uses the "wrong" kind of thread locals (FlsAlloc), like for rethrow so does not at present work here, but it could, or maybe omit the rethrow functionality; exceptions on Windows at least do not require heap, unlike other systems; they can be used to indicate out of memory).

 - I think Apple could/should merge libSystem and dyld and therefore ease

   the development of dyld, but there may be reasons they are split,

   some functionality I am unaware of. Or maybe it is just too much

   work at this point. Maybe they have static executables that do not use dyld,

   or even libSystem?

not all control paths return a value

 struct Blah {};

#define TRY     {                                                \

                    bool e = true; /* really should be false. */ \

                    try                                          \

                    {                                            \

#define CATCH       }                                           \

                    catch (...)                                 \

                    {                                           \

                        try                                     \

                        {                                       \

#define END             }                                       \

                        catch(const Blah&)                      \

                        {                                       \

                            e = true;                           \

                        }                                       \

                    }                                           \

                    if (e)                                      \

                        throw;                                  \

                }

__declspec(dllexport) int (*g)();

__declspec(dllexport)

int f()

{

    TRY

        return g();

    CATCH

        return g();

    END

}

C:\s>cl /c /GX eh3.cpp

Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29915 for x64

C:\s\eh3.cpp(36) : warning C4715: 'f': not all control paths return a value

Huh? They sure do.

Reserved identifiers in C++.

 Reserved identifiers in C++: Leading underscore followed by capital letter.

Who are they reserved for? Microsoft?

That's a lot of people. Who is arbitrating them? Nobody?

Innocent user code, uses no reserved identifiers:

  #include <unordered_map>

  using namespace std;

  #import "C:\windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.tlb" no_namespace

Older compiler/library:

C:\s>cl /c i.cpp

Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64

c:\s\mscorlib.tlh(2511) : error C2872: '_Mutex' : ambiguous symbol

        could be 'c:\s\mscorlib.tlh(1321) : _Mutex'

        or       '...\9.0\VC\Include\yvals.h(723) : std::_Mutex'

Newer compiler/library:

c:\s>cl /c i.cpp

Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29915 for x64

i.cpp

C:\s\mscorlib.tlh(2686): error C2872: '_Hash': ambiguous symbol

C:\s\mscorlib.tlh(1661): note: could be '_Hash'

C:\...\MSVC\14.28.29910\include\xhash(340): note: or       'std::_Hash'

Rust and the C preprocessor.

I'm a long time C and C++ programmer.

And I enjoy using the C preprocessor also.

Trying out Rust.

But I must have those "tables", and the Rust macros are not good, so I must have the C preprocessor.

Cargo.toml:

[build-dependencies]
cc = "1.0"

build.rs:

use std::env;
use std::fs::File;
use std::io::Write;
use std::path::Path;
extern crate cc;

fn main() {
 let out_dir = env::var("OUT_DIR").unwrap();
 let dest_path = Path::new(&out_dir).join("wasm_instructions.rs");
 let mut f = File::create(&dest_path).unwrap();

 let c = &mut cc::Build::new();
 c.flag("-I.").file("src/wasm_instructions.h");
    
 // Preprocess w/o line numbers
 // -EP for Visual C++
 // -E -P for gcc
 //
 // TODO Compiler detection.
 //
 // Nothing works, so remove line directives ourselves.

 for s in String::from_utf8(c.expand()).unwrap().lines() {
  let bytes = s.as_bytes();
  if bytes.len() > 0 && bytes [0] != b'#' {
   f.write(bytes).unwrap();
   f.write(b"\n").unwrap();
  }
 }
}

The .h file looks like this:

#ifdef FOO


FOO(...)
FOO(...)



#else

#undef FOO
#define FOO(...)
#include __FILE__



#undef FOO
#define FOO(...)
#include __FILE__

#endif

C++ disappointment? Member functions adding parameters?

This seems like such a basic thing:

int foo(int local, int context);​

​

struct A​

{​

    int context;​

​

    int foo (int local)​

    {​

        //return foo (local, context);​ // would be nice, but not legal

        return ::foo (local, context);​ // workaround

    }​

};​

And then wrap it in a namespace:

​

namespace n1​

{​

​

int foo(int local, int context);​

​

struct A​

{​

    int context;​

​

    int foo (int local)​

    {​

        return ::foo (local, context);​ // no longer correct

    }​

};​

​

}​

it seems to me there are two principles violated here:

1. I shouldn't have to disambiguate in the first place. The lexical scopes should be merged. Too ambiguous?

2. If I have to disambiguate, code should not have to know what namespace it is in.

I should be able to wrap existing code in namespaces w/o breaking it.

Epilogues and esp. prologues are probably not what you think.

  The NT/amd64 ABI speaks of function prologues and epilogues, and the rest of the function. 
  Epilogues might not be what you think and prologues almost definitely are not what you think. 


  First, the easier clarification, is that a function an have any number of epilogues. 
  It can have zero epilogues, it can have one epilogue at the end, it can have 
  one epilogue not at the end, and it can have any number of epilogues. 

  An epilogue is not the code located at the end of the function, 
  it is the last thing a function runs -- it is about dynamic execution, 
  not static location. 

  A function will have zero epilogues if it never returns: 

  type no_epilogue.c 
  cl /LD /O2 /GL /GS- no_epilogue.c /link /incremental:no /export:no_epilogue /nod /noentry
  link /dump /disasm no_epilogue.dll  

  C:\> type no_epilogue.c
        cl /LD /O2 /GL /GS- no_epilogue.c /link /incremental:no /export:no_epilogue /nod /noentry
        link /dump /disasm no_epilogue.dll


   void no_epilogue(void (*f)()) { while (1) f(); } 

  Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64 

    0000000180001000: 40 53              push        rbx 
    0000000180001002: 48 83 EC 20        sub         rsp,20h 
    0000000180001006: 48 8B D9           mov         rbx,rcx 
    0000000180001009: 0F 1F 80 00 00 00  nop         dword ptr [rax+0000000000000000h] 
                      00 
    0000000180001010: FF D3              call        rbx 
    0000000180001012: EB FC              jmp         0000000180001010 


   A function can have multiple epilogues if it has an "early return":  
    
  C:\> type multiple_epilogues.c 
       cl /LD /O2 /GL /GS- multiple_epilogues.c /link /incremental:no /export:multiple_epilogues /nod /noentry
       link /dump /disasm multiple_epilogues.dll 

   int multiple_epilogues(int i, int (*f)(void), int (*g)(void)) 
   { 
     if (i) 
      return f(); 
     return g() + g() + g(); 
   } 

  Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64 
    0000000180001000:   push        rdi 
    0000000180001002:   sub         rsp,20h 
    0000000180001006:   mov         rdi,r8 
    0000000180001009:   test        ecx,ecx 
    000000018000100B:   je          0000000180001015 
    000000018000100D:   add         rsp,20h            <== possibly epilog  
    0000000180001011:   pop         rdi                <== epilog 
    0000000180001012:   jmp         rdx                <== epilog  
    0000000180001015:   mov         qword ptr [rsp+30h],rbx  
    000000018000101A:   call        rdi  
    000000018000101C:   mov         ebx,eax  
    000000018000101E:   call        rdi  
    0000000180001020:   add         ebx,eax  
    0000000180001022:   call        rdi  
    0000000180001024:   add         eax,ebx  
    0000000180001026:   mov         rbx,qword ptr [rsp+30h]  
    000000018000102B:   add         rsp,20h               <== possibly epilog  
    000000018000102F:   pop         rdi                   <== epilog 
    0000000180001030:   ret                               <== epilog 

   And this multiple epiloge case accidentally demonstrates the next point. 

   Just as epilogue is not instructions located at the end of a function, 
   prologue is not instructions located at the start of a function. 

   The prologue *instructions* are the instructions that save nonvolatile 
   registers, or adjust rsp (prior to frame pointer establishment -- not alloca), 
   or establish the frame pointer (mov x, rsp). 

   The prologue instructions can and are interleaved with somewhat arbitrary 
   other instructions. The critical requirement is that nonvolatiles be saved 
   before nonvolatiles are changed -- as well as recording rsp adjustment 
   and frame pointer establishment -- such recording being a function 
   of executing the instruction marked as such in the "xdata". 

   The multi-prologue example above has such "dispersed" prologue. 
   Let's look at it again in more detail: 

  C:\> link /dump /unwindinfo /disasm multiple_epilogues.dll 

    Microsoft (R) COFF/PE Dumper Version 14.00.24215.1 

    0180001000:  push rdi               <=== prologue instruction, unsurprising  
    0180001002:  sub  rsp,20h           <=== prologue instruction, unsurprising  
    0180001006:  mov  rdi,r8 
    0180001009:  test ecx,ecx 
    018000100B:  je   0000000180001015 
    018000100D:  add  rsp,20h 
    0180001011:  pop  rdi 
    0180001012:  jmp  rdx 
    0180001015:  mov  qword ptr [rsp+30h],rbx  <=== also a prologue instruction  
    018000101A:  call rdi               <=== offset 1A in the unwind info below  
    018000101C:  mov  ebx,eax 
    018000101E:  call rdi 
    0180001020:  add  ebx,eax 
    0180001022:  call rdi 
    0180001024:  add  eax,ebx 
    0180001026:  mov  rbx,qword ptr [rsp+30h] 
    018000102B:  add  rsp,20h 
    018000102F:  pop  rdi 
    0180001030:  ret
  Function Table (1) 

             Begin    End      Info      Function Name 
    00000000 00001000 00001031 0000208C 
      Unwind version: 1 
      Unwind flags: None 
      Size of prologue: 0x1A     <== This is also telling.
      Count of codes: 4 
      Unwind codes: 
        1A: SAVE_NONVOL, register=rbx offset=0x30 
        06: ALLOC_SMALL, size=0x20 
        02: PUSH_NONVOL, register=rdi 
       /*\ 
        * 
        * 
        * look here 

   The critical information we want to look at is the left most column 
   of the unwind codes. These are the offsets just after prologue instructions. 
   They are reverse sorted by offset, and the underlying data is not fixed 
   size per line shown -- you must always walk them linearly from the start. 

   Offset 2 and 6 are what you expect -- the first two instructions. 
   But offset 1A is quite a bit into the function -- that is a bit surprising when you first see it.

 And another thing. While the specification is that the offsets are just after the instruction that does the nonvolatile save, etc., the requirement and reality are looser. The offset can be later than the save, as long as it is before a change. As well, the location of a save might change between the save and the recorded offset. For example, the compiler will move nonvolatiles into home space, and then adjust rsp, and then or at the same place record that the nonvolatile was saved.

This can be achieved with the multiple_prologue.c example just by compiling with /O1 instead of /O2. Let's see:


  cl /LD /O1 /GL /GS- multiple_epilogues.c /link /incremental:no /export:multiple_epilogues /nod /noentry 
  link /dump /disasm /unwindinfo multiple_epilogues.dll
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Microsoft (R) COFF/PE Dumper Version 14.00.24215.1

  0180001000:   mov   qword ptr [rsp+8],rbx   <==== rbx saved here
  0180001005:   push  rdi
  0180001006:   sub   rsp,20h                 <==== but recorded here
  018000100A:   mov   rdi,r8
  018000100D:   test  ecx,ecx
  018000100F:   je    0000000180001015
  0180001011:   call  rdx
  0180001013:   jmp   0000000180001021
  0180001015:   call  rdi
  0180001017:   mov   ebx,eax
  0180001019:   call  rdi
  018000101B:   add   ebx,eax
  018000101D:   call  rdi
  018000101F:   add   eax,ebx
  0180001021:   mov   rbx,qword ptr [rsp+30h]
  0180001026:   add   rsp,20h
  018000102A:   pop   rdi
  018000102B:   ret
Function Table (1)
           Begin    End      Info      Function Name
  00000000 00001000 0000102C 0000208C
    Unwind version: 1
    Unwind flags: None
    Size of prologue: 0x0A
    Count of codes: 4
    Unwind codes:
      0A: SAVE_NONVOL, register=rbx offset=0x30     <=== rbx save
      0A: ALLOC_SMALL, size=0x20                    <=== two unwind codes with same offset
      06: PUSH_NONVOL, register=rdi


And see how rbx is saved at rsp+8 but recorded as rsp+30, because
rsp changes by 28 between the save and the recorded position.

And this is all ok. If you take an exception between the save and recorded
position of the save, rbx has not been changed, and need not be restored.
Such an exception is rare -- maybe stack overflow -- but the ABI accounts
for exceptions and stack walks from arbitrary instructions.

DLL_THREAD_ATTACH and DLL_THREAD_DETACH are not what they sound like (but are documented correctly)

 DllMain gets called four specific reasons:
  DLL_PROCESS_ATTACH
  DLL_PROCESS_DETACH
  DLL_THREAD_ATTACH
  DLL_THREAD_DETACH

 Ostensibly, THREAD_ATTACH is where you initialize
 any thread locals and THREAD_DETACH is where you clean them up.

 However they do not work as they sound.

 They are documented correctly however.

 DLL_THREAD_ATTACH is only called for threads created after a dll is loaded.
 DLL_THREAD_DETACH is only called for threads that exit while a dll is loaded.

 That leaves DLL_THREAD_ATTACH not called for threads that exist
 before the dll is loaded, and DLL_THREAD_DETACH not called for threads
 that still exist when a dll is unloaded.

 Therefore, if you have a thread local with a constructor, it can be used without being constructed.
 If you have a thread local with a destructor, it might not be called.

 Here is an example:

F:\1>type dll.cpp dll.h dll.def exe.cpp
dll.cpp

 #include <windows.h>
 #include <stdio.h>

 int constructs;
 int destroys;
 __declspec(thread) bool constructed;

 struct A
 {
 A() { constructed = true; printf(" constructed:%d on thread:%d \n", ++constructs, GetCurrentThreadId()); }

 ~A() { printf(" destroyed:%d on thread:%d \n", ++destroys, GetCurrentThreadId()); }

 void F1() { printf(" called on thread:%d constructed:%d \n", GetCurrentThreadId(), (int)constructed); }
 };

 __declspec(thread) A a;

 extern "C" void F1() { a.F1(); }

 PCSTR ReasonString(ULONG Reason)
 {
     switch (Reason)
     {
     case DLL_PROCESS_ATTACH: return "DLL_PROCESS_ATTACH";
     case DLL_PROCESS_DETACH: return "DLL_PROCESS_DETACH";
     case DLL_THREAD_ATTACH: return "DLL_THREAD_ATTACH";
     case DLL_THREAD_DETACH: return "DLL_THREAD_DETACH";
     }
     return "unknown";
 }

 BOOL __stdcall DllMain(HINSTANCE dll, ULONG  reason, PVOID reserved)
 {
     printf(" DllMain dll:%p, reason:%s reserved:%d thread:%d \n", dll, ReasonString(reason), !!reserved, GetCurrentThreadId());
     if (reason == DLL_PROCESS_DETACH)
         printf(" unloading with %d leaked constructions \n", constructs - destroys);
     return TRUE;
 }

dll.h

extern "C" void F1();

dll.def

EXPORTS
F1
exe.cpp

 #include <stdio.h>
 #include <windows.h>
 #include "dll.h"

 decltype(&F1) pf1;

 HANDLE thread[3];
 ULONG threadid[3];
 HANDLE event[3];

 ULONG __stdcall Thread(PVOID p)
 {
     size_t i = (size_t)p;

     // wait for dll/function to be available
     while (!pf1)
         Sleep(1);
     pf1();

     // Indicate this thread is done with the dll.
     SetEvent(event[i]);

     // keep doing more work on this thread -- at least pretend
     if (i == 2)
     {
         printf(" not exiting thread:%d \n", GetCurrentThreadId());
         Sleep(INFINITE);
     }

     printf(" exiting thread:%d \n", GetCurrentThreadId());
     return 0;
 }

 int main()
 {
     size_t i = 0;

     printf(" initial thread:%d \n", GetCurrentThreadId());

     // create thread before loading dll (current thread as well)
     event[i] = CreateEvent(0, 0, 0, 0);
     thread[i] = CreateThread(0, 0, &Thread, (PVOID)i, 0, &threadid[i]);
     printf(" created thread:%d \n", threadid[i]);
     ++i;

     auto const dll = LoadLibrary("dll");
     (PROC&)pf1 = GetProcAddress(dll, "F1");
     pf1();

     // create threads after loading dll
     event[i] = CreateEvent(0, 0, 0, 0);
     thread[i] = CreateThread(0, 0, &Thread, (PVOID)i, 0, &threadid[i]);
     printf(" created thread:%d \n", threadid[i]);
     ++i;

     event[i] = CreateEvent(0, 0, 0, 0);
     thread[i] = CreateThread(0, 0, &Thread, (PVOID)i, 0, &threadid[i]);
     printf(" created thread:%d \n", threadid[i]);
     ++i;

     // Wait for all calls to the dll to be done.
     WaitForMultipleObjects(3, event, TRUE, INFINITE);

     // Wait for some of the threads to exit.
     WaitForMultipleObjects(2, thread, TRUE, INFINITE);

     FreeLibrary(dll);
 }

F:\1>cl /LD /MD /Zi dll.cpp /link /def:dll.def
F:\1>cl  /MD /Zi exe.cpp

F:\1>.\exe.exe
 initial thread:80676
 created thread:70020
 constructed:1 on thread:80676
 DllMain dll:00007FFBDEFD0000, reason:DLL_PROCESS_ATTACH reserved:0 thread:80676
 called on thread:80676 constructed:1
 created thread:66288
 constructed:2 on thread:66288
 DllMain dll:00007FFBDEFD0000, reason:DLL_THREAD_ATTACH reserved:0 thread:66288
 created thread:49672
 called on thread:70020 constructed:0 called on thread:66288 constructed:1
 exiting thread:66288
 exiting thread:70020
 constructed:3 on thread:49672
 DllMain dll:00007FFBDEFD0000, reason:DLL_THREAD_ATTACH reserved:0 thread:49672
 called on thread:49672 constructed:1
 not exiting thread:49672
 destroyed:1 on thread:66288
 DllMain dll:00007FFBDEFD0000, reason:DLL_THREAD_DETACH reserved:0 thread:66288
 DllMain dll:00007FFBDEFD0000, reason:DLL_THREAD_DETACH reserved:0 thread:70020
 destroyed:2 on thread:80676
 DllMain dll:00007FFBDEFD0000, reason:DLL_PROCESS_DETACH reserved:0 thread:80676
 unloading with 1 leaked constructions 

 - Jay