fork() on Windows?

See also:
   http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
   http://www.evanjones.ca/fork-is-dangerous.html

 Perhaps I need not say anything more.

 People often ask me about fork() on Windows.


 It has always been there.


 It is documented in "Inside Windows NT" that the NT kernel developers
  were faced with providing a kernel that could support OS/2, Posix, Win16, and Win32.
  Rather than implement four things, or various special cases, they implemented
  essentially the union of what they needed. [citation needed].

  (For example, where Posix wanted case sensitivity and the others did not, case sensitivity is a flag when opening objects. Another example, that I lack details on, is unusual OS/2 "mutex" properties, that lead to the NT kernel "mutex" being called a "mutant", since it supports OS/2's requirements.)

 Therefore, fork has been there all along.

 The Posix subsystem used it.

 The later Interix/SFU/SUA product used it. (I ported some code both to Cygwin and Interix, and fast/reasonable fork() made the Interix port much more pleasent to use.)

 No doubt the current WSL Windows Subsystem for Linux uses it.

 So, great, it is there. I can just use it?

 Not so fast.

 There are problems with using it.
 So Windows isn't very good or something?
 People are quick to jump to strong negative conclusions.
 Just what are the problems?

 There are at least two problems.
 The first kind is that it isn't exposed via a public API.

 But that isn't the larger problem.

 The larger problem is that in my opinion, to large amounts of reasonable programmers' and code,
 the semantics of fork() are wierd, and things won't work.

 Again, we wonder, Bad programmers? Bad assumptions? Bad code?
 I don't think so.


 Backing up for a short time.
 Consider that most uses of fork are fork + exec.
 This is also known as spawn or posix_spawn or CreateProcess.
 This works fine -- just jump ahead to spawn/CreateProcess.
 If you call fork, and you are going to call exec, the system
 can't know if you if you are going to call exec or not.


 This leaves us with, what is fork without exec for?
 It is for "forking servers", that could be multi-threaded servers,
 but might want to trade performance for separation/reliablity.

 It is for threads prior to the existance of pthreads. Use pthreads.


 It is, I suspect, for creating unused children to diagnose the parent.
 For example, dumping what files it has open.
 "Reliable walk of handle table of live process is racy. Reliable walk of handle table of non-executing forked child is not."

 So, I claim, fork without exec does have some uses cases, but
 the overwhelming use is fork + exec, which you can do otherwise.


 Ok, going back to what is wrong with fork.


 Let's imagine -- this isn't true, but let's imagine, that getpid
 or GetCurrentProcessId() is slow. This is approximately two instructions
 on Windows -- disassemble kernel32!GetCurrentProcessId. But let's assume
 the opposite. This is a valid though exercise, as it leads us into
 real world patterns.



 So, we want to mitigate the assumed slowness of GetCurrentProcessId().


 We write:

   static int fast_pid;

   int FastGetCurrentProcessId()
   {
      int pid = fast_pid;
      if (pid == 0)
         fast_pid = pid = GetCurrentProcessId(); // incorrectly assumed slow
      return pid;
   }

 Note: This is correct and thread safe and everything.
 It assumes a pid of 0 isn't valid, which is incorrect, but it works anyway, just slower, for pid == 0.
 In a race conditon where FastGetCurrentProcessId is called at about the same
 time on multiple threads, that is ok, they all do the same thing, just all "slowly"..


 Now, enter fork().

 In the face of fork(), this function is incorrect.
 Its cache of the pid will  survive fork() and be incorrect.


 Now, while this function is ridiculous, in optimizing something that isn't slow, it is a representative of a reasonable pattern:
   Compute something once; cache it; the result might be process-dependent.
 (And by the way the result might be thread or other-dependent, and still the cache
 will be used across later calls -- be aware and be careful.)


The more general thing is the assumption that a "process"'s globals
are "initialized" "as expected" at the start of the "process" (or when a dll loads, rather).
For some definition of "process".

This assumption, which is a reasonable and useful idea to bake into your
mental model, permeating much of your code, is violated by fork().

But, it gets worse.

If you read various man pages and standards, you discover more problems that fork()
brings to the unsuspecting coder. [citation needed]

Apparently at some point, a question was raised as to how fork() interacts with pthread_create_thread.
Does the new process contain just one thread, that called fork(), or all the threads?

The decision was apparently so unclear that Solaris has fork1 and forkall, so both options
are available to the explicit caller, and fork()'s behavior has varied between them
depending on operating system version (changed in 5.10 to be fork1()).


The Posix standard mandates "fork1".
[citation needed -- try "Open Group Posix standard fork", and for that matter,
read their documentation of pthread_atfork and see the references to forkall]


But wait, there is more.


These days, we have threads, we use them.
Arbitrary libraries loaded into our process might create worker threads on their behalf.
This is a fine thing for them to do. It is a buried implementation detail.


And then you come along and fork. What happens?


It depends. What were the worker threads in the middle of doing?
Did they hold any locks?


Lock holding may or may not be inherited by the new process, but only one thread
will be inherited. So inheriting held locks leads typically to deadlock.
The thread holding the lock is gone and will never release it.


Apple's documentation covers this problem, in a fashion, by saying
that if you fork without exec, and use almost any of their libraries,
it is unsupported. [citation needed]


Posix covers this also in a fashion.
They provide the function pthread_atfork. [citation needed]
This takes three callback functions -- ParentBeforeFork, ParentAfterFork, ChildAfterFork.


The idea is, in the parent before the fork, acquire all locks (blocking).
In the parent and child after the fork, release all locks.
And in child reinitialize globals such as "fast_pid" to zero?


So apparently the problem is solved, but Apple did not bother to use the solution.
Who else does not?


So, in answer to the question: Yes, Windows has fork. You can't really use it,
and you don't really want to. On systems that leave it for you to use, be careful.

Easing maintenance of class template parameter lists by deriving from them.

 This seems to me related to "template traits parameters"
 and "the curiously recurring template pattern", but it is a bit different than both.

 So I have just a few explict template instantiations,
 and my code is mostly not in headers, so previously I had:

 template <typename T1, bool f1, bool f2, etc.>
 class foo_t
 {
   void Function1();
   void Function2();
   etc. a fair number
 };


 and then

 template <typename T1, bool f1, bool f2, etc.>
 void class foo_t<T1, f1, f2>::Function1() { ... }

 template <typename T1, bool f1, bool f2, etc.>
 void class foo_t<T1, f1, f2>::Function2() { ... }

 etc. a fair number

 This bugged me for two reasons:

 1. The usual unnamed parameter problem, what does true/false mean:

 foo_t<x, true, false> mean.

 and then in the debugger:
   foo_t<x, 1, 0>

 Functions have the same problem.

 Coworker highlighted this actually.

 2. As there were a fair number of functions, changing the template parameter list was tedious. Because, again, I had a fair number of functions, in a separate .cpp file.

 So, I finally decided to use variadic templates and I now have:

 template <typename ...Types> 
 struct foo_t : Types... { ... }; 


 foo_t inherits from its template parameters.

 Now changing the parameter list is not tedious.

 Instantiations are like:

 foo_t<fooarg1<true>, fooarg2<false>>

 The factoring of the template parameter-dependantness can take multiple forms.

 They include:

 template <bool f> struct fooarg1;

 template <> struct fooarg1<false> { implement stuff here };
 template <> struct fooarg1<true> { implement stuff here };

 template <bool f> struct fooarg2 { enum { arg2 = f }; };


 template <typename ...Types>
 void foo_t<Types...>::function1() { if (arg2) ... }


 This second case might look like a conditional branch, but any half decent
 compiler will optimize it.


 Debugger now says foo_t<x, fooarg1<0>, fooarg2<1>> etc.

 and changing the template parameter list requires only localized edits.
 (Again, I have very few instantiations).


 There were other details as to how this affected my interaction with assembly,
 this it shouldn't matter for typical applications (Yes, I use C++ templates and assembly).


 - Jay

combing git repositories with git and svn

 - Combining GIT repositories.
 - Combining GIT repositories and SVN repositories.
 - Keeping git svn working on Mac OS X.

 I have:
 
  git1/stuff/something.c
  git2/other/blah.c
  svn3/project/foo.c
 
In particular, I have three repositories, with no overlap
 in file paths.
 
  I wish to have:
 
   git1/stuff/something.c
   git1/other/blah.c
   git1/project/foo.c
 
  I do NOT wish to push stuff under a new root:
      gitnew/git1/stuff/something.c I do NOT want this.
      gitnew/git2/other/blah.c      I do NOT want this.
      gitnew/svn3/project/foo.c     I do NOT want this.
     
  NOT even picking one of the git repositories
  to house the others in subdirectories:
      git1/stuff/something.c    This is ok.
      git1/git2/other/blah.c    I do NOT want this.
      git1/svn3/project/foo.c   I do NOT want this.

 I will likely have no further use of git2 and svn3.
 This is a one time one direction conversion.
 
  All the repositories are small.
  All have a small number of changelists.
  All have no branches.
  All have just one committer -- me.
  All have little commercial interest.
 
  If I messed up, it would be ok.
  If I lost history, it would be ok.
 
  But ideally, no mess up, and history is preserved.

 I have very little experience with either git or subversion.

 This stuff is mostly documented online by others.

  Some things I did not do, or tried but gave up on:
    http://stackoverflow.com/questions/10014054/git-2-svn-migration (NOT)
   
    http://jasonkarns.com/blog/merge-two-git-repositories-into-one/     (NOT)
      Close, but several of these steps didn't work or weren't needed.
      My git doesn't have a "ci" command.
      read-tree -- more obscure stuff, that I didn't use.

 I tried this briefly, no luck:
   http://www.subgit.com/  (NOT)

 I wanted to use a minimal of commands that made a maximum amount of sense to me.
 I acknowledge that I don't understand git much and haven't use svn much.
 I use Perforce usually.


 The basic steps are:
   convert svn to new temporary git repository
   add new svn-converted git repository as a remote, and pull it
   add "git2" as a remote, and pull it


  You have to say --allow-unrelated-histories.
 
  git svn is typically broken on Mac OS X.
  Some error about perl SVN::Core.pm.
 

  Everyone says to make some links, but that didn't work.
  I only had the developer tool command line tools installed.
  Everyone else uses Xcode.

 
  Solution is to adjust the links, or give in and install Xcode.
 
  I also tried to install svn, but it failed to build because
  something about my httpd. I was going to build that, but, ended up
  just installing Xcode.
 

  I did not use anything related submodules or subtrees or grafting.
  I don't know what they are, so it is nice to not use them.
 

  Github automates the svn to git conversion and that conversion
  worked for me, but I also had done it myself on the command line
  and that worked. I used the local command line conversion instead
  of the github one.
 
 
  Resembling: http://john.albin.net/git/convert-subversion-to-git
 
  I did something like this:
 
  git svn clone [SVN repo URL] gitsvn
  git clone ... git2
  cd git1
  git remote git2 ../git2
  git remote add gitsvn ../gitsvn
  git pull git2 master --allow-unrelated-histories
  git pull gitsvn master --allow-unrelated-histories


It is quite possible I would have been better off with the github
gui for part of this. In particular, it prompts for the author stuff.