[kernel32.dll]CreateHardLink fails with "Cannot create a file when that file already exists" on Azure Website during RavenDb Embedded backup

I'm running RavenDb.Embedded v2.0.2370 inside an Azure Web site.

All of this is working as expected except for the backup. The backup routine initiated with EmbeddableDocumentStore.DocumentDatabase.StartBackup(...) used to work perfectly until I noticed that the last successful backup, dates from around the 30th of January (don't ask :-)) around 6 PM UTC0 and that the next backup initiated on the 11th of February around 8 AM UTC0 failed, as do all the other ones initiated afterwards up until now.

So it seems that something must have been changed in between those 2 dates on the side of Azure, as there have been no changes or deployments of the web application since May 2013

So I did some investigation and here are my findings:

• The RavenDb backup status shows the following message after each initiated backup:

  Failed to complete backup because: Cannot create a file when that file already exists

• After enabling logging for RavenDb the log file shows the following error when running the backup

  2014-04-12 10:38:20.5797,Raven.Storage.Esent.Backup.BackupOperation,Error,Failed to complete backup,"System.ComponentModel.Win32Exception (0x80004005): Cannot create a file when that file already exists at Raven.Database.Backup.DirectoryBackup.Prepare() at Raven.Storage.Esent.Backup.BackupOperation.Execute(Object ignored)

• According to the Azure Web Sites forum there has been no planned maintenance of the Azure Web Sites in the period between the last successful and the first failing backup. There have been between 9-15 December 2013 and between 10-14 March 2014

• I did notice on the Azure Service Dashboard an issue with the Web Sites on the 31th of January in the West European Region but the site is hosted in the North European Region

• When looking (even the last version) at the faulting RavenDb code Raven.Database.Backup.DirectoryBackup.Prepare it seems that the exception is being thrown when calling CreateHardLink which is an external method defined as

  [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
  private static extern bool CreateHardLink(string lpFileName, string lpExistingFileName, IntPtr lpSecurityAttributes);
  

  The strange thing is that the code actually checks for the specific exception shown above (0x80004005), yet the exception is still thrown...

  if (Marshal.GetLastWin32Error() != 0x80004005)
    throw new Win32Exception();
  

• So to exclude RavenDb from the equation, I created the following small program that simulates what RavenDb is actually doing during its backup routine, and after deploying the executable and executing it (through the Console feature found on the new azure portal) on the Azure Web Site it fails as well:

  using System;
  using System.ComponentModel;
  using System.IO;
  using System.Runtime.InteropServices;
  
  namespace HardLinkCreationTester
  {
      public class Program
      {
          [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
          private static extern bool CreateHardLink(string lpFileName, string lpExistingFileName, IntPtr lpSecurityAttributes);
  
          static void Main()
          {
              const string sourcePath = "TestSource";
              const string targetPath = "TestTarget";
              const string tempPath = "TestTemp";
  
              try
              {
                  Directory.CreateDirectory(sourcePath);
                  Directory.CreateDirectory(targetPath);
                  Directory.CreateDirectory(tempPath);
                  File.WriteAllText(Path.Combine(sourcePath, "testfile.txt"), "Test content");
  
                  string[] sourceFilesSnapshot = Directory.GetFiles(sourcePath);
                  for (int index = 0; index < sourceFilesSnapshot.Length; index++)
                  {
                      var sourceFile = sourceFilesSnapshot[index];
                      var destFileName = Path.Combine(tempPath, Path.GetFileName(sourceFile));
  
                      if (!CreateHardLink(destFileName, sourceFile, IntPtr.Zero))
                      {
                          // 'The system cannot find the file specified' is explicitly ignored here
                          if (Marshal.GetLastWin32Error() != 0x80004005)
                              throw new Win32Exception();
                      }
                  }
              }
              finally
              {
                  Directory.Delete(sourcePath);
                  Directory.Delete(targetPath);
                  Directory.Delete(tempPath);
              }
          }
      }
  }
  

• To exclude my specific web application from the equation as well, I created a new Azure Web Site and deployed the same executable mentioned above and it fails as well

• It's fair to assume that they are not yet running on ReFS (which does not support hard links) but are still using NTFS?

tl;dr

• What has changed on Azure Web Sites that makes a call to [kernel32.dll]CreateHardLink fail while it used to work before?

• How can this be fixed outside RavenDb code or inside RavenDb code...?