How can I install a macOS VM Guest under VirtualBox on a Mac host?

Here's the configuration I installed:

Late 2015 Retina iMac (probably any new enough Mac will work)
----- VirtualBox 5.2.19 (no foreseeable reason future revs of VirtualBox won't work)
--------- macOS High Sierra 10.13.6 (probably works with several macOS release versions)

1. First download High Sierra installer from app store. (App store app puts it into /Applications and launches it. The autorun installer can be quit so that it can be converted to an ISO)

2. You can convert the installer DMG to an ISO, using the script at the end of this answer. Some other approaches work too. You may need or want to tweak the file names or the name of the OS/Installer app depending on what OS you're trying to install, etc...

3. After creating the ISO, create a Virtual Machine with the following configuration settings:

OS Type: macOS 10.13 High Sierra (64-bit)
Base Memory: 4096 GB RAM
Disk Size: 32 GB disk size
Video RAM: 128 MB
2 CPUs
disable floppy
disable serial port
disable audio

From command line create the following VirtualBox extra data values for the VM.

NOTE: These seemed to be critical for me to get the installer to boot and run.

These extra data settings I found to be omitted from most or all of the Mac installation instructions that I found online. I borrowed these values from Windows VirtualBox install instructions, and modified the system product to match my iMac's (iMac17,1). Without these extra data settings, the installer failed mid-boot, infinitely looping with some error pertaining to trying to reboot MACH).

    VBoxManage setextradata  {vm name or ID} "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac17,1"
    VBoxManage setextradata  {vm name or ID} "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
    VBoxManage setextradata  {vm name or ID} "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
    VBoxManage setextradata  {vm name or ID} "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
    VBoxManage setextradata  {vm name or ID} "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1

4. Point the VM to the ISO you created in step 2, and boot it.

When the recovery partition's GUI comes up (wherein you have the option to reinstall (also used for initial install)), first choose Disk Utility, and find and format (e.g. Erase), the virtual disk you created during the initial creation of the VirtualBox Guest VM. In Disk Utility, the hard disk will show up as a disk on the left. Erase/format it as Mac OS Extended Journaled.

If you don't first explicitly format the virtual disk, the macOS installer will not find any disks to install macOS on. If you haven't formatted the disk, the installer will not err out but will just sit there expecting you to choose a disk, with none listed, and no option to continue with the installation.

After the first reboot during installation, I got stuck at the UEFI shell prompt, which is obtuse and confusing for the uninitiated (i.e. I didn't know what to do about it).

The reason I got dumped in the UEFI boot shell is that macOS wasn't installed yet in the boot partition, and the boot loader doesn't automatically boot into the recovery partition if the first partition isn't valid.

To manually boot/execute code in recovery partition in order to complete the installation, do the following (credit to VirtualBox forums for this information):

1. At UEFI shell prompt: Type exit
2. You'll be brought into an EFI text-mode GUI.
3. Select Boot Maintenance Manager and click.
4. Select Boot From File and click

You should see a list populated with two cryptic looking entries (PCI bus paths)

The first is the PCI path to a boot partition that is not valid because you haven't installed macOS on it yet. After installing macOS the boot should find that partition and boot normally, without dumping you out at the UEFI shell.

The 2nd partition is the recovery partition, the one you need to boot from to do the macOS installation.

• Click the 2nd entry, you should see (and then click): macOS Install Data
• Then click: Locked Files
• And finally click: boot.efi

At that point the installer boot will continue and you should get to the point where macOS can be installed.

Bash script to convert macOS installer app downloaded from app store to ISO

#!/bin/bash
installer="/Applications/Install macOS High Sierra.app"    
hdiutil create -o /tmp/HighSierra.cdr -size 5500m -layout SPUD -fs HFS+J
hdiutil attach /tmp/HighSierra.cdr.dmg -noverify -mountpoint /Volumes/install_build
sudo "$installer"/Contents/Resources/createinstallmedia --volume /Volumes/install_build
mv /tmp/HighSierra.cdr.dmg ~/Downloads/InstallSystem.dmg
hdiutil detach /Volumes/Install\ macOS\ High\ Sierra
hdiutil convert ~/Downloads/InstallSystem.dmg -format UDTO -o ~/Downloads/HighSierra.iso