How does git know which ssh key to use for its operations?

Git does not know, or care. It just runs ssh.

How does ssh know? It looks at your ~/.ssh/config file (edit: or gets it from ssh-agent; see below):

Host github.com
    # IdentitiesOnly yes # see below to decide if you want this
    IdentityFile ~/.ssh/github_id_file

Host domain.com
    IdentitiesOnly yes # again, see below
    IdentityFile ~/.ssh/another_id_file

Edit: here is a link to a Linux version of the ssh_config documentation. While each system (MacOS, Linux, the various BSDs, even the Windows ports) has its own flavor of ssh config handling, they all share most of these configurables. Note these two items in particular (I have adjusted formatting slightly for StackOverflow markdown):

IdentitiesOnly

      Specifies that ssh(1) should only use the authentication identity files configured in the ssh_config files, even if ssh-agent(1) or a PKCS11Provider offers more identities. The argument to this keyword must be “yes” or “no”. This option is intended for situations where ssh-agent offers many different identities. The default is “no”.

IdentityFile

      Specifies a file from which the user's DSA, ECDSA, ED25519 or RSA authentication identity is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa for protocol version 2. Additionally, any identities represented by the authentication agent will be used for authentication unless IdentitiesOnly is set. ssh(1) will try to load certificate information from the filename obtained by appending -cert.pub to the path of a specified IdentityFile.

      The file name may use the tilde syntax to refer to a user's home directory or one of the following escape characters: ‘%d’ (local user's home directory), ‘%u’ (local user name), ‘%l’ (local host name), ‘%h’ (remote host name) or ‘%r’ (remote user name).

      It is possible to have multiple identity files specified in configuration files; all these identities will be tried in sequence. Multiple IdentityFile directives will add to the list of identities tried (this behaviour differs from that of other configuration directives).

      IdentityFile may be used in conjunction with IdentitiesOnly to select which identities in an agent are offered during authentication.

As Alexey Ten noted in a comment, IdentityFile is peculiar in that it is additive (rather than one-setting-overrides-another).

You can also run ssh (manually) with additional -v options to trace the connection. In Git, you can set GIT_SSH to the name of a script that runs ssh -vvv for a temporary trace (or fuss with the log level in your ~/.ssh/config file). I've found this useful to debug occasionally. (Note that you cannot pass options to ssh via GIT_SSH, you need a one-line script such as ssh-vvv with one line reading ssh -vvv $@.)