How to access domain users' calendar using service account in .net?

I am using service account with key.p12 cert to access google calendar API. However, it cannot access any user's calendar in the domain. I did follow the steps to Delegating domain-wide authority to the service account https://developers.google.com/accounts/docs/OAuth2ServiceAccount

It does not have code sample for .net. And it seems I only get ServiceAccountCredential in the google .net client library. Here is my code

    static void Main(string[] args)
    {
        string serviceAccountEmail = "[email protected]";
        var certificate = new X509Certificate2(@"clientPrivateKey.p12", "notasecret",  X509KeyStorageFlags.Exportable);

        Console.WriteLine("service account: {0}", serviceAccountEmail);

        ServiceAccountCredential credential = new ServiceAccountCredential(new
        ServiceAccountCredential.Initializer(serviceAccountEmail)
        {
            Scopes = new[] { CalendarService.Scope.Calendar }
        }.FromCertificate(certificate));

        BaseClientService.Initializer initializer = new  BaseClientService.Initializer();            

        initializer.HttpClientInitializer = credential;           
        initializer.ApplicationName = "Google Calendar Sample";
        CalendarService calservice = new CalendarService(initializer);

        // list all the calendars it can see
        try
        {
            var list = calservice.CalendarList.List().Execute();

            if (list.Items.Count > 0)
            {
                foreach(var item in list.Items)
                {
                    Console.WriteLine("Found calendar for account {0}", item.Id);
                }
            }
            else
            {
                Console.WriteLine("Calendar list for this service account is empty");
            }
        }
        catch(Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }

The calendar list is always empty. If I manually share my domain account calendar with this service account in the calendar setting, then this code returns my domain account calendar successfully.

Is there a way to make this service account access all the user's calendar in the domain?