Check if the function has been called in gtest

without gmock, use gtest only

That's a very strict restriction. In general, you can't tell if a function was called. Gmock gets around this by generating mock functions that record the calls, arguments and can fake behavior based on runtime parameters.

Without this, you only have two options:

Observe a side effect of the function in question.

This is straightforward, but brittle: if you know there is a observable side effect of the function, you can check that:

class a
{
public: 
    a() : aa_flag(false) {}

    void aa()
    {
        aa_flag = true;
    }

    bool aa_flag;
};

TEST(FuncCalled, CheckSideEffectFlag)
{
    a dut;
    dut.aa();
    EXPECT_TRUE(dut.aa_flag);
}

You don't need to restrict yourself to flags that a function sets. Log messages and other side effects are also workable.

class a
{
public: 
    a() : aa_flag(false) {}

    void aa()
    {
        LOG_INFO("aa called");
    }

    bool aa_flag;
};

TEST(FuncCalled, CheckSideEffectLog)
{
    a dut;
    dut.aa();
    EXPECT_TRUE(LogContains("aa called"));
}

As mentioned above, this is a brittle solution, because you may be checking a side effect that randomly changes in the future. However, sometimes this is good enough.

Hotpatch the function to trace calls

This is nasty and I can't provide a complete example because the way to do this depends on your compiler and target. Basically, you instruct the compiler to generate the functions starting with a few no-op (not necessarily NOP) instructions. This allows you to take the address of the function, change these instructions to jump somewhere and then back. This is very useful because you can call a function that registers the return address and from that you can tell if a function was called or not. It was called iif the return address of aa() is registered.

You will need OS-specific calls to hotpatch your code and some knowledge of the CPU instructions you are running on. You obviously lose portability. I also don't know your requirements, but this probably isn't worth the trouble.

All in all, your best bet is gmock, if you want to stay in the boundaries of the standard. Virtual functions are the standard way of dynamic dispatch (which is what you would do if you were hotpatching your image).