C++ How can I return an unknown derived class?

Once an object is created, it is not possible to morph its type (e.g. based on information about a button obtained at run time). Returning a Rectangle as a Shape has the effect of slicing the object, so the caller only receives a copy of the Shape part of the Rectangle, but not the remaining parts.

Assuming Shape is a polymorphic base class (i.e. it provides virtual functions that may be specialised by derived classes), a Shape * (pointer to shape) can point at a Rectangle. However, it is then necessary to manage lifetime of the object correctly.

You can deal with all this by using a smart pointer such as, in C++11 and later, std::unique_pointer<Shape>.

std::unique_pointer<Shape> updateShape(int button,
        std::unique_pointer<Shape> active, bool leftClick)
{
    switch(button)
    {
        case 1:
             break;    // don't change active
        case 2:
            active = new Line(active->getFillColor(), active->getBorderColor(), active->getP1(), active->getP2(),false);
            break;
        case 3:
            active = new Rectangle(active->getFillColor(), active->getBorderColor(), active->getP1(), active->getP2(),false);
            break;
        case 4:
            active = new FilledRectangle(active->getFillColor(), active->getBorderColor(), active->getP1(), active->getP2(),false);
            break;
        case 5:
            active = new Ellipse(active->getFillColor(), active->getBorderColor(), active->getP1(), active->getP2(),false);
            break;
        case 6:
            active = new FilledEllipse(active->getFillColor(), active->getBorderColor(), active->getP1(), active->getP2(),false);
            break;
        default:
            if(leftClick)
            {
                active->setColor(getEnumColor(button), active->getBorderColor());
            }
            else
            {
                active->setColor(active->getFillColor(), getEnumColor(button));
            }
            break;
     }
     return active;
}

The reason this works is that std::unique_pointer manages the lifetime of a dynamically allocated (created with operator new) object. It does this by storing a pointer to the object, and assigning to a std::unique_pointer<Shape> changes the pointer (to make it point at a different object). Importantly, assigning to a smart pointer also releases the object it previously managed.

Note that, since std::unique_pointer will use operator delete to destroy the contained object when its lifetime is over, Shape MUST have a virtual destructor. Failure to do that will cause undefined behaviour on the use of operator delete.

The usage of this would go something like

 std::unique_pointer<Shape> activeShape(new Rectangle( whatever_parameters));

 activeShape = updateShape(button, activeShape, leftClick);

Bear in mind that activeShape is a smart pointer. So use of the contained Shape object requires pointer syntax (activeShape->whatever) not member syntax (activeShape.whatever).

Because the active parameter is passed (by value) to the function and returned, it is NECESSARY to assign the return value. If, instead of

activeShape = updateShape(button, activeShape, leftClick);

you simply

updateShape(button, activeShape, leftClick);   // returned object is discarded

(i.e. don't assign the return value to anything), the net effect is that the object held by activeShape will be destroyed, and any attempt to use it will give undefined behaviour.