camera frame world coordinates relative to fiducial

What do you mean under world coordinates? If you mean object coordinates then you should use the inverse transformation of solvepnp's result. Given a view matrix [R|t], we have that inv([R|t]) = [R'|-R'*t], where R' is the transpose of R. In OpenCV:

cv::Mat rvec, tvec;
cv::solvePnP(objectPoints, imagePoints, intrinsics, distortion, rvec, tvec);

cv::Mat R;
cv::Rodrigues(rvec, rotation);

R = R.t();          // inverse rotation
tvec = -R * tvec;   // translation of inverse

// camPose is a 4x4 matrix with the pose of the camera in the object frame
cv::Mat camPose = cv::Mat::eye(4, 4, R.type());
R.copyTo(camPose.rowRange(0, 3).colRange(0, 3)); // copies R into camPose
tvec.copyTo(camPose.rowRange(0, 3).colRange(3, 4)); // copies tvec into camPose

Update #1:

Result of solvePnP

solvePnP estimates the object pose given a set of object points (model coordinates), their corresponding image projections (image coordinates), as well as the camera matrix and the distortion coefficients.

The object pose is given by two vectors, rvec and tvec. rvec is a compact representation of a rotation matrix for the pattern view seen on the image. That is, rvec together with the corresponding tvec brings the fiducial pattern from the model coordinate space (in which object points are specified) to the camera coordinate space.

That is, we are in the camera coordinate space, it moves with the camera, and the camera is always at the origin. The camera axes have the same directions as image axes, so

• x-axis is pointing in the right side from the camera,
• y-axis is pointing down,
• and z-axis is pointing to the direction of camera view

The same would apply to the model coordinate space, so if you specified the origin in upper right corner of the fiducial pattern, then

• x-axis is pointing to the right (e.g. along the longer side of your pattern),
• y-axis is pointing to the other side (e.g. along the shorter one),
• and z-axis is pointing to the ground.

You can specify the world origin as the first point of the object points that is the first object is set to (0, 0, 0) and all other points have z=0 (in case of planar patterns). Then tvec (combined rvec) points to the origin of the world coordinate space in which you placed the fiducial pattern. solvePnP's output has the same units as the object points.

Take a look at to the following: 6dof positional tracking. I think this is very similar as you need.