Box rotation around multiple axises using Matrix4f

Billboards are a very common application of computer graphics (as I'm sure you've noticed, since you're asking the question!)

Ultimately I think you are over complicating the problem, based on:

as in rotate the origin position then rotate the origin position plus the size..

For computer graphics, the most common transformations are Scaling, Translating, and Rotating, and you do these in an order to achieve a desired effect (traditionally you scale, then rotate about the origin, then translate the vertex's position).

Additionally, you will have three main matrices to render a model in 3d: World Matrix, View Matrix, and Projection Matrix. I believe you are having misunderstandings of transforming from Model Space to World Space.

Graphics TRS and Matrix info. If you are having conceptual problems, or this answer is insufficient, I highly recommend looking at this link. I have yet to find a better resource explaining the fundamentals of computer graphics.

So right at the moment, you have your three angles (in degrees, in a Vector3) corresponding to the angle difference in the X,Y, and Z coordinate spaces from your billboard and your camera. With this information, we generate the View matrix by first gathering all of our matrix transformations in one place.

I'm going to assume that you already have your Translation and Scaling matrices, and that they both work. This means that we only need to generate our Rotation matrix, and then transform that matrix with the scaling matrix, and then transforming that matrix by our translation matrix.

X Rotation Matrix

Y Rotation Matrix

Z Rotation Matrix

(Images taken from CodingLabs link above)

So you will generate these three matrices, using the X,Y, and Z angles you calculated earlier, and then transform them to consolidate them into a single matrix, transform that matrix by the scaling matrix, and then transform that matrix by the translation matrix. Now you have your awesome matrix that, when you multiply a a vertex by it, will transform that vertex into the desired size, rotation, and position.

So you transform every single vertex point by this generated matrix.

And then after that, you should be done! Using these techniques will hopefully simplify your code greatly, and set you on the right path :)

So now how about some code?

//I do not guarantee that this code compiles! I did not write it in an IDE nor did I compile it
float angleToRotX = 180f;
float angleToRotY = 90f;
float angleToRotZ = 0f;

// example vertex
Vector4f vertex = new Vector4f(0, 1, 0, 1);

// Rotate vertex's X coordinates by the desired degrees
Matrix4f rotationXMatrix = new Matrix4f();
rotationXMatrix.rotX(angleToRotX);

Matrix4f rotationYMatrix = new Matrix4f();
rotationYMatrix.rotY(angleToRotY);

Matrix4f rotationZMatrix = new Matrix4f();
rotationZMatrix.rotZ(angleToRotZ);

//now let's translate it by 1.5, 1, 1.5 in the X,Y,Z directions
Matrix4f translationMatrix = new Matrix4f();
translationMatrix.setTranslate(new Vector3f(1.5, 1, 1.5));

/*
Now we have our three rotational matrices. So we multiply them (transform them) to get a single matrix to transform all of the points in this model to the desired world coordinates

*/
Matrix4f rotationMatrix = new Matrix4f();
rotationMatrix.mul(rotationXMatrix);
rotationMatrix.mul(rotationYMatrix);
rotationMatrix.mul(rotationZMatrix);

Matrix4f worldMatrix = translationMatrix;
worldMatrix.mul(rotationMatrix);

//now worldMatrix, when applied to a vertex, will rotate it by X,Y,Z degrees about the origin of it's model space, and then translate it by the amount given in translationMatrix

worldMatrix.transform(vertex);

//now vertex should be (1.5, 0, 1.5, 1) with (x,y,z,1)

Now this code could really be simplified, and it is excessively verbose. Try it out! I don't have java downloaded on my machine, but I grabbed the methods from the java documentation Here

Here is an image of what is happening (again, taking from coding labs):

(Advanced Info: Quaternions. These are really cool way of orienting a model in 3d space, however I don't quite understand them to the degree I need to in order to explain it to someone else, and I also believe that your problem is more fundamental)