opencv - Optimal measurement model for Kalman in Augmented Reality

I am developing an augmented reality SDK that uses Kalman for tracking a planar marker. My state is composed of 3D position, a quaternion, velocity and angular velocity.

\begin{bmatrix}{\vec{p}}\\{\vec{q}}\\{\vec{v}}\\{\vec{\omega }}\end{bmatrix}

For some reason Kalman is not tracking properly. I think that the problem is our measurement model.

We use points (we tried 4 and 8) detected in the real-world marker (proyected points) as measurements $ z_k$. Kalman predicted measurements $h(x_k^-)$ are the matches of those points in the trained-marker, and proyected with the homography predicted by Kalman. Kalman doesn't predict homography directly, but can be extracted from position and quaternion. Homography from Kalman should proyect the matched pairs close to the measured pairs.

Is this a good measurement model? Does anybody have a better idea?

Here is a link of the expected result of project of this type with Kalman softening the tracking. In our project the 3D projections are shaking, it is not precise.