Another reason is that there is no disagreement that Ja’far was not infallible, while it is proved that an Imam has to be infallible. His actions contradict his being infallible.
ومنها أنه لا خلاف أنه لم يكن معصوما، وقد بينا أن من شرط الإمام أن يكون معصوما، وما ظهر من أفعاله ينافي العصمة.
Similar narrations
Kitāb al-Ghayba #379As for the believers in the Imamate of Ja’far bin Ali [Ja’far Kazzab] after his brother, their belief is wrong, because an Imam has to be...