Is it forbidden to say “ya Muhammad”?

In prayer, it is legitimate to say “ya Allah” and not “ya Muhammad”. Saying “ya Muhammad” in order to call on him, ask him and seek his response is shirk and forbidden in Islam. Since dua’ is worship, we have to do it only in the form sanctioned by the Shari’ah and not as we wish.

An-Nu’man ibn Basheer reported: I heard the Prophet, (ﷺ)  saying, “Supplication is worship.” Then he recited the verse, “Your Lord says: Call upon Me and I will respond to you, for verily, those who disdain My worship will enter Hell in humiliation.” [1]

Reference:

[1] Surah Ghafir, 40:60; Sunan At-Tirmidhi 3247;Grade: Sahih (authentic) according to At-Tirmidhi