You're special is a term that can mean both positive and negative things.
A saying that's meaning depends on its user.
Example:
If your Girlfriend says "You're special" it means "I love you"
But
If your Friend says "You're special" it's most likely a remake of you being mentally disabled in some way such as autistic