Freedom of speech/censorship?

I think my freedom of speech extends only so far as it does not interfere with another person's rights. For example I do not have the right to persecute another person or impinge on their rights to free movement. I also do not have the right to make defamatory statements about another person if that may affect their right to a fair trail.