Latency

The time it takes a conversation to travel from the speaker to the listener is the end-to-end delay, or latency.

End-to-end delay is actually made up of four components:

• Propagation delay— The time to travel across the network from end to end. It is based on the speed of light and the distance the signal must travel.
• Transport delay— The time to get through the network devices along the path. Networks with many firewalls, many routers, congestion, or slow WANs introduce more delay than an overprovisioned LAN on one floor of a building.
• Packetization delay— The time for the codec to digitize the analog signal and build frames—and undo it at the other end.
• Jitter buffer delay— The delay introduced by the receiver as it holds one or more datagrams to reduce variations in arrival times.

Many VoIP engineers don't know how much latency is too much. A simple answer is 150 ms.

Source: Taking Charge of Your VoIP Project