Asterisk audio inbound or outbound not working when firewall enabled

Due to attempted hacks on VOIP boxes we need to increase security by allowing SIP access only from the relevant IPs using iptables firewall

In the case below the Asterisk VOIP server is sitting behind a NAT firewall and has the relevant config set for this. I found upon enabling the linux firewall that while audio outbound was working, it was not working inbound.

In this case I was running RTP ports on 40000:40100 and the voip box was on 192.168.0.20 on the LAN. The UDP rules at the top are what got the audio flowing inbound properly.

Accept If protocol is UDP and source is 0.0.0.0/0 and destination is 0.0.0.0/0 and destination port is 40000:40100
Accept If protocol is UDP and source is sipprovider.com and destination is 0.0.0.0/0
Accept If source is sipprovider.com
Accept If source is 192.168.0.0/24 and destination is 192.168.0.20
Accept If source is 192.168.0.20
Accept If source is 127.0.0.1/8