Where should check_multi run? On the local or on the remote server?

There is often the discussion, if monitoring checks should be run remotely or locally on the server, which has to be monitored. The decision is sometimes easy, when the ressource to be monitored is not available remotely, e.g. logfiles or disks.
But there are plenty of cases where you can do both, e.g. applications and services, which are accessed via network. Please don't think, that network services have to be monitored remotely, because otherwise there's no proof that it's working over the network. You can give exactly one proof with your nagios check, and that is for the nagios server. Where the servers users normally not reside
So why not executing all checks on the remote server?

There are indeed some reasons for this approach:

• All checks are consistent.
• The transport problem has only to be solved once.
• The customer is paying in terms of performance for his own monitoring. The nagios server does not need to bear the load of multiple checks, it receives only the results.
  

But the disadvantage of this approach lies just here - when the customer comes and asks:

• what the heck are all these 'nagios' processes doing on my server?
• And why are they running so often?
  

Generally we won't find a generic solution for all cases. I have prepared a small table to help you finding the criteria for your specific solution:

	1. check_multi running on Nagios server	2. check_multi running on client server
Basic concept	check_multi runs on the Nagios server and accesses the remote server with each child check independently	check_multi is running on the remote server (here called client) and executes the child services locally
Resources Load	1. The Nagios servers bears most of the execution load and needs more power. 2. The footstep on the client server is not that big.	1. The load on the Nagios server is very small. 2. The major work is done on the client, therefore it takes the burden of its own monitoring.
Network	The network load is higher, especially with SSH transport mechanisms the effort is much higher due to multiple authentication steps.	The network load is very small, there is only one connection to trigger the startup and transport back the results over the network.
Configuration	Configuration is easily accessible, there is no need to distribute configuration files	The remote client needs configuration to be distributed and updated.
Plugins	As for configuration, its sufficient to provide plugins only on the central Nagios server.	Plugins have to be distributed and updated.
Local monitoring vs. remote monitoring	There are particular local checks like disk monitoring, which need high efforts to be run remotely	Every remote check can also be run as a local check. That means vice versa, that all checks can be run from the remote server, even checks for network services. A plus in terms of homogenity of monitoring