BigFix Client Troubleshooting
Whether you're new to the BigFix Platform, or a seasoned administrator, client troubleshooting can sometimes be difficult. This post will serve as a short how-to to get you started on client troubleshooting.
BES Server cannot notify the BES clients of new actions:
The BES platform is technically a pull technology, meaning that the BES Clients are always the one downloading instructions and retrieving software payloads. However, when new actions are issued, the BES Server will notify the BES Clients via UDP. By default, the BES Clients will go to the BES Server to poll or "gather" new instructions once every 24 hours.
This is described as traffic pattern "B" in the BES Network Traffic Guide.
So, if the BES Server/Relays are not able to send UDP messages to the BES Clients due to network constraints the Gather Interval can be modified from the default of once in 24 hours.
This can be configured via the BES Client settings so this can be done on a client basis:
Please see the following BES Support Site content which can be used to enable this setting:
'BES Client Setting: Enable Command Polling "
To enable this setting manually please see the following setting:
The _BESClient_Comm_CommandPollEnable of value "1" will enable this mechanism and a value of "0" will disable it. The _BESClient_Comm_CommandPollIntervalSeconds setting determines how often the client will check with its BES Server/Relay for any new gathers or refreshes. It is specified in seconds, such as 14400 for every 4 hours (14,400 seconds).
Please refer to the following document for how to create a computer setting: Configuration Settings
Important: You should not set the Gather Interval less than 10 minutes, otherwise the BES Clients may not function properly because they will constantly interrupt themselves to gather the latest actions.
Note: You can also change the Gather Interval value in the masthead using BES Administration Tool so that all BES Clients (including any BES Clients that will be installed in the future) will use the specified Gather Interval. Please note that the masthead changes the gather interval for the BES Client's Actionsite (commands issued by the Master Operators). It does not change the gathering interval for the Opsites (i.e., operator sites) or Fixlet sites. To change the gather interval for all subscribed sites, use the _BESClient_Comm_CommandPollEnable setting described above.
Another reason to use these settings would be if the BES Clients are in the DMZ and cannot receive UDP messages from a BES Relay.
BES Clients are too busy evaluating the content:
The BES Client is in a constant loop of evaluating each Fixlet message, Task, Baseline, etc. to see if it is relevant. Sometimes, due to the excessive number of Fixlets subscribed, or inadvertently written Fixlets that spend an inordinate amount of time, BES Clients can appear unresponsive. For example, if a Fixlet is written to scan the entire file system or frequently querying network resources that might not be available.
Determine the Client Evaluation Cycle:
There is a new Relevance Inspector in BES 8.0 that can provide useful information regarding the Client Evaluation Cycle. An Evaluation Cycle represents a complete run through all the content available on the BES Client, measured in milliseconds. The Inspector returns statistics based on the time sampled whenever the client returns to the beginning of its content set.
The Relevance "Average of evaluation cycle of the client" will return the average evaluation cycle for the last 10 loops. Anything in the 15-minute range is considered excellent. Anything over 30-45 minutes, is worthy of investigation. The evaluation cycles will depend heavily on the amount of content-enabled in your environment, so these times are just a rough recommendation. Use the Client Profiler to debug Fixlets taking too long to evaluate.
Client Usage Profiler:
The Client Usage Profiler is an important tool (enabled via setting) that will log time spent on evaluating content. This information allows you to see which Fixlet messages, tasks, actions, or properties consume most of the Client's time. After the usage profiler has been enabled, the Client tracks the top 100 activities that take the longest time and stores them in a file called "usageprofiler.txt.XXXX", located in the same folder as the Client.
Note: After enabling the usage profiler, you may have to restart the Client before it can produce the log file "usageprofiler.txt.XXXX" and begin tracking activities.
Logging can be enabled and disabled using the following Tasks from the BES Support Site in your
TROUBLESHOOTING: Enable BES Client Usage Profiler
TROUBLESHOOTING: Disable BES Client Usage Profiler
Here is an example of a Usage file that contains unusually long evaluation times. A health item would take less than a second to evaluate. Items that take longer should be investigated further.
Start:Thu, 30 Feb 2011 15:21:40 -0800
Tracking: Top 100
Elapsed Evaluation Time:18:18:32
1) Time in Milliseconds:Site.Item ID:Activity
2) 16478.126: actionsite.2147453787:Analysis Relevance
3) 16478.124: actionsite.2146599782:Analysis Relevance
4) 16478.119: actionsite.2142786964:Analysis Relevance
5) 16478.118: BES Support.521:Verify Fixlet Relevance
Client Debug Logging:
For advanced troubleshooting purposes, it is sometimes useful to enable the debug log on a BES Client so it will log every activity.
To turn on debug logging through the BES Console, follow the instructions in the Task in the BES Console titled:
BES Client Setting: Enable Debug Logging
BES Client Setting: Disable Debug Logging
To turn on Message debug logging manually, go to the computers tab in the BES Console and create a BES Client custom setting (choose the computers, right-click and go to "Edit Computer Settings"). Create two custom settings with the names "_BESClient_EMsg_Detail" and "_BESClient_EMsg_File" and the values as specified below:
_BESClient_EMsg_Detail - This configuration setting will enable the BES Client debug log that will give extended information about BES Client activity. The higher the level of detail, the greater the detail of the messages that are put into the log. The value should be set between 0 and 10000. Use 10,000 by default.
_BESClient_EMsg_File - Full path to file to store activity messages. The _BESClient_EMsg_Detail setting must be greater than 0 to use this option. The value should be the full path to the log (Example: C:\BESClientEMsg.log). If you change the name or the path of this file, to avoid character display problems, ensure to use names that have only standard ASCII characters and not extended ASCII characters.
Note: If the BES Client does not appear in the BES Console, you can still turn on the Message debug Log by manually creating the client settings in the registry: