Web Plugin Framework (web admin UI) stopped working on a FAS HA Cluster

Description

FAS is installed in HA mode, i.e.  Master node and at least one Slave node is installed. 

NOTE: This can be seen during a singlebox installation (when the drive space has been filled)

While accessing web plugin framework using following URL:

https://<fas-server-cluster-address>:8443/web_plugin_framework/webcontroller

User fails to use web admin UI with error message being displayed based on web page being loaded.

Typically, other pages, such as Assist or User Credentials load correctly.

Server log on master node would contain ERROR messages related to "PerformanceLogCache". Below is an example:

Other Possible Errors when loading console:

java.lang.NoClassDefFoundError: Could not initialize class com.alicecallsbob.porth.gateway.rtpproxy.PerformanceLogCache

ERROR [com.alicecallsbob.porth.admin.web.plugin.gateway.GatewayWebPlugin] (http-connector-threads - 284)  | Creating view for Gateway Admin: com.alicecallsbob.web.framework.webplugin.WebPluginException: Failed to prepare call log

Caused by: com.alicecallsbob.command.line.exception.LoadException: Failed to load historic calls

WARN  [com.alicecallsbob.web.framework.webplugin.WebPluginCollection] (http-connector-threads - 284)  | Plugin failed to service request: com.alicecallsbob.web.framework.webplugin.WebPluginException: Error handling event /calls/: No such object

Caused by: com.alicecallsbob.web.framework.webplugin.WebPluginException: Failed to prepare call log

Cause

Issue is caused by time not being in sync on between Master and Slave(s) node of the HA Cluster.

It can also be caused when the drive space has become full and the performance log can no longer be written to.

This result in infinispan cache being corrupted hence cluster fails to respond correctly.

Resolution

Resolution of this issue require 2 steps:

Step 1 Ensure time and date are in sync between all FAS cluster elements:  

It is recommended to use NTP service for this purpose. Please check with your network administrator if you are unsure about your NTP server.

Below are some example commands to help you with NTP server config:

  • NTP Server configurationNTP server can be configured in "/etc/ntp.conf". The conf file "/etc/ntp.conf" should contain following line:

server <your ntp server address>

  • NTP ServiceThe ntpd service needs to be started to synchronise date and time with configured NTP server. Use following command to check status of ntpd service:

# service ntpd status
Note:Please restart ntpd service after changing ntp.conf, "service ntpd restart"

  • Test NTP Config:  To verify that time has been successfully synchronised use ntpstat command. Allow couple of minutes for service to get synchronised. Below is an example of successful run of ntpstat command:

# ntpstat

synchronised to <ntp server name> at stratum 3
time correct to within 463 ms
polling server every 64 s

Step 2 Delete corrupted cache files:

Following steps can be taken to delete corrupted cache files:

  • Stop all services on master as well as slave node(s)

service fas stop

  • Delete cache files from master & slave(s). below are example commands:

cd  <fas server install directory>/domain/servers/appserver-<node address>/data/infinispan/fcsdk/PerformanceStatistics/

rm *.

Note: If the files names start with a - e.g. -12345 you need to run rm -- -12345 to remove it.

  • Once cache files are removed start fas service on all nodes.

service fas start

 

 

 

 

 

 

 

Comments are disabled on these articles if you require help contact support@cafex.com.

 

Have more questions? Submit a request

Comments

Powered by Zendesk