Skip to main content
MediaBeacon University

R3Search Clustering in Practice

Installation

In this example, we will configure a sample system with a core MediaBeacon server, 3 Zookeepers (which also run R3Search), and 3 R3Search servers. These steps can be adjusted to suit various system configurations, but the basic progression is the same:

  1. Configure core MediaBeacon server.
  2. Configure Zookeepers.
  3. Configure standalone R3Search servers.
  4. Launch core MediaBeacon server.
  5. Launch Zookeepers.
  6. Launch standalone R3Search servers.

Prior to installation, download the correct R3Search components to each server in the cluster.

Port Information

  • Zookeepers must run on port 31000.
  • R3Searches must run on port 30000.
  • Zookeepers need to be able to communicate with each other and all R3Searches.
  • R3Searches need SQL access.

Core MediaBeacon Server

  1. For the R3Search cluster to function, the IP addresses of all servers in the cluster must be white listed on the core MediaBeacon server. On the Enterprise tab, click “Applications” and click “Add”. Enter the IP addresses of all servers in the cluster, including the address of the core MediaBeacon server.
    R3CM-R3Search_Cluster_1.png
  2. From the R3Search tab, set the following options:
    R3CM-R3Search_Cluster_3.png
    1. Local R3Search server
    2. R3Search Only
    3. Number of shards: 3
    4. Zookeeper Hosts: List the IP addresses of all Zookeepers for your system. Note: This is different than the white list from Step 1—only list the Zookeeper servers. All Zookeepers run on port 31000. R3CM-R3Search_Cluster_2.png
    5. Uncheck “Automatically start local R3Search”.
    6. Click “Apply”.
    7. Make sure R3Search isn’t running.
  3. Navigate to the Enterprise tab. If you haven’t done so, set a remote password by clicking “Change Remote Password”.
  4. Stop the core MediaBeacon server and shut down the application.
  5. Delete the contents of the following folder (if it exists), which contains the R3Search cache:
    C:\Users\Administrator\AppData\Local\Solr
  6. Do not restart the core MediaBeacon server until all R3Search servers have been configured.

Zookeeper Server

  1. On the Zookeeper servers, set the following options:
    R3CM-R3Search_Cluster_4.png
    1. HTTP port: 30000 (default)
    2. Memory limit: appropriate value for your system
    3. Number of shards: 3
    4. R3Search + Zookeeper
    5. Zookeeper Hosts: List the IP addresses of all Zookeepers for your system. Note: This is different than the white list on the core MediaBeacon server. Only list the Zookeeper servers.
  2. Click “Save”.
  3. Stop R3Search (if running) and quit the application.
  4. Delete the contents of the following folder (if it exists), which contains the R3Search cache: C:\Users\Administrator\AppData\Local\Solr
  5. Do not restart the core MediaBeacon server until all R3Search servers have been configured.

R3Search Server

  1. On the R3Search servers, set the following options:
    R3CM-R3Search_Cluster_5.png
    1. HTTP port: 30000 (default)
    2. Memory limit: appropriate value for your system
    3. Number of shards: 3
    4. R3Search Only
    5. Zookeeper Hosts: List the IP addresses of all Zookeepers for your system. Don’t forget to include the port number. Note: This is different than the white list on the core MediaBeacon server. Only list the Zookeeper servers.
  2. Click “Save”.
  3. Stop R3Search (if running) and quit the application.
  4. Delete the contents of the following folder (if it exists), which contains the R3Search cache: C:\Users\Administrator\AppData\Local\Solr
  5. Do not restart the core MediaBeacon server until all R3Search servers have been configured.

Starting up the Cluster

Once the configuration above is in place, start up the cluster by launching the following components in order.

  1. Start the core MediaBeacon server.
  2. On each Zookeeper server:
    1. Click “Attach” and select the core MediaBeacon server.
    2. Enter the remote password.
    3. Click “Start”.
      R3CM-R3Search_Cluster_6.png
  3. On each R3Search server:
    1. Click “Attach” and select the core MediaBeacon server.
    2. Enter the remote password.
    3. Click “Start”.
  4. The R3Search cluster is now running. To see the status of the system, go to the MediaBeacon web interface and click “Admin > R3Search Administration”. R3CM-R3Search_Cluster_7.png

R3Search Administration: Troubleshooting

There are a few notable places in the Admin area that are especially useful to troubleshooting that we recommend investigating should any issues arise.

Logging

R3CM-Screen_Shot_2015-08-11_at_10.26.18_AM.png

The Logging section is useful to see any warnings or issues that are detected.

Cloud > Graph

This section of the Dashboard shows how the servers are structured and the status of each. The status is indicated by a circle icon to the left of each component.

  • Leader: Filled in green circle: Things are running fine and all is well with the leader.
  • Active: Empty green circle: Indicates a slave. It’s running just fine, and replicating the data its leader has.
  • Recovering: Empty yellow circle: The shard went down or had an issue and is now working to be functional again.
  • Down: Empty orange circle: The shard is not functional.
  • Recovery Failed: Empty red circle: The shard attempted to recover from an issue and failed. Human interjection is probably required to correct the issue.
  • Gone: Empty gray circle: The shard is not communicating with the cluster at all. The system does not know the shard’s status.

When a system encounters an outage, the status of the affected components is updated in the dashboard. Take a look at the following example to see what happens when we lose a shard leader, which is then restarted.

All shards were running and operating normally. Then, we suddenly lose one of the leader shards (10.6.0.77).

R3CM-Screen_Shot_2015-07-07_at_6.11.24_PM.png

Notice the lost shards is indicated by an empty gray circle next to the IP of the server. The system intelligently responds to this loss by converting the slave shard of the leader into a leader. It’s important to resolve the issue shortly because if the new leader were to encounter an issue, search functionality could potentially begin encountering issues.

We correct the issue and begin bringing the lost shard back up.

R3CM-Screen_Shot_2015-07-07_at_6.16.15_PM.png

Notice the shard indicates it is recovering, indicated by the empty yellow circle next to it.

Once it is up, the shard that failed, which was once a leader, is now a slave of the new leader.

R3CM-Screen_Shot_2015-07-07_at_6.11.48_PM.png

Cloud > Dump

On rare occasions, the MediaBeacon team may request this data be provided if they are troubleshooting a reported issue.

Core Admin > Index Core

This section is useful to see the number of documents in a shard’s R3Search index.

Core Admin > Index Core > Documents

In the t on the bottom left, select Index. Next, select Documents. Sometimes when troubleshooting, the MediaBeacon team will instruct you to use this area for manually issuing Solr updates.

R3CM-Screen_Shot_2015-08-11_at_10.52.47_AM.png

Core Admin > Index Core > Query

This section is useful for issuing queries to the Solr index. The MediaBeacon team may instruct you to use this area when troubleshooting.

An example query we can run here: id:29113

R3CM-Screen_Shot_2015-08-11_at_11.07.32_AM.png

Port Information

Below are the default ports used to connect.

R3Search

  • Connects to MediaBeacon R3volution core: 44444, 44445, and 44446.
  • Connects to R3Search node(s): 30000.
  • Connects to Zookeeper(s): 31000.
  • Connects to the database.

Zookeeper

  • Connects to MediaBeacon R3volution core: 44444, 44445, and 44446.
  • Connects to R3Search node(s): 30000.
  • Connects to Zookeeper(s): 31000.
  • Connects to the database.
  • Was this article helpful?