Virtual Server Administration
Linux Virtual Server (LVS) for Red Hat Enterprise Linux
- Red Hat Enterprise Linux Installation Guide — Provides information regarding installation.
- Red Hat Enterprise Linux Introduction to System Administration — Provides introductory information for new Red Hat Enterprise Linux system administrators.
- Red Hat Enterprise Linux System Administration Guide — Provides more detailed information about configuring Red Hat Enterprise Linux to suit your particular needs as a user.
- Red Hat Enterprise Linux Reference Guide — Provides detailed information suited for more experienced users to reference when needed, as opposed to step-by-step instructions.
- Red Hat Enterprise Linux Security Guide — Details the planning and the tools involved in creating a secured computing environment for the data center, workplace, and home.
- Red Hat Cluster Suite Overview — Provides a high level overview of the Red Hat Cluster Suite.
- Configuring and Managing a Red Hat Cluster — Provides information about installing, configuring and managing Red Hat Cluster components.
- LVM Administrator's Guide: Configuration and Administration — Provides a description of the Logical Volume Manager (LVM), including information on running LVM in a clustered environment.
- Global File System: Configuration and Administration — Provides information about installing, configuring, and maintaining Red Hat GFS (Red Hat Global File System).
- Using Device-Mapper Multipath — Provides information about using the Device-Mapper Multipath feature of Red Hat Enterprise Linux 4.
- Using GNBD with Global File System — Provides an overview on using Global Network Block Device (GNBD) with Red Hat GFS.
- Red Hat Cluster Suite Release Notes — Provides information about the current release of Red Hat Cluster Suite.
Chapter 1. Linux Virtual Server Overview
- To balance the load across the real servers.
- To check the integrity of the services on each real server.
1.1. A Basic LVS Configuration
Figure 1.1. A Basic LVS Configuration
eth0:1. Alternatively, each virtual server can be associated with a separate device per service. For example, HTTP traffic can be handled on
eth0:1, and FTP traffic can be handled on
1.1.1. Data Replication and Data Sharing Between Real Servers
- Synchronize the data across the real server pool
- Add a third layer to the topology for shared data access
126.96.36.199. Configuring Real Servers to Synchronize Data
rsyncto replicate changed data across all nodes at a set interval.
1.2. A Three-Tier LVS Configuration
Figure 1.2. A Three-Tier LVS Configuration
1.3. LVS Scheduling Overview
1.3.1. Scheduling Algorithms
- Round-Robin Scheduling
- Distributes each request sequentially around the pool of real servers. Using this algorithm, all the real servers are treated as equals without regard to capacity or load. This scheduling model resembles round-robin DNS but is more granular due to the fact that it is network-connection based and not host-based. LVS round-robin scheduling also does not suffer the imbalances caused by cached DNS queries.
- Weighted Round-Robin Scheduling
- Distributes each request sequentially around the pool of real servers but gives more jobs to servers with greater capacity. Capacity is indicated by a user-assigned weight factor, which is then adjusted upward or downward by dynamic load information. Refer to Section 1.3.2, “Server Weight and Scheduling” for more on weighting real servers.Weighted round-robin scheduling is a preferred choice if there are significant differences in the capacity of real servers in the pool. However, if the request load varies dramatically, the more heavily weighted server may answer more than its share of requests.
- Distributes more requests to real servers with fewer active connections. Because it keeps track of live connections to the real servers through the IPVS table, least-connection is a type of dynamic scheduling algorithm, making it a better choice if there is a high degree of variation in the request load. It is best suited for a real server pool where each member node has roughly the same capacity. If a group of servers have different capabilities, weighted least-connection scheduling is a better choice.
- Weighted Least-Connections (default)
- Distributes more requests to servers with fewer active connections relative to their capacities. Capacity is indicated by a user-assigned weight, which is then adjusted upward or downward by dynamic load information. The addition of weighting makes this algorithm ideal when the real server pool contains hardware of varying capacity. Refer to Section 1.3.2, “Server Weight and Scheduling” for more on weighting real servers.
- Locality-Based Least-Connection Scheduling
- Distributes more requests to servers with fewer active connections relative to their destination IPs. This algorithm is designed for use in a proxy-cache server cluster. It routes the packets for an IP address to the server for that address unless that server is above its capacity and has a server in its half load, in which case it assigns the IP address to the least loaded real server.
- Locality-Based Least-Connection Scheduling with Replication Scheduling
- Distributes more requests to servers with fewer active connections relative to their destination IPs. This algorithm is also designed for use in a proxy-cache server cluster. It differs from Locality-Based Least-Connection Scheduling by mapping the target IP address to a subset of real server nodes. Requests are then routed to the server in this subset with the lowest number of connections. If all the nodes for the destination IP are above capacity, it replicates a new server for that destination IP address by adding the real server with the least connections from the overall pool of real servers to the subset of real servers for that destination IP. The most loaded node is then dropped from the real server subset to prevent over-replication.
- Destination Hash Scheduling
- Distributes requests to the pool of real servers by looking up the destination IP in a static hash table. This algorithm is designed for use in a proxy-cache server cluster.
- Source Hash Scheduling
- Distributes requests to the pool of real servers by looking up the source IP in a static hash table. This algorithm is designed for LVS routers with multiple firewalls.
1.3.2. Server Weight and Scheduling
1.4. Routing Methods
1.4.1. NAT Routing
Figure 1.3. LVS Implemented with NAT Routing
1.4.2. Direct Routing
Figure 1.4. LVS Implemented with Direct Routing
188.8.131.52. Direct Routing and the ARP Limitation
iptablespacket filtering tool for the following reasons:
arptables_jfprevents ARP from associating VIPs with real servers.
iptablesmethod completely sidesteps the ARP problem by not configuring VIPs on real servers in the first place.
1.5. Persistence and Firewall Marks
1.5.2. Firewall Marks
1.6. LVS — A Block Diagram
Figure 1.5. LVS Components
pulsedaemon runs on both the active and passive LVS routers. On the backup router,
pulsesends a heartbeat to the public interface of the active router to make sure the active router is still properly functioning. On the active router,
lvsdaemon and responds to heartbeat queries from the backup LVS router.
lvsdaemon calls the
ipvsadmutility to configure and maintain the IPVS routing table in the kernel and starts a
nannyprocess for each configured virtual server on each real server. Each
nannyprocess checks the state of one configured service on one real server, and tells the
lvsdaemon if the service on that real server is malfunctioning. If a malfunction is detected, the
ipvsadmto remove that real server from the IPVS routing table.
send_arpto reassign all virtual IP addresses to the NIC hardware addresses (MAC address) of the backup node, sends a command to the active router via both the public and private network interfaces to shut down the
lvsdaemon on the active router, and starts the
lvsdaemon on the backup node to accept requests for the configured virtual servers.
1.6.1. LVS Components
pulse” shows a detailed list of each software component in an LVS router.
/etc/rc.d/init.d/pulsescript. It then reads the configuration file
/etc/sysconfig/ha/lvs.cf. On the active router,
pulsestarts the LVS daemon. On the backup router,
pulsedetermines the health of the active router by executing a simple heartbeat at a user-configurable interval. If the active router fails to respond after a user-configurable interval, it initiates failover. During failover,
pulseon the backup router instructs the
pulsedaemon on the active router to shut down all LVS services, starts the
send_arpprogram to reassign the floating IP addresses to the backup router's MAC address, and starts the
lvsdaemon runs on the active LVS router once called by
pulse. It reads the configuration file
/etc/sysconfig/ha/lvs.cf, calls the
ipvsadmutility to build and maintain the IPVS routing table, and assigns a
nannyprocess for each configured LVS service. If
nannyreports a real server is down,
ipvsadmutility to remove the real server from the IPVS routing table.
lvsdaemon sets up and administers LVS by calling
ipvsadmto add, change, or delete entries in the IPVS routing table.
nannymonitoring daemon runs on the active LVS router. Through this daemon, the active router determines the health of each real server and, optionally, monitors its workload. A separate process runs for each service defined on each real server.
184.108.40.206. Piranha Configuration Tool
/etc/sysconfig/ha/lvs.cfLVS configuration file.
Chapter 2. Initial LVS Configuration
2.1. Configuring Services on the LVS Routers
chkconfig, the ncurses-based program
ntsysv, and the graphical Services Configuration Tool. All of these tools require root access.
su -command followed by the root password. For example:
su - root password
piranha-guiservice (primary node only)
chkconfig, type the following command for each service:
/sbin/chkconfig --level 35 daemon on
chkconfigdoes not actually start the daemon. To do this use the
/sbin/servicecommand. See Section 2.3, “Starting the Piranha Configuration Tool Service” for an example of how to use the
ntsysvand the Services Configuration Tool, refer to the chapter titled "Controlling Access to Services" in the Red Hat Enterprise Linux System Administration Guide.
2.2. Setting a Password for the Piranha Configuration Tool
2.3. Starting the Piranha Configuration Tool Service
piranha-guiservice located in
/etc/rc.d/init.d/piranha-gui. To do this, type the following command as root:
/sbin/service piranha-gui start
/sbin/service piranha-gui restart
/usr/sbin/piranha_gui -> /usr/sbin/httpd. For security reasons, the
httpdruns as the piranha user in a separate process. The fact that
httpdservice means that:
- The Apache HTTP Server must be installed on the system.
- Stopping or restarting the Apache HTTP Server via the
servicecommand stops the
/sbin/service httpd stopor
/sbin/service httpd restartis issued on an LVS router, you must start the
piranha-guiservice by issuing the following command:
/sbin/service piranha-gui start
piranha-guiservice is all that is necessary to begin configuring LVS. However, if you are configuring LVS remotely, the
sshdservice is also required. You do not need to start the
pulseservice until configuration using the Piranha Configuration Tool is complete. See Section 4.8, “Starting LVS” for information on starting the
2.3.1. Configuring the Piranha Configuration Tool Web Server Port
Listen 3636in Section 2 of the
piranha-guiWeb server configuration file
http://localhost:3636. You can reach the Piranha Configuration Tool from anywhere via Web browser by replacing localhost with the hostname or IP address of the primary LVS router.
piranhain the Username field and the password set with
piranha-passwdin the Password field.
2.4. Limiting Access To the Piranha Configuration Tool
/etc/sysconfig/ha/web/secure/.htaccess. After altering the file you do not have to restart the
piranha-guiservice because the server checks the
.htaccessfile each time it accesses the directory.
Order deny,allow Allow from all
.htaccessfile to allow access from only the loopback device (127.0.0.1). For more information on the loopback device, see the chapter titled Network Scripts in the Red Hat Enterprise Linux Reference Guide.
Order deny,allow Deny from all Allow from 127.0.0.1
Order deny,allow Deny from all Allow from 192.168.1.100 Allow from 172.16.57
.htaccessfile limits access to the configuration pages in the
/etc/sysconfig/ha/web/secure/directory but not to the login and the help pages in
/etc/sysconfig/ha/web/. To limit access to this directory, create a
.htaccessfile in the
denylines identical to
2.5. Turning on Packet Forwarding
net.ipv4.ip_forward = 0in
/etc/sysctl.confto the following:
1, then IP forwarding is enabled. If it returns a
0, then you can turn it on manually using the following command:
/sbin/sysctl -w net.ipv4.ip_forward=1
2.6. Configuring Services on the Real Servers
httpdfor Web services or
xinetdfor FTP or Telnet services.
sshddaemon should also be installed and running.
Chapter 3. Setting Up LVS
3.1. The NAT LVS Network
- Network Layout
- The topology for LVS using NAT routing is the easiest to configure from a network layout perspective because only one access point to the public network is needed. The real servers pass all requests back through the LVS router so they are on their own private network.
- The NAT topology is the most flexible in regards to hardware because the real servers do not need to be Linux machines to function correctly. In a NAT topology, each real server only needs one NIC since it will only be responding to the LVS router. The LVS routers, on the other hand, need two NICs each to route traffic between the two networks. Because this topology creates a network bottleneck at the LVS router, gigabit Ethernet NICs can be employed on each LVS router to increase the bandwidth the LVS routers can handle. If gigabit Ethernet is employed on the LVS routers, any switch connecting the real servers to the LVS routers must have at least two gigabit Ethernet ports to handle the load efficiently.
- Because the NAT topology requires the use of
iptablesfor some configurations, there can be a fair amount of software configuration outside of Piranha Configuration Tool. In particular, FTP services and the use of firewall marks requires extra manual configuration of the LVS routers to route requests properly.
3.1.1. Configuring Network Interfaces for LVS with NAT
eth0) will be on the 192.168.26/24 network (I know, I know, this is not a routable IP, but let us pretend there is a firewall in front of the LVS router for good measure) and the private interfaces which link to the real servers (
eth1) will be on the 10.11.12/24 network.
/etc/sysconfig/network-scripts/ifcfg-eth0, could look something like this:
DEVICE=eth0 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.26.9 NETMASK=255.255.255.0 GATEWAY=192.168.26.254
/etc/sysconfig/network-scripts/ifcfg-eth1for the private NAT interface on the LVS router could look something like this:
DEVICE=eth1 BOOTPROTO=static ONBOOT=yes IPADDR=10.11.12.9 NETMASK=255.255.255.0
3.1.2. Routing on the Real Servers
/etc/sysconfig/network-scripts/ifcfg-eth0file could look similar to this:
DEVICE=eth0 ONBOOT=yes BOOTPROTO=static IPADDR=10.11.12.1 NETMASK=255.255.255.0 GATEWAY=10.11.12.10
GATEWAY=line, the first one to come up will get the gateway. Therefore if both
eth1are configured and
eth1is used for LVS, the real servers may not route requests properly.
ONBOOT=in their network scripts within the
/etc/sysconfig/network-scripts/directory or by making sure the gateway is correctly set in the interface which comes up first.
3.1.3. Enabling NAT Routing on the LVS Routers
pulseservice as shown in Section 4.8, “Starting LVS”. Once
pulseis up and running, the active LVS router will begin routing requests to the pool of real servers.
3.2. LVS via Direct Routing
- Network Layout
- In a direct routing LVS setup, the LVS router needs to receive incoming requests and route them to the proper real server for processing. The real servers then need to directly route the response to the client. So, for example, if the client is on the Internet, and sends the packet through the LVS router to a real server, the real server must be able to go directly to the client via the Internet. This can be done by configuring a gateway for the real server to pass packets to the Internet. Each real server in the server pool can have its own separate gateway (and each gateway with its own connection to the Internet), allowing for maximum throughput and scalability. For typical LVS setups, however, the real servers can communicate through one gateway (and therefore one network connection).
ImportantIt is not recommended to use the LVS router as a gateway for the real servers, as that adds unneeded setup complexity as well as network load on the LVS router, which reintroduces the network bottleneck that exists in NAT routing.
- The hardware requirements of an LVS system using direct routing is similar to other LVS topologies. While the LVS router needs to be running Red Hat Enterprise Linux to process the incoming requests and perform load-balancing for the real servers, the real servers do not need to be Linux machines to function correctly. The LVS routers need one or two NICs each (depending on if there is a back-up router). You can use two NICs for ease of configuration and to distinctly separate traffic — incoming requests are handled by one NIC and routed packets to real servers on the other.Since the real servers bypass the LVS router and send outgoing packets directly to a client, a gateway to the Internet is required. For maximum performance and availability, each real server can be connected to its own separate gateway which has its own dedicated connection to the carrier network to which the client is connected (such as the Internet or an intranet).
3.2.1. Direct Routing and
arptables_jf, each real server must have their virtual IP address configured, so they can directly route packets. ARP requests for the VIP are ignored entirely by the real servers, and any ARP packets that might otherwise be sent containing the VIPs are mangled to contain the real server's IP instead of the VIPs.
arptables_jfmethod, applications may bind to each individual VIP or port that the real server is servicing. For example, the
arptables_jfmethod allows multiple instances of Apache HTTP Server to be running bound explicitly to different VIPs on the system. There are also significant performance advantages to using
arptables_jfmethod, VIPs can not be configured to start on boot using standard Red Hat Enterprise Linux system configuration tools.
- Create the ARP table entries for each virtual IP address on each real server (the real_ip is the IP the director uses to communicate with the real server; often this is the IP bound to
arptables -A IN -d <virtual_ip> -j DROP arptables -A OUT -d <virtual_ip> -j mangle --mangle-ip-s <real_ip>This will cause the real servers to ignore all ARP requests for the virtual IP addresses, and change any outgoing ARP responses which might otherwise contain the virtual IP so that they contain the real IP of the server instead. The only node that should respond to ARP requests for any of the VIPs is the current active LVS node.
- Once this has been completed on each real server, save the ARP table entries by typing the following commands on each real server:
service arptables_jf save
chkconfig --level 2345 arptables_jf onThe
chkconfigcommand will cause the system to reload the arptables configuration on bootup — before the network is started.
- Configure the virtual IP address on all real servers using
ifconfigto create an IP alias. For example:
ifconfig eth0:1 192.168.76.24 netmask 255.255.252.0 broadcast 192.168.79.255 upOr using the
ip, for example:
ip addr add 192.168.76.24 dev eth0As previously noted, the virtual IP addresses can not be configured to start on boot using the Red Hat system configuration tools. One way to work around this issue is to place these commands in
- Configure Piranha for Direct Routing. Refer to Chapter 4, Configuring the LVS Routers with Piranha Configuration Tool for more information.
3.2.2. Direct Routing and
iptablesfirewall rules. To configure direct routing using
iptables, you must add rules that create a transparent proxy so that a real server will service packets sent to the VIP address, even though the VIP address does not exist on the system.
iptablesmethod is simpler to configure than the
arptables_jfmethod. This method also circumvents the LVS ARP issue entirely, because the virtual IP address(es) only exist on the active LVS director.
iptablesmethod compared to
arptables_jf, as there is overhead in forwarding/masquerading every packet.
iptablesmethod. For example, it is not possible to run two separate Apache HTTP Server services bound to port 80, because both must bind to
INADDR_ANYinstead of the virtual IP addresses.
iptablesmethod, perform the following steps:
- On each real server, run the following command for every VIP, port, and protocol (TCP or UDP) combination intended to be serviced for the real server:
iptables -t nat -A PREROUTING -p <tcp|udp> -d <vip> --dport <port> -j REDIRECTThis command will cause the real servers to process packets destined for the VIP and port that they are given.
- Save the configuration on each real server:
service iptables save#
chkconfig --level 2345 iptables onThe commands above cause the system to reload the
iptablesconfiguration on bootup — before the network is started.
3.3. Putting the Configuration Together
eth0connects to public network and
eth1connects to the private network, then these same devices on the backup LVS router must connect to the same networks.
3.3.1. General LVS Networking Tips
- Bringing Up Real Network Interfaces
- To bring up a real network interface, use the following command as root, replacing N with the number corresponding to the interface (
WarningDo not use the
ifupscripts to bring up any floating IP addresses you may configure using Piranha Configuration Tool (
eth1:1). Use the
servicecommand to start
pulseinstead (see Section 4.8, “Starting LVS” for details).
- Bringing Down Real Network Interfaces
- To bring down a real network interface, use the following command as root, replacing N with the number corresponding to the interface (
- Checking the Status of Network Interfaces
- If you need to check which network interfaces are up at any given time, type the following:
/sbin/ifconfigTo view the routing table for a machine, issue the following command:
3.4. Multi-port Services and LVS
iptables, outside of Piranha Configuration Tool.
3.4.1. Assigning Firewall Marks
iptablesrule to assign marks to the network packets.
/sbin/service iptables status
iptablesis not running, the prompt will instantly reappear.
iptablesis active, it displays a set of rules. If rules are present, type the following command:
/sbin/service iptables stop
/etc/sysconfig/iptablesand copy any rules worth keeping to a safe place before proceeding.
/sbin/iptables -t mangle -A PREROUTING -p tcp -d n.n.n.n/32 --dport 80 -j MARK --set-mark 80
/sbin/iptables -t mangle-A PREROUTING -p tcp -d n.n.n.n/32 --dport 443 -j MARK --set-mark 80
iptablesbefore issuing rules for the first time.
iptablescommands, n.n.n.n should be replaced with the floating IP for your HTTP and HTTPS virtual servers. These commands have the net effect of assigning any traffic addressed to the VIP on the appropriate ports a firewall mark of 80, which in turn is recognized by IPVS and forwarded appropriately.
3.5. Configuring FTP
3.5.1. How FTP Works
- Active Connections
- When an active connection is established, the server opens a data connection to the client from port 20 to a high range port on the client machine. All data from the server is then passed over this connection.
- Passive Connections
- When a passive connection is established, the client asks the FTP server to establish a passive connection port, which can be on any port higher than 10,000. The server then binds to this high-numbered port for this particular session and relays that port number back to the client. The client then opens the newly bound port for the data connection. Each data request the client makes results in a separate data connection. Most modern FTP clients attempt to establish a passive connection when requesting data from servers.
3.5.2. How This Affects LVS Routing
3.5.3. Creating Network Packet Filter Rules
iptablesrules for FTP service, review the information in Section 3.4.1, “Assigning Firewall Marks” concerning multi-port services and techniques for checking the existing network packet filtering rules.
21in the Firewall Mark field. See Section 4.6.1, “The VIRTUAL SERVER Subsection” for details.
220.127.116.11. Rules for Active Connections
iptablescommand allows the LVS router to accept outgoing connections from the real servers that IPVS does not know about:
/sbin/iptables -t nat -A POSTROUTING -p tcp -s n.n.n.0/24 --sport 20 -j MASQUERADE
iptablescommand, n.n.n should be replaced with the first three values for the floating IP for the NAT interface's internal network interface defined in the GLOBAL SETTINGS panel of Piranha Configuration Tool.
18.104.22.168. Rules for Passive Connections
/etc/vsftpd.confto override the real server IP address to the VIP, which is what the client sees upon connection. For example:
10000:20000in the commands below to
iptablescommands have the net effect of assigning any traffic addressed to the floating IP on the appropriate ports a firewall mark of 21, which is in turn recognized by IPVS and forwarded appropriately:
/sbin/iptables -t mangle -A PREROUTING -p tcp -d n.n.n.n/32 --dport 21 -j MARK --set-mark 21
/sbin/iptables -t mangle -A PREROUTING -p tcp -d n.n.n.n/32 --dport 10000:20000 -j MARK --set-mark 21
iptablescommands, n.n.n.n should be replaced with the floating IP for the FTP virtual server defined in the VIRTUAL SERVER subsection of Piranha Configuration Tool.
3.6. Saving Network Packet Filter Settings
iptables, type the following command:
/sbin/service iptables save
/etc/sysconfig/iptablesso they can be recalled at boot time.
/sbin/servicecommand to start, stop, and check the status (using the status switch) of
/sbin/servicewill automatically load the appropriate module for you. For an example of how to use the
/sbin/servicecommand, see Section 2.3, “Starting the Piranha Configuration Tool Service”.
Chapter 4. Configuring the LVS Routers with Piranha Configuration Tool
/etc/sysconfig/ha/lvs.cf. This chapter describes the basic operation of the Piranha Configuration Tool and how to activate LVS once configuration is complete.
lvs.cfand therefore prevent software failures.
4.1. Necessary Software
piranha-guiservice must be running on the primary LVS router to use the Piranha Configuration Tool. To configure LVS, you minimally need a text-only Web browser, such as
links. If you are accessing the LVS router from another machine, you also need an
sshconnection to the primary LVS router as the root user.
sshconnection in a terminal window. This connection provides a secure way to restart
pulseand other services, configure network packet filters, and monitor
/var/log/messagesduring trouble shooting.
4.2. Logging Into the Piranha Configuration Tool
piranha-guiservice is running and an administrative password has been set, as described in Section 2.2, “Setting a Password for the Piranha Configuration Tool”.
http://localhost:3636in a Web browser to access the Piranha Configuration Tool. Otherwise, type in the hostname or real IP address for the server followed by
:3636. Once the browser connects, you will see the screen shown in Figure 4.1, “The Welcome Panel”.
Figure 4.1. The Welcome Panel
piranhafor the Username and the administrative password you created in the Password field.
pulsedaemon, the LVS routing table, and the LVS-spawned
Figure 4.2. The CONTROL/MONITORING Panel
- Auto update
- The status display on this page can be updated automatically at a user configurable interval. To enable this feature, click on the Auto update checkbox and set the desired update frequency in the Update frequency in seconds text box (the default value is 10 seconds).It is not recommended that you set the automatic update to an interval less than 10 seconds. Doing so may make it difficult to reconfigure the Auto update interval because the page will update too frequently. If you encounter this issue, simply click on another panel and then back on CONTROL/MONITORING.The Auto update feature does not work with all browsers, such as Mozilla.
- You can manually update the status information manually by clicking this button.
- Clicking this button takes you to a help screen with information on how to change the administrative password for the Piranha Configuration Tool.
4.4. GLOBAL SETTINGS
Figure 4.3. The GLOBAL SETTINGS Panel
- Primary server public IP
- In this field, enter the publicly routable real IP address for the primary LVS node.
- Primary server private IP
- Enter the real IP address for an alternative network interface on the primary LVS node. This address is used solely as an alternative heartbeat channel for the backup router and does not have to correlate to the real private IP address assigned in Section 3.1.1, “Configuring Network Interfaces for LVS with NAT”. You may leave this field blank, but doing so will mean there is no alternate heartbeat channel for the backup LVS router to use and therefore will create a single point of failure.
NoteThe private IP address is not needed forconfigurations, as all real servers as well as the LVS directors share the same virtual IP addresses and should have the same IP route configuration.
NoteThe primary LVS router's private IP can be configured on any interface that accepts TCP/IP, whether it be an Ethernet adapter or a serial port.
- Use network type
- Click thebutton to select NAT routing.Click thebutton to select direct routing.
- NAT Router IP
- Enter the private floating IP in this text field. This floating IP should be used as the gateway for the real servers.
- NAT Router netmask
- If the NAT router's floating IP needs a particular netmask, select it from drop-down list.
- NAT Router device
- Use this text field to define the device name of the network interface for the floating IP address, such as
NoteYou should alias the NAT floating IP address to the Ethernet interface connected to the private network. In this example, the private network is on the
eth1:1is the floating IP address.
Figure 4.4. The REDUNDANCY Panel
- Redundant server public IP
- Enter the public real IP address for the backup LVS router node.
- Redundant server private IP
- Enter the backup node's private real IP address in this text field.If you do not see the field called Redundant server private IP, go back to the GLOBAL SETTINGS panel and enter a Primary server private IP address and click .
- Heartbeat Interval (seconds)
- This field sets the number of seconds between heartbeats — the interval that the backup node will check the functional status of the primary LVS node.
- Assume dead after (seconds)
- If the primary LVS node does not respond after this number of seconds, then the backup LVS router node will initiate failover.
- Heartbeat runs on port
- This field sets the port at which the heartbeat communicates with the primary LVS node. The default is set to 539 if this field is left blank.
4.6. VIRTUAL SERVERS
Figure 4.5. The VIRTUAL SERVERS Panel
4.6.1. The VIRTUAL SERVER Subsection
Figure 4.6. The VIRTUAL SERVERS Subsection
- Enter a descriptive name to identify the virtual server. This name is not the hostname for the machine, so make it descriptive and easily identifiable. You can even reference the protocol used by the virtual server, such as HTTP.
- Application port
- Enter the port number through which the service application will listen. Since this example is for HTTP services, port 80 is used.
- Choose between UDP and TCP in the drop-down menu. Web servers typically communicate via the TCP protocol, so this is selected in the example above.
- Virtual IP Address
- Enter the virtual server's floating IP address in this text field.
- Set the netmask for this virtual server with the drop-down menu.
- Firewall Mark
- Do not enter a firewall mark integer value in this field unless you are bundling multi-port protocols or creating a multi-port virtual server for separate, but related protocols. In this example, the above virtual server has a Firewall Mark of 80 because we are bundling connections to HTTP on port 80 and to HTTPS on port 443 using the firewall mark value of 80. When combined with persistence, this technique will ensure users accessing both insecure and secure webpages are routed to the same real server, preserving state.
WarningEntering a firewall mark in this field allows IPVS to recognize that packets bearing this firewall mark are treated the same, but you must perform further configuration outside of the Piranha Configuration Tool to actually assign the firewall marks. See Section 3.4, “Multi-port Services and LVS” for instructions on creating multi-port services and Section 3.5, “Configuring FTP” for creating a highly available FTP virtual server.
- Enter the name of the network device to which you want the floating IP address defined the Virtual IP Address field to bind.You should alias the public floating IP address to the Ethernet interface connected to the public network. In this example, the public network is on the
eth0:1should be entered as the device name.
- Re-entry Time
- Enter an integer value which defines the length of time, in seconds, before the active LVS router attempts to bring a real server back into the pool after a failure.
- Service Timeout
- Enter an integer value which defines the length of time, in seconds, before a real server is considered dead and removed from the pool.
- Quiesce server
- When the Quiesce server radio button is selected, anytime a new real server node comes online, the least-connections table is reset to zero so the active LVS router routes requests as if all the real servers were freshly added to the pool. This option prevents the a new server from becoming bogged down with a high number of connections upon entering the pool.
- Load monitoring tool
- The LVS router can monitor the load on the various real servers by using either
ruptime. If you select
rupfrom the drop-down menu, each real server must run the
rstatdservice. If you select
ruptime, each real server must run the
WarningLoad monitoring is not the same as load balancing and can result in hard to predict scheduling behavior when combined with weighted scheduling algorithms. Also, if you use load monitoring, the real servers must be Linux machines.
- Select your preferred scheduling algorithm from the drop-down menu. The default is
Weighted least-connection. For more information on scheduling algorithms, see Section 1.3.1, “Scheduling Algorithms”.
- If an administrator needs persistent connections to the virtual server during client transactions, enter the number of seconds of inactivity allowed to lapse before a connection times out in this text field.
ImportantIf you entered a value in the Firewall Mark field above, you should enter a value for persistence as well. Also, be sure that if you use firewall marks and persistence together, that the amount of persistence is the same for each virtual server with the firewall mark. For more on persistence and firewall marks, refer to Section 1.5, “Persistence and Firewall Marks”.
- To limit persistence to particular subnet, select the appropriate network mask from the drop-down menu.
NoteBefore the advent of firewall marks, persistence limited by subnet was a crude way of bundling connections. Now, it is best to use persistence in relation to firewall marks to achieve the same result.
4.6.2. REAL SERVER Subsection
Figure 4.7. The REAL SERVER Subsection
Figure 4.8. The REAL SERVER Configuration Panel
- A descriptive name for the real server.
NoteThis name is not the hostname for the machine, so make it descriptive and easily identifiable.
- The real server's IP address. Since the listening port is already specified for the associated virtual server, do not add a port number.
- An integer value indicating this host's capacity relative to that of other hosts in the pool. The value can be arbitrary, but treat it as a ratio in relation to other real servers in the pool. For more on server weight, see Section 1.3.2, “Server Weight and Scheduling”.
4.6.3. EDIT MONITORING SCRIPTS Subsection
Figure 4.9. The EDIT MONITORING SCRIPTS Subsection
- Sending Program
- For more advanced service verification, you can use this field to specify the path to a service-checking script. This functionality is especially helpful for services that require dynamically changing data, such as HTTPS or SSL.To use this functionality, you must write a script that returns a textual response, set it to be executable, and type the path to it in the Sending Program field.
NoteTo ensure that each server in the real server pool is checked, use the special token
%hafter the path to the script in the Sending Program field. This token is replaced with each real server's IP address as the script is called by the
nannydaemon.The following is a sample script to use as a guide when composing an external service-checking script:
#!/bin/sh TEST=`dig -t soa example.com @$1 | grep -c dns.example.com if [ $TEST != "1" ]; then echo "OK else echo "FAIL" fi
NoteIf an external program is entered in the Sending Program field, then the Send field is ignored.
- Enter a string for the
nannydaemon to send to each real server in this field. By default the send field is completed for HTTP. You can alter this value depending on your needs. If you leave this field blank, the
nannydaemon attempts to open the port and assume the service is running if it succeeds.Only one send sequence is allowed in this field, and it can only contain printable, ASCII characters as well as the following escape characters:
- \n for new line.
- \r for carriage return.
- \t for tab.
- \ to escape the next character which follows it.
- Enter a the textual response the server should return if it is functioning properly. If you wrote your own sending program, enter the response you told it to send if it was successful.
NoteTo determine what to send for a given service, you can open a
telnetconnection to the port on a real server and see what is returned. For instance, FTP reports 220 upon connecting, so could enter
quitin the Send field and
220in the Expect field.
4.7. Synchronizing Configuration Files
/etc/sysconfig/ha/lvs.cf— the configuration file for the LVS routers.
/etc/sysctl— the configuration file that, among other things, turns on packet forwarding in the kernel.
/etc/sysconfig/iptables— If you are using firewall marks, you should synchronize one of these files based on which network packet filter you are using.
/etc/sysconfig/iptablesfiles do not change when you configure LVS using the Piranha Configuration Tool.
/etc/sysconfig/ha/lvs.cf, is created or updated, you must copy it to the backup LVS router node.
lvs.cffiles. Mismatched LVS configuration files between the LVS router nodes can prevent failover.
sshdmust be running on the backup router, see Section 2.1, “Configuring Services on the LVS Routers” for details on how to properly configure the necessary services on the LVS routers.
lvs.cffiles between the router nodes:
scp /etc/sysconfig/ha/lvs.cf n.n.n.n:/etc/sysconfig/ha/lvs.cf
sysctlfile is only modified once in most situations. This file is read at boot time and tells the kernel to turn on packet forwarding.
4.7.3. Synchronizing Network Packet Filtering Rules
iptables, you will need to synchronize the appropriate configuration file on the backup LVS router.
scp /etc/sysconfig/iptables n.n.n.n:/etc/sysconfig/
sshsession to the backup router or log into the machine as root and type the following command:
/sbin/service iptables restart
4.8. Starting LVS
sshsessions to the primary LVS router.
tail -f /var/log/messages
/sbin/service pulse start
pulseservice's startup in the terminal with the kernel log messages. When you see the following output, the pulse daemon has started properly:
gratuitous lvs arps finished
/var/log/messages, type Ctrl+c.
Appendix A. Using LVS with Red Hat Cluster
Figure A.1. LVS with a Red Hat Cluster
- First tier — LVS routers performing load-balancing to distribute Web requests.
- Second tier — A set of Web servers to serve the requests.
- Third tier — A Red Hat Cluster to serve data to the Web servers.
Appendix B. Revision History
|Revision 1.0-0||Fri Apr 24 2009|
- /etc/sysconfig/ha/lvs.cf file, /etc/sysconfig/ha/lvs.cf
- arptables_jf, Direct Routing and arptables_jf
- direct routing
- and arptables_jf, Direct Routing and arptables_jf
- job scheduling, LVS, LVS Scheduling Overview
- least connections (see job scheduling, LVS)
- /etc/sysconfig/ha/lvs.cf file, /etc/sysconfig/ha/lvs.cf
- components of, LVS Components
- daemon, lvs
- date replication, real servers, Data Replication and Data Sharing Between Real Servers
- direct routing
- initial configuration, Initial LVS Configuration
- ipvsadm program, ipvsadm
- job scheduling, LVS Scheduling Overview
- lvs daemon, lvs
- LVS routers
- multi-port services, Multi-port Services and LVS
- FTP, Configuring FTP
- nanny daemon, nanny
- NAT routing
- overview of, Linux Virtual Server Overview
- packet forwarding, Turning on Packet Forwarding
- Piranha Configuration Tool, Piranha Configuration Tool
- pulse daemon, pulse
- real servers, Linux Virtual Server Overview
- routing methods
- NAT, Routing Methods
- routing prerequisites, Configuring Network Interfaces for LVS with NAT
- scheduling, job, LVS Scheduling Overview
- send_arp program, send_arp
- shared data, Data Replication and Data Sharing Between Real Servers
- starting LVS, Starting LVS
- synchronizing configuration files, Synchronizing Configuration Files
- Red Hat Cluster Manager, A Three-Tier LVS Configuration
- using LVS with Red Hat Cluster, Using LVS with Red Hat Cluster
- lvs daemon, lvs
- multi-port services, Multi-port Services and LVS
- (see also LVS)
- packet forwarding, Turning on Packet Forwarding
- (see also LVS)
- Piranha Configuration Tool, Piranha Configuration Tool
- CONTROL/MONITORING, CONTROL/MONITORING
- EDIT MONITORING SCRIPTS Subsection, EDIT MONITORING SCRIPTS Subsection
- GLOBAL SETTINGS, GLOBAL SETTINGS
- limiting access to, Limiting Access To the Piranha Configuration Tool
- login panel, Logging Into the Piranha Configuration Tool
- necessary software, Necessary Software
- overview of, Configuring the LVS Routers with Piranha Configuration Tool
- REAL SERVER subsection, REAL SERVER Subsection
- REDUNDANCY, REDUNDANCY
- setting a password, Setting a Password for the Piranha Configuration Tool
- VIRTUAL SERVER subsection, The VIRTUAL SERVER Subsection
- VIRTUAL SERVERS, VIRTUAL SERVERS
- piranha-gui service, Configuring Services on the LVS Routers
- piranha-passwd, Setting a Password for the Piranha Configuration Tool
- pulse daemon, pulse
- pulse service, Configuring Services on the LVS Routers
- weighted least connections (see job scheduling, LVS)
- weighted round robin (see job scheduling, LVS)