LVS and keepalived – An example

LVS

The purpose of the post is just show an example of a Linux box operating as a loadbalancer with LVS and keepalived together.

Briefing:

LVS, or Linux Virtual Server, is a feature of the Linux Kernel for loadbalance services in a Linux box. Check the official site for more.

keepalived is a routing software that implement VRRP in order to manage dynamic gateway and failure. Here you can read more.

The components:

  • Two Linux box with LVS and keepalived, one master and another slave, in case of failure of the master.
  • Two Webservers. I will assume you already have the webservers configured and working.
  • LVS overview

    LVS overview

    Configuration

    LVS is a kernel feature, but in order to handle it, ipvsadm package is needed. In CentOS, you can resolve with:

    yum install -y ipvsadm keepalived
    

    keepalived is also shipped with most of the distros.

    Once installed both, ipvsadm and keepalive, lets configure them. Below follows the /etc/keepalived/keepalived.conf commented:

    global_defs {
       router_id LVS_DEVEL
    }
    

    router_id is just an string to identify.

    vrrp_instance VI_1 {
        state MASTER
        interface eth0
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            100.10.20.50
        }
    }
    

    Above is defined the instance of the virtual router. In the first machine it will have the state as MASTER. In the second machine, use SLAVE instead. Another parameter should be different in both, the virtual_router_id. The number does not matter, just keep it different. The priority will make sense when you have 2 slaves. It defines what SLAVE should become MASTER first.
    The authentication section is needed to the keepalived servers trust each other.
    The virtual_ipaddress is the IP that the clients will connect to.

    virtual_server 100.10.20.50 80 {
        delay_loop 15
        lb_algo rr
        lb_kind DR
        persistence_timeout 50
        protocol TCP
    
        real_server 100.10.20.55 80 {
            HTTP_GET {
                path /
                status_code 200
            }
        }
        real_server 100.10.20.56 80 {
            HTTP_GET {
                path /
                status_code 200
            }
        }
    }
    

    The delay_loop defines in seconds the timer for polling service. lb_algo is round-robind. lb_kind is direct routing, no need tunneling or nat. persistence_timeout is the persistence time in seconds for LVS. The real_server sections define the webservers. Both have the HTTP_GET checker. keepalived will do a http get at the ‘/’ path and expect a 200 code. If it’s not ok, keepalived will disable forwarding till receive 200 status code.

    As soon as keepalived is started, it will configure the VIP in the interface defined at vrrp_instance.

    Now in the webservers, a dummy interface must be configured. So they can receive packages destinated to the VIP 100.10.20.50. First load the dummy module:

    modprobe dummy
    

    Create the dummy interface and configure it:

    ip link add dummy0 type dummy
    ip link set dummy0 up
    ip addr add 100.10.20.50/32
    

    CentOS 7 login with Samba4 LDAP

    Login

    Centralized authentication may be a good solution for large environments. Sysadmin can better manage users’ logins and permissions. Here I list a few steps in order to implement authentication of a CentOS 7 server against an Samba4 LDAP service.

    My example environment

    • Samba4 server = ldap1.example.com
    • CentOS 7 client = localhost.example.com

    My user
    *I will assume your Samba4 server is already running.

    dn: CN=Eduardo de Lima Ramos,OU=people,DC=example,DC=com
    uidNumber: 10000
    unixHomeDirectory: /home/eduardo.ramos
    gidNumber: 10
    loginShell: /bin/bash
    ...
    

    Instalation and configuration

    [root@localhost ~]# yum install -y nss-pam-ldapd 
    

    Now, configure the PAM. Make sure that the following lines exist in these files
    /etc/pam.d/system-auth:

    auth        sufficient    pam_ldap.so use_first_pass
    account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
    password    sufficient    pam_ldap.so use_authtok
    session     optional      pam_ldap.so
    

    /etc/pam.d/password-auth:

    auth        sufficient    pam_ldap.so use_first_pass
    account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
    password    sufficient    pam_ldap.so use_authtok
    session     optional      pam_ldap.so
    session     required      pam_mkhomedir.so umask=0027
    

    That will make PAM use ldap users and create homedir when it does not exist.

    Now, we need to configure the nslcd daemon. Use the following model.
    /etc/nslcd.conf

    uid nslcd
    gid ldap
    
    uri ldap://ldap1.example.com
    ldap_version 3
    base dc=example,dc=com
    
    binddn LOCAL\Administrator
    bindpw super$ecret
    
    pagesize 1000
    referrals off
    idle_timelimit 800
    filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
    map    passwd uid              sAMAccountName
    map    passwd homeDirectory    unixHomeDirectory
    map    passwd gecos            displayName
    filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
    map    shadow uid              sAMAccountName
    map    shadow shadowLastChange pwdLastSet
    filter group  (objectClass=group)
    
    ssl no
    tls_cacertdir /etc/openldap/cacerts
    

    Now, just enable and start the nslcd and nscd daemons:

    [root@localhost ~]$ sudo systemctl enable nslcd
    [root@localhost ~]$ sudo systemctl enable nscd
    [root@localhost ~]$ sudo systemctl start nslcd
    [root@localhost ~]$ sudo systemctl start nscd
    

    Try to login.

    A little dev with op

    github

    I’m not a great developer, but I use to write some in order to help me with the operation. I little ‘dev’ with ‘op’.

    So, you can see on my github some of my prodution:

    check_disckactivity: A NRPE plugin that gets the level of activity of a disk/partition. The data source is sysstat.

    imaptest: That script simulate IMAP clients interacting with server. It can be useful in lab/tests environment (it is for me).

    check_proc_blocked: Another NRPE plugin that checks for processes in uninterruptible state in Linux. A high number could indicate problems with I/O.

    check_nimap: That NRPE plugin show the number of dovecot’s IMAP connections and the number of unique users.

    rebuildpkg: If you need to backup the current version of a package which is installed on your Slackware, you can regenerate it. It is useful when you can’t find that version on the web, but has it installed on your system.

    oVirt plugin – shell in a box

    OVirt logo

    I would like to share my experience implementing this great oVirt UIPlugin.

    I followed the Derez Blog but with some changes in order to work in oVirt 3.4.

    oVirt Engine

    First, I created the plugin directory:

    # mkdir /usr/share/ovirt-engine/ui-plugins/shellbox-files
    

    Then I put the start.html file into it, with that content:

    # cat << EOF > /usr/share/ovirt-engine/ui-plugins/shellbox-files/start.html
    <!DOCTYPE html>
    <html>
    <head>
    <script type='text/javascript'>
      var api = parent.pluginApi('ShellBoxPlugin');
    
      api.register({
        UiInit : function() {
          // Add 'Shell Box' sub-tab under 'Hosts' main-tab
          api.addSubTab('Host', 'Shell Box', 'shell-box', '');
    
          // Add 'Shell Box' button (+ context menu)
          // to 'Hosts' main-tab
          api.addMainTabActionButton('Host', 'Shell Box', {
            onClick : function() {
              window.open(getShellBoxUrl(arguments), '_blank');
            },
            isEnabled : function() {
              // The button is enabled only when a
              // single host is selected
              return arguments.length == 1;
            },
            isAccessible : function() {
              // The button is always visible
              return true;
            }
          });
        },
        HostSelectionChange : function() {
          if (arguments.length == 1) {
            // Update iframe URL on host selection
            api.setTabContentUrl(
              'shell-box', getShellBoxUrl(arguments));
          }
        },
      });
      api.ready();
    
      // Get 'Shell Box' URL using specified host address
      var getShellBoxUrl = function(arguments) {
        var hostAddress = arguments[0].name;
        var port = '4200';
        var shellUrl = 'https://' + hostAddress + ':' + port;
    
        return shellUrl;
      }
    </script>
    </head>
    <body>
    </body>
    </html>
    EOF
    

    Now, create the plugin definition:

    # cat << EOF > /usr/share/ovirt-engine/ui-plugins/shellbox.json
    {
      "name": "ShellBoxPlugin",
      "url": "/ovirt-engine/webadmin/plugin/ShellBoxPlugin/start.html",
      "resourcePath": "shellbox-files"
    }
    EOF
    

    Restart oVirtn Engine:

    # service ovirt-engine restart
    

    oVirt Host

    The backend of this plugin is the shellinabox package. Install it with:

    # yum install shellinabox -y
    

    Edit OPTS option in /etc/sysconfig/shellinaboxd like this:

    OPTS="--service /:SSH"

    Shellinabox will generate https certificated automatically, but invalid. I’ve found the solution here.

    Now start shellinaboxd and test the access.

    # service shellinaboxd start
    oVirt Shell

    oVirt Shell

    oVirt host – iptables

    OVirt logo

    When you add a new host to your oVirt Engine, your iptables rules are overwritten by oVirt deploy. The new rules might not meet your needs. But you can change this.

    oVirt 3.4

    Using engine-config command in Engine host, get the default rules:

    $ sudo engine-config -g IPTablesConfig
    IPTablesConfig: 
    # oVirt default firewall configuration. Automatically generated by vdsm bootstrap script.
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    
    -A INPUT -i lo -j ACCEPT
    # vdsm
    -A INPUT -p tcp --dport @VDSM_PORT@ -j ACCEPT
    # SSH
    -A INPUT -p tcp --dport @SSH_PORT@ -j ACCEPT
    # snmp
    -A INPUT -p udp --dport 161 -j ACCEPT
    
    @CUSTOM_RULES@
    
    # Reject any other input traffic
    -A INPUT -j REJECT --reject-with icmp-host-prohibited
    -A FORWARD -m physdev ! --physdev-is-bridged -j REJECT --reject-with icmp-host-prohibited
    COMMIT
    

    To set new rules, copy the lines returned above and add your rules just after @CUSTOM_RULES@, for example:

    $ sudo engine-config -s IPTablesConfig="
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [0:0]
    -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    
    -A INPUT -i lo -j ACCEPT
    # vdsm
    -A INPUT -p tcp --dport @VDSM_PORT@ -j ACCEPT
    # SSH
    -A INPUT -p tcp --dport @SSH_PORT@ -j ACCEPT
    # snmp
    -A INPUT -p udp --dport 161 -j ACCEPT
    
    @CUSTOM_RULES@
    -A INPUT -m comment --comment 'new rule '-j LOG --log-prefix='new rule '
    
    # Reject any other input traffic
    -A INPUT -j REJECT --reject-with icmp-host-prohibited
    -A FORWARD -m physdev ! --physdev-is-bridged -j REJECT --reject-with icmp-host-prohibited
    COMMIT"
    

    oVirt 3.5

    New version has a proper variable for this. Follow the example:

    $ sudo engine-config --set IPTablesConfigSiteCustom="
    -A INPUT -m comment --comment 'new rule '-j LOG --log-prefix='new rule '
    "

    That new rule will be set in place of @CUSTOM_RULES@.

    Problem compiling on Slackware64 multilib

    compile

    Some times when I am installing packages from SlackBuilds, the compilation stops with the following error:

    error adding symbols: File in wrong format
    

    In order to fix it, I alter the SlackBuild script specifying the library directory in CFLAGS variable:

    CFLAGS="$SLKCFLAGS -L/usr/lib64"
    

    Running again the script, it terminates with success.

    Source based routing

    Routes

    Most of network routing is based on the destination. But sometimes you may need to forward packets to different gateways depending on the source.

    In Linux you can do this using the iproute2 package. It uses netlink socket interface in order to handle addressment, routing, queuing and scheduling of Linux network subsystem. Follow an example:

    Define a lable for a table to be used:

    echo "10 foo" >> /etc/iproute2/rt_tables
    

    Insert a route into foo table:

    ip route add default via 10.10.10.1 dev eth1 table foo
    

    Insert a rule with low priority in order to a host consult the new table foo:

    ip rule add prio 10 from 192.168.16.7 lookup foo
    

    You can check the rules with:

    ip rule show list
    

    Use the man for more information.

    Nagios plugin – check_proc_blocked

    Nagios Logo

    I have written another plugin using python to get the number of process in uniterruptive state (D). This state means that the process is waiting for a I/O operation, and a high number, it can mean problem. The code you can get here.

    For more information of Nagios plugin development, read the guidelines.

    Nagios plugin – check_nimap.py

    Nagios Logo

    In order to learn python and monitor my mail service, I have written a Nagios plugin that gets the number of imap clients connected to a dovecot imap server. The code you can get here.

    You can see that check_nimap.py uses doveadm so as to get connections information. So, you need to configure NRPE to run as nagios user. A good guide can be found here.

    For more information of Nagios plugin development, read the guidelines.

    Internal/Isolated networks on oVirt

    OVirt logo

    Although virt-manager provides easily isolated network, oVirt haven’t so evident configuration. In fact, we need some commands on terminal.

    You can use dummy module to get internal networks. First of all, make sure your host load dummy module at startup.
    Create /etc/sysconfig/modules/dummy.modules:

    modprobe dummy >/dev/null 2&1
    exit0
    

    Manually, you can run modprobe to load on a running machine. It will appear a dummy0 network interface. Done this, create /etc/sysconfig/network-scripts/ifcfg-dummy0 with this content:

    DEVICE=dummy0
    BOOTPROTO=none
    ONBOOT=yes
    NM_CONTROLLED=no
    PROMISC=yes
    

    Now comes the oVirt configuration. In webadmin portal, go to the ‘Network’ tab and click new:

    New network

    New network

    The definition could be simple. Just give a name and match ‘VM network':

    New network

    New network

    With the virtual switch created, we need to link our dummy interface on it. Go to the network configuration of host:

    Configure network on host

    Configure network on host

    Configure network on host

    Configure network on host

    Drag internal network and drop in dummy0 interface

    Configure network on host

    Configure network on host

    Check ‘Save network configuration’ and click ok.

    Configure network on host

    Configure network on host

    Now, for each virtual machine you want to use internal network, you can create a virtual NIC and attach to internal virtual switch.

    Configure network on host

    Configure network on host