Alex Scoble's IT Notes - An Information Technology Blog

Networking for OpenStack

When using Mirantis Fuel Web to build up an OpenStack cloud, it requires 6 separate networks to be created. Simplest way to do this is by using virtual networks (VLANs), however this presents a few challenges because of the complexities of configuring VLANs.

This post is specific to Cisco switches, but may be relevant for other networking equipment as well.

For test purposes, this was all done on one switch and each bare metal server had one interface cabled to the switch. In production environments, it is likely that, at a minimum, a completely separate storage network would also exist, but more complicated networking schemes are quite possible.

For each port on the switch that will be connected to one of the bare metal servers, the following needs to be done:

switchport mode trunk
switchport trunk allowed vlan 10,100-104
switchport trunk native vlan 10
spanning-tree portfast trunk
switchport nonegotiate

Trunking mode must be used because each interface needs to be able to handle all 6 of the virtual networks. In the normal access mode, an interface can only communicate over a single VLAN.

For each interface, it's a good practice to specify which vlans it is allowed to handle.

Trunk native tells the switch that untagged traffic goes over this vlan. VLAN tags are the mechanism that allows a network interface to communicate over multiple discrete subnets. So in this case, only traffic going over vlans 100 through 104 will be tagged.

Spanning-tree portfast trunk tells the switch that this port is a trunk connected to a server or PC and to bypass the usual network loop checks.

Switchport nonegotiate forces the switch to treat this interface as a trunk because the connected device does not understand the protocol the switch uses to negotiate trunking vs access modes.

In the switch config the VLANs also need to be activated using:

vlan10,100,101,102,103,104

For those who normally work with vlans in switchport access mode, the switch automatically does this, but apparently, when working with trunks, adding a vlan to a port does not automatically create and enable that vlan.

I am not a networking expert so some particulars here may have been missed, but I hope this is helpful.

Restoring a MySQL database from files

If you are a relative MySQL newbie like me, and someone tells you to backup your databases, you might make the same mistake that I did and copy the database files (all of those .frm, MYD and .MYT files) to a backup folder instead of using MySQL's built in backup method.

Of course, when you make this kind of mistake, often you are required to restore from backups, so you copy the files back to their original location and MySQL bombs on you when you try to start it.

In my case, I backed up the entire /var/lib/mysql directory to /tmp/mysql-backups. When I copied the /tmp/mysql-backups stuff back to /var/lib/mysql, nothing worked. Doing a 'cat /var/log/mysqld.log' showed a "Can't find file: './mysql/plugin.frm' (errno: 13)" error.

Turns out that you have to reset permissions on those files and do some work with SELinux to get that to allow the mysql account to access the databases.

This help is specific to MySQL using InnoDB on CentOS 6.x. I have no way of knowing if it will work for any other configurations.

Here's how to restore your databases and reset the permissions assuming that you have a backup of the entire /var/lib/mysql directory. I'm going to do a bunch of similar chown/chmod statements below just to be sure. You could do the same thing by doing
'chown -R mysql:mysql <directory>' and you'll want to do the separate commands for chmod since the directories and files should have different allowed operations.

rm -rf /var/lib/mysql
mkdir /var/lib/mysql
cp -r /tmp/mysql-backups/* /var/lib/mysql
chown mysql:mysql /var/lib/mysql
chown mysql:mysql /var/lib/mysql/*
chmod 660 /var/lib/mysql/*
chown mysql:mysql /var/lib/mysql/mysql
chmod 700 /var/lib/mysql/mysql
chown mysql:mysql /var/lib/mysql/mysql/*
chmod 660 /var/lib/mysql/mysql/*

Do the chown and chmod commands as shown for any other directories that you have under /var/lib/mysql

If you tried to start mysql right now, you'd still get the same error though if you have SELinux turned on, so now we have to fix that.

chcon -v --type=mysqld_db_t /var/lib/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/*
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql/*

And so forth for any directories that you have under /var/lib/mysql

If you started mysql now, you'd see this error now "[ERROR] Can't start server : Bind on unix socket: Permission denied". SELinux is still blocking access so now you have to do the following:

semanage fcontext -a -t mysqld_db_t "/var/lib/mysql(/.*)?"
restorecon -Rv /var/lib/mysql

It is likely that you will only have to do the 'restorecon' command above, but I'm being thorough here.

You can now start the mysql service ('service mysqld start') and it should work without issues.

Installing Zabbix agent on Debian

I could not find a tutorial that showed how to install the precompiled Zabbix 2.0 agents on a 64 bit Debian box, so I'm making one here.

The latest downloads for precompiled agents and source code for building servers is here. http://www.zabbix.com/download.php

On the Debian system that is getting the agent (remember this is for 64 bit version of Debian, go to the Zabbix download site for the link to the 32 bit agent if your system is 32 bit):

We need both the precompiled agent and the source files

You should be logged in to a root sudo or su session for the following

cd $HOME

wget http://www.zabbix.com/downloads/2.0.0/zabbix_agents_2.0.0.linux2_6_23.amd64.tar.gz

wget http://downloads.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/2.0.0/zabbix-2.0.0.tar.gz

mkdir zabbix

cd zabbix

tar -xzvf ../zabbix_agents_2.0.0.linux2_6_23.amd64.tar.gz

tar -xzvf $HOME/zabbix-2.0.0.tar.gz

cp $HOME/zabbix/bin/* /usr/bin

cp $HOME/zabbix/sbin/* /usr/sbin

groupadd zabbix

useradd -g zabbix zabbix

mkdir /var/log/zabbix-agent

chown zabbix:zabbix /var/log/zabbix-agent

nano /usr/local/etc/zabbix_agent.conf

Change "Server=127.0.0.1" option to the IP address of your Zabbix server, for example:

Server=192.168.1.10

Exit and save the file

nano /usr/local/etc/zabbix_agentd.conf

Change "LogFile=/tmp/zabbix_agentd.log" to:>/P>

LogFile=/var/log/zabbix_agentd.log

Change "Server=127.0.0.1" option to the IP address of your Zabbix server, for example:

Server=192.168.1.10

Change "ServerActive=127.0.0.1" option to the IP address of your Zabbix server, for example:

ServerActive=192.168.1.10

Rem out the "Hostname=Zabbix server" line:

#Hostname=Zabbix server

Exit and save the file

nano /etc/init.d/zabbix-agent

#! /bin/sh
  counter=0
  zabbix_counter=$(ps -ef | grep zabbix_agentd | grep -v grep | wc -l)
  
  start(){
  echo "-------------------------------------------------------------------"
  echo " LAUNCHING ZABBIX AGENT"
  
  if [ $zabbix_counter -gt 0 ]; then
  echo " * Zabbix agent was previously running"
  echo " * Number of Zabbix agentd instances= $zabbix_counter"
  echo "-----------------------------------------------------------------"
  fi
  
  # Checking if the user is able to start the agent.... if the user is not able to, script performs su to
  # the user zabbix and starts the agent
  
  if [ $(whoami) != "zabbix" ];then
  sudo -u zabbix zabbix_agentd
  else
  # Script is acting as the zabbix user, so it can start the agent.
  zabbix_agentd
  fi
  sleep 10
  
  zabbix_counter=$(ps -ef | grep zabbix_agentd | grep -v grep | wc -l)
  if [ $zabbix_counter -gt 0 ]; then
  echo " * Zabbix agent succesfully started"
  echo " * Number of zabbix agentd instances= $zabbix_counter"
  echo "-------------------------------------------------------------------"
  else
  echo " * Zabbix agent couldn't be started, check Zabbix logs"
  echo "-------------------------------------------------------------------"
  fi
  
  }
  
  stop(){
  # Checking if the user is able to stop the agent.... if the user is not able to, script performs su to
  # the user zabbix and kills the agent. Also script tries to kill zabbix-agent processes 5 times using a counter, if at
  # the fith try the agent is still there, script outputs a message to the console.
  
  echo "-------------------------------------------------------------------"
  echo " STOPPING ZABBIX AGENT"
  
  if [ $zabbix_counter -eq 0 ]; then
  echo " * Zabbix agent was not running on this machine"
  echo "-------------------------------------------------------------------"
  fi
  
  while [ $zabbix_counter -gt 0 ] && [ $counter -lt 5 ] ; do
  
  let counter=counter+1
  
  echo " * Number of Attempts (Max 5)=$counter"
  echo " * Stopping zabbix.."
  echo " * Number of zabbix agentd instances= $zabbix_counter"
  
  if [ $(whoami) != "zabbix" ];then
  sudo -u zabbix killall zabbix_agentd > /dev/null &
  else
  killall zabbix_agentd > /dev/null &
  fi
  
  sleep 10
  # Script has a 10 second delay to avoid attempting to kill a process that is still shutting down. If the script
  # can't kill the processes, an error will appear.
  # After 10 seconds script checks again the number of zabbix_agentd processes running,
  # if it's 0, script will exit the loop and continue on
  
  zabbix_counter=$(ps -ef | grep zabbix_agentd | grep -v grep | wc -l)
  
  done
  
  if [ $zabbix_counter -gt 0 ]; then
  echo " * Zabbix agent couldn't be stopped, check Zabbix logs"
  echo "-------------------------------------------------------------------"
  fi
  
  if [ $zabbix_counter -eq 0 ]; then
  echo " * Zabbix agent successfully stopped"
  echo "-------------------------------------------------------------------"
  fi
  
  }
  
  restart(){
  stop
  # Gives system some time to stop processes before script restarts service
  sleep 10
  # Now script can start the agent again
  start
  }
  
  case "$1" in
  start)
  start
  ;;
  stop)
  stop
  ;;
  restart)
  restart
  ;;
  *)
  echo "Usage: zabbix {start|stop|restart}"
  exit 1
  esac
  
  exit 0

Exit and save the file

chmod 744 /etc/ini.d/zabbix-agent

ln -s /etc/init.d/zabbix-agent /sbin/zabbix-agent

update-rc.d zabbix-agent defaults

Go to your Zabbix server's web interface and add the new system to hosts as you normally would and configure accordingly and you are done.

References:

http://www.zabbix.com/forum/showthread.php?p=103244#post103244

Enterprise Linux tools need packaging

With few exceptions (Puppet being one of them, because it just rocks) open source enterprise tools need to be packaged and available in one of the mainstream repositories (even EPEL is good enough! The bar isn't that high!) or your tool just isn't ready for primetime.

Looking at you Plone, LogAnalyzer, nxLog and LogStash! OSSEC gets a pass, but just barely because they are using the funky Atomic repo.

This is where tools like Zabbix and OSSIM come out ahead. The last thing you want to be doing when managing an enterprise, particularly in the industrial control space because of the funky requirements, is worrying about manually updating ANYTHING.

Really, the only reason why Puppet gets a pass is because it's so hugely useful in an enterprise.

And no, I don't want a bunch of excuses on why your tool isn't in a repository, whether it's for Debian, Ubuntu or the Red Hat side of things. If it's not available as a managed package, chances are your tool isn't going to be deployed by me or other IT people like me...and we tend to be a very stodgy group.