Alex Scoble's IT Notes - An Information Technology Blog

Networking for OpenStack

When using Mirantis Fuel Web to build up an OpenStack cloud, it requires 6 separate networks to be created. Simplest way to do this is by using virtual networks (VLANs), however this presents a few challenges because of the complexities of configuring VLANs.

This post is specific to Cisco switches, but may be relevant for other networking equipment as well.

For test purposes, this was all done on one switch and each bare metal server had one interface cabled to the switch. In production environments, it is likely that, at a minimum, a completely separate storage network would also exist, but more complicated networking schemes are quite possible.

For each port on the switch that will be connected to one of the bare metal servers, the following needs to be done:

switchport mode trunk
switchport trunk allowed vlan 10,100-104
switchport trunk native vlan 10
spanning-tree portfast trunk
switchport nonegotiate

Trunking mode must be used because each interface needs to be able to handle all 6 of the virtual networks. In the normal access mode, an interface can only communicate over a single VLAN.

For each interface, it's a good practice to specify which vlans it is allowed to handle.

Trunk native tells the switch that untagged traffic goes over this vlan. VLAN tags are the mechanism that allows a network interface to communicate over multiple discrete subnets. So in this case, only traffic going over vlans 100 through 104 will be tagged.

Spanning-tree portfast trunk tells the switch that this port is a trunk connected to a server or PC and to bypass the usual network loop checks.

Switchport nonegotiate forces the switch to treat this interface as a trunk because the connected device does not understand the protocol the switch uses to negotiate trunking vs access modes.

In the switch config the VLANs also need to be activated using:

vlan10,100,101,102,103,104

For those who normally work with vlans in switchport access mode, the switch automatically does this, but apparently, when working with trunks, adding a vlan to a port does not automatically create and enable that vlan.

I am not a networking expert so some particulars here may have been missed, but I hope this is helpful.

A few months ago Microsoft released security patch MS12-034 Because of this patch and the way that it installs, I ran into problems installing it on Win XP and 2003 systems.

Specifically, the patch does a check of a system's keyboard layout registry keys and makes sure that each layout key has a resulting dll file. If no file is found, it adds the missing dll to a file named C:\Windows\faultykeyboard.log and stops the install until either the registry keys are removed, or the missing dll is readded to the file system.

In my case, I found that all of the systems having problems installing the patch were failing because kbdsf.dll and kbdcr.dll were missing from C:\Windows\System32. The offending registry objects are:

Value: HKLM\System\CurrentControlSet\Control\Keyboard Layout\DoskeybCodes, value 0000100c
Key: HKLM\System\CurrentControlSet\Control\Keyboard Layouts\0000046e
Key: HKLM\System\CurrentControlSet\Control\Keyboard Layouts\0000141a

The two files, incidentally are for the "Luxembourgish" and Slovakian keyboard layouts.

Since we use neither layout here, the simple fix was to remove the registry keys before running the patch.

Because we use a patch management system and because that system is able to run VB scripts in order to do things like remove registry keys, it was a fairly simple matter of researching the VBScript code required to delete the keys.

I made a custom patch in the system that first checks to make sure that the offending registry objects are not in the registry, and if they are then proceeds to run the following VBScript to remove the keys:

Const HKEY_LOCAL_MACHINE = &H80000002

strComputer = "."

Set objRegistry=GetObject("winmgmts:\\" & _
 strComputer & "\root\default:StdRegProv")

strKeyPath1 = "System\CurrentControlSet\Control\Keyboard Layout\DoskeybCodes"
strKeyPath2 = "System\CurrentControlSet\Control\Keyboard Layouts\0000046e"
strKeyPath3 = "System\CurrentControlSet\Control\Keyboard Layouts\0000141a"
strValueName = "0000100c"

objRegistry.DeleteValue HKEY_LOCAL_MACHINE, strKeyPath1, strValueName
objRegistry.DeleteKey HKEY_LOCAL_MACHINE, strKeyPath2
objRegistry.DeleteKey HKEY_LOCAL_MACHINE, strKeyPath3
ReportRepairResult True,"nice job!"

Here's a quick rundown of what the script does... It sets up a constant &H80000002 as HKEY_LOCAL_MACHINE (go here to find out what constants apply to which registry hive)
It then sets up a number of variables to be used later, including the registry keys and value to be deleted. It then deletes the registry objects one at a time using the previously defined variables to complete the function. Lastly we echo True and nice job! out so that the system running the script knows that the script completed successfully. Without that last bit, our patch management system would keep reporting that the custom patch failed.

We then set up a custom patch cloned from the patch management system's own definition for MS12-034 and add our registry fix as a dependency so that it runs before MS12-034 is installed.

And there you have it. The things we go through to fix Microsoft's messes, eh?

Restoring a MySQL database from files

If you are a relative MySQL newbie like me, and someone tells you to backup your databases, you might make the same mistake that I did and copy the database files (all of those .frm, MYD and .MYT files) to a backup folder instead of using MySQL's built in backup method.

Of course, when you make this kind of mistake, often you are required to restore from backups, so you copy the files back to their original location and MySQL bombs on you when you try to start it.

In my case, I backed up the entire /var/lib/mysql directory to /tmp/mysql-backups. When I copied the /tmp/mysql-backups stuff back to /var/lib/mysql, nothing worked. Doing a 'cat /var/log/mysqld.log' showed a "Can't find file: './mysql/plugin.frm' (errno: 13)" error.

Turns out that you have to reset permissions on those files and do some work with SELinux to get that to allow the mysql account to access the databases.

This help is specific to MySQL using InnoDB on CentOS 6.x. I have no way of knowing if it will work for any other configurations.

Here's how to restore your databases and reset the permissions assuming that you have a backup of the entire /var/lib/mysql directory. I'm going to do a bunch of similar chown/chmod statements below just to be sure. You could do the same thing by doing
'chown -R mysql:mysql <directory>' and you'll want to do the separate commands for chmod since the directories and files should have different allowed operations.

rm -rf /var/lib/mysql
mkdir /var/lib/mysql
cp -r /tmp/mysql-backups/* /var/lib/mysql
chown mysql:mysql /var/lib/mysql
chown mysql:mysql /var/lib/mysql/*
chmod 660 /var/lib/mysql/*
chown mysql:mysql /var/lib/mysql/mysql
chmod 700 /var/lib/mysql/mysql
chown mysql:mysql /var/lib/mysql/mysql/*
chmod 660 /var/lib/mysql/mysql/*

Do the chown and chmod commands as shown for any other directories that you have under /var/lib/mysql

If you tried to start mysql right now, you'd still get the same error though if you have SELinux turned on, so now we have to fix that.

chcon -v --type=mysqld_db_t /var/lib/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/*
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql/*

And so forth for any directories that you have under /var/lib/mysql

If you started mysql now, you'd see this error now "[ERROR] Can't start server : Bind on unix socket: Permission denied". SELinux is still blocking access so now you have to do the following:

semanage fcontext -a -t mysqld_db_t "/var/lib/mysql(/.*)?"
restorecon -Rv /var/lib/mysql

It is likely that you will only have to do the 'restorecon' command above, but I'm being thorough here.

You can now start the mysql service ('service mysqld start') and it should work without issues.

Patching is killing Windows (AKA the problem that Microsoft MUST solve in order to remain competitive)

I wrote a blog post back in 2005 on my Computerworld blog detailing 4 things that I thought that Microsoft needed to fix in Windows in order to remain competitive. Unfortunately, the biggest thing on that list, the land of a thousand reboots, hasn't gotten much better.

Right now, one of my main functions at my job is to manage patches for nearly a thousand Windows systems. "Pain in the ass" doesn't even begin to describe the difficulties of patching that many Windows servers and workstations.

First off, you can't do it without an enterprise tool like LANDesk, Secunia, GFI LANGuard, Microsoft SCCM, etc. These tools are big, complex, hard to manage and typically require agents which often have their own issues.

Second, the number of third party apps that you have to track and patch is significant and growing. Because of compliance requirements, some organizations have to assess every single "security patch" for applicability. To do this, you have to know every piece of software that you have installed in areas requiring compliance and you have to regularly search for security updates to all of those applications. Yes, it is as laborious and mind numbing as it sounds. I have a lot of fun at my job, but this area is totally not intellectually stimulating at all.

Third, because of the fact that Windows patches require a reboot, scheduling downtime of critical production systems can be a total nightmare.

I should mention right now that Linux does all of this far better. If you pick your tools right (limit the tools that can't be installed from a repository), you manage all of your patching from one place. Yum for RedHat/CentOS or Aptitude for Debian. If your server is headless (who needs a GUI that takes up all of those resources and adds another 200 packages that need to be updated), patching a brand new server up to current takes minutes and requires ZERO reboots. It's as easy as typing "yum update". Since it can be used to track what security patches are applicable for each system, there's no research that needs to be done on just about anything other than the stuff that you installed from source.

Linux is quite simply light years ahead of Windows in this area. It is probably the main reason that I'm getting into Linux so much of late.

It sometimes takes over an hour to install a service pack on a Windows server. I can build up a new Linux server and fully patch it in less time. In fact, I can build up a new server with the LAMP stack, install an app on top of that (for log management, for instance) AND fully patch it in less time.

It's no wonder that Microsoft's share of datacenter systems has remained weak and will continue to slide. This situation is unacceptable. Even with clustering, which is expensive, hard to maintain and requires a lot more hardware, Microsoft can't maintain the kind of uptime that Linux is able to. Because guess what, patching invariably BREAKS your cluster.

On Windows 7, it can take over an hour to fully patch a new laptop or workstation and thats with SP1 already installed.

Even a shark with a congested nose can smell the blood in the water.

Linux administrators are able to perform patch management on their systems without buying expensive tools, without wasting a lot of time performing CVE and NVD searches for their apps, with a minimal amount of downtime and with significantly lower risk to their systems. Not to mention that it's much easier to deal with patching of systems isolated from the internet because you can build your own repositories.

Microsoft will continue to bleed marketshare and loyal customers like me until they solve this problem.

It will likely take a complete rewrite of Windows to do it, but given that the survival of the company depends on it, at least in my opinion, it would be stupid of them not to do it and soon.

I won't hold my breath though.

Dear Bill Gates, features and fixes I'd like to see in next Windows

This was originally posted to my ComputerWorld blog back in 2005, but as some of it is still pertinent today, I'm recreating it here. 

I'm sure that anyone who's used Windows has at least one pet peeve that they'd like to see fixed. As much as I like Windows (at least in its latest incarnation), I'm no exception.

What's more annoying is that some issues have been around for several versions of Windows (Office too).

OK, so let's get to it! Here's my list of things I'd like to see changed in Windows Mountain View (sorry, still can't call it what Microsoft wants to call it -psst...Windows Vista-...hurts my fingers just thinking about typing it).

1. Focus, focus focus! 

The way Windows changes focus, or how it designates what is the current active window, has been an annoyance of mine for a while now. 

For example, you are working on something, a Word document for instance, you need to reference a web page, so you click on the Firefox icon (or IE) in the Quick Launch bar and get back to typing your current thought stream on the Word document, when all of a sudden mid-sentence, you realize that you are now on your homepage in Firefox. Exasperated, you sigh, click back on your Word document and retype the thought stream from the point where Windows, thoughtfully, changed the active window.

Hey Microsoft, here's a hint for you in as plain english as I can make it: Never, ever, never, ever, ever, ever, never, ever, ever, ever, ever change focus from the window that I'm WORKING on until I EXPLICITY COMMAND Windows to do so!!!!

I don't CARE what your usability testing shows you, I work in specific ways and it would be MOST APPRECIATIVE if I was given the choice on how Windows handles change of focus.

2. I'm the boss!! 

I'm trying to delete files, a lot of them, from the Temp directory for instance. Windows pops up an error window telling me that the file is in use and can't be deleted. It doesn't tell me the specific app that is holding the file open. It doesn't ask me if I want to delete it anyways and it doesn't give me the choice to continue the file delete with the next file. LAME! 

Give me the choice, I don't care how hidden it is, as long as it's there and documented. Unless the file is a required system file, ask me if I want to disconnect the app's connection to the file and delete the file. 

Let me continue the delete from the next file chosen. 

Give me choice! Give me control! Give me more information! Gimme, gimme, gimme!

3. Major Tom to Ground Conrol.

An error comes up when I'm running an application. An error is in the event viewer. I'm having problems performing a specific task (such as trying to connect to console session of a Windows 2003 Server using RDP).

Where's the info in the event logs? Why are you making me click on a link to find out what the error means? Why is there nothing in the event logs at all? Why is the error not listed anywhere in the knowledge base with the same context as my error?

Windows should do a much better of job of telling the user (and an even better job of telling the admin) what's going on under the hood. What does the error mean, why is that application not starting, what's causing my inability to connect to another system's event logs.

I shouldn't have to download support tools in order to drill down to the lower levels of what's going on in Windows. It should tell me.

All error codes should translate into real english messages in the event viewer. All problems should be logged somewhere, or Windows should ask the user if they'd like the problem to be thoroughly added to the event log and Microsoft should at least let customers know that their problem is known even if there's not currently a fix.

Microsoft could even go the extra mile and ask the user if they'd like to be notified if a fix is created for their particular issue.

4. Welcome to the land of a thousand reboots.

So you install a new app, Windows asks you to reboot. You install a new driver for your printer, Windows asks you to reboot. You install a new security update, Windows asks you to reboot. You sneeze, Windows asks you to reboot. You run into a problem, so you reboot. Reboot, reboot, reboot after reboot.

I think you get the idea even if I have exaggerated the issue, but not by much. Sometimes it feels like I spend half my life waiting for systems to reboot. Perhaps it's not quite that much, but it's still too much. 

Even with all the improvements that Microsoft has made in this area it's still not enough. Linux only needs to be rebooted for kernel updates (as far as I know, perhaps Martin MC Brown could shed more light on this), in other words, very very rarely.

With the amount of updates, patches and security fixes that Microsoft puts out for Windows, the reboot requirements are way too high. The only way to get ultra high reliability with Windows currently is to run it in clusters. This is simply not acceptable for small businesses that can't afford double the servers and licenses.

Because we can't afford double the hardware and software is no excuse for us not to have high levels of uptime.

These are just 4 of the issues that I have with Microsoft. To some they might seem minor, but in my line of work, I am bumping up against these problems almost every day.

Microsoft is starting to bump up against competitors in the consumer sector and business sector. In light of this, I would suggest to Microsoft and all of its employees, that good enough isn't. New features aren't enough. You have to be better than your competitors, some of whom offer their products for free, far better. You just about have to be perfect.

You have a long way to go. And Windows Vista better be the next big leap instead of the next small step.

Linux would not have protected the Iranians against Stuxnet

Interesting discussion going on over at Google+ about the ramifications of Stuxnet and Flame. 

One part of the conversation is completely *facepalm*/smh (shaking my head) worthy, though. Someone brought up the extremely ludicrous notion that Linux would have protected the Iranians against Stuxnet and Flame. Linux totally would not have given the Iranians any extra protection.

The Stuxnet attacks are highly sophisticated and targeted and were able to get through every layer of the Iranian infrastructure to get embedded within the centrifuge controls. Flame, at least as far as I can tell, may have been used to provide the reconnaissance needed in order to build Stuxnet. It wouldn't have mattered what types of computers and OSes the Iranians used. The attackers would have just found vulnerabilities in any of them and developed exploits accordingly.

These hacks were very difficult to defend against and would have required a very thorough "defense in depth" strategy with a dedicated staff of security analysts to monitor and analyze network, OS and application logs and health. Even then, given the fact that spies were involved, It would have been an uphill battle to stop these attacks from succeeding.

There are certainly advantages that Linux gives to organizations within datacenters. Protecting them against highly advanced and targeted threats that exploit unknown weaknesses is not one of them, in my opinion.

How to install Snort on CentOS 5.6 from source files

It took me a while and a lot of searching around to find all the pieces to do this install so I figured that I might as well document the whole process step by step in case someone else needs help and to help me remember what I did.

They do have RPMs for Snort if you'd rather go that route, but Sourcefire (the makers of Snort) recommend that you install by compiling from source, which is what this instructional blog post is about.

The first thing you'll want to do is install CentOS 5.6 and run the autoupdater.

Next you'll want to download the following source packages:

Snort http://www.snort.org/downloads/867

Daq http://www.snort.org/downloads/860

Libpcap http://www.tcpdump.org/release/libpcap-1.1.1.tar.gz

TCPDump http://www.tcpdump.org/release/tcpdump-4.1.1.tar.gz

Libpcre http://sourceforge.net/projects/pcre/files/pcre/8.12/pcre-8.12.tar.gz/download

Libdnet http://libdnet.googlecode.com/files/libdnet-1.12.tgz

After downloading those files, you'll want to extract them. It's your choice where to extract them, but I just do it in the same folder that I downloaded them to.

Open up a terminal (command line) window and log in as root using

su - root

and enter your root password.

Make sure that gcc and the c++ module for gcc are installed:

yum install gcc

yum install gcc-c++

Change directories to where the libpcap source files were extracted to using the cd command, in my case the command is:

cd /home/alex/downloads/cd libpcap-1.1.1

Follow that with the following commands (these commands assume that libpcap 0.9.4 was installed per the default CentOS 5.6 install):

./configure

make

make install

That finishes the install of libpcap, so now we have to remove the links to the old version and create the new symbolic links.

cd /usr/lib

rm libpcap.so.0

rm libpcap.so.0.9

ln -s /usr/local/lib/libpcap.so.1.1.1 /usr/lib/libpcap.so.1.1.1

ln -s /usr/lib/libpcap.so.1.1.1 /usr/lib/libpcap.so.1

ln -s /usr/lib/libpcap.so.1 /usr/lib/libpcap.so

And now to the extracted directories from above for the daq source files and to do the install of daq:

cd /home/usr/alex/downloads/daq-0.5

./configure

make

make install

Next we need to install libpcre:

cd /home/usr/alex/downloads/pcre-8.12

./configure

make

make install

Next we need to install libdnet:

cd /home/usr/alex/downloads/libdnet-1.12

./configure

make

make install

And finally we can install snort:

cd /home/usr/alex/downloads/snort-2.9.0.5

./configure

make

make install

And you are done with the installation of Snort. You can now run it by doing this command:

/usr/local/bin/snort -v

You should now see information on the console for any packets that are received by your system's network adapters. You use ctrl-c command to exit from Snort.

Linux Guide for Windows Admins

A while back I posted on my Computerworld blog about my experiences with several flavors of Linux and my thoughts of each.

Today, over on Mark's (We)blog, Mark Wilson posts his thoughts on recent experiences with different *NIX flavors including Solaris 10, Red Hat 4 and SUSE Linux 10.

Good read for anyone interested in getting their toes wet in the Linux pool.

Of particular interest is the information on the "root" account, run levels and how *NIX systems set up a disk hierarchy (partitions, folder and such).

Microsoft releases WMF exploit patch early

Well if you were wondering if Microsoft pays attention to the grumblings of its userbase, you can stop wondering.

They released the official patch for the WMF exploit that has been widely publicized of late today. This is five days ahead of the expected and previously announced 10 Jan 2006 date.

Go here to download the Microsoft patch for the WMF exploit for Windows.

From Computerworld blog: Possible bug in Windows patch 902400

The following is excerpted completely from my Computerworld blog. Normally, I'd just give the highlights, but felt this was sufficiently important to give the full scoop on either blog.

A week ago, Microsoft released fix under KB article 902400 for security bulletin MS05-051. The patch is to fix a known vulnerability with the COM+ and Microsoft Distributed Transaction Coordination (MSDTC) subsystems in Windows.

Unfortunately, as many have found out, in certain circumstances the patch can break various network applications and prevent some tools from working properly.

A fix for some problems that may arise out of installation of the patch is detailed in MS KB article 909444.

There is also some information on turning on Transaction Internet Protocol (TIP) in Win 2000 machines after the patch in KB article 908620. TIP was disabled by default on XP and 2003 machines, so the article is probably not relevant for most XP or 2003 machines unless the administrator turned on TIP.

I think I have also stumbled onto an as yet undocumented problem related to this patch. I can no longer use Remote Desktop Protocol (RDP) to connect to the console session of the two servers that received the patch. So I'm using UltraVNC for these servers for the time being until I can fix the problem.

As good as UltraVNC is versus other flavors of VNC, it's still not as good performance wise as RDP.

Anyhow, so far I haven't found a solution to the problem. And Server 2003 isn't logging any information on why RDP can't connect to console in Event Viewer either.

The only errors that I have seen are on pop-ups that appear after the attempt to connect fails. I get either "Error [7051]: The requested session is not configured to allow Remote   Control." if I try connecting from Terminal Services Manager, or if I attempt to connect using RDP "Error connecting to existing session for <username> (Id 0). The operation completed successfully." pops up.

All the fixes I've found on Microsoft or on Google (there's that search problem rearing it's ugly head again) to date, haven't resolved the issue.

Oh, and the fact that this latest round of updates requires a restart is further validation that Windows has room to improve in this area with the next version.

If anyone has any ideas, let me know.