Alex Scoble's IT Notes - An Information Technology Blog

Restoring a MySQL database from files

If you are a relative MySQL newbie like me, and someone tells you to backup your databases, you might make the same mistake that I did and copy the database files (all of those .frm, MYD and .MYT files) to a backup folder instead of using MySQL's built in backup method.

Of course, when you make this kind of mistake, often you are required to restore from backups, so you copy the files back to their original location and MySQL bombs on you when you try to start it.

In my case, I backed up the entire /var/lib/mysql directory to /tmp/mysql-backups. When I copied the /tmp/mysql-backups stuff back to /var/lib/mysql, nothing worked. Doing a 'cat /var/log/mysqld.log' showed a "Can't find file: './mysql/plugin.frm' (errno: 13)" error.

Turns out that you have to reset permissions on those files and do some work with SELinux to get that to allow the mysql account to access the databases.

This help is specific to MySQL using InnoDB on CentOS 6.x. I have no way of knowing if it will work for any other configurations.

Here's how to restore your databases and reset the permissions assuming that you have a backup of the entire /var/lib/mysql directory. I'm going to do a bunch of similar chown/chmod statements below just to be sure. You could do the same thing by doing
'chown -R mysql:mysql <directory>' and you'll want to do the separate commands for chmod since the directories and files should have different allowed operations.

rm -rf /var/lib/mysql
mkdir /var/lib/mysql
cp -r /tmp/mysql-backups/* /var/lib/mysql
chown mysql:mysql /var/lib/mysql
chown mysql:mysql /var/lib/mysql/*
chmod 660 /var/lib/mysql/*
chown mysql:mysql /var/lib/mysql/mysql
chmod 700 /var/lib/mysql/mysql
chown mysql:mysql /var/lib/mysql/mysql/*
chmod 660 /var/lib/mysql/mysql/*

Do the chown and chmod commands as shown for any other directories that you have under /var/lib/mysql

If you tried to start mysql right now, you'd still get the same error though if you have SELinux turned on, so now we have to fix that.

chcon -v --type=mysqld_db_t /var/lib/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/*
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql
chcon -v --type=mysqld_db_t /var/lib/mysql/mysql/*

And so forth for any directories that you have under /var/lib/mysql

If you started mysql now, you'd see this error now "[ERROR] Can't start server : Bind on unix socket: Permission denied". SELinux is still blocking access so now you have to do the following:

semanage fcontext -a -t mysqld_db_t "/var/lib/mysql(/.*)?"
restorecon -Rv /var/lib/mysql

It is likely that you will only have to do the 'restorecon' command above, but I'm being thorough here.

You can now start the mysql service ('service mysqld start') and it should work without issues.

Enterprise Linux tools need packaging

With few exceptions (Puppet being one of them, because it just rocks) open source enterprise tools need to be packaged and available in one of the mainstream repositories (even EPEL is good enough! The bar isn't that high!) or your tool just isn't ready for primetime.

Looking at you Plone, LogAnalyzer, nxLog and LogStash! OSSEC gets a pass, but just barely because they are using the funky Atomic repo.

This is where tools like Zabbix and OSSIM come out ahead. The last thing you want to be doing when managing an enterprise, particularly in the industrial control space because of the funky requirements, is worrying about manually updating ANYTHING.

Really, the only reason why Puppet gets a pass is because it's so hugely useful in an enterprise.

And no, I don't want a bunch of excuses on why your tool isn't in a repository, whether it's for Debian, Ubuntu or the Red Hat side of things. If it's not available as a managed package, chances are your tool isn't going to be deployed by me or other IT people like me...and we tend to be a very stodgy group.

Yet Another Poor Vendor Support Experience

Had a disappointing exchange of emails with a support person from Versys today.

Blogged about it on my ComputerWorld blog.

Pretty much bummed me out because I had previously had good experiences with the person who I was dealing with.

I wrote her an email about a problem and she wrote back this long email stating that their systems couldn't possibly be at fault because it wasn't designed that way. Based on her email, she hadn't even seemed to connect to our system to check out the facts of the matter.

Anyhow, it's just another example of how not to treat your customers if you are in any support position. *Hint: the customer is always right until you can PROVE otherwise and even then you are better off trying to do what's possible to make them happy, unless you want unhappy people going around blogging about how your company sucks*

How to Manually Uninstall Veritas Backup Exec 9.x

Sorry for the length of this post, but this is verbatim from what the Veritas support tech sent me and since it's not available on their website, I figured I'd post it here for your use as well as mine.


How to properly uninstall Backup Exec 9.x if unable to uninstall using Add/Remove Programs.

---

Details:

This procedure should only be used if uninstalling Backup Exec (tm) 9.x for Windows Servers fails from Add/Remove Programs and subsequent installs of Backup Exec also fail due to a corrupt installation.

To manually uninstall Backup Exec 9.x, follow the instructions given below:

1. Download the Microsoft Installer (MSI) Cleanup utility, and install it on the system in question.

The MSI Installer Cleanup utility can be currently obtained from the Microsoft Web site:    http://support.microsoft.com/default.aspx?kbid=290301 .

2. After the application is installed, run it from Start | Programs | Windows Install Clean Up

3. Select VERITAS Update and VERITAS Backup Exec for Windows Servers, and then click Remove (Figure 1)

Figure 1

4. Click Exit to end the clean up process

5. Open Services from Control Panel in Windows NT 4.0, or from Administrative Tools in Windows 2000. Stop all Backup Exec services and the MSSQL$BKUPEXEC service (Figures 2 and 3). If Desktop and Laptop Option (DLO) is installed, stop the MSSQL$BKUPEXECDLO service as well

Figure 2

Figure 3

6. Click Start | Run, type regedt32, and click OK

Warning: Incorrect use of the Windows Registry Editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

7. Remove the following VERITAS registry keys (Figure 4):

a. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Adamm

b. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Agent Accelerator for Windows NT/2000

c. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Backup Exec

d. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Common

e. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\DLO

f. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\Exec View

g. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\SigLogger

h. HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS (only if empty)

i. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\VERITAS Backup Exec

Figure 4

8. If the Backup Exec Web Administration Console is installed (BEWAC), remove the following keys:

a. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ElementManager\WebElementDefinitions\TabBE* (all keys that start with TabBE)

b. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ElementManager\WebElementDefinitions\ContextHelpBE* (all keys that start with ContextHelpBE)

c. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ElementManager\WebElementDefinitions\AboutBoxBE

d. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ElementManager\WebElementDefinitions\HELPTOC_* (all keys that start with HELPTOC_. The underscore is important.)

e. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ApplianceManager\ObjectManagers\Microsoft_SA_Task\BackupJobTask

f. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ApplianceManager\ObjectManagers\Microsoft_SA_Task\RestoreJobTask

g. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ServerAppliance\ApplianceManager\ObjectManagers\Microsoft_SA_Task\TemplateJobTask

9. Remove the Backup Exec services by removing the following registry keys (Figure 5):

a. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BackupExecAgentAccelerator

b. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BackupExecAgentBrowser

c. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BackupExecDeviceMediaService

d. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BackupExecJobEngine

e. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BackupExecNamingService

f. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BBackupExecRPCService

g. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ECM Service

h. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\besakalert (BEWAC only)

i. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DLOAdminSvcu (DLO only)

j. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SCSIChanger

k. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SCSIChanger

Figure 5

10. Remove all Microsoft SQL Server Desktop Engine (MSDE) and Backup Exec associated SQL components by removing the following registry keys:

a. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\BKUPEXEC

b. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\BKUPEXECDLO (only when DLO is installed)

c. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\Component Set\InstanceComponentSet.1 or InstanceComponentSet.2.

Remove only the value where it matches one of the following global unique identifiers (GUIDs):

{44FE936D-2B47-40D3-9109-FC1360BFB85D} - Backup Exec 9.0 revision 4367

{78B54B0C-5280-4236-8DF2-7EC6DC45C364} - Backup Exec 9.0 revision 4454

{E09B48B5-E141-427A-AB0C-D3605127224A} - Backup Exec 9.1 (Backup Exec Instance)

{689404D2-1C94-44B3-9203-BEC5594FDA7A} - Backup Exec 9.1 (DLO Instance)

d. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server (only if there is no other Microsoft SQL Server Instances installed)

e. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSSQL$BKUPEXEC

f. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SQLAgent$BKUPEXEC

g. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ MSSQL$BKUPEXECDLO (only when DLO is installed)

h. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SQLAgent$BKUPEXECDLO (only when DLO is installed)

11. Reboot the computer

12. Remove all files from the Backup Exec install directory, that is C:\Program Files\VERITAS\Backup Exec\NT

13. If BEWAC is installed, remove the following files:

a. %SystemRoot%\system32\ServerAppliance\web\Admin\BE

b. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0C0A\BE

c. %SystemRoot%\system32\ServerAppliance\web\Admin\help\040C\BE

d. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0404\BE

e. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0407\BE

f. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0409\BE

g. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0410\BE

h. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0411\BE

i. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0412\BE

j. %SystemRoot%\system32\ServerAppliance\web\Admin\help\0804\BE

Note: For the folders given below (k-r), remove the files listed after the folder list.

k. %SystemRoot%\system32\ServerAppliance\mui\0411

l. %SystemRoot%\system32\ServerAppliance\mui\0412

m. %SystemRoot%\system32\ServerAppliance\mui\0804

n. %SystemRoot%\system32\ServerAppliance\mui\0C0A

o. %SystemRoot%\system32\ServerAppliance\mui\040C

p. %SystemRoot%\system32\ServerAppliance\mui\0404

q. %SystemRoot%\system32\ServerAppliance\mui\0407

r. %SystemRoot%\system32\ServerAppliance\mui\0409

i. besakfirstwizres.dll

ii. BESigSchedulerRes.dll

iii. BEUtilityJobRes.dll

iv. BEUtilityJobUIRes.dll

v. BEUtilsRes.dll

vi. BEActLogRes.dll

vii. bealertconfigres.dll

viii. BEAlertsRes.dll

ix. BEAuditLogRes.dll

x. BEBackupJobOptionsRes.dll

xi. BEBackupJobOptionsSummaryRes.dll

xii. BECreateJobRes.dll

xiii. BECreateMiscRes.dll

xiv. BEDeviceMgmtRes.dll

xv. BEDevMediaRes.dll

xvi. BEDiagsLogRes.dll

xvii. BEDrivePoolMgmtRes.dll

xviii. BEErrorRulesRes.dll

xix. BEHelpFilesRes.dll

xx. BEJobMonitorRes.dll

xxi. BEJobPropsRes.dll

xxii. BEJobSettingsRes.dll

xxiii. BEJobsetupRes.dll

xiv. BELicenseRes.dll

xv. BeLogonAccountsRes.dll

xvi. BEMediaMgmtRes.dll

xvii. BEMediaRotationRes.dll

xviii. BEMediaSetMgmtRes.dll

xxix. BENotification.dll

xxx. BENotificationConfigRes.dll

xxxi. BEOptionsResource.dll

xxxii. beoverviewres.dll

xxxiii. BEReportsRes.dll

xxxiv. BEResource.dll

xxxv. BEResourceDiscoveryRes.dll

xxxvi. BERestoreJobOptionsRes.dll

14. Open Task Manager, and end the process VxTaskBarMgr.exe (Figure 6)

Figure 6

15. Remove the directory C:\Program Files\VERITAS\VxUpdate

16. Remove the directory C:\Program Files\VERITAS, only if there is no other VERITAS Applications installed on the server

17. Remove the directory C:\Program Files\Common Files\VERITAS Shared, only if there is no other VERITAS Application installed on the computer

18. Remove the directory C:\Program Files\Microsoft SQL Server\MSSQL$BKUPEXEC

19. Remove the directory C:\Program Files\Microsoft SQL Server\MSSQL$BKUPEXECDLO (only when DLO is installed)

20. Remove the directory C:\Program Files\Microsoft SQL Server, only if there is no other Microsoft SQL Instance installed on the computer

Fix For Veritas Backup Exec 9.1 Issue

Yet another inelegant fix for a problem. The higher tier tech first had me make a backup of the database and then had me manually uninstall of Veritas BE. I will post the exact procedure for that later.

Then we had to fix the problem with Remote Server Manager and then reinstall BE 9.1 and SP2.

Now it seems to be working 100% again.

And there was much rejoicing...yay.

Update to Veritas Backup Exec 9.1 Issue

Two Wednesdays ago I posted about an issue I was having with Backup Exec. I finally found time to call Veritas about it today.

Mostly the tech person had me try a few things and send her some log files. Nothing worked so the call was escalated. Will get a call tomorrow from a higher level tech to work on the issue.

About the only thing she had me do that I hadn't tried was install the Microsoft XML 4.0 SP2 patch. Didn't help the problem, but seeing as how the problem I'm having is pretty obscure, a quick fix at this juncture wasn't expected.

Problems With Backup Exec

Was going to have Backup Exec generate new Intelligent Disaster Recovery ISO images today for each server, but BE kept telling me that the files had been generated, but nothing was done. Should take 40 minutes or so for each bootable CD ISO image to be created, but it was indicating that it was done immediately.

I first wanted to check to see that IDR was still installed, so I tried to go to Serial Numbers and Installation in Tools menu in BE. Nothing was happening there either.

Found this article at Veritas that fixed that problem.

Then I found out that Veritas had released SP2 for BE 9.1 on March 1st, which is odd because the Veritas Update Service didn't detect it.

Anyhow, I downloaded that and tried to install it, but I keep getting an Error 1606: Could Not Access Network Location Ÿ    . I've tried googling this error and looking for solutions on Microsoft.com as well as Veritas's support site, but haven't found anything yet. Also tried searching the registry for that odd Y character with the umlaut.

Found some items that suggest fixing some registry settings which I'm trying now, but I don't have much confidence that it will solve the problem.

I guess if you don't like solving problems, you shouldn't be in this business.

How WORLDOX Handles Document Versions

Got an email from someone at another law firm that uses WORLDOX today and figured that I would blog about what they asked as there is a scarcity of information on this Document Management System, at least that I could find.

The question was that if you have a document with say 3 versions and you accidentally create a 4th, how do you delete the newest version? The simple answer is that you can't. WORLDOX won't let you.

The more convoluted answer is that you can cut the text from version 3 into 4 and delete 3, leaving 3 versions, 1, 2 and 4.

Another possibility would be to save version 1 as a new document and then cut and paste the text from versions 2 and 3 into new versions of the new document.

Kind of unwieldy and not what the person wants, but it's a limitation of the software. I personally have run into this issue as well and wish that the latest version could be deleted. Perhaps in WORLDOX 2005 which is due to be out soon...we shall see.

I also recommended to the person that new versions of documents not be saved as such, but rather as completely separate documents. I recently found out that WORLDOX only indexes the text of the latest version of a document, so if you have text that should be indexed but is different between versions 1 and 2, WORLDOX will only index the text in the latest version. So when you look for the text that is in version 1, it won't find version 1 of the document.

Now this might not come up often, but there are instances when it will come up, unless you don't use the Save As New Version feature at all, but rather use the document Relations feature that they added to WORLDOX 2002 instead.

Big Time Blackberry Outage

Well it looks like there's a nationwide outage of Blackberry service at the moment. Some people can send, but most people cannot receive messages.

*Note* Guess I should have looked closer at the source that I just posted and saw that the date of that linked article was 1 Dec 2004.

Might have something to do with their rollout of the version 4 software? Too many people downloading it at once, perhaps? At any rate...not good considering we just rolled ours out.

KI Template Package and Remote Systems

I wrote a short blog entry about 8 months back or so that we had chosen to use KI Systems Law Office Automation Tools as our Word template/macro package. It's a pretty good bunch of software that has been customized by Kristi at KI to meet our needs. It's got a SQL back end for a functional client/matter based address book and to hold all of the user data and favorites as well. The templates are all stored on each user's local machine and each user still gets a customized normal.dot specific to them and not to the machine (although there was some debate as to whether there should be a normal.dot per each user or just one for every machine).

The software has some usability issues that make it a bit complicated to use for some things such as labels. Our secretaries much prefer the WordPerfect macro that they have for our "letterhead" labels as it allows you to select a block of text that has the addressee's name and address in it, click a keystroke combo and the label is automatically filled in. Whereas with KI, they have to click the KI menu button, select the label that they want, either tell it to just create one page of blank labels with the "letterhead" or for each label they want filled in, they have to select persons from the address book which adds further steps...You get the picture.

Took a lot of work, testing and more testing to get the customized templates looking and working how we wanted, but I can't imagine that it's any different from any other customizable package out there. I don't think that a turnkey package even exists out there and it certainly wouldn't meet the needs of the average lawfirm.

Kristi at KI, who's the founder/president/head programmer, is knowledgable about her product, is easy to work with and provides good support. For the most part, she's been able to give us everything we asked for (a lot of it at $135 an hour) and been able to resolve the issues that we have had.

The software is integrated with our WORLDOX DMS system, which is nice as it's able to do certain things such as automatically add the document number where it's supposed to be in a template...but that integration also has it's drawbacks as you will soon learn.

Anyhow, for users that work remotely and aren't always connected to the network, either physically while in the office or via VPN, things get a bit more complicated. If you leave the KI software and Word as is and open Word while you aren't connected to the network, the KI software "macro" template bombs out while trying to connect to the SQL server. This in turn causes Word to disable the KI template, which in turn causes WORLDOX to not function properly and actually makes it so that users cannot save documents that they are working on, if they don't realize that this has happened.

To prevent this from happening, a piece of software called KI Remote is installed on each remote system which basically points the KI template to either the SQL server database or makes a copy of the database and stores it on the local PC.

As we have found out, this whole process is too complicated for our userbase. It's not that lawyers can't handle complication, it's that time is money for them. Time is fleeting for most of us these days, but when you are trying to squeeze as much billable work as possible into an hour, complications really mess with things, especially when they don't work right or you forget all the steps, etc.

My question and the whole point of this long post is if it's possible for add-in templates in Word to programatically check for the existance of the SQL server and switch to a local copy or simply switch off in a way that allows Word to function normally.

Because the biggest issue that I have with this whole mechanism is that if it doesn't work or the user forgets, Word just does not function properly. The user isn't able to save, the autosave function doesn't work at all and it can cause an attorney to lose a lot of work...which is very bad.