Instructions on how to set up Linux modules needed to get a LogAnalyzer log aggregation/analysis server up and running and collecting logs.
Prerequisites
These instructions are specific to CentOS 6.2. If you are using a different distro, many of the installation commands and paths to files will be different from what I've documented below. I strongly suggest that you document the steps to perform a similar install for your distro.
You will need to install the prerequisites by using the following commands:
yum install httpd
yum install mysql
yum install mysql-server
yum install php
yum install php-mysql
yum install php-gd
yum install rsyslog
yum install rsyslog-mysql
/usr/bin/updatedb
The '/usr/bin/updatedb' command updates the file index so that the 'find' and 'locate' commands work properly. If you've already properly set up your system to index the files daily, this will be unnecessary.
If your distro of Linux is using a different syslog server such as syslog-ng or sysklogd, you'll need to remove it.
MySQL
Set up MySQL
/sbin/chkconfig --levels 235 mysqld on
/etc/init.d/mysqld start
/usr/bin/mysql_secure_installation
Hit enter key after last command has run since no password has yet been set for root MySQL account. Hit 'y' and enter when asked to set up a root password and type in a strong password. Hit 'y' and enter for the following questions: "Remove anonymous users?", "Disallow root login remotely?", "Remove test database and access to it?", and "Reload privilege tables now?"
Set up database and tables
Create the user/database/table and table schema:
Log in to mysql:
mysql -u root -p
Create a user:
CREATE USER rsyslog;
SET PASSWORD FOR rsyslog= PASSWORD('yourpasswordgoeshere');
Set up database and table schema:
CREATE DATABASE rsyslogdb;
USE rsyslogdb;
Paste contents below to mysql to set up the schema:
CREATE TABLE SystemEvents
(
ID int unsigned not null auto_increment primary key,
CustomerID bigint,
ReceivedAt datetime NULL,
DeviceReportedTime datetime NULL,
Facility smallint NULL,
Priority smallint NULL,
FromHost varchar(60) NULL,
Message text,
NTSeverity int NULL,
Importance int NULL,
EventSource varchar(60),
EventUser varchar(60) NULL,
EventCategory int NULL,
EventID int NULL,
EventBinaryData text NULL,
MaxAvailable int NULL,
CurrUsage int NULL,
MinUsage int NULL,
MaxUsage int NULL,
InfoUnitID int NULL ,
SysLogTag varchar(60),
EventLogType varchar(60),
GenericFileName VarChar(60),
SystemID int NULL
);
CREATE TABLE SystemEventsProperties
(
ID int unsigned not null auto_increment primary key,
SystemEventID int NULL ,
ParamName varchar(255) NULL ,
ParamValue text NULL
);
Next, we need to grant permissions to the rsyslog account we created earlier:
GRANT ALL PRIVILEGES ON `rsyslogdb`.* TO 'rsyslog'@'%' IDENTIFIED BY 'yourpasswordgoeshere';
flush privileges;
Leave MySQL:
exit
Configure rsyslog
Setting up
How to configure rsyslog:
nano /etc/rsyslog.conf
Make your #### Modules #### section the same as the following:
#### MODULES ####
$ModLoad ommysql # provides support for MySQL
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 514
Just above ### begin forwarding rule ### section add info similar to the following line to limit IP addresses that can send syslog info to the server, for each class C subnet the server will be collecting from, you'll need to enter the subnet info followed by /24 (such as 172.18.22.0/24) to allow that subnet to send syslog data. Alternatively, you can limit by single IP addresses. The 127.0.0.1 is necessary so the server can send logs to itself:
$AllowedSender TCP, 127.0.0.1, 172.18.22.0/24
$AllowedSender UDP, 127.0.0.1, 172.18.22.0/24
Add the following line to the ### begin forwarding rule ### section. Replace the "<yourrsyslogpasswordhere>" bit with the password you set for rsyslog MySQL user above:
*.* :ommysql:127.0.0.1,rsyslogdb,rsyslog,<yourrsyslogpasswordhere>
When done modifying the file, hit Ctrl+x, then y and then enter to save the file.
Restart the rsyslog service:
service rsyslog restart
Test rsyslog
Check if messages are arriving at the syslog server:
tail -f /var/log/messages
Check if messages are being stored in mysql database:
mysql -u root -p
use rsyslogdb;
select * from SystemEvents;
If you see anything other than “empty set” it’s working. Exit out of MySQL:
exit
Configure Apache
Configure CentOS to start the web server at bootup and manually start the service:
chkconfig --levels 235 httpd on
service httpd start
modify 2 lines to match your server's respective ip and fqdn in /etc/httpd/conf/httpd.conf
nano /etc/httpd/conf/httpd.conf
from:
Listen 80
to:
Listen ip.address.of.server:80
and from:
#ServerName www.example.com:80
to:
ServerName fully.qualified.domian.name:80
Hit CTRL+x, then Y and then enter to save and exit the file.
Restart the server:
/etc/init.d/httpd restart
Set up IPTables
Edit the iptables file:
nano /etc/sysconfig/iptables
Add these lines to the /etc/sysconfig/iptables file (before the COMMIT line):
-I INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT -I OUTPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
You'll need to enter lines similar to the following based on your network environment. For more info on how to use IPTables in CentOS see http://wiki.centos.org/HowTos/Network/IPTables:
-I INPUT -p tcp --dport 514 -s 172.18.22.0/24 -j ACCEPT
-I INPUT -p udp --dport 514 -s 172.18.22.0/24 -j ACCEPT
Restart the network service and IPTables:
/etc/init.d/network restart
/etc/init.d/iptables restart
Configure LogAnalyzer
Install LogAnalyzer
Check for the latest stable release by going to http://loganalyzer.adiscon.com/downloads in a browser. Current latest release is http://loganalyzer.adiscon.com/downloads/loganalyzer-3-4-2-v3-stable
Download it on your CentOS server by doing the following:
cd /tmp
wget http://download.adiscon.com/loganalyzer/loganalyzer-3.4.2.tar.gz
Uncompress the file:
tar -xvzf loganalyzer-3.4.2.tar.gz
Copy the source directory to the Apache html directory and create config.php file:
cd loganalyzer-3.4.2/src
rm -R -f /var/www/html
mkdir /var/www/html
cp -R * /var/www/html
cd /tmp/loganalyzer-3.4.2/contrib/
cp * /var/www/html
cd /var/www/html
chmod +x configure.sh secure.sh
./configure.sh
The last line will create a blank “config.php” file, and will give everyone write access to it. It won´t generate any output.
Check if the config.php file has been created (initial setup via browser will make changes to this file):
ls
Create LogAnalyzer MySQL user and database:
mysql -u root -p
create database loganalyzerdb;
CREATE USER loganalyzer;
SET PASSWORD FOR loganalyzer= PASSWORD('yourpasswordgoeshere');
GRANT ALL PRIVILEGES ON `loganalyzerdb`.* TO 'loganalyzer'@'%' IDENTIFIED BY 'yourpasswordgoeshere';
flush privileges;
exit
Initial setup of Log Analyzer, Step One:
On a client system go to the Log server's URL using a web browser (http://yoursystemnamehere.blah.org).
A message stating "Critical Error Occurred: Error main configuration file is missing! Click here to install Adiscon LogAnalyzer!" will appear in browser. Click on the word "here" to start the install.
Click "Next" twice and you should get to the "Basic Configuration" screen. The recommend settings are:
- Number of syslog messages per page: 200 (set this lower if the log server is on a slow system)
- Message character limit for main view: 80 (default)
- Character display limit for all string fields: 80
- Show message details popup: Yes (default)
- Automatically resolved IP Addresses (inline): Yes (default)
- Enable User Database: Yes
- Database Host: localhost (default)
- Database Port: 3306 (default)
- Database Name: loganalyzerdb
- Table prefix: logcon_
- Database User: loganalyzer
- Database Password: <enter in the loganalyzer database user password that you set earlier here>
- Require user to be logged in: Yes
Click "Next".
Initial setup of Log Analyzer, Step Two:
Click "Next" on the "Create Tables" page, then click "Next" on the "Check SQL Results" page and then set up the admin user:
- Username: <enter in the username here that you want>
- Password: <enter in the user password that you want to use>
- Repeat Password: <re-enter in the user password that you want to use>
Click "Next".
Initial setup of Log Analyzer, Step Three:
The recommended settings for the "Create the first source for syslog messages" page are:
- Name of the source: All Syslog Sources
- Source type: MySQL Native
- Select view: Syslog Fields (default)
- Table type: MonitorWare
- Database host: localhost (default)
- Database name: rsyslogdb
- Database table name: SystemEvents
- Database user: rsyslog
- Database password: <enter in the rsyslog database user password that you set earlier here>
- Enable row counting: "Yes"
Click "Next" and then click "Finish".
The install of LogAnalyzer has now been completed. Now other users can be created and there are many settings that can be tweaked as needed.
Point all of the syslog capable devices to the new log server and begin analyzing the aggregated logs.
References
The following sites were used to help figure this all out
http://en.tiagomarques.info/2011/07/centos-syslog-server-rsyslog-mysql-and-loganalyzer/
http://www.pantz.org/software/mysql/mysqlcommands.html
http://www.linuxhelp.in/2010/10/how-to-configure-syslog-server-or.html
http://www.beguelin.com/2009/05/locate-and-updatedb-on-centos.html
Chris Borte's brain
Recent Comments