Thursday, March 25, 2021

Send Custom Application Log to Central Syslog/SIEM Server (e.g. graylog/QRadar)

Let's say, our application which is running on a Windows 2016 server which is generating customs logs and we want to send those logs to our central syslog or SIEM server. For log collection and forwarding we will use NXLog Community Edition. NXLog Community Edition is an open source log collection tool available at no cost.

Let's assume our syslog server (e.g. graylog or Qradar) is installed and configured. We will just collect and forward logs from a Windows Server to destination server. To achieve that, let's follow below steps:


1) Install NXLog Community Edition.

2) Depending on this installation location, update the configuration file nxlog.conf. In my case, the file location is C:\Program Files (x86)\nxlog\conf

3) Let's say we wanna forward access and error logs. Add below configuration in the above mentioned configuration file. Beforehand, make sure the input methods are properly configured and service is running on respective server. In case of graylog, GELF input method is configured and running (Picture-1)


# For sending logs to graylog, we'll use xm_gelf module. Add below in configuration file.
# Assume graylog server IP is 10.10.100.100 and QRadar server IP is 10.10.100.200.
# Application is generating custom logs at a shared folder under directory \\10.20.30.40\LOG\WebApp\

<Extension _gelf>
    Module      xm_gelf
</Extension>

# send logs to graylog

<Input application_accesslog_graylog>
Module im_file
        File '\\10.20.30.40\LOG\WebApp\Information\\accesslog_*'
#File 'C:\Program Files (x86)\nxlog\data\\*.log'
PollInterval 1
SavePos True
ReadFromLast True
Recursive False
RenameCheck False
        Exec $FileName = file_name();
</Input>

<Input application_errorlog_graylog>
Module im_file
File '\\10.20.30.40\LOG\WebApp\Error\\errorlog_*'
#File 'C:\Program Files (x86)\nxlog\data\\*.log'
PollInterval 1
SavePos True
ReadFromLast True
Recursive False
RenameCheck False
Exec $FileName = file_name();
</Input>

<Output gelf>
Module om_tcp
Host 10.10.100.100
Port 12201
OutputType  GELF_TCP
</Output>

<Route graylog>
Path application_accesslog_graylog , application_errorlog_graylog => gelf
</Route>

# send logs to QRadar

<Input application_accesslog_qradar>
Module im_file
        File '\\10.20.30.40\LOG\WebApp\Information\\accesslog_*'
#File 'C:\Program Files (x86)\nxlog\data\\*.log'
ReadFromLast False
Exec parse_syslog();
Exec log_info("Input Event: " + $raw_event);
</Input>

<Input application_errorlog_qradar>
Module im_file
        File '\\10.20.30.40\LOG\WebApp\Error\\errorlog_*'
#File 'C:\Program Files (x86)\nxlog\data\\*.log'
ReadFromLast False
Exec parse_syslog();
Exec log_info("Input Event: " + $raw_event);
</Input>

<Output event-out-qradar>
Module om_tcp
Host 10.10.100.200
Port 514
</Output>

<Route qradar>
    Path application_accesslog_qradar, application_errorlog_qradar  => event-out-qradar
</Route>

############################
#End of configuration file
############################

4) Save the configuration file and restart nxlog service from services.msc


After login to respective portal, verify the results.


Picture-2: graplog



Picture-3: QRadar



Cheers :-)



Sunday, March 21, 2021

Create Logical Volume Manager (LVM) in Linux

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume. Heinz Mauelshagen wrote the original LVM code in 1998 (Source: wiki)

Advantages of LVM: (Source: wiki)

  • Volume groups (VGs) can be resized online by absorbing new physical volumes (PVs) or ejecting existing ones.
  • Logical volumes (LVs) can be resized online by concatenating extents onto them or truncating extents from them.
  • LVs can be moved between PVs.
  • Creation of read-only snapshots of logical volumes (LVM1), leveraging a copy on write (CoW) feature, or read/write snapshots (LVM2)
  • VGs can be split or merged in situ as long as no LVs span the split. This can be useful when migrating whole LVs to or from offline storage.
  • LVM objects can be tagged for administrative convenience.
  • VGs and LVs can be made active as the underlying devices become available through use of the lvmetad daemon
LVM Elements: (Source: wiki)


Physical volumes (pv) are regular storage devices. LVM writes a header to the device to allocate it for management.

Volume Groups (vg) is the combination of physical volumes into storage pools known as volume groups.

Logical Volumes (lv) is the sliced portion of volume group

Create LVM:
Usually fdisk is used for drives that are smaller than 2TB and either parted or gdisk is used for disks that are lasger than 2TB. Here is a very good article that covers the differences, titled: The Differences Between MBR and GPT.

Find out the harddisk:
server1:~# fdisk -l

As there is no partitions yet on /dev/sdb, let's create a partition on it and assume the harddisk is larger than 2TB, we will use parted tool

parted /dev/sdb
(parted) mklabel gpt
(parted) unit TB
(parted) print
(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]? xfs # other known filesystems are ext3, ext4
Start? 0
End? 11TB
(parted) print # print newly created disk
(parted) quit

Now check the changes again:
server1:~# fdisk -l

Now create PV, VG and LV. Once this is done, format the partition in desired partition type and mount the volume in under a mount point. For persistent, update fstab file accordingly.

server1:~# pvcreate /dev/sdb1        # create physical volume
server1:~# pvs                         # view physical volume
server1:~# vgcreate vg_data /dev/sdb1  # create volume group
server1:~# vgs                         # view volume group
server1:~# vgdisplay -v vg_data        # display volume group details with FREE PE (Physical Extent) number
server1:~# lvcreate -l +2621437 -n lv_data vg_data    # create logical volume group with FREE Physical Entent (PE) number
server1:~# vgdisplay -v vg_data        # view the volume group
server1:~# lvs                         # view the logical volume
server1:~# mkfs.ext4 /dev/vg_data/lv_data # format partition with ext4
OR
server1:~# mkfs.xfs /dev/vg_data/lv_data # format partition with xfs
server1:~# mkdir /data     # create a mount point
server1:~# mount /dev/vg_data/lv_data /data/ # mount newly created partition

server1:~# df -Th

server1:~# vim /etc/fstab

/dev/vg_data/lv_data /data/ default 0 0 # add fstab entry for a persistent automount after reboot, first 0 option is for skip backup and second 0 option is for no filesystem check (fsck) at boot time

Let's assume the harddisk /dev/sdb is < 2TB. So for this, we will use fdisk instead of parted

server1:~# fdisk -l
server1:~# fdisk /dev/sdb
server1:~# fdisk> m # for help
fdisk> n # create new partition
fdisk> Primary/ Extended? Primary
fdisk> Partition ID: 1
fdisk> Start: [enter for default]
fdisk> End: [enter for default]
fdisk> p       # print newly created partition
fdisk> t                              # change the partition ID
fdisk> L [Print all types]       # change the partition ID
fdisk> 8e           # 8e is the partition code for Linux LVM
fdisk> w       # write and exit
server1:~# fdisk -l

Repeat the same process again starting from creation of PV, VG and LV. Format the partition and mount as above process.

Cheers :-)

Wednesday, February 17, 2021

NIC Bonding/Teaming in RHEL 6-7/CentOS 6-7

Linux allows administrators to bind multiple network interfaces together into a single channel using the bonding kernel module and a special network interface called a channel bonding interface. Channel bonding enables two or more network interfaces to act as one, simultaneously increasing the bandwidth and providing redundancy. Network Bonding is a kernel feature and also known as NIC teaming. 

Let’s assume we are configuring bond0 with interfaces ifcfg-enp25s0f0 and ifcfg-enp25s0f1

We need to create a channel bonding interface configuration file on /etc/sysconfig/network-scripts/ directory called ifcfg-bond<N> replacing <N> with the number for the interface, such as 0 and specify the bonding parameters on the file. Here we are creating ifcfg-bond0 file with following contents:


# cat ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.20.10.11
NETMASK=255.255.255.0
GATEWAY=10.20.10.1
BONDING_OPTS="mode=4 miimon=200"

or 

# cat ifcfg-bond1
DEVICE=bond1
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.20.10.11
NETMASK=255.255.255.0
GATEWAY=10.20.10.1
MTU=9000
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=slow xmit_hash_policy=layer2+3"



Below are the bonding modes:
  • mode=0 (Balance Round Robin)
  • mode=1 (Active backup)
  • mode=2 (Balance XOR)
  • mode=3 (Broadcast)
  • mode=4 (802.3ad)
  • mode=5 (Balance Transmit Load Balance (TLB))
  • mode=6 (Balance Adaptive Load Balance (ALB))

After the channel bonding interface is created, the network interfaces to be bound together and configured by adding the MASTER= and SLAVE= directives to their configuration files. Below are the interface files: 


# cat ifcfg-enp25s0f0 
DEVICE=
enp25s0f0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
IPV6INIT=no
MASTER=bond0 #or bond1
SLAVE=yes



#cat ifcfg-enp25s0f1
DEVICE=enp25s0f1
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
IPV6INIT=no
MASTER=bond0 
#or bond1
SLAVE=yes



Now, load the bond driver, bring up the newly created bond0 or bond1 interface and verify the same by following commands:

# modprobe bonding
# ifconfig bond0 up
# ifconfig
# ip a
# cat /proc/net/bonding/bond0



Cheers :-) 

Thursday, October 17, 2019

Configure SNMP on Force10 MXL 10/40GbE Switch I/O Module


Simple network management protocol (SNMP) is supported on the MXL switch platform. For SNMPv1 and SNMPv2, create a community string to enable the community-based security in the Dell Networking OS.

Step 1: SSH to MXL Switch using putty/x-shell

Step 2: Enter EXEC Privilege mode
Dell(conf)> conf t

Step 3: View SNMP configuration
Dell(conf)# show running-config snmp

Step 4: Create a Community string by below command
Dell(conf)# snmp-server community community-string ro

Step 5: Add Contact and Location information
Dell(conf)# snmp-server contact sysadmin@mydomain.com
Dell(conf)# snmp-server location DC-DHK-BD-Rack1

Step 5: Verify Configuration
Dell(conf)# show running-config snmp

Step 5: Save Configuration and Copy Running configuration in Startup configuration
Dell(conf)# crtl+z
Dell(conf)# write memory
Dell(conf)# copy running-config startup-config

Step 5: Verify from Monitoring Server
#snmpwalk –v 2c –c community-string 10.20.10.10

Cheers J

SNMP Configuration in VMware ESXi Host


Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior.

Let's configure SNMP in VMware ESXi host for monitoring purpose. It's pretty straight forward steps to follow.

Step 1: Connect to ESXi Host via SSH


Step 2: Configure SNMP Community String and Enable Service on ESXi

#esxcli system snmp set --enable true
#esxcli system snmp set --communities Community_String

Step 3: Let's add Contact & Location

#esxcli system snmp set --syscontact sysadmin@mydomain.com
#esxcli system snmp set --syslocation "DC_DHK_BD_RACK1"



Step 4: Configure ESXi Firewall exclusion for SNMP and Allow Incoming Traffic

Allow incoming traffic from any host
#esxcli network firewall ruleset set --ruleset-id snmp --allowed-all true
#esxcli network firewall ruleset set --ruleset-id snmp --enabled true



Allow incoming traffic from specific host/network block. Lets assume 10.20.10.10 is our monitoring server
#esxcli network firewall ruleset set --ruleset-id snmp --allowed-all false
#esxcli network firewall ruleset allowedip add --ruleset-id snmp --ip-address 10.20.10.10/32
#esxcli network firewall ruleset set --ruleset-id snmp --enabled true




Step 5: Restart SNMP Service

#/etc/init.d/snmpd restart



Step 6: Verify SNMP Configuration

From ESXi Host
#esxcli system snmp get

From Monitoring Server
# snmpwalk –v 2c –c Community_String ESXi_Host_IP



Cheers J




Sunday, October 15, 2017

Data Center Site Selection Criteria

Now a days data center plays the most crucial and vital role irrespective to business segment. The cost of data center downtime has increased significantly for companies in the last three years, according to results of a recent study by Ponemon Institute published on February 1, 2011, sponsored by Emerson Network Power. Read the study report: "Cost of Data Center Outages: Sponsored by Emerson Network Power"

Research indicates "data center outages have serious financial consequences for an organization. According to the study, the cost of a data center outage ranges from a minimum cost of $38,969 to a maximum of $1,017,746 per organization, with an overall average cost of $505,502 per incident."

Actual cost of downtime in these days would be much more than the above calculation. Could you ever imagine if Amazon went down for a minute just before eve of Christmas or New Year or Alibaba had an interruption for a minute before Chinese New Year? I am sure that you need a large scientific calculator to compute this. 

According to the above study report, 37% of the unplanned outages occurred related to site or location. Thus data center site selection is one of the major decision that a management should take. There are several criterion for selecting site for your future business. Depending your choice, your business might be bloomed or doomed.

Selection criteria can be divided into below broad categories:
  • Potential Natural Hazardous Area
  • Potential Man-made Hazardous Area
  • Proximity Evaluation
  • Building Evaluation

For selecting a site, Potential Natural Hazardous Area should be avoided for below reasons:
       Lightning
       Flooding
       Typhoon
       Forest Fires
       Seismic Prone Area

Sometimes Man-made Hazardous Area also impacts data center overall performance, thus should be avoided.
       Flight path
       Tunnels, lakes
       Train/airport
       RF towers
       Power distribution network
       Industrial pollution

Proximity Evaluation is one of the premier selection norm for selecting site.
       Emergency Services e.g. Fire, Police, Medical Facilities
       neighborhood
       Public transport and public roads
       High risk targets e.g. Embassies, Govt. Building, Power Stations, Radio/TV Station

And lastly Building Evaluation is required for below reason:
       Rent/Buy/Build
       History of the building and/or area (flood, fire etc.)
       National building code
       24x7 access
       Level/floor within the building
       Space required/ potential expansion
       Floor loading capacity
       Slab to slab height (min. 3.8meter)
       Power Capabilities: Redundancy & Capacity
       Network Capabilities: Redundancy & Diversity
       External Supply Capabilities
       Utility duct
       Secured perimeter
       Grounding space
       Standby generator, external generator provision
       Delivery of heavy/big equipment's and route to data center

Don't forget your prime selection criteria which is your Budget, Budget, & Budget!!


References:


Tuesday, April 18, 2017

Backup & Restore MySQL Stored Procedures and Triggers

There are two ways to backup MySQL routine procedures and triggers. Either it could be with data and tables or  without data and tables.

Backup:

DB Host: 172.16.16.29
DB Name: reportdb
DB Username: root
DB User Pass: abc123

Backup stored procedures with data and tables:
==================================
# mysqldump -h 172.16.16.29  -u root -p --routines reportdb > reportdb_proc_with_data_table.sql
When prompted, enter root password

Backup only Stored Procedures and Triggers:
=================================
# mysqldump -h 172.16.16.29 -u root -p  --routines --no-create-info --no-data --no-create-db --skip-opt reportdb > reportdb_routine_proc.sql
When prompted, enter root password

Restore:
mysql -u root -p reportdb  < reportdb_proc_with_data_table.sql
When prompted, enter root password
or

mysql -u root -p reportdb  < reportdb_routine_proc.sql
When prompted, enter root password

Restore Archived Log into VMware Aria Operations for Logs (formerly known as vRealize Log Insight - vRLI)

As we cannot keep all logs in searchable space in vRLI production system due to performance and slowness issue, it is always recommended to ...