In this How To we’ll setup a catch-all Postfix server and use it to Splunk all of your system generated email.

Overview

• Install and start Splunk.
• Install and start Postifx.
• Configure Postfix catch-all.
• Configre your system(s) to use the Postfix catch-all.
• Configure Splunk to consume the catch-all email.

Note: Splunk need not be installed on the same system as Postfix, but for the purposes of this How To, they are co-existent.

Steps

1. Download Splunk Here.
2. Postfix can be installed on Ubuntu using apt, or in FreeBSD using ports:
   • Ubuntu: $ sudo apt-get install postfix -f
   • FreeBSD: $ cd /usr/ports/mail/postfix; sudo make install
3. In Postfix’s main.cfg file:
   • Set virtual_alias_domains to all hosts from which you’re going to accept mail: virtual_alias_domains = sfeserv01.splunk.com,sfeserv31.splunk.com
   • Set virtual_alias_maps to your virtual alias map file: virtual_alias_maps = hash:/etc/postfix/virtual
4. In Postfix’s virtual_alias_map file create a catch-all alias for each host from which you’re going to accept mail:
@sfeserv01.splunk.com catch-all
@sfeserv31.splunk.com catch-all

5. In Postfix’s aliases file create a catch-all alias and redirect it to a Maildir: catch-all: /var/mail/catch-all/
6. Refresh aliases, rehash maps, and reload Postfix configs:
$ sudo newaliases
$ sudo postmap /etc/postfix/virtual
$ sudo postfix reload

7. In Splunk’s inputs.conf file configure batch monitor of the catch-all Maildir:
# $SPLUNK_HOME/etc/system/local/inputs.conf
[batch:///var/mail/catch-all]
interval = 300
disabled = false
index = admin_mail
source = admin_mail
move_policy = sinkhole
sourcetype = admin_mail

8. In Splunk’s props.conf file configure email event parsing:
# $SPLUNK_HOME/etc/system/local/props.conf
[admin_mail]
TRUNCATE = 0
MAX_EVENTS=200000
TIME_PREFIX = Date:\s
LINE_BREAKER = tacotacotacotaco
BREAK_ONLY_BEFORE = tacotacotacotaco

9. In Splunk’s indexes.conf file configure the email index:
# $SPLUNK_HOME/etc/system/local/indexes.conf
[admin_mail]
homePath = $SPLUNK_DB/admin_mail/db
coldPath = $SPLUNK_DB/admin_mail/colddb
thawedPath = $SPLUNK_DB/admin_mail/thaweddb

10. Restart Splunk: $ splunk restart
11. Now configure your system(s) to use the Postfix catch-all mail server. In Postfix this can be accomplished in main.cfg:
    relayhost = mail-relay.splunk.com

Search

You can now search Splunk for system emails: index="admin_mail" ERROR
Which should return results like these:

Dear Fellow Amateurs,
You may have seen the news that Interop has returned its IP address
block to ARIN. See
http://arstechnica.com/business/news/2010/10/embargoed-interop-gives-back-a-months-worth-of-ipv4-addresses.ars

This was done as a means of prompting other organizations that hold
large, mostly-unused blocks – that means us – to return them now that we
are approaching the exhaustion of available IPV4 addresses.

Amateur Radio holds a block of 16 million IP addresses that are mostly a
relic of past operation. When TCP/IP over 1200 baud packet was
interesting, the IP address pool was far from exhaustion and holding
that block had no cost to the general public. Now, Amateur Radio is a
very significant contributor to the problem of global IPV4 address
exhaustion.

Obviously it is true that everybody must convert to IPV6. As Amateurs,
technically competent and in complete control of our own networking
infrastructure, this is an easy place for us to lead. It isn’t so for
the global internet. Commercial internet providers must struggle with a
tremendous technically-naive user pool who must be guided through
conversion or provided with address translation kludges that will cause
service problems, routing hardware that can’t be converted to IPV6, and
a tremendous expense of converting all of this infrastructure and
training users and their own staff that has come at a really bad time
economically.

Thus, I suggest that Amateurs would be fulfilling their social duty to
the public by returning an address pool that they no longer need as soon
as possible, and leading in conversion of their remaining and future
TCP/IP operations to IPV6.

This isn’t like giving up a frequency band that will never be returned -
equivalent IPV6 address blocks are available to us, and the IPV6 address
space is astronomical in size compared to IPV4.

Many Thanks

Bruce Perens K6BP

Caregivers –  must have current CPR Pro or BLS certification, in addition to professional license / certifications.

Recorders -  must have current lay CPR certification and First Aid.

Communicators -  need to be FCC licensed HAM radio operators and have a dual band HAM hand held radio.

WHAT: Nike Women’s Marathon, San Francisco

WHEN: Sunday, October, 17th 2010

3:30 AM – (time to be  confirmed) until 2:30 pm

Note:    This is 3:30 AM Sunday morning until 2:30 p.m. Sunday afternoon

WHERE: Staging Area: Pier 54, San Francisco

Free T-shirt, Breakfast, Lunch & much more!

REGISTRATION IS TO BE COMPLETED ONLINE :

http://www.surveymonkey.com/s/2010NikeMarathon

Please scan & email copies (front and back) of certificates and licenses to NikeStaffing@redcross-bayarea.org or FAX them to (415) 963-4430.

Any questions, email Nikestaffing@redcross-bayarea.org or call (415) 427-8092.

…and I’m only publishing this here because it’s near impossible to find via Google on an iPhone.

From the Solaris 10 OS Companion Software CD install the packages below, this will get you up and running with Ruby 1.3.5:

1. SFWruby
2. SFWrline
3. SFWncur
4. SFWcoreu

Install & Update RubyGems

In this step we’ll install a version of RubyGems compatible with our version of Ruby, 1.3.5.

cd /tmp
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.5.tgz
gtar -zxf rubygems-1.3.5.tgz
cd rubygems-1.3.5
/opt/sfw/bin/ruby setup.rb
/opt/sfw/bin/gem install rubygems-update
/opt/sfw/bin/gem update --system

Who the hell is Steve?

To avoid the error message below, run:

mkdir -p /export/home/steve/work/usr/src/tools
ln -s /usr/sfw/bin/gcc /export/home/steve/work/usr/src/tools/gcc

make
/export/home/steve/work/usr/src/tools/gcc -I/usr/sfw/include -I/export/home/steve/work/proto/root_i386/opt/sfw/include -I. -I/opt/sfw/lib/ruby/1.8/i386-solaris2.10 -I/opt/sfw/lib/ruby/1.8/i386-solaris2.10 -I. -fPIC -g -O3 -Wall -c generator.c
sh: /export/home/steve/work/usr/src/tools/gcc: not found

Install Chef with RubyGems

gem install chef

Oh no!

At this point you’ll get this error when attempting to run ‘chef-client’:

ld.so.1: ruby: fatal: relocation error: file /opt/sfw/lib/ruby/gems/1.8/gems/json-1.4.2/ext/json/ext/json/ext/parser.so: symbol RSTRING_PTR: referenced symbol not found
Killed

Worry not! See below.

Replace json with json_pure

gem uninstall json
gem install json_pure --version 1.4.2
cat /opt/sfw/lib/ruby/gems/1.8/specifications/json_pure-1.4.2.gemspec | sed s/json_pure/json/g > /opt/sfw/lib/ruby/gems/1.8/specifications/json-1.4.2.gemspec
cp -pr /opt/sfw/lib/ruby/gems/1.8/gems/json_pure-1.4.2 /opt/sfw/lib/ruby/gems/1.8/gems/json-1.4.2

Victory!

chef-client

[Wed, 08 Sep 2010 17:21:09 -0700] INFO: Client key /etc/chef/client.pem is not present – registering
[Wed, 08 Sep 2010 17:21:10 -0700] WARN: HTTP Request Returned 404 Not Found: Cannot load node stress10.
[Wed, 08 Sep 2010 17:21:11 -0700] INFO: Starting Chef Run (Version 0.9.8)
[Wed, 08 Sep 2010 17:21:11 -0700] WARN: Node stress10. has an empty run list.
[Wed, 08 Sep 2010 17:21:11 -0700] INFO: Chef Run complete in 0.830921 seconds
[Wed, 08 Sep 2010 17:21:11 -0700] INFO: Running report handlers
[Wed, 08 Sep 2010 17:21:11 -0700] INFO: Report handlers complete

Done!

The American Red Cross Bay Area (ARCBA) San Francisco Disaster Communications Team (DComms) is holding a Communications Treasure Hunt on Saturday, September 18th from 9:00AM to 1:00PM at Pier 54, San Francisco, CA. The goal if this exercise is to gain experience in these areas:

1. Radio communications in a dense urban environment.
2. Formal message handling.
3. Establishing and operating a radio traffic net.
4. Working with communications volunteers of different skill levels.
5. Incident Management techniques.
6. Radio programming and theory.

THIS EVENT IS OPEN TO ALL RED CROSS VOLUNTEERS. All existing Red Cross volunteers are invited to attend, prior radio experience nor possession of a radio license is required. This will be a field exercise, so please bring comfortable walking shoes or a personal mode of transportation (bicycle, skates, scooter, etc). Lunch will be served!

If you are interested in attending this exercise, please RSVP by Friday, September 17th:

• RSVP by email: arcba.dcomms@gmail.com
• RSVP by phone: 1-415-504-2721

For more information please contact the ARCBA SF Disaster Communications Team:

• Email: arcba.dcomms@gmail.com
• Phone: 1-415-504-ARC1 (1-415-504-2721)
• Twitter: http://twitter.com/dcomms
• Web: http://arcba.net/teams/sfo
• Calendar: (web): http://bit.ly/dcomms-cal
• Calendar (iCal): http://bit.ly/dcomms-ical

Sincerely,
Greg Albrecht & Lawrence Lin
SF Disaster Communications Leads
American Red Cross Bay Area
85 Second Street, 8th Floor
San Francisco, CA 94105

1. A Master Boot Record (MBR) partition table is not necessary to boot Ubuntu.
2. The hard drive must be blessed after installation.

Prerequisites

1. MacOS X Install DVD (any version, OEM or full)
2. Ubuntu 10.04 Install CD (server or desktop)

Overview

• Using a MacOS X Install DVD format the disk with a single partition. (Phase I)
• Boot from the Ubuntu 10.04 Install CD and install Ubuntu. (Phase I)
• Boot from the MacOS X Installation DVD once more and bless the hard drive. (Phase II)

Steps, Phase I

1. Boot a MacOS X Install DVD (Don’t worry about the version of the DVD or if it’s OEM, Upgrade or whatever, we won’t actually be installing the OS.)
2. Once booted choose your language and select Utilitites > Disk Utility from menu bar.
3. Select your hard disk drive and click Partition.
4. Under Volume Scheme select 1 Partition.
5. Click Apply then Partition.
6. Quit Disk Utility, Quit Mac OS X Installer and reboot.
7. On reboot hold down the first (left or only) mouse button to eject the Mac OS X Install DVD.
8. Insert the Ubuntu 10.04 Install CD.
9. Hold down the C key to boot from cd.
10. Walk through the installer to get Ubuntu 10.04 setup on your system.
11. When installation completes the CD will eject and the system will reboot.

At this point you will have Ubuntu installed on your system but will be unable to boot. If you attempt to boot you’ll be presented with a flashing question mark folder. Do not fret!

Steps, Phase II

1. Once more, Boot the MacOS X Install DVD.
2. Once booted choose your language and select Utilitites > Terminal from menu bar.
3. In Terminal, enter:

   bless --device /dev/disk0s2 --setBoot --legacy --verbose

4. Quit Terminal, Quit Mac OS X Installer and reboot.
5. On reboot hold down the first (left or only) mouse button to eject the Mac OS X Install DVD.
6. Done!

You are now ready to enjoy your single boot Ubuntu 10.04 MacBook Pro. Have fun!

Sources: Greg Albrecht, Ubuntu Forum, iLinux, Ubuntu Community Documentation.

“Health Care Is A Right, Not A Privilege”

Outside of my day job I spend a vast amount of my time providing volunteer disaster response and communications for the American Red Cross Bay Area. This past weekend, however, I was invited to embed with the Rock Medicine team at San Francisco’s Outside Lands 2010 festival in Golden Gate Park. This volunteer group of medical professionals and care takers provide free-of-charge emergency medical services at large events throughout California. (For more information on Rock Medicine please see the Haight Ashbury Free Clinic or the San Francisco Medical Society.) What follows are observations I made while in the field with Rock Med.

1. When using a repeater, always monitor the input frequency. I can’t count the number of times I’ve been unable to hear a station because they didn’t have a strong signal into the repeater. If the station is far enough away there’s not much to be done, but in our case we were closer to the transmitting station than we were to the repeater tower, so it would’ve made sense to either switch to simplex or always monitor the input frequency.
2. Patient advocacy and patient privacy require that you learn some medical lexicon. BLS, IPR, AMS, CSPINE? You’re going to be getting a lot of traffic like this as a radio operator, it’s not important that you know how to respond to an IPR patient, only that you know you’re calling them IPR for their own safety.
3. Bring your mobile phone charger. Even with every mode of radio communication at your disposal, people are going to revert to mobile phone usage, and your phone, or your supervisors phone, will die half way through the day. (see Keith Robertory’s article ‘Will text messaging work in disaster?‘)
4. Use a team lead’s fist name as the team’s tactical callsign. First, everyone on a medical team should know who’s in charge: “I’m on team Robert.” Second, and this happens a lot, people tend to forget magic numbers: “Am I on team 1 or team 2?“
5. Have a backup land-line phone. I thought it was interesting that every facility at this event had a hard wired phone. Rock Med didn’t use theirs, but it was nice to know one could call a facilities office directly should the need arise.
6. Teach your team how to properly hold a radio. Numerous times I witness people holding radios sideways, upside down, laying them flat on tables, and doing other inappropriate things with their antennas. Using a radio doesn’t require an electrical engineering degree, so take the time to show your volunteers how to hold a radio upright and away from your mouth :).
7. Put everyone on one frequency. Having both the medical dispatch and the medical teams listening to one frequency made a huge difference in response times. Teams in the field could hear dispatch describe the location of a medical incident and the closest team could respond. This was opposed to the traditional method of a central controller dispatching whichever team they assumed was closest.

There was blood, there was vomit, there were stitches. I came away knowing more than I ever wanted to know about alcohol dehydration (seriously, drink some water people). All in all I had a really good time, big thanks to Eric Elliot for dragging me along.

Oh, and shirtless dude: At least put on some shoes!

Overview

1. Setup a Notifo account.
2. Install the Notifo app on your iPhone.
3. Install the notifo.py Python module.
4. Install the splunknotifo.py Python alert script.
5. Setup splunknotifo.py
6. Setup saved search.

Assumptions

• This process assumes that you’ve got Splunk installed and monitoring a file containing sshd log messages.

Steps

1. Browse to https://notifo.com/user/register to setup a Notifo account.
2. Browse to https://notifo.com/user/login, login, and visit the Settings page.
3. Locate and record your API Secret (screenshot)
4. From your desktop or your iPhone browse to the iTunes App Store to install and configure the Notifo app.
5. On the system running Splunk, download and install the notifo.py Python module (screenshot):
~$ cd /usr/local/src
/usr/local/src$ git clone git://github.com/mrtazz/notifo.py.git
/usr/local/src$ cd notifo.py
/usr/local/src/notifo.py$ $SPLUNK_HOME/bin/splunk cmd python setup.py install

6. On the system running Splunk, download the splunknotifo.py Python alert script (screenshot):
~$ cd $SPLUNK_HOME/bin/scripts
/opt/splunk/bin/scripts$ get http://github.com/ampledata/soss/raw/master/splunknotifo/splunknotifo.py
/opt/splunk/bin/scripts$ get http://github.com/ampledata/soss/raw/master/splunknotifo/splunknotifo_conf-default-.py

7. Configure splunknotifo_conf.py with your Notifo APIUsername and APISecret (see step #3 above):
~$ cd $SPLUNK_HOME/bin/scripts
/opt/splunk/bin/scripts$ mv splunknotifo_conf-default-.py splunknotifo_conf.py
/opt/splunk/bin/scripts$ vim splunknotifo_conf.py

8. Using the Splunk web interface, search for the term(s) you’d like to match and click Actions >> Save search… (screenshot).
9. Enter the parameters for your Saved Search:
   
10. Done!

To Test

1. Generate some sshd log messages.
2. You should get an alert on your iPhone like this:
   

while true; do 
  ssh -o PreferredAuthentications=publickey bob$RANDOM@localhost;
  sleep 240;
done

This should generate some messages in your logs (/var/log/secure.log under MacOS) like:


Aug 10 10:06:03 jupiter sshd[73325]: Invalid user bob30582 from ::1