The Big Brother Systems and Network Monitor

| help

# Installation and Configuration
    	       The Big Brother Systems and Network Monitor
    ------------------------------------------------------------------------
    		  Installation and Configuration Manual
    		    Version 1.9c - May 15th 2002
    ------------------------------------------------------------------------

    1.0	Quick and Dirty Install for Big Brother

    	1.1	Installing Big Brother
    	1.2	Upgrading BB
    	1.3	Compiler directives

    2.0	Configuring the BB hosts file

    	2.1	The etc/bb-hosts file
    	2.2	Basic format of etc/bb-hosts
    	2.3	etc/bb-hosts special directives
    	2.3.1		Display results in another HTML page
    	2.3.2		Display "page" results into an HTML subpage
    	2.3.3		Grouping hosts on the display (HTML tables)
    	2.3.4		Sending a summary to another BBDISPLAY
    	2.3.5		Testing Modem Banks
    	2.3.6		Specifying DHCP hosts (no fixed IP address)
    	2.4	Validating your etc/bb-hosts file
    	2.5	Pointers for writing entries in bb-hosts

    3.0	Customizing your BB installation

    	3.1	The runbb.sh script
    	3.2	Configuring the etc/bbdef.sh file
    	3.3	Localizing programs paths
    	3.4	Creating BB clients
    	3.5	Configuring the Web Stuff
    	3.6	History
    	3.7	Adding your own tests
    	3.8	Starting BB at system boot
    	3.9	Summary of BB protocol

    4.0	Configuring the Notification feature

    	4.1	A Little History
    	4.2	Setting up notification
    	4.3	Creating notification rules
    	4.4	Acknowledging notification messages
    	4.5	Manual notification (Web based)
    	4.6	SMS notification
    	4.7	Using kermit
    	4.8	Adding a custom notification
    	4.9	Notification/acknowledgement logs
    	4.10	Disabling notifications temporarely
    	4.11	Acknowledging a notification by e-mail
    	4.12	Displaying acknowledgements

    5.0 	Customizing your Big Brother display

    	5.1	HTMLized status logs
    	5.2	headers / footers (for reports too)
    	5.3	Creating skins
    	5.4	Inserting notes about your hosts
    	5.5	Extending the contents of the main HTML pages
    	5.6	Extending the contents of the history HTML pages
    	5.7	Working with historical logs
    	5.8	Hiding the bb.html/bb2.html page header
    	5.9	Pre-defined anchors in bb.html

    6.0 	Miscellaneous Big Brother configuration options & tools

    	6.1	Specifying filesystem specific values in the disk test
    	6.2	Enabling security
    	6.3	Using bbnet to test TCP services
    	6.4	Sending custom one line status with bb
    	6.5	Changing the 'purple' interval
    	6.6	Specifying processes to look for
    	6.7	Sending arbritary data to a BBDISPLAY(s)
    	6.8	Disabling the messages file(s) zero-length test
    	6.9	Setting a Time To Live to a status message
    	6.10	Disabling the connectivity test
    	6.11	Reducing connectivity errors
    	6.12	Setting up BBPAGERs and BBNETs host in failover mode
    	6.13	Speeding up the network tests
    	6.14	Enabling WML ouput (WAP browsing)
    	6.15	Having BB notify you to reboot a server after is has been up XXX days
    	6.16	Using an alias name for the current host in etc/bb-hosts
    	6.17	Specifying CPU threshold values
    	6.18	Configuring messages file testing
    	6.19	Using the combo message
    	6.20	Tools to manage Big Brother data
    	6.21	Sending a BB message in cron
    	6.22	Defining characters that BB accepts in a status message
    	6.23	Integrating a LARRD graph with a status log

    7.0 	Other documentation sources

    ------------------------------------------------------------------------
    This Installation and Configuration Guide is
    © Copyright Quest Software, Inc.  1997-2002  All rights reserved.
    ------------------------------------------------------------------------

    ------------------------------------------------------------------------

    Section 1: Quick and Dirty Install for Big Brother

    1.1	Installing Big Brother

    WARNING: For security reasons, it is best to install and run
    	 BB as its own user and not as root.   (but use the
             root account during the install process, at the end
             you'll change ownership to the chosen BB user ID)

    *** READ THE README.SECURITY FILE BEFORE PROCEEDING ***

    You've extracted the BB archive and this is the directory
    structure that was created:

    bbnewversion/		(i.e.  bb18/) referred as $BBHOME
    bbvar/acks/		referred as $BBACKS
         /data/		referred as $BBDATA
         /disabled/		referred as $BBDISABLE
         /hist/		referred as $BBHIST
         /histlogs/		referred as $BBHISTLOGS
         /logs/		referred as $BBLOGS

    1.      cd bb<newversion>		(i.e.: cd bb18)
    	This is often referred to BBHOME or $BBHOME
    	cd ./install
    	./bbconfig <OS-NAME>
            where OS-NAME is  bsdi sco3 sco freebsd solaris hpux9 hpux
                                    linux sunos netbsd osf ultrix irix
                                    unixware redhat aix dynix debian dgux
    				caldera mandrake
    	Note that <OS-NAME> is optional, BB will try to figure it out.
    	If you are running Linux you may have to provide the proper
    		distribution name:

    		./bbconfig readhat
    		./bbconfig debian
    		./bbconfig caldera
    		./bbconfig mandrake
    		./bbconfig linux

            If you're not running one of the above, read the install/README
            ./bbconfig will ask you questions about your setup
            be ready with:
                    BBHOME: directory where BB resides
                            (usually the choice given by BB is correct)
    		Should the old-style structure be kept
                    If you intend to use FQDN(Fully qualified domain names)
                    Which host(s) is(are) the BBDISPLAY(s)
                    Which host(s) is(are) the BBPAGER(s)
                    If the current host is a BBDISPLAY/BBPAGER
                    Default e-mail recipient for notification
                    URL you intend to view BB with
                    URL of the BB CGI scripts
    		The user id of your web server

    2.      cd ../src
            type "make"
            then type "make install"

            If you have trouble compiling, refer to the README file
            located at src/README

            for those who are upgrading then go to "upgrading BB" section

    3.      cd ../etc

            edit bb-hosts, put your hosts names in there.  Refer to install/README.
            This is the core of Big Brother.  You must read the docs here.
            edit bbdef.sh, set alarm levels and things.  If you want to use
            fully qualified domain name hosts then make sure you set FQDN=TRUE
            in etc/bbdef.sh.  If this is the first host you install and intend
            to use it as the display/notification server as well as the server
            that tests the network services then your bb-hosts should contain
            this line:

            xxx.xxx.xxx.xxx this.host.name  # BBDISPLAY BBPAGER BBNET

    4.      ./bbchkcfg.sh
            Checks the bbdef.sh/bbinc.sh/bbsys.sh source scripts for invalid entries

            ./bbchkhosts.sh
            Checks the bb-hosts file for errors

    5.      cd ../..
            ln -s bbdir bb		where bbdir is the new version
    				directory (e.g. ln -s bb13a bb)
            This is useful as you don't have to change the directory in your
            startup script. (see section 11)

    	cd bb
    	chown -R bbuser	.	where bbuser is the user you defined
    				in the install process. This makes sure
    				that the bbuser can write/read into
    				the BB directory structure as you will
    				probably install it while in the root account.

    	cd ..
    	chown -R bbuser bbvar

    6.      ln -s /full_path_to_bb/www /WWW/bb (where /WWW is the Document Root dir).
            Make sure the permissions are correct.  Configure your web server
            for this directory if need be.
            Also make sure that your web server follows symbolic links

    7.      cd bbdir		where bbdir is the new version directory (e.g. bb13a)
    	or
    	cd bb			if you've followed section 5
            ./runbb.sh start
            examine the BBOUT file for any errors. By default the BBOUT log file
    	is located in $BBHOME.  The location of BBOUT is defined by the value
    	of the BBOUT variable in bbdef.sh

            NB The HTML summary pages (bb.html/bb2.html) should be available
            2 minutes after the startup.  Don't panic the'll be there if
            you are patient. In your browser, you should be able to
            see the results at http://yourwebhost/bb/ or
            http://yourwebhost/bb/bb.html  (assuming you used /bb as BB URL root)

    8.      Debug, and look at all the docs.

    9.      Check the online documentation!  It lives under:
            http://bb4.com/bb/bb-help.html

    10.     This will have enabled you to have a display/pager server set up.
            At this point no clients are running.  When you have all of
            your hosts defined in etc/bb-hosts then
            use install/bbclient to create a tarball for BB clients of the same
            OS/HW type.  If you have different OS/HW platforms then
            reinstall BB on each one (then use the bbclient to create a tarball
            for each identical OS/HW client) and don't forget to copy your master
            bb-hosts file to it. Run through the install procedure to make
            sure that the clients are also installed properly(bbchkcfg.sh/bbchkhosts.cfg)

    11.     If you wish to start BB automatically at startup, we suggest the
            following command in your startup script:

            su - <bbuser> -c "cd <BBHOME>;./runbb.sh start"

            <bbuser> is the user you chose at install
            <BBHOME> is the directory in which you installed BB
                     or where you've created the bb directory link
                     as per section 5. (e.g. bb18/)

    If you're new to BB, we suggest that you  subscribe to the
    BB mailing list.  To do so, follow these instructions:

    Send an e-mail to MajorDomo@bb4.com
    In the body of the e-mail message (not the Subject line),
    place the statement:

        subscribe bb

    Nick Silberstein has made an archive of the Big Brother
    mailing list available.  It can be found at the URL:
    http://www.tpdinc.com/~bb/

    When you have finished installing BB on all of your
    monitored hosts, you might want to check out the FTP archive
    that contains many contributed scripts.  Check it out
    at ftp://ftp.deadcat.net/pub/BB

    Sans '98 & '99 presentations are available at
    http://bb4.com/propaganda.html.  These presentations
    may fill in some details.

    1.2	Upgrading BB

    	If you are upgrading from a previous version then do these steps after
    	the preliminary install of BB:

    	If your previous version of BB was pre-1.5 then follow these next steps
    	otherwise jump to 1.2a.

    	We assume that the BBVAR/BBLOGS/BBHIST/BBHISTLOGS/BBDATA/BBDISABLE/BBNOTES
    	weren't modified in $BBHOME/etc/bbinc.sh.

    	cd BBTOP/bbvar	(BBTOP is where you unpacked BB)
    	rm -r logs acks hist histlogs notes data disabled
    	mv OLDBBHOME/www/logs .
    	mv OLDBBHOME/www/acks .
    	mv OLDBBHOME/www/hist .
    	mv OLDBBHOME/www/histlogs .
    	mv OLDBBHOME/www/data .
    	mv OLDBBHOME/disabled .

    1.2a	BB upgrade steps

    	cd <bbdir>      where <bbdir> is the directory of the new install
                                        (where BBHOME points to)
    	cd www
    	rm -r notes
    	# Replace OLDBBHOME by the full path of BBHOME defined in your previous install
    	mv OLDBBHOME/www/notes .
    	cd ..

    	cd ext                          If you have external scripts
    	cp OLDBBHOME/ext/* .		!!!!! Only copy your scripts
    	cp OLDBBHOME/ext/hist/* hist/	!!!!! Do not copy the basic scripts
    	cp OLDBBHOME/ext/rep/* rep/	!!!!! included in the archive
    	cp OLDBBHOME/ext/mkbb/* mkbb/
    	cd ..

    	cd etc
    	cp OLDBBHOME/etc/bb-hosts .             Depending of your current setup
    	cp OLDBBHOME/etc/bbwarnrules.cfg .      copy and edit appropriate config files
    	cp OLDBBHOME/etc/bbwarnsetup.cfg .
    	cp OLDBBHOME/etc/bb-dftab .
    	cp OLDBBHOME/etc/bb-proctab .
    	cp OLDBBHOME/etc/bb-cputab .
    	cp OLDBBHOME/etc/bb-bbexttab .
    	cp OLDBBHOME/etc/bb-msgstab .
    	cp OLDBBHOME/etc/security .

    	vi bbdef.sh			Set environment variables for your setup
    					like FQDN, ...
    	vi bbinc.sh			It is always useful to run a diff getween
    					your old version of bbdef.sh/bbinc.sh
    					to  view your own changes
    	cd ../..
    					if you already have a link called
    					bb used to access your BB install
    	rm bb
    	ln -s <bbdir> bb          Setup bb as a link to the new bbdir
    					directory such that it's easier
    					to refer rc scripts to bb/ without
    					having to change them every time
    					you upgrade.

    	Finally don't forget to move over the extensions you've installed
    	(lardd, "The State", ...)

    1.3	Compiler directives

    	You change the behavior of BB  by compiling it
    	with different compiler options.  All the options are
    	defined in the CFLAGS variable on the Makefile

    	-DENABLE_DISABLE
    		Include code to temporarely disable notifications on a host
    	-DDATAMSG
    		To allow "data" messages type
    	-DNOTESMSG
    		To allow "notes" messages type
    	-DBZERO
    		If your OS doesn't support the bzero() function
    	-DZOMBIE
    		If zombie processes are generated while running BB
    		This will get rid of them
    	-DGETTIMEOFDAY
    		To get duration statistics during TCP service checks
    	-DREGEXEC
    		If your OS supports regexec(),regcomp()
    	-DREGEX
    		If your OS support regex(),regcmp()
    	-DRE_EXEC
    		If your OS supports re_exec(),re_comp()
    	-DTIMEH
    		If your OS has time.h instead of sys/time.h
    	-DSIGSETJMP
    		If your OS supports sigsetjmp() and
    		does not handle signals properly with setjmp()
    	-DDONTCLEANSTRING
    		If you don't want BB to remove dubious
    		characters in incoming messages: `$;|&\
    	-DNOTAMP
    		If you don't want BB to remove '&' in an incoming
    		message. You'll require this if you install the LARRD
    		extension.

    ------------------------------------------------------------------------

    Section 2: Configuring the Big Brother hosts file

    2.1	The etc/bb-hosts file

    	The etc/bb-hosts file controls where Big Brother looks for things
    	and the actions that are taken.  The format is identical to the
    	standard /etc/hosts file, except with additional directives
    	for Big Brother.

    	The etc/bb-hosts file should be kept identical on all BB client
    	hosts as to reduce the risk of misconfigured bb-hosts file.
    	Even though clients only require entry lines for BBDISPLAY,
    	BBPAGER and the local client, it is recommended that you distribute
    	a master bb-hosts file to all clients.  The only time you should
    	not have an identical bb-hosts file is if you decide to have more
    	than one BBNET host, in this case each BBNET host should only
    	have tested entries in its bb-hosts file and those entries
    	should not appear in any other BBNET host.

    2.2	Basic format of etc/bb-hosts

    	Lines are of the format:

    	IP-ADDR		HOSTNAME	# DIRECTIVES

    	IP-ADDR: XXX.XXX.XXX.XXX

    	HOSTNAME: host.domain.com	if FQDN="TRUE"
    		  host			if FQDN=""

    		FQDN (display with Fully Qualified Domain Name) is a
    		variable that you set in the etc/bbdef.sh file.

    	Directives the Big Brother knows about are:

    		BBDISPLAY	This host displays the HTML results

    			This host receives incoming status reports, generates the
    			HTMLized status reports, the bb.html and bb2.html web pages.
    			There may be more than one BBDISPLAY if you want redundancy.
    			BBDISPLAY host execute the bbd daemon to collect reports, and run
    			the web/mkbb.sh & web/mkbb.sh scripts to generate the bb.html
    			and bb2.html pages.  You'll need a Web server on this host
    			to display the results.

    		BBPAGER		This host acts as the notification server

    			This host receives requests to notify admins and processes
    			those requests.  There may be more than one BBPAGER as well,
    			for redundancy.  Of course, you'll be notified more than once
    			per problem.  BBPAGER hosts likeweise run the bbd daemon so they
    			can receive notification requests.

    			To requests more than one BBDISPLAY/BBPAGER host, put the
    			BBDISPLAY/BBPAGER directive on the config line for each host
    			that acts as BBDISPLAY/BBPAGER.

    		BBNET		Indicates that this host checks
    					the ip network services

    			Many hosts can act as a BBNET.  If you do have more than
    			one BBNET, make sure that the bb-hosts file only
    			contains the hosts that you will test against.  You must
    			be very careful because if a host is tested by many
    			BBNETs then the status display for the services on
    			that host will only show that last test status.
    			You may get errors for that host even though it shows
    			green, it's just that another BBNET send a status message
    			afterwards and that message returned OK. On the other
    			hand you may want to have tests from different location :)

    		BBRELAY:<hostname or IP address> 	The host defined is a relay host

    			This host will be relayed all incoming messages from
    			any BBDISPLAY/BBPAGER hosts that has this directive defined
    			on the same line as the BBDISPLAY/BBPAGER.
    			So if a BBDISPLAY/BBPAGER entry in the bb-hosts file
    			also contains a BBRELAY:<hostname|IP addr> directive
    			then that host will be relayed all incoming messages
    			from the current BBDISPLAY/BBPAGER. This is useful
    			if you are moving a BBDISPLAY or a BBPAGER and you
    			don't want to change all the BB clients bb-hosts files
    			at once.

    			1.2.3.4   host1.domain99.com # BBDISPLAY BBPAGER BBRELAY:4.3.2.1

    			Host 4.3.2.1 will be sent all incoming messages that 1.2.3.4 receives.

    		http://www_path	The host is tested for http connections
    					using the www_path
    		https://www_path The host is tested for https connections
    					using the www_path
    			(requires lynx: http://lynx.browser.org/ or curl:  http://curl.haxx.se/ )

    			Note that you can specify multiple URLs by joining the
    			URLs with '|':
    				http://www_path|http://www_path1
    			or by specifying them individually:
    				http://www_path   http://www_path1

    		ftp		Test for ftp service
    		smtp		Test for smtp server
    		pop3		Test host for pop3 server
    		telnet		Check telnet service
    		ssh		Test ssh server
    		nntp		Test nntp news server
    			Check /etc/services for proper service name
    			especially pop3 (sometimes referred as pop-3).
    			These directives names MUST appear in /etc/services
    			of the BBNET host that is doing the testing.
    			Any text based service can be checked by putting
    			the /etc/services name as the directive and adding
    			in the list of services in the BBNETSVCS variable
    			of etc/bbdef.sh. (See below for TCP ports that are
    			note in /etc/services) By default, the test sends
    			the "QUIT" string the to the tested service.

    			The '!' can be prefixed to a service such it will
    			be checked that the service is NOT running (!ftp).

    			The '?' can also prefix a service.  This prefix
    			indicates that the service is considered a
    			dialup service: it will generate a clear button
    			if the service is down. NOTE: the ! and the ?
    			prefixes can only be used with the basic TCP services
    			that are handled by the bb-network.sh script.
    			http* and dns/dig tests cannot use these prefixes.

    			The :s , :q and :Q tags can be added to the directive
    			by appending them to the directive: ftp:s or ftp:s:q
    			This will enable special options to the bbnet
    			program that does the TCP tests.
    			:s - Just connect to the port, do not engage in
    				a converstation with the service tested (silent)
    			:q - Do not return timing statistics if you've
    				enabled them (quiet)
    			:Q - Do not return error messages usually returned
    				by the bbnet program.

    			You can also add a custom TCP port to test. You
    			have to define the network service name in the
    			BBNETSVCS variable of etc/bbdef.sh.
    			In the bb-hosts you declare that new TCP test by
    			putting in entry in this format: svcname:portnumber
    			i.e.  mytest:1234 - Test port 1234 and return
    			it with the mytest status name

    		dns		Checks for name resolution server
    		dig		Same check as dns but using the dig
    				command if the command is available.

    		noping		Don't do ping test for this host
    					and display a clear dot.
    		noconn		Don't do ping test for this host and
    					don't generate any colored dot
    		dialup		If host is down then display a clear dot
    					otherwise display a green status

    		testip		Use the IP address defined to execute network
    				test.  Don't use the hostname given.

    		bbd	Test a remote BBDISPLAY/BBPAGER

    		You can add your own directives that you access through an
    			external test.  Known directives from user contributed
    			tests: oracle/fping/trap/sybase/...  These contributions
    			can be found on the user contributions archive.
    			(www.deadcat.net)

    	This is a sample bb-hosts file:
    	#
    	# BIG BROTHER HOSTS FILE
    	#
    	192.168.110.102	bobo	 	# BBDISPLAY BBPAGER ftp smtp pop3
    	192.168.110.95	admin		# http://admin/
    	192.168.110.108	mynet		# BBNET !nntp

    	bobo is the display and notification server.
    	mynet is the host that runs ip network services checks
    		it will check the ftp/smtp/pop3 services on bobo,
    		then the http service on admin and finally it makes sure
    		that nntp is NOT running on mynet

    	You can also group hosts within a seperate HTML table.  There
    	are three group tags defined: group, group-compress and group-only.
    	Results can also be saved to a specific page using the 'page' directive.

    	Changes to the etc/bb-hosts file are automatic.  You do not
    		have to restart BB to initialize the changes except if you
    		make changes to BBDISPLAY/BBPAGER/BBNET locations.

    2.3	etc/bb-hosts special directives

    	2.3.1	Display results in another HTML page
    		page nyrouters New York Routers

    		BB always creates two HTML pages: bb.html/bb2.html.
    	When the "page" directive is encountered, BB will then create a new
    	HTML page until another "page" directive is encountered and will save
    	the output from that point on to that new HTML page.  It will keep a pointer
    	of the new HTML page in bb.html.  The first argument is the directive,
    	the second one is the name of the page (i.e. nyrouters.html
    	as the .html tage is automatically added) and the remaining
    	arguments is the caption that will appear in bb.html when it
    	creates the link to that page.  Always place your page directives
    	after all hosts that you want to appear in the bb.html page.
    	When you use a page directive then all subsequent output will
    	be in an HTML page (the current one or a subsequent page directive)
    	Each HTML page will be created in it's own directory to enable
    	password protection (using your web server's features) based
    	on the directory.

    	N.B. summaries and dialup directives will always appear on
    		bb.html/b2.html and not in any HTML subpages

    	2.3.2	Display "page" results into an HTML subpage
    		page nyrouters New York Routers
    		subpage manhattan Manhattan Routers
    		...
    		subpage queens Queens Routers
    		...

    		BB allows up to 3 levels of HTML display pages.
    	The top pages bb.html and bb2.html.  Another level using the
    	"page" directive and finally the last level defined by
    	the "subpage" directive.  This will create a link in the
    	current HTML page (created with "page") to a subpage.
    	The subpage is created in it's own directory (as in "page").
    	It allows you to create a directory structure where there
    	can be up to 3 different views.  A topmost with bb.html/bb2.html.
    	The next level with the "page" directive and the lowest level
    	created with the "subpage" directory.  This should be ample to
    	secure the different views with user/password protection.

    	2.3.3	Grouping hosts on the display (HTML tables)

    		The Web display may be broken into tables to create
    	a more aesthetic and sensible display.  It's also much faster
    	to load small tables, than one giant table.  So consider
    	grouping your etc/bb-host entries logically and separating
    	them using the "group" , "group-compress" or "group-only" directives.

    		group Intranet Servers

    		The group directive defines a block of hosts to be grouped
    	in the same HTML table.  All hosts lines following the group
    	directive, until a new group/group-compress is defined, belong
    	to that group.

    		group-compress California Servers

    		The group-compress is identical to the "group" directive
    	except it will only display services (columns) containing data
    	for that group.

    		group-only  conn|cpu|disk   Restricted Services

    		The group-only will create a table with only the colums
    	defined in the directive.  The columns are '|' delimited.

    	Note that the text that follows the prevous directives is the
    	title given to the HTML table.  Note that you can embed HTML
    	code in the title: Italic/H3/etc..., but use with caution:

    	group <I>Intranet Servers</I>
    	group-compress <I><H3>California Servers</H3></I>
    	group-only  conn|cpu|disk   <B>Restricted Services</B>

    	2.3.4	Sending a summary to another BBDISPLAY
            	summary quebec.bb 255.255.255.255 http://cafe.domain.com/bb/bb.html
            	summary quebec.bb2 255.255.255.255 http://cafe.domain.com/bb/bb2.html

    		This indicates to the BBDISPLAY machine that
    	summary information of the bb.html, bb2.html, an HTML page
    	or an HTML subpage be forwarded to the IP address noted
    	by the 3rd item on summary line line.  The summary can be
    	sent to more than one BBDISPLAY.
    	N.B. summary directive from other BBDISPLAYs (or the current
    		one) will always appear at the bottom of the
    		 bb.html/b2.html and not in any HTML pages/subpages.

    	2.3.5	Testing Modem Banks
    		dialup modem-bank 204.19.116.20 4

    		The dialup directive (not to be confused with the other
    	dialup tag which displays a clear button if the host is down)
    	is used to specify connectivity for a bank of modems.  The
    	2nd parameter is the name to be displayed on the display page.
    	The 3rd argument is the starting IP address of the modem bank.
    	The last argument is the number of modems on that bank.
    	A green dot will be displayed for IP addresses that are currently
    	used and a clear dot will be displayed if the IP address
    	is not used.  Note that this test can take a long time if
    	your modem banks have many unused modems as the test will
    	time out to determine the connection status.
    	N.B. dialup directive output will always appear at the
    		bottom of bb.html/b2.html and not in any HTML
    		page/subpages.

    	2.3.6	Specifying DHCP hosts (no fixed IP address)
    		0.0.0.0	dhcphost.domain	#

    		The 0.0.0.0 indicates that this host is a DHCP
    	host and that the BBNET host will not try to run the
    	connection test on that host using the IP address but only
    	the hostname.

    2.4	Validating your etc/bb-hosts file

    	You can validate the etc/bb-hosts file by executing the

    	etc/bbchkhosts.sh

    	command.  Note that it is a simple check and sometimes
    	may complain about invalid qualifiers especially if they
    	are associated with user contributed scripts.  Also in some
    	cases the script will complain but also mention that the
    	error might not apply.  Read the output carefully.

    2.5	Pointers for writing entries in bb-hosts

    	If you are going to use the same hostname in multiple
    	entries then use the "noconn" directive for all entries
    	for that hostname except the first entry.

    	1.2.3.4  some1.host # ftp smtp
    	2.3.4.5  some2.host # ftp smtp
    	...
    	group-only ftp All ftp servers
    	1.2.3.4 some1.host # noconn
    	2.3.4.5 some2.host # noconn

    	The first entry indicates all the tests to run while
    	the second entry is used only as a display definition.

    ------------------------------------------------------------------------

    Section 3:  Customizing your BB installation

    3.1	The runbb.sh script

    	The runbb.sh script is the master BB script.  It sets up the
    	BB environment with the values from the configuration files.

    	You MUST set the BBHOME variable to the directory path where
    	you have installed BB.  Without it, BB will not run properly.

    	cd $BBHOME  (to your BB install directory)

    	To start BB:

    	./runbb.sh   or    ./runbb.sh start

    	To stop BB:

    	./runbb.sh stop

    	To restart BB:

    	./runbb.sh restart

    	When BB starts/restarts it checks that there is no processes
    	already running.  If this is the case, BB will complain but
    	will continue to start as it cannot determine if the process
    	was really started by BB.  BB checks for any processes that
    	contain the BBHOME in its definition.

    	If you ever have to add a script that runs as a deamon use
    		the following template to add it in runbb.sh:

    	echo "Starting snmptrapd"
    	echo "Starting snmptrapd" >> $BBOUT
    	MIBFILE=$BBHOME/cmu/etc/mib.txt
    	export MIBFILE
    	$BBHOME/cmu/bin/snmptrapd -B | $BBHOME/bin/bb $BBDISP - &
    	set `$PS | $GREP snmptrapd | $GREP -v "$GREP"` > /dev/null
    	echo $1 >> $BBPID

    	This is from an SNMP trapd extension available at the FTP
    	archive site (ftp://ftp.deadcat.net/pub/BB).  You'll notice
    	that the PID is added to $BBPID,  This way, you'll be
    	able to stop the extension at the same time as BB when
    	executing "runbb.sh stop".

    3.2	Configuring the etc/bbdef.sh file

    	etc/bbdef.sh is where you configure how Big Brother reacts to a
    	variety of situations that may arise.  You can establish levels
    	for warnings(yellow), panics(red) values.

    	Please note that any modifications to this file do not come
    	into effect until the next restart of BB.  It is best to
    	stop/start BB immediately after this file has been modified.

    	BBUSER is the user name that BB should run as.  If you
    	try to start BB using a different used than the one
    	specified in BBUSER, BB won't start.

    	STOPROOT prevents you from starting BB as root.  It is
    	recommended to you set STOPROOT=TRUE.

    	Use FQDN (fully qualified domain naming)
    	FQDN="" - Host names rendered on the BB status pages will only
    			include the host name, not including the domain
    			name.
    	FQDN="TRUE" - Host names rendered on the BB status pages will
                            include the host name and the domain name.

    	Which program to use to check the HTTP connectivity:
    	LYNX="/usr/local/bin/lynx -dump -head" - Use lynx if you need to
    		check password protected pages.  Please use the proper
    		directory path as per your host installation. cUrl
    		(another program) can also be used to check for
    		protected pages.

    	NONETPAGE defines which IP services not to notify on if an error
    		is encountered.  Include the services as they are named
    		in the etc/bb-hosts file.  This is only valid on a host
    		defined as a BBNET host.

    	BBTMP indicates the location of the BB temporary directory. You
    		may want to locate it outside of the BB structure.

    	BBREPWARN specifies the value at which the reporting feature
    		returns red or yellow.  Reporting will return green
    		if 100% availability is determined, will be yellow
    		until it reaches the value of BBREPWARN and then
    		will return red below that value

    	The DFWARN and DFPANIC variables contain the % levels at which
    		the disk test will fail.  By default DFWARN is 90% and
    		DFPANIC is 95%.  These values can be overriden by values
    		in the etc/bb-dftab file described in section 5.2.

    	CPUWARN and CPUPANIC are default levels based on the load
    		average.  CPUWARN is set at 150 and CPUPANIC at 300.
    		These values are the load average (from uptime)
    		multiplied by 100 (the dot is stripped from the value
    		returned by uptime).  You can also set these values
    		in the etc/bb-cputab file.

    	If you want the load average to be displayed in its original
    		format (not stripped of its decimal point) then set
    		the DISPREALLOADAVG to TRUE.

    	You can define the processes for which you want to monitor if
    		they are up and running.  You do this by setting the
    		PROCS and PAGEPROC variables.  The processes included
    		in PROCS will lead to a yellow condition if one or more the
    		processes is down.  Processes in PAGEPROC will generate
    		a red condition if a process is down.  But the preferred
    		configuration method is the etc/bb-proctab file.

    	MSGS and PAGEMSG are used to monitor error messages in the
    		system logs.  MSGS contain keywords to look for in
    		the system logs.  If a keyword is found then this
    		will create a yellow condition and if the keyword
    		is also in the PAGEMSG variable then it will be upgraded
    		to a red condition. Note that the values of these
    		variables can be overridden using the etc/bb-msgstab file

    	You can also discard certain messages by defining portions of
    		that message in the IGNMSGS variable.  You can define
    		multiple messages by delimiting each message by a ';'.
    		The value can also be set in etc/bb-msgstab.

    	MSGEXPIRE defines the amount of time the yellow/red conditions
    		will last.  Yellow conditions will last by the number
    		of minutes defined by the first number will red conditions
    		are defined by the 2nd number.  i.e.  30:60, yellow=30m
    		and red=60m.

    	The CHKMSGLEN determines if the message file zero-length test
    		is run. Set to TRUE or FALSE

    	Define how many MSGS and PAGEMSG entries that are returned
    		in a messages file(s) check. YELLOWMSGSLINES determines
    		the number of yellow entries while REDMSGSLINES indicates how
    		many red entries are returned.

    	The OLDSTYLEBBMSGS variable indicates if the message file(s) check
    		should not expire message just like pre-1.6d versions. You
    		had to expire messages by clearing them by hand in the
    		message file(s) by modifying the token string so it wouldn't
    		match anymore.

    	BBWEB="/bb"
    		This variable defines the path to the root page of BB.
    		Make sures it never ends with "/".  All locations
    		in BB are referenced from that location.

    	BBWEBHOST is the full path to your BBDISPLAY server except
    		the directory location: http://www.bb4.com

    	BBWEBHTMLLOGS="${BBWEB}/html"
    		This variable defines the path to the HTMLized status logs.
    		This path is concatenated with the name of the HTMLized
    		status file and is appended to the e-mail notification
    		message for a quickpath to the error information.

    	CGIBINURL is the directory location of the CGI scripts.

    	The PAGELEVELS variable indicates to the 'bb' process at which
    		level(s) it should send a notification message.  The 'bb'
    		program is used to send status messages and if the status
    		level of the message is found in the PAGELEVELS variable
    		then a notification message is also sent to the BBPAGER
    		host.  This is the default value.  It can be overridden
    		by the 'pagelevels:' token in the etc/bbwarnsetup.cfg
    		file.  The PAGELEVELSMAIL contains the color level(s) on
    		which only e-mail is to be sent.  The color level must also
    		be found in PAGELEVELS.

    	The WEBHISTORY defines if you want to use the web-based history
    		display script.  This script gives you a 24hr HTML chart
    		and the last 50 status changes.

    	BBHIST_IGNOREBLUE tells the reporting to ignore maintenance
    		time in statistics compiling if it's set to TRUE.

    	The SAVESTATUSLOG variable defines if the historical log of
    		the host.service is kept or not.  The logs are saved
    		in the $BBHISTLOGS directory.

    	The BBLOGSTATUS variable determines if each log received is
    		rendered into a static HTML page, dynamically by CGI or
    		not at all (it appears as plain text).  If it is
    		set to DYNAMIC then a CGI will create the HTMLized status
    		log only when requested.  That's when you click on a
    		colored dot on the BB display pages.  The dynamic CGI
    		is called bb-hostsvc.sh. If is set to DYNAMIC then
    		a static HTML status log will be generated the www/html
    		directory.  If it is set to TEXT then no HTMLized status
    		logs are available.  The status will appear as plain text.

    	You can define how many pings are executed before a connectivity
    		test fails using the PINGTESTS variable.  You can also
    		set the sleep factor between tests using PINGSLEEPFCT. Each
    		sleep within a single test is incremented by that factor.
    		PINGWARN defines how many failed pings are required before
    		the connectivity test yellow.  If all pings fail then
    		it returns red.

    	NOPINGCLEAR lets you decide if you want clear dots for host that
    		have the noping directive.  If you set to TRUE then clear
    		dots will be displayed for hosts with the noping as
    		well as hosts with IP address 0.0.0.0 (DHCP hosts or
    		hosts accessed only with its full name).  You can use the
    		noconn directive in conjunction with NOPINGCLEAR to select
    		which hosts you don't want a clear dot.

    	PURPLEDELAY sets the default expiry time stamp on an incoming status
    		log.  It is expressed in minutes.

    	The IPTEST_2_CLEAR_ON_FAILED_CONN variable, if set to TRUE,  defines
    		that if a network test fails and that the connectivity (ping)
    		test also failed then display the test results as clear.  You
    		can turn off this setting individually by test by using the
    		~ modifier.

    		in etc/bbdef.sh:
    			 IPTEST_2_CLEAR_ON_FAILED_CONN=TRUE

    		in etc/bb-hosts
    			123.123.123.123.host1 # ftp

    		If the connectivity test to 123.123.123.123 failed and
    		the ftp test also failed (very likely) then display the
    		test with a clear dot.  If you had set the directive to
    		~ftp then it would have been displayed as red.

    		Note that this will work only if the basic connectivity
    		is not disabled (CONNTEST=TRUE - enabled).

    	SUMMARY_SET_BKG value determines if the summary dots
    		are to be taken in account when determining the background color.
    		Set it to anything but TRUE if you don't want summaries to
    		set background.

    	BBHISTEXT contains scripts names located in BBHOME/ext/hist
    		that are to be executed when bb-hist.sh is run.
    		This is to add site specific display information.

    	BBMKBBEXT and BBMKBB2EXT defines scripts to execute at the
    		end of the mkbb.sh and mkbb2.sh script respectively.
    		Same basic idea as BBHISTEXT, you may extend
    		their functionality that way.  The scripts should
    		be located in the ext/mkbb directory.

    	BBEXT contains the list of external scripts found in BBHOME/ext
    		that are to be executed. *** NOTE *** etc/bb-bbexttab
    		should be configured instead of using the BBEXT variable.
    		The BBEXT variable has been kept for backward
    		compatibility.  BBDEFSLEEP is the lifecycle
    		in seconds of an external script while STARTSLEEP is
    		then initial interval between each external script
    		at startup.

    	The CONNTEST variable determines if the default connectivity
    		testing is executed.  Set to FALSE if you don't want
    		the standard connectivity test.  This is usually
    		done when the fping.sh external script is used.

    	BBHOSTHISTLOG & BBALLHISTLOG are used when specifying if
    		host based and system based historical information
    		should be kept.  This may be disabled when you
    		are not interested in historical data.

    	The BBNETTHREADS variables defines how many conccurent
    		bb-network.sh tests should run to reduce
    		the length of time of the network testing

    	BBNETSVCS contains the valid TCP network tests that BB
    		can perform.  If you want to test a non-supported
    		port then add the port name here.  Read this
    		section for more details.

    	If you want WAP enabled display then set the WML_OUTPUT
    		variable to TRUE.  Depending on your device, you
    		can set the size of the bb.wml/bb2.wml pages
    		using WMLMAXCHARS.

    	BBTOOMANYDAYSUP is used to notify if a host has been running
    		for more than so many days.  Some people use this
    		to remind them to reboot some servers after a certain
    		amount of time.

    	WARNMINSONREBOOT and WARNCOLORONREBOOT determines which color
    		the cpu column will bear the color defined in
    		WARNCOLORONREBOOT and for how long after a server reboots.
    		Note that only red/yellow are accepted and that the duration
    		has to be less than 60 minutes.

    	You can change the timeout values used by bbnet. Change the
    		BBNETTIMER1/BBNETTIMER2 and BBNETTIMER3 to the
    		appropriate values.

    	You can disable local test(s) by setting one or all of
    		DOCPU/DOMSGS/DOPROCS/DODISK to FALSE.  If you disable
    		a test you can decide if you still want to send a message
    		to the BBDISPLAY by setting the color in
    		NOCPUCOLOR/NOMSGSCOLOR/NODISKCOLOR/NOPROCSCOLOR to the
    		color of your choice or omit the color if you don't
    		want any message sent to the BBDISPLAY.

    3.2	Localizing programs paths

    		Sometimes the OS specific programs paths found in the
    	installation might be wrong for a particular host(s) OS.
    	Change the paths in the etc/bbsys.local file.  The variables
    	contained in this file all have default values from the
    	bbsys.OS file.  The original bbsys.local is copied from the
    	etc/bbsys.OS file during installation (bbconfig); so keep track
            of your changes as they will automatically be overwritten if you
            re-run bbconfig.

    3.4	Creating BB clients

    	Creating BB clients is very simple:
    	cd $BBHOME
    	cd install
    	./bbclient <client-hostname>

    	 <client-hostname> must be found in the etc/bb-hosts file.

    		This will create a tarball for the client that has the same
    	OS/HW platform type as the host on which the bblient command
    	is executed. The tarball is found in the directory above
    	$BBHOME.

    		 Don't forget to configure the PROCS and PAGEPROC
            variables in etc/bbdef.sh on the client after you've untarred the
    	archive such that the correct processes are monitored.

    		If you have different OS/HW platforms then install BB,
    	as a client,  on each OS/HW platform (then use the bbclient to
    	create a tarball for each identical OS/HW client) and don't forget
    	to copy your master bb-hosts file to it. Run through the install
    	procedure to make sure that the clients are also installed
    	properly (bbchkcfg.sh/bbchkhosts.cfg).

    3.5	Configuring the Web Stuff

    		All Web-based things live in the $BBHOME/www directory.
    	In order for the bb stuff to work correctly, this directory must
    	be linked into your Web site somewhere.  I suggest something like
    	the following:

    	ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir
    	of your web server).

    	You should then be able to access BB with the URL
    		http://.../bb/bb.html
    	where ... is your webserver address.

    		You also have to configure the script which generates the
    	web-based history page if you set the WEBHISTORY variable to TRUE in
    	etc/bbdef.sh.  This page is accessible from the "History" button
    	when you click on a colored dot in one of the display pages
    	(bb.html/bb2.html).  You must copy the web/bb-hist.sh script to your
    	/cgi-bin directory and set the BBHOME.
    	Make sure the script has the proper permissions to execute: system and
    	web server permissions.  If you want only a text based history display
    	then set the WEBHISTORY variable to "FALSE" in etc/bbdef.sh.
    	Set the BBHIST_IGNOREBLUE variable to TRUE in etc/bbdef.sh if you
    	don't want the blue state to be used in the calculations of daily % use.

    		By default the CGIBINURL (in etc/bbdef.sh) points to /cgi-bin
    	but if your web server cofiguration has a different CGI directory
    	then change it here or your history feature won't work.

    		If you intend to enable manual notification/acknowledgement
    	("PAGE/ACK"button in bb.html/bb2.html) then copy the web/bb-ack.sh
    	script to you /cgi-bin directory and set the BBHOME
    	variable in the script.  Make sure the script has the proper
    	permissions to execute: system and web server permissions.
    	Acknowledgments generate a log entry in $BBACKS/acklog file.

    3.6	History

    	Each time a status log is received it is checked against its
    	previous status to see if it has changed.  If it has, then
    	the duration of the previous status is recorded and a new line
    	is appended to the history file with the timestamp of the
    	new status.

    	There's a history file for each host.service (status log) received.
    	The history files are located in the $BBHIST directory.

    	Here's a sample:

    	Sun Jan 24 18:30:45 1999 green 917220645 232068
    	Wed Jan 27 10:58:33 1999 red 917452713 1560
    	Wed Jan 27 11:24:33 1999 green 917454273

    	Notice that last line doesn't have a duration because it is
    	the current status.  The duration will be appended only when
    	the status changes.

    	If you do not wish to have historical data then rename
    	$BBHIST to something else or remove it.

    3.7	Adding your own tests

    	You can easily add your own tests.  Start with the template
    	available at ext/ext-proto, add your code.  Look at bb-local.sh,
    	bb-network.sh for an example on how to send data to BB.
    	Specify the name of your script in the etc/bb-bbexttab
    	file. This file defines all scripts to run for each hosts.
    	Start with the etc/bb-bbexttab.DIST file, copy to
    	etc/bb-bbexttab and configure. Typical entries in
    	etc/bb-bbexttab:

    	<host> : <options> : scripts...
    	www.bobo.com : : script1 script2;3600 script3;900 script4
    	www.baba.com : : script5 script6

    	Note that <options> is not yet implemented.

    	make sure each defined script exists in the ext directory
    	of the client install.  See below for an explanation of
    	the script2;3600 entry.

    	Restart BB and your test should be running.  But
    	before you use it within BB, I suggest you test it for errors
    	by using the method:

    	cd /home/bb (or wherever your BB is located)
    	BBHOME=/home/bb
    	export BBHOME
    	. ./etc/bbdef.sh
    	cd ext
    	./yourexternaltest

    	Look for errors, fix them, rerun your test until you're
    	satisfied, then update bbdef.sh.  Note that all temporary
    	files should be created in $BBTMP and make sure you remove them
    	after use.   Also, remember that you don't have to deal with
    	sending notification messages, the 'bb' process send a 'page'
    	type message to the BBPAGER host when the status color is
    	found in the PAGELEVELS variable defined in bbdef.sh.

    	You can also set the frequency of the test by specifying
    	the interval it should run at.  In ext/bb-bbexttab, set an
    	interval qualifier with the script name by appending the
    	qualifier to the script name. Here's an example entry in
    	etc/bb-bbexttab:

    	www.bobo.com : : script1 script2;3600 script3;900 script4

    	script1 and script4 will run every 5mins (the default)
    	while script2 and and script3 will run every 60 and 15
    	minutes respectively.  You specify the interval in seconds.
    	Note that the delimiter is ';'. It used to be ':' in BBEXT
    	but it had to change due to the line entry format.

            When it's ready don't forget to update the svcerrlist token
            in the bbwarnsetup.cfg file on your BBPAGER host.  You must
            assign a numeric code to your column name.

    	You can also set the lifetime of the status sent by your
    	script by following the instructions described in
    	section 6.9: "Setting a Time To Live to a status message".

    3.8	Starting BB at system boot

    	Depending on U*X operation system version (Linux/BSD/Solaris/...)
    	your startup procedures will vary from OS to OS.  In short, you
    	have to create a startup script that has a start/stop
    	capabilities.  Under a lot of OSes, you'll want to create your
    	script in the init.d directory and create an S89bb link in rc3.d that
    	links to init.d/S89bb (you may also want to create a K11bb link in
    	rc3.d that'll be used when the system shuts down).  Use an existing
    	startup script has an example and substitute with these commands:

    	To start BB, use this command:

    	su - <bbuser> -c "cd <BBHOME>;./runbb.sh start"
    		or
    	su - <bbuser> -c "cd <BBHOME>;./runbb.sh restart"

    	To stop BB, use this command:

    	su - <bbuser> -c "cd <BBHOME>;./runbb.sh stop"

    	<bbuser> is the user that BB will execute has
    		make sure that bbuser has all permissions under BBHOME
    	<BBHOME> is the location of your BB install

    	e.g. su - bb -c "cd /home/bb;./runbb.sh start"

    3.9	Summary of BB protocol

    	Big Brother is based on a client/server paradigm.  BB Client
    	software sends a message to a BB daemon (either a BBDISPLAY,
    	BBPAGER or a BBRELAY) over a TCP connection using port 1984.
    	The client connects to the BB port socket writes its data and
    	closes the port.  The server listens on a that port socket
    	then reads the incoming messages and closes the socket.

    	Here's a quick summary of valid messages using an example:

    	Status message:

    	status www,bb4,com.disk red Wed Mar  7 21:16:33 EST 2001 Disk partitions on www,bb4,com OK
    	/dev/wd0s1a    496053    25081   431288     5%    /
            /dev/wd0s4e   1998122  1094955   743318    60%    /home
            /dev/wd0s2e   1489830  1094928   275716    80%    /usr

    	*** Note that the status message can be also be sent within
    	    a combo message.  You should only send "green" status thru a combo
    	    message.  Each status message should be delimited by at least 2 newlines

    		combo

    		status ...

    		status ...

    		status ...

    	Error message:

    	page www,bb4,com.cpu red Thu Jan 25 17:36:30 EST 2001 up: 16 mins, 3 users, 105 procs, load=363

    	For both messages,

    		host can be fully qualified or not
    			www,bb4,com.disk
    			www.disk
    		  Note that ',' replaces the '.' in the hostname
    		  for historical reason (I know, I know...)
    		color can be red,green,yellow,purple,clear,blue
    		text can be anything but usually on the first line
    		it's just a summary while the rest of the text is
    		a more detailed status.  Usually the date is inserted
    		as the first entry in the text.

    	Summary message:

    	summary quebec.mtl 255.255.255.255 red http://cafe.domain.com/bb/

    		quebec.mtl is the row/col definition
    			quebec is the row
    			mtl is the column
    		255.255.255.255 is the BBDISPLAY to send the summary to
    		http://cafe.domain.com/bb/ is the URL that will be displayed
    		  when the red dot will be clicked on

    	Acknowledgement messages:

    	ack add_event 1234567 120 Rob acked this

    		This is to acknowledge an event from a script

    		1234567 is the acknowledgement ID
    		120 is the amount in minutes of the TTL of the ack
    		"Rob acked this" is an optional message

    	ack rm_event www.bb4.com.disk 345

    		This is to signify that event has recovered.
    		This should only be sent by a BBDISPLAY server daemon

    		www.bb4.com.disk is the host.service combination
    			that has recovered from an event. Note
    			that commas are not needed in the hostname
    			(confusing isn't it)

    	ack addtag_event www.b4.com.disk robert@bb4.com 1234567 120 999999999 Rob acked this

    		This message is used to create a tag file on BBDISPLAYs
    		to enable a checkmark to be displayed in the web pages.
    		This should only be sent by a BBPAGER server daemon

    		www.bb4.com.disk is the host.service combination
    			to be acked.
    		robert@bb4.com is the recipient that is acking the event
    		1234567 is the ack number
    		120 length in minutes of the ack
    		99999999 is the current time in integer format
    		"Rob acked this" is an optional message

    	ack rmtag_event www.bb4.com.disk

    		This message is used to remove a tag file on BBDISPLAYs
    		that was created by an addtag_event message.

    		www.bb4.com.disk is the host.service combination to remove

    	Maintenance mode messages: (these message types are disabled by default)

    	disable www.bb4.com* 120 Disk Removal

    		This message will notify BB to disable all notifications
    		for the host.service combinations that match the
    		regular expression specified.  Only a BBPAGER
    		should receive this message

    		www.bb4.com* is a regular expression that will match
    		  all services for www.bb4.com
    		120 is how long notifications will be disabled in minutes
    		"Disk Removal" is an optional message

    	enable www.bb4.com*

    		This message will notify BB to reenable all notifications
    		for the host.service combination specified. Only a BBPAGER
    		should receive this message.

    		www.bb4.com* is a regular expression that will match
    		  all services for www.bb4.com.

    	offline www.bb4.com.disk

    		This message instructs BB to create the tagfile in the
    		bbvar/disable directory.  A BBPAGER automatically
    		sends this message to a BBDISPLAY.  It shouldn't
    		be used by scripts.

    		www.bb4.com.disk is the host.service combination

    	online www.bb4.com.disk

    		This message instructs BB to remove the tagfile in the
    		bbvar/disable directory.  A BBPAGER automatically
    		sends this message to a BBDISPLAY.  It shouldn't
    		be used by scripts.

    		www.bb4.com.disk is the host.service combination

    	Data transfer messages: (these message types are disabled by default)

    	data www.bb4.com.data ....
    	....
    	....

    		The data message appends to the bbvar/data/www.bb4.com.data
    		file the text specified by ....  Note that the data text
    		is appended.

    	notes www.bb4.com.html ....
    	....
    	....

    		The notes message write the text defined by .... to the
    		www/notes/www.bb4.com.html file.  Note the the data message
    		overwrites the current file. This file can then be
    		display by clicking on the hostname in the web display
    		(bb.html/bb2.html).

    ------------------------------------------------------------------------

    Section 4:  Configuring the Notification feature (BBWARN)

    4.1	A little History

    	In the Beginning :), BB could only e-mail and notify by pager.
    	The configuration required that all BB hosts (servers and
    	clients) have the PAGER variable set with the recipients of the
    	notifications.  There were no possibilities of customizing the
    	notification feature.   This led to the:

    	The Pager Protection Act of 1997
    	================================

    	Use of BB has been linked to the untimely death of at least one
    	pager in Texas.  In order to protect pagers from their users, a
    	notification scheme has been created to allow to specify a delay
    	between the notifications.

    	But this wasn't enough.  BB netizens got restless and finally a
    	solution was delivered by Robert-Andre Croteau, and described in
    	an article in SysAdmin magazine.  It was called BBWARN and it was
    	good.  It allowed an admin to specify rules which enable him/her
    	to get notified based on the source of the error (host) and the
    	service involved and also based on the day/time.  Different rules
    	could be set up and different recipients could be specified
    	depending on the characteristics of the problem
    	(host/service/day/time).

    	Numerous changes have been made and various configuration
    	options are available to the BB admin(s).

    	Notification methods

    	You can currently get notified by email, numeric pager and
    	SMS communication device.  Users have reported custom
    	notification by alpha-numeric pagers and other PCS devices
    	(some are available on the FTP archive or a search of the
    	mailing list archives will lead you to them).

    	To use the email feature, your BBPAGER host (in bb-hosts)
    	must have an email program.  This is not a problem on UNIX:
    	the mail (mailx) program is available.

    	Sending a notification message to a numeric pager requires
    	kermit and a modem.  The path of kermit should be defined in
    	the KERMIT variable of bbsys.sh/bbsys.local.

    4.2	Setting up notification

    	All of the BB notification configuration is specified in the
    	etc/bbwarnsetup.cfg file.  Here are the various options that
    	can be set. Note that instructions are included in the
    	configuration file as comments.

    	All tokens in the etc/bbwarnsetup.cfg file must be defined
    	on the BBPAGER host.  ***** The 'pagelevels:' token must be
    	defined on all BB hosts (servers & clients) because the 'bb'
    	executable sends the notification and that is done locally
    	on each client host. *****

    	bbwarn: Set to "TRUE" is you want to enable the notification
    		feature

    	svcerrlist: This contains a list of service/code pairs.  The service
    			name is the column found in the HTML display and
    			the code is the value displayed on a numeric message.
    			If you add a custom check then the column name
    			must have a corresponding code.

    	ignforall: Is a regular expression that is used to temporarely
    			disable notification a host.service combinations
    		e.g.  .*.cpu|.*.msgs|host11.*
    		Don't notify for any cpu/msgs errors and any errors
    			concerning host11.

    		N.B. This feature can be duplicated but the ! rule (below).

    	pagehelpcode: Numeric code to use when a user sends a manual
    		notification.  See section 4.5.

    	ttyline: List of modems devices.  You can specify more than one.

    	prefix: Prefix to use when dialing out
    	suffix: Suffix to use when dialing out (like a hangup)

    	pagelevels: At which color level to send a notification.
    		This token MUST be set on every host (clients & servers).
    		You must restart BB if you modify it.

    	pagelevelsmail: if a color defined here is also defined in pagelevels
    		then only e-mail recipient will receive notifications
    		when an error occurs at the pagelevelsmail defined color.
    		Note that only recipient in the form rcpt@some.mail will
    		be notified, if a recipient is prefixed with ep-, ext-XXX-
    		or any other valid prefix, it will be ignored as they are
    		not considered e-mail recipients.

    	pagerecovered:  Set to TRUE if you want to be notified
    		when a problem has been fixed.  This feature is only available
    		when 'pagetype:' is set to EVENT.

    	pagetype: Defines how the pager delay is handled.  There are
    		4 choices (RCPT|EVENT|HOST|GROUP).
    		RCPT: the recipient is not notified until pager delay
    			expires.
    		EVENT: the recipient is not notified for a particular
    			host.service combination until the pager delay
    			expires.
    		HOST: the recipient is not notified for a particular
                            host until the pager delay expires.
    		GROUP:  the recipient is not notified for a particular
                            host.service within the same etc/bb-hosts group
    			combination until the pager delay expires.

    	pagemaster: Recpient(s) to receive an e-mail notification if
    			a page notification could not be sent.

    	pageaddhtmlpath:  Set to TRUE if you want the HTML path of
    		the status log to be appended to an e-mail
    		notification.  If this is set to TRUE, make sure the
    		BBWEBHTMLLOGS variable is set correctly in etc/bbdef.sh.

    	cfgdelim: entry delimiter in the etc/bbwarnrules.cfga

    	briefrcpt: define the recipients that should receive a brief
    		notifications message.  You can you use regular expressions:
    		ie. ep-*  (all e-page recipients)
    		This expression will be expanded to this regular expresssion: ep-.*
    		The message will have this format:
    			hostname.service - XXXYYYYYYYYYYYY
    			<link to host.svc page in bb>

    		Note: XXXYYYYYYYYYYY - Service code and IP address

    	hg-xxxxxxx: You can have multiple hg-xxxxxxx tokens that
    		define a group of hosts/devices.  So you can
    		e.g.  create hostgroup 'hg-routers' as an alias or
    			shorthand of a list your routers.
    		These tokens can then be specified in the bbwarnrules.cfg
    		file in the host-fields columns (1st/2nd column) instead
    		of having to type all of them individually in all rules lines.

    	pg-yyyyyyy: You can have multiple pg-yyyyyyy tokens that define a
    		group of recipients. These tokens are expanded in the rules
    		lines inthe recipients definition.

    	On the client make sure that PAGELEVELS is set (etc/bbdef.sh) and
    	on the server, if appropriate, that PAGELEVELSMAIL(etc/bbdef.sh) is set.
    	Refer to section 3.2 for more details.

    4.3	Creating notification rules

    	You define the notification rules in the etc/bbwarnrules.cfg file.

    	Rules are written in the following format:

    	hosts;exhosts;services;exservices;day;time;recipients

    	hosts: match on these hosts (* is a wildcard for all hosts)
    	exhosts: exclude these hosts
    	services: match on these services (* is wildcard for all hosts)
    	exservices: exclude these services
    	day: 0-6 (sunday-saturday)
    	time: 0000-2359
    	recipients: email address, numeric pager, sms number,
    		qpage recipients are defined with qp-<recipient>
    		sendpage recipients are defined with sp-<recipient>
    		Hylafax sendpage recipients are defined with hsp-<recipient>
    		   (note that Hylafax sendpage arguments will be added
    		    with the -p construct, regular sendpage don't)
    		e-pager recipients are defined with ep-<recipient>
    		SNMP traps are sent with trap-<destination IP address>
    		(note that SNMP trap support is preliminary and may be subject to change)

    	N.B. Please specify all recipients that do not allow long messages
    		in the briefrcpt token in etc/bbwarnsetup.cfg.

    	There's also a special format of the rule line:

    	!hosts;exhosts;services;exservices;day;time;recipients

    	If a rule line starts with ! the event thats matches
    	the rule line will disable notification to any
    	recipient defined on that rule line.
    	If the recipients field is '*' then no notification
    	will occur for that event.  Here's an example

    	!*;;*;;*;*;*

    	This will in effect disable all notifications and render
    	useless any other rule that you have defined :)

    	!*;;*;;*;*;robert@localhost 9999999

    	This will remove robert@localhost and 9999999 from the
    	list of recipients for the current event if they were
    	defined on another rule that matched the same event.

    	The value set by 'cfgdelim' (in bbwarnsetup.cfg) is your
    	field delimiter. In this case the value is ';'
    	Even though egrep regular expressions are allowed, do not use
    	the .* construct, just use '*'.  It will be replaced with
    	.* in the regexp.  It's just that * is more readable than .*

    	Also the default 'pagedelay' value (see bbwarnsetup.cfg)
    	which indicates how long before the next notification occurs
    	can be overridden on the rule line for a specific recipient
    	by appending the time value to the recipient:

    		recipient:XX   where XX is the value in minutes.

    	For examples and complete information please refer to the
    	etc/bbwarnrules.cfg and the etc/bbwarnsetup.cfg files.

    	Escalation

    	To escalate a notification, you use the following format
    	for the recipient:

    		recipient:^XX[-YY]

    		XX is the initial wait before sending the notification
    		YY is the delay for each subsequent call.  If it is
    		not specified then the 'pagedelay' value is used.

    		An escalation can only be acknowledged by the
    		recipient.

    	Initial delay

    	An initial delay can be specified when configuring a recipient.
    	The format is:

    		recipient:~XX[-YY]

    		XX is the initial delay before sending the notification
    		YY is the delay for each subsequent call.  If it is
    		not specified then the 'pagedelay' value is used.

    		An initial delay can be reset if a recipient
    		acknowledges a notification for all recipients of
    		that notification (see below).

    	There's a simple checking program that validates the contents
    	of your bbwnarnrules.cfg file.  It is located in $BBHOME/etc.
    	cd $BBHOME/etc
    	./bbchkwarnrules.cfg

    	Note that it only checks if your config file is not missing
    	a column.

    4.4	Acknowledging a notification

    	When an admin is notified, the admin is always sent an
    	acknowlegement tag with the message.  This tag number
    	is seven digits (XXXXXYY): the first five digits are a unique
    	number and the last two digits are the recipient's ID for that
    	notification (the recipient's ID can change, it is valid
    	only for the current notification). Note that you can
    	acknowledge using XXXXX99 to signify to acknowledge
    	for all recipients of the XXXXX ack ID message except
    	for escalated recipients(^rcpt) which can only ack themselves.

    	In an e-mail notification, the ack number is in the
    	subject and the body of the message: in the subject
    	it is found at the beginning as in "!BB - XXXXXXX!",
    	and in the body of the message it is at the beginning of
    	the body delimited by [XXXXXXX].

    	In a numeric page, the acknowledgement is always after
    	the BB numeric code.  It is the last 7 digits of the
    	numeric message.  NOTE, if your pager doesn't support
    	more than 21 digits, then you're out of luck, you'll
    	be missing some digits. You could always try to
    	shorten the ack number but that is left as an exercise
    	for the reader ;-).  If you get the first 5 digits
    	then you can always use recipient 99 to acknowledge
    	for all recipients but that may not fly at your site
    	especially if you have multiple recipients.

    4.5	Manual notification (Web based "PAGE/ACK" button)

    	You can enable the manual notification feature and
    	let the user reach you by pager/e-mail if they can't get
    	a hold of you.  If this is enabled, then you'll
    	receive a numeric message starting with 911 (pagecode
    	in etc/bbwarnsetup.cfg) followed by the phone
    	number to call.  Copy the web/bb-ack.sh script (if
    	it was not copied during the installation)
    	to your cgi-bin directory and make sure all permissions
    	are correct.

    	You can setup who gets called when, by adding rules
    	to the etc/bbwarnrules.cfg file.  The pseudo host to use
    	is notify-admin*.  There's an example in the config
    	file.

    4.6	SMS notification

    	This a user custom notification scheme for SMS devices.
    	What follows is the README file for his custom hack.

    	Last updated: 1997-01-11 by jaclu@ibk.se

    	This section describes the usage of a kermit script that
    	sends SMS messages.

    	It is used as a mail -> SMS gateway, and also for the "Big Brother"
    	network monitor(obviously).

    	The kermit script for SMS can be found in etc/sms.scr

    	My operator uses NOKIA server software, hopefully you can use
    	this also on Motorola servers, without too much hassle.

    	Terms
    		SMS     Short Message Service
    		SMSC    Short Message Service Center
    		CIMD    Computer Interface to Message Distribution

    	Syntax

    		kermit sms.scr device file number [number [number [...]]]
    			device is modem device like /dev/cua0

    	The file should contain the message, only first line of file
    	will be sent, and it MUST be terminated with LF, otherwise the
    	kermit script won't be able to extract the message.

    	SMS messages can only be 160 chars, and since nobody would want
    	a lengthly message on a 8 character display, I haven't bothered
    	to handle message splitting. If message is too long, it is truncated.

    	If one or more SMS phonenumbers can be specified, the message will be
    	sent to each one of them.

    	Configuration

    		I have done this the easy way, that is I start CIMD with
    	faked checksums, if anybody manages to generate correct checksums,
    	please mail me.

    	Log in to SMSC

    		You have to set two variables in sms.scr:

    		\%b 	phonenumber to you operators SMSC
    		\%c	login to access CIMD

    		For Europolitan (Swedish operator) you can use:

    		\%b 46708222902
    		\%c cimd3  # (use faked checksums)

    		This starts the CIMD server, and from now we must ACK
    		all responses.

    		Don't forget to include your modem prefix to \%b!!

    		I don't use the "prefix" token from BB, since I also use
    		this script as a mail->SMS gateway. If you are going to
    		use this, you must anyhow configure this script, so I hope
    		you don't mind.

    	Start a CIMD session

    		The first step in a CIMD session is to identify yourself

    		You have to set two variables in sms.scr:

    		\%d	CIMD account
    		\%e	CIMD password

    		Unfourtunately, I cant give you theese values ;)

    		Now you have configured your script and can start to use it!

    		Send to all recipients

    		BB calls the script one time for each recipient, that's
    		a bit of a waste, but it would just be to much work to
    		get around.  If you call it manually or from sendmail,
    		you can use multiple recipients.

    	More info

    		The (english) spec of NOKIAS CIMD implementation can
    		be found on:

    		http://www.europolitan.se/europolitan/fick/tjanst/cimdspec.htm

    		If you happen to know swedish, you can read more on Europolitans
    		implementation  on:

    		http://www.europolitan.se/europolitan/fick/tjanst/faqcimd.htm

    	Feedback

    		Please let me know if you use this with a non NOKIA server,
    		or if you fix real checksums!

    	Jacob Lundqvist <jaclu@ibk.se> (SMS:+46-708-555 456)

    	Thanks Jacob !!!

    4.7	Using kermit

    		By default, BB expects kermit 5 to be installed on your
    	BBPAGER host.  On certain installations it is V6 and up that is
    	installed.  If that's your case then you must (might) modify which
    	kermit script is used:

    		copy numeric-k6.scr to numeric.scr in the script.
    		(don't forget to make a backup of numeric.scr)

    		But try with numeric.scr at first and if it doesn't work
    	then replace it with numeric-k6.scr. Don't forget to set KERMIT in
    	bbsys.sh.

    4.8	Adding/modifying for a custom notification procedure

    		You can create your own notification by editing your own
    	notification script.  For examples see: ext/pg/ex1
    	or ext/pg/ex2 .  Your script must be saved in the $BBHOME/ext/pg
    	directory.

    		You would use a special prefix in the etc/bbwarnrules.cfg
    	file.  For example, you could use the recipient ext-ex1-robert@bb4.com
    	to send a notification to the recipient called "robert@bb4.com"
    	using the "ex1" script.

    		You create your own look for the notification message
    	by using the following environment variables in your script
    	in order to customize it.

    	BBALPHAMSG - is the text body
    	ACKCODE - is the security associated with the notification
    	RCPT - is the recipient name (stripped of the ext-name- prefix)
    	BBHOSTNAME - is the hostname in  host.name.com format
    	BBHOSTSVC - is the host.name.com.service tag
    	BBHOSTSVCCOMMAS - is the host,name,com.service tag
    		(notice that the '.' were replaced by ',')
    	BBNUMERIC - is a numeric value made up of 22+ digits
    		XXXYYYYYYYYYYYYZZZZZZZ
    		XXX - corresponding code of service defined in bbwarnsetup.cfg
    		YYYYYYYYYYYY - IP address
    		ZZZZZZZ - security code - first 5 digits: event ID code
    					  last  2 digits: user ID code
    	MACHIP - is the IP address of host in normalized 12 digits AAABBBCCCDDD
    	BBSVCNAME - Name of the service of the event
    	BBSVCNUM - numeric equivalent of the service name
    		(from svcerrlist in bbwarnsetup.cfg)
    	BBCOLORLEVEL - Color level causing the notification
    	RECOVERED - 1 in recovery mode, anything else non-recovery mode
    	DOWNSECSMSG - Default recovery message
    	DOWNSECS - # of seconds to recovery

    		After you've created your own script, make sure the
    	permissions allow only the authorized users to modify it.

    4.9	Notification/acknowledgement logs

    		You'll find logs of who got notified for what reasons,
    	logs about recoveries, who got a recovery message and logs about
    	who acknowledged what notification.   They are all located in the
    	$BBACKS directory:

    		acklog
    		notifications.log
    		recoveries.log
    		recoverymsgs.log

    4.10	Disabling notifications temporarely

    	This feature is not automatically available.  It must be
    	explicitely compiled in. You must add the -DENABLE_DISABLE
    	directive to the CFLAGS variable of the Makefile:

    	CFLAGS=-DENABLE_DISABLE -O

    	Please be warned that if you enable this feature, a hacker
    	could disable notifications while cracking into your systems.

    	You can temporarely disable notifications without having to
    	modify the etc/bbwarnrules.cfg file.

    	All you have to do is send a 'disable' message to the
    	BBDISPLAY(s) using the following format:

    	MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO
    	USE 'bb' directly

    	./bb $BBDISP "disable 'host regular expression' 'duration' [reason]"

    	You can match multiple hosts/services by specifying a regular
    	expression instead of a real host name.  The duration is
    	in minutes. You can also add an optional reason that will be
    	displayed in the status.

    	AGAIN, MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO
    	USE 'bb' directly

    	e.g.

    	cd $BBHOME/bin
    	./bb $BBDISP "disable www.bb4.com.disk 240"

    		Disable notifications for the disk event of www.bb4.com
    		for 240 minutes.

    	cd $BBHOME/bin
    	./bb $BBDISP "disable www.bb4.com* 240"

    		Disable notifications for all events for host
    			www.bb4.com for 240 minutes

    	cd $BBHOME/bin
    	./bb $BBDISP "disable www.bb4.com* 240 Taking www.bb4.com offline for a new disk"

    		Disable notifications for all events for host
    			www.bb4.com for 240 minutes and specify the reason

    	NOTE: duration can be expressed in seconds/minutes/hours/days

    		30s: 30 seconds
    		15m: 15 minutes
    		1h:  1 hour
    		1d:  1 day
    		by default, minutes are used if a qualifier is not specified

    	To reenable a disabled host(s), send the "enable" message

    	./bb $BBDISP "enable 'host regular expression'"

    	As an example:

    	cd $BBHOME/bin
    	./bb $BBDISP "enable www.bb4.com*"

    		Enable notifications on all events for host www.bb4.com

    	Note that when you use the enable message, the colored dot
    	will stay blue until a new status is received by the BBDISPLAY(s)

    4.11	Acknowledging a notification by e-mail

    	You can acknowledge a notification by e-mail by setting
    	up the bin/bb-mailack.sh in the .forward file of the used id
    	that sent the notification message.

    	On the BBPAGER host, set up a .forward file in the user
    	account that's running the BB instance.  The contents of
    	the .forward file should be:

    	|/bin/bb-mailack.sh

    	Replace  by the real BBHOME path,
    	i.e.
    	|/home/bb/bb/bin/bb-mailack.sh

    	Then in the bb-mailack.sh script, set up these variables
    	BBUSER - user id of the BB instance
    	BBHOME - self describing ;)
    	MAILFILE - Location of the user id mailbox in case the
    		   message is not an acknowledgement
    	REPLYPFX - Prefix used to identify a reply. If you intend
    		   to use multiple mail programs to ack and if they
    		   don't provide the same prefix, I suggest you
    		   use your own prefix and enter it on the subject
    		   line instead of the one given by the mail program.
    	DELAY - Default duration of the delay

    	When you receive an e-mail notification, you can simply reply
    	to the mail message to ack the notification and to delay the
    	next notification for DELAY minutes if the problem persists.

    	As long as the subject of the reply mail message is in the
    	following format(It assumes the REPLYPFX is "Re:"):

    	Re: !BB - XXXXXXX!

    	You can specify the DELAY in the reply message also

    	Re: !BB - XXXXXXX! DELAY=120

    	This will set the delay to 120 mins

    	You can also specify a message that is saved in bbvar/acks/acklog

    	Re: !BB - XXXXXXX! MSG=This is a message

    	Anything after the MSG= is considered a message

    	Finally, you can also tell the bb-mailack.sh to return a
    	confirmation of the replied message (but it doesn't guarantee that
    	the actual acknowledgement worked).

    	Re: !BB - XXXXXXX! ACK=Y

    	You can use all 3 directives on a subject line

    	Re: !BB - XXXXXXX! ACK=Y DELAY=120 MSG=This is a message

    	If the mail message does not start with "Re: !BB - XXXXXXX", then
    	the message is saved at the location pointed by MAILFILE.

    4.12	Displaying acknowledgements

    	When an acknowledgement is sent to BB, it displays that
    	fact by changing the colored dot of the acknowledged event
    	to a checkmark of the same color.  It also adds a status
    	line at the bottom of the individual status page.

    	You can also have the last 25 acknowledgements listed in the
    	bb2.html page.  This feature is enabled by default. To disable
    	it, you must remove the acklog.sh script name in the BBMKBB2EXT variable
    	in etc/bbdef.sh.  You need to restart BB to have the change take
    	effect.

    ------------------------------------------------------------------------

    Section 5:  Customizing your Big Brother display

    5.1      Customizing the HTMLized status logs

    	If you don't like the blinking gifs, you can change them to
    	non-blinking by copying over the files found in www/gifs.
    	The non-blinking gifs start with the nb- prefix.

    	You can change Sean's mean face in the bb.html/bb2.html and
    	HTMLized status logs by replacing the bb.gif file.  This only
    	applies to pre-1.3 versions.

    	You can change the header/footer for the HTMLized status pages.
    	The files are located in the /web directory.  Note that
    	your customized version can contain special tags which can be
    	replaced by values:

    		&BBDATE: current date/time
    		&BBBACKGROUND: with the current background color
    					(red/green/yellow/clear/purple)
    		&BBRELDATE: Release date of the current version
    		&BBREL: Current version number
    		&BBCOLOR: Color of this status file
    		&BBHOST: Name of host of current status file
    		&BBSVC: Service name of current status file.
    		&BBIPNAME: IP address of host, if 0.0.0.0 then use hostname
    		&BBIP: IP address of host, always return IP address unless not found

    		You can add tags by modifying the bbd.c file. Every
    	time a status log is received by bbd, an HTML version of the
    	status log is created by prefixing the status with the header
    	file and suffixing with the footer file.

    		You can put tags also in the status logs:

    		&red	&green	&yellow	&clear	&purple

    		These tags are replaced with a corresponding image
    	source tag for the HTMLized version.  Just like the bb-local.sh
    	script does for the disk and procs tests.  So, if you create
    	your own custom check, enter these tags if you want to see a
    	colored dot in the HTMLized status.

    	NOTE:  If you do not wish to have HTMLized status files, then
    		rename the www/html directory and the main BB display
    		pages will use the plain text status logs which are
    		located in the $BBLOGS directory.  This will only
    		work if you had the install program create the
    		old-style directory structure.

    5.2     Customizing headers / footers (for reports too)

    	bb.html/bb2.html and various HTML page generation
    	can be customized by modifying their respectable
    	headers/footers. These are located in the $BBHOME/web
    	directory.

    	The bb.html uses bb_header/bb_footer and bb2.html uses
    	bb2_header/bb2_footer.  You can also customize the output
    	of the HTML file defined by the page directive in bb-hosts
    	by creating <page name>_header/<page name>_footer
    	files and you can also customize the HTML file defined by
    	the subpage directive in bb-hosts with the files
    	<page name>_<subpage name>_header and
    	<page name>_<subpage name>_footer. So an example is
    	division1_department3_header and division1_department3_footer.

    	The reporting feature (bb_rep.html) also uses the same format.
    	Except that bb.html uses bbrep_header/bbrep_footer. (bb2.html
    	has no equivalent in reporting).  As for the page directive,
    	use <page name>rep_header/<page name>rep_footer.
    	And for the subpage directive, use <page name>_<subpage name>rep_header
    	and <page name>_<subpage name>rep_footer.

    	Do not use the < and > characters in the filename,
    	they were used for readability.

    	NOTE:  The link to bb4.com MUST be kept in the
    		footer page as per your agreement of the
    		license agreement.

    5.3     Creating skins

    	You can create your own look and feel by customizing
    	various HTML aspects of Big Brother.

    	You can use your own gifs by creating a new directory
    	under $BBHOME/www.  As an example, a directory 'psy'
    	has been created with more psychedelic colored dots
    	It lives in $BBHOME/www/psy.  To use those images then
    	set the BBSKIN variable in etc/bbinc.sh to "psy".

    	To further modify the appearance of your display,
    	modify the header/footer files found in $BBHOME/web.

    	You also have some control over the bb.html/bb2.html
    	and the status logs HTML pages by modifying the
    	following variables in etc/bbinc.sh:

    	MKBBLOCAL
    	MKBBREMOTE
    	MKBBTITLE
    	MKBBROWFONT
    	MKBBCOLFONT
    	MKBBMANAGE
    	DAYS
    	HOURS
    	MINS
    	STATUNCHNMSG
    	RECVFROMMSG
    	DOTHEIGHT
    	DOTWIDTH
    	BBSKIN

    5.4     Inserting notes about your hosts

    	You can setup HTML links on the host name in the
    	bb.html/bb2.html pages to point to an information
    	page for that particular host.

    	If a file exists in the www/notes directory that
    	matches the system name as displayed on the bb pages,
    	then they are linked into both the bb.html page and the
    	bb2.html summary page.

    	These files can end in .htm, .html, .shtml, .php3
    	or nothing :)

    	You can create these files manually or you can use the
    	"notes" messages type:

    	"notes <name of file> <data to write to the file>"

    	Here's an example:

    	cd $BBHOME/bin
    	./bb $BBDSIP "notes www.bb4.com.html <HTML><BODY>It's a hit!</BODY></HTML>"

    	This feature is not automatically available.  It must be
    	explicitly compiled in. You must add the -DNOTESMSG
    	directive to the CFLAGS variable of the makefile:

    	CFLAGS=-DNOTESMSG -O

    	Please be warned that if you enable this feature,  a hacker
    	could fill the disk with dummy notes file.  Also note that
    	the file size is limited to the incoming buffer of BB
    	and by the size of the environment buffer.

    	Also note that if you source the BB environment prior
    	to sending the "notes" message and you have multiple BBDISPLAYs
    	then the "notes" message will be sent to all BBDISPLAYs.  If you
    	want to send it to only one BBDISPLAY, then use that IP address
    	as the command line argument and set BBDISPLAYS to "" just
    	before sending your message.

    5.5     Extending the contents of the main HTML pages

    	You can tack on some extra HTML code at the bottom of the
    	bb.html /bb2.html pages before the insertion of the footer.

    	You save the scripts that generate the HTML code in the
    	$BBHOME/ext/mkbb directory.  And in etc/bbdef.sh, you specify
    	the name of those scripts in the BBMKBBEXT variable for output on
    	the bb.html page and in the BBMKBB2EXT variable for output in
    	bb2.html.

    	There's an example of a script in $BBHOME/ext/mkbb called
    	eventlog.sh that displays the last global events in a table
    	of the bb2.html page.

    	Make sure that the scripts are only writable by the owner
    	to prevent someone to execute malicious code.

    5.6     Extending the contents of the history HTML pages

    	You can tack on some extra HTML code at the bottom of a
    	history display page before the insertion of the footer.

    	You save the scripts that generate the HTML code in the
    	$BBHOME/ext/hist directory.  And in etc/bbdef.sh, you specify
    	the name of those scripts in the BBHISTEXT variable.

    	Make sure that the scripts are only writable by the owner
    	to prevent someone to execute malicious code.

    5.7     Working with historical logs

    	There are many historical logs available with BB.  These logs
    	contain all events that occured based on the host.service
    	combination, on the host itself or for all hosts.  These logs
    	are found in the $BBHIST directory.  You can use these logs to
    	view that last events that occured.  By default, if you view
    	a status log, you can click the history button to view the
    	last 50 events for that host.service combination.  You can
    	also use the ext/mkbb/eventlog.sh to view the last 20
    	events that happened for all hosts (you can view more by changing
    	the NUMEVENTS variable in eventlog.sh to the size you want).

    	You can disable the logging of events by renaming the
    	$BBHIST directory.  You can set BBHOSTHISTLOG=FALSE
    	or BBALLHISTLOG=FALSE to prevent the logging of host or all
    	hosts events.  You set those values in etc/bbdef.sh

    	There's a lot of interesting information available in those
    	logs.  Check the FTP site for scripts that take advantage
    	of that information.

    	There's also a directory that keeps a copy of the status logs
    	when they change state such that they can be viewed later on.
    	The status logs are saved in the $BBHISTLOGS directory.  A directory
    	is created with the name of the hosts and then each service has its
    	own directory where the status logs are saved based on the time of
    	the occurance.

    5.8     Hiding the bb.html/bb2.html page header

    	You can position the bb.html/bb2.html pages at the beginning
    	of the first block by accessing the page as follows:

    	http://.../bb.html#begindata

    	You can bookmark this position to hide the header.

    5.9     Pre-defined anchors in bb.html

    	http://.../bb.html#hosts-blk
    		Go to the first host (same anchor as begindata)
    	http://.../bb.html#pages-blk
    		Go to the subpages table
    	http://.../bb.html#summaries-blk
    		Go to the summaries table
    	http://.../bb.html#dialups-blk
    		Go to the dialups table

    	http://.../bb.html#hostname
    		Go to the host line by using the hostname
    		as the anchor: i.e.
    		http://www.bb4.com/bb/bb.html#www.bb4.com

    ------------------------------------------------------------------------

    Section 6:  Miscellaneous Big Brother configuration options

    6.1     Specifying filesystem specific values in the disk test

    		You can define warning and panic levels for specific
    	filesystems.  They are defined in the etc/bb-dftab file. This file
    	contains definitions for all hosts all in a single file.  You just need
    	to redistribute across your hosts. The format is as follows:

    		[hostname]:/filesystem:warning level:panic level

    		example:

    		/:85:95
    		www.maclawran.ca:/oradata:98:99

    		/ uses 85% for warning level and 95% for panic, this is
    			valid for all hosts (as no host name was define
    			in the definition)
    		Only on www.maclawran.ca /oradata uses 98% for warning
    			and 99% for panic
    		All other filesystems use the default values defined
    		in the etc/bbdef.sh file.

    	Use the etc/bb-dftab.DIST file as a starting point.

    6.2     Enabling security

    		When BB is installed, it accepts connections from
    	any hosts.  You can specify from which hosts only to accept
    	status logs using the etc/security file.

    		You can define individual hosts or networks in the
    	etc/security file.  Using this format:

    		192.168.1.0
    		192.168.2.125
    		10.0.0.0

    		Accept from network 192.168.1.0, for 192.168.2.125 host
    	and the 10.0.0.0 network.

    6.3     Using bbnet to test TCP services

    	You can test any text based TCP services using bbnet.
    	To do so add the test in bb-network.sh.  If you need
    	just a basic test, then just add the service to
    	the list of other services.  If you need to send a
    	special command to the service daemon, then add a case
    	statement entry like the imap test.  Set change the
    	textmsg variable to the text to be sent to the daemom.

    	You can also use special options for bbnet. The -q/-Q/-s
    	options are available.  The -q option does not return
    	the elasped time of test if bbnet was compiled with the
    	-DGETTIMEOFDAY switch (which is only valid if
    	the gettimeofday() function is available).  The -Q option
    	prevents the display of error messages. Finally the -s
    	option only connects to the port to verify if the port
    	can be opened.

    	See the section on the etc/bb-hosts configuration file
    	on how to tell the network test to use these options.

    6.4     Sending custom one line status with bb

    	The bb executable used to send statuses/notifications
    	to the BBDISPLAY/BBPAGER can also be used by a custom
    	check using standard input:

    	custom_check | $BB $BBDISP -

    	The custom_check program just needs to send a single
    	line status to stdout (use the exact format of status
    	files found in $BBLOGS).  This is very useful if the
    	custom check program is a daemon.

    6.5     Changing the 'purple' interval

    	By default, BB will send a 'purple' status to indicate
    	that a status has not been received in the last 30
    	minutes (default). You can change this value to a more
    	suitable value by modifying  PURPLEDELAY in etc/bbdef.sh

    6.6    Specifying processes to look for

    	The PROCS & PAGEPROC variables in etc/bbdef.sh can
    	be set with the names of the processes to monitor.

    	A more sensible way ... is to use the etc/bb-proctab file.
    	This file contains definitions for all hosts all in a single file.
    	You just need to redistribute this file across your hosts.
    	Host entries can be defined on multiple line entries for clarity
    	in case you have many processes to check.

    	The format is as follows:

    		[hostname1]:process list for yellow : process list for red
    		[hostname1]:process list for yellow : process list for red
    		[hostname2]:process list for yellow : process list for red
    		[hostname3]:process list for yellow : process list for red
    		[hostname3]:process list for yellow : process list for red

    		example:

    		localhost:smtp:httpd
    		www.maclawran.ca:!oracle "sleep 30":smtp

    		!process - check that host does not exists
    		"sleep 30" - use '"' when you want to check on a
    				process that spawns multiple words

    		You can also add logic to define min/max instances
    		of a running process:
    			httpd;>15  - more than 15 httpd must be running
    			httpd;<=10 - less or equal to 10 hhtpd must be running
    			httpd;5 - exactly 5 instances must be running

    		if you don't specify the # of instances then it checks
    		that at least one instance is running.  For processes
    		defined with "" as in "sleep 30", you must add the logic
    		within the "": i.e. "sleep 30;>6"

    	Use the etc/bb-proctab.DIST file as a starting point.
    	If bb-proctab exists, then it overrides the PROCS/PAGEPROC
    		values in bbdef.sh

    6.7    Sending data to the BBDISPLAY to be handled by an external script

    	You can send data to your BBDISPLAY host to be
    	processed by an external script, a cron job or
    	a command line program/script.

    	The data is appended to the specified file name into the
    	$BBDATA directory.

    	To use this feature you have to send a "data" message
    	to the BBDISPLAY in this format:

    	"data <file name> <file data>"

    	Here's an example:

    	cd $BBHOME
    	./bb $BBDISP "data www.bb4.com.hits 952281437 10m 5000"

    	The "952281437 10m 5000" would be appended to the
    	$BBDATA/www.bb4.com.hits file.

    	This feature is not automatically available.  It must be
    	explicitly compiled in. You must add the -DDATAMSG
    	directive to the CFLAGS variable of the makefile:

    	CFLAGS=-DDATAMSG -O

    	Please be warned that if you enable this feature,  a hacker
    	could fill the disk with dummy data files.

    	Also note that if you source the BB environment prior
    	to sending the "data" message and you have multiple BBDISPLAYs
    	then the "data" message will be sent to all BBDISPLAYs.  If you
    	want to send it to only one BBDISPLAY, then use that IP address
    	as the command line argument and set BBDISPLAYS to "" just
    	before sending your message.

    6.8   	Disabling the messages file(s) zero-length test

    	By default Big Brother verifies that none of the messages file
    	specified in the MSGFILE variable is of zero-length.  This is
    	a simple test to make sure that a hacker hasn't linked your
    	messages file(s) to /dev/null.

    	Unfortunately, on some systems, when the logs are rotated
    	they are left empty by the procedure that rotates the logs.

    	If this is your case then you have 2 choices:

    	Immediately have the log rotation, add this command:
    	echo "`date`" >> <message file(s)>
    	or use a syslogd feature to add a line in the messages file(s)
    	or simply disable the zero-length check by setting the
    	CHKMSGLEN variable to FALSE in bbdef.sh.

    	Note, IMHO, the zero-length should always be there.  Simply
    	echo the data to the messages file(s) or use syslogd features
    	to add an entry in the message file(s).  Security is too
    	important to ignore.  Your best bet is to install a log
    	scanner to analyze your messages file(s).  Which one ?
    	It is our of BB's scope to tell which one(s) but ask
    	around for opinions of people who use log scanners.

    6.9	Setting a Time To Live to a status message

    	When you create external scripts, you can now tell how
    	long the status is valid for.  Just specify the length
    	in minutes of the status.  This is to prevent purple status
    	on certain service that are not run in the regular interval
    	of 5 minutes.

    	To use this feature just follow this example

    	$BB $BBDISP "status+1560 www,bb4,com.backup green `date` Backup Successful"

    	A purple indicator will be triggered only if no status is received
    	within 1560 minutes (26hours).

    6.10	Disabling the connectivity test

    	In some cases you may want to disable the connectivity test
    	altogether because you don't need it or because you want to
    	use the fping.sh external script available at ftp://ftp.deadcat.net .

    	To disable it, just set the CONNTEST variable in etc/bbdef.sh,
    	on the BBNET host, to FALSE.  Also take in consideration that by
    	disabling this test you will lose the ability to generate clear
    	conditions for the other network tests because you will not have
    	access to then connectivity status for that host.

    6.11	Reducing connectivity errors

    	If you are prone to a lot of red status for your connectivty
    	tests even though the host responds when you ping by hand.
    	To possibly reduce these false alarms, you may want to add/change
    	the PINGPAR1 or PINGPAR2 parameters in etc/bbsys.local on
    	the BBNET host.  Adjust according to the BBNET's ping
    	parameters. PINGPAR1 are options that goes before the hostname/
    	IP address in the ping command while PINGPAR2 are options that
    	goes after the hostname/IP address.

    	Adjust the number of packets sent for the ping test. Start with
    	sending 2 packets and analyze the results. If it did not make
    	a difference, try with 3 packets.  Note that when you specify
    	more than one packet to be sent, you inherently add a 1 second
    	sleep on each new packet. This will delay, in seconds, your
    	whole network test by (n-1) * hosts.  If you test a large number
    	of hosts, you will see a noticable difference in how often your
    	hosts gets tested.

    6.12	Setting up BBPAGERs and BBNETs host in failover mode

    	You can declare failover BBPAGER and BBNET hosts
    	and have them be in standby mode by checking for a failure
    	on the master BBPAGER or BBNET host and when that happens
    	they start behaving like the master BBPAGER or BBNET host
    	until the master host is back online.

    	To setup failover BBPAGER/BBNET hosts, define the primary
    	BBPAGER/BBNET in the bb-hosts file with the BBPAGER or BBNET
    	directive and enable the failover external script on the
    	failover hosts by putting the failover script name into the
    	etc/bb-bbexttab file entry for each host that is designated
    	as a failover host:

    	failover1.host.com : : failover
    	failover2.host.com : : failover

    	Do not put failover in etc/bb-bbexttab for the primary
    	BBPAGER or BBNET host !  For complete details please
    	refer to the failover script in $BBHOME/ext.

    	Each failover host will generate a "fo" status log.  When a host
    	will be in standby mode, it will generate a clear status and
    	when failover mode is enabled it will send a yellow status.

    6.13	Speeding up the network tests

    	If you have a faily large number of servers to monitor there's
    	a very good possibility that the network tests takes a while
    	to complete.

    	You can speed up the tests by parallelizing the network tests
    	from a single bb-network.sh script to running multiple
    	copies of bb-network.sh at once.

    	All you have to do is to increase the value of BBNETTHREADS in
    	etc/bbdef.sh.  Indicate how many bb-network.sh you'd like to
    	have working concurrently.

    	You can also speed up the network tests by using the fping
    	program to do the connectivity test.  You'll have to download
    	fping and also get the fping.sh external script at the FTP
    	archive site: http://www.deadcat.net.  You can disable the
    	regular ping test by setting CONNTEST to FALSE in etc/bbdef.sh.

    6.14	Enabling WML ouput (WAP browsing)

    	By default, BB does not generate WML files.  YOu can enable
    	the generation of the WML files by setting WML_OUTPUT to
    	TRUE in etc/bbdef.sh.  The files will be saved in the www/wml
    	directory.

    	You'll be able to view the output on a WAP browser by viewing
    	the <BBWEB>/wml/bb.wml URL.

    	Make sure your web server can render wml files.  In your
    	web server's mime.types file, make sure the following line
    	is defined:

    	text/vnd.wap.wml             wml

    	Add it if need be and stop/start your web server. Also note
    	that some WAP device accepts can't accept large page size
    	and you may have to reduce the value of WMLMAXCHARS to
    	control the size of the bb.wml/bb2.wml pages.  On the other
    	hand, if your WAP device accepts large pages, then you can
    	increase the value such that more info can appear on a
    	single page.

    6.15	Having BB tell you to reboot a server after is has been up XXX days

    	You can have BB display the "cpu" column as yellow dot when a
    	server has been up more than a pre-determined value.  You
    	set the BBTOOMANYDAYSUP value in etc/bbdef.sh to the number
    	of days of uptime after you wish to schedule a reboot.
    	You can also set this value in etc/bb-cputab.

    6.16	Using an alias name for the current host in etc/bb-hosts

    	Sometimes you may want to use an alias for the hostname
    	instead of the hostname returned by the 'uname -a' command.
    	To do so, on the BB client you wish to use an alias,
    	create an etc/bbaliasname file that you want with the alias
    	hostname as its contents and BB will use that hostname.
    	Then use that alias hostname in the all bb-hosts files (BBDISPLAY,
    	BBPAGER, BBNET).  Also, you must declare the "testip"
    	directive in the bb-hosts file of the BBNET host because
    	the alias is unlikely resolvable by DNS otherwise the
    	network tests will likely fail.

    6.17    Specifying CPU threshold values

    	BB uses the 5min load average value (from uptime) multiplied
    	by 100 (the dot is stripped from the value returned by uptime).

    	The CPUWARN & CPUPANIC variables in etc/bbdef.sh can
    	be set with the load average thresholds to monitor.  This was
    	explained in section 3.2.

    	A more sensible way ... is to use the etc/bb-cputab file.
    	This file contains definitions for all hosts all in a single file.
    	You just need to redistribute this file across your hosts.

    	The format is as follows:

    	hostname: misc settings : threshold value for yellow : threshold value for red
    	localhost: misc settings : threshold value for yellow : threshold value for red
    	: misc settings :threshold value for yellow : threshold value for red

    	examples:

    	localhost: : 150 : 300
    	www.maclawran.ca: : 250 : 350
    	www.bb4.com: daysup=999 : 75 : 150

    	Note that the order for matching is the following:
    	1) try to match for the hostname
    	2) use the localhost values is defined
    	3) use the ": XX : YY" line values if defined
    	4) use the CPUWARN/CPUPANIC values in etc/bbdef.sh if no match is found

    	Use the etc/bb-cputab.DIST file as a starting point.
    	If bb-cputab exists and if a match is found then the values
    		override the CPUWARN/CPUPANIC values in bbdef.sh

    	The "misc settings" field is reserved for other CPU test checks.
    	Currently on "daysup" is valid and the number associated with it
    	overrides the BBTOOMANYDAYSUP variable.

    6.18    Configuring messages file testing

    	The PAGEMSG, MSGS & IGNMSGS variables in etc/bbdef.sh can
    	be used to define keywords to search for in messages file(s).

    	Well, that's the old way.  You can now define entries in
    	the etc/bb-msgstab file to set those values.

    	This file contains definitions for all hosts all in a single file.
    	You just need to redistribute this file across your hosts.

    	The format is as follows:

    	hostname: filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore
    	localhost: filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore
    	: filename(s) : misc settings : yellow string(s) : red string(s) : matched strings to ignore

    	example:

    	www.bb4.com: /var/log/messages : : WARNING : NOTICE : not this message ; and this one either
    	www1.bb4.com: /var/log/maillog : : refused : ERROR ; error ; BAD SU : from www.bb4.com

    	To define more than one string in each field, the strings must
    	be seperated by a ';'.

    	The <misc settings> is currently not implemented.

    	You can define multiple entries for the host in the
    	etc/bb-msgstab file.  If you want different settings for different files
    	on the same host, use different lines.

    	If a matched entry for the yellow/red string(s) contains the
    	"matched strings to ignore" then the current match is ignored.

    	Use the etc/bb-msgstab.DIST file as a starting point.

    	If bb-msgstab exists and if a host matches an entry then the values
    	override the PAGEMSG/MSGS/IGNMSGS  values in bbdef.sh. If there's
    	no matches for the current host then the PAGEMSG/MSGS/IGNMSGS
    	values are used.

    	You can reset the colored dot from red/yellow to green by removing
    	the corresponding tmp/MSG.red.* and tmp/MSG.yellow.* files.  Those
    	files contain previous error messages and by default they are removed
    	when they expire.

    6.19    Using the combo message

    	The combo message enables BB to send more than one status message
    	at a time.  This is useful when you create an external script
    	that may have to send at lot of status messages (i.e. doing some
    	network tests against all hosts).  There are 3 steps in using
    	the combo message:

    	1) Initialize combo message processing

    		Before you can start using the combo message in your
    		script, you must initialize its processing, add these
    		lines at the beginning of your script:

    		BBCOMBOID="$$"
    		export BBCOMBOID
    		$BBHOME/bin/bb-combo.sh start

    		The BBCOMBOID variable acts as a unique tag for a particular
    		run of your script.  (Many scripts might be executing
    		at the same time).

    	2) Building a combo message

    		Instead of calling $BBHOME/bin/bb to send each status
    		message to a BBDISPLAY, you replace that call with
    		a bb-combo.sh call:

    		Basic method:

    		$BB $BBDISP "status $MACHINE.cpu $COLOR `date`  DATA..."

    		combo method:

    		$BBHOME/bin/bb-combo.sh add "status $MACHINE.cpu $COLOR `date` DATA..."

    		This will build a combo message. When the message reaches
    		the MAXLINE size then the message is sent to the BBDISPLAY(s)
    		and starts a new message.

    		Note that only "green" status are saved in a combo message.
    		All non-green status are sent immediately.

    	3) Ending combo message processing

    		A partial combo message might be required to be sent
    		to a BBDISPLAY(s) when your script is done.  At the end
    		of your script, add this line:

    		$BBHOME/bin/bb-combo.sh end

    		This will also remove temporary files that were used.

    	If you need examples, take a look at the bb-local.sh and bb-network.sh
    	scripts, they use the combo message.

    6.20    Tools to manage Big Brother data

    	bin/bbrm Removes data associated with a hostname or a service
    		that is no longer monitored.
    	bin/bbmv Renames data associated with a hostname or a service.
    	bin/bbprune Archives historical data up to and including the
    			month given as arguments.  This tool creates
    			archive files in the bbvar/archive directory.

    6.21    Sending a BB message in cron

    	If you want to use bin/bb send a single message to a
    	BBDISPLAY/BBPAGER in cron, remember that bin/bb requires the
    	BB environment to be loaded before it is executed. This loads
    	environment variables required by bin/bb to execute properly.

    	Use a crontab line similar to the following entry:

    	0 8-20 * * * BBHOME=/home/bb/bb;export BBHOME;. $BBHOME/etc/bbdef.sh;$BBHOME/bin/bb $BBDISP "status test.test red `date` This is a test"

    	BBHOME=/home/bb/bb;export BBHOME;. $BBHOME/etc/bbdef.sh
    		This sets the BB environement
    	$BBHOME/bin/bb $BBDISP "status test.test red `date` This is a test"
    		This sends the actual command. $BBDISP is the BBDISPLAY location.

    6.22    Defining characters that BB accepts in a status message

    	By default BB removes dubious characters from an incoming
    	status log.  The `,$,;,|,&,\ characters are removed.  If you
    	don't want that feature, add -DDONTCLEANSTRING in CFLAGS
    	of the Makefile.

    	If you use LARRD, you'll require to at least compile BB
    	using the -DNOTAMP entry in CFLAGS of the Makefile.  You can
    	also use -DDONTCLEANSTRING with LARRD.

    6.23    Integrating a LARRD graph with a status log

    	If you have installed LARDD along with your BB installation,
    	you can have a status log associated LARRD graph displayed when
    	you view the HTML generated page of a status log.

    	To enable that feature, add the -DLARRD directive to the CFLAGS
    	entry in src/Makefile and recompile (make clean;make;make install
    	make sure BB was stopped) After the BB restart, when you'll view
    	the status log, the LARDD graph will also appear if it's available.

    ------------------------------------------------------------------------

    Section 7: 	Other documentation sources

    	$BBHOME/README
    	$BBHOME/LICENSE
    	$BBHOME/README.INSTALL
    	$BBHOME/README.CHANGES
    	$BBHOME/install/README
    	$BBHOME/src/README

###### © Copyright Quest Software, Inc. 1997-2002 All rights reserved.