The NodeBrain program, nb, is an interpreter of a declarative rule-based language designed for use in construction of state and event monitoring applications. For the purpose of constructing event streams and performing rule administration, it supports authenticated and encrypted peer-to-peer communication over TCP/IP, including encrypted file transfers, store-and-forward message transmission, and remote command execution.

1.1 Operating Modes

NodeBrain is capable of processing commands from multiple sources and issuing commands to multiple destinations. Four basic operating modes enable the use of NodeBrain for various types of application components.

Command Input and Output	Operating Mode
	Non-Persistent			Persistent
	Interactive	Batch	Servant	Service
Source:
User prompts	Yes	No	No	No
Files	Yes	Yes	Yes	Yes
Predefined rules	Yes	Yes	Yes	Yes
System command output	Yes	Yes	Yes	Yes
Peer TCP/IP connections (inbound)	No	No	Yes	Yes
NodeBrain queues	No	No	Yes	Yes
FIFO	No	No	Yes	Yes
System or application log file	No	No	Yes	Yes
Clock interrupts	Delayed	Delayed	Yes	Yes
Destination:
NodeBrain queues	Yes	Yes	Yes	Yes
System shell commands	Yes	Yes	Yes	Yes
Peer TCP/IP connections (outbound)	Yes	Yes	Yes	Yes
Component Type:	Shell	Script	Script	Agent
In Relation to an Agent:	Client	CLI	Child	Peer

We call an executing nb program a "brain". Each brain is potentially a node in a network of brains. An interactive brain may function as a primitive expert system shell to solve an isolated problem, or function as a terminal by directing commands to an agent brain and displaying the output. A batch brain executes a NodeBrain script that may perform an isolated function or provide a command line interface (CLI) to an agent. A servant is a temporary child process spawned to handle a requested conversation, system shell command or other function. A service is a persistent brain (Unix daemon or Windows service) that functions as an application agent or manager. Services may monitor multiple inputs of one or more of the highlighted input types, depending on their configuration. The set of possible inputs may be extended by the creation of servant programs and skill modules (plug-ins).

1.2 Application Scenarios

This section illustrates, at a conceptual level, how NodeBrain runs in combinations of the various modes described in the previous section to address communications requirements of an event monitoring application.

We are deliberately downplaying the role of NodeBrain rules in this chapter, because we want to talk about rules later, but we need to be careful here. The purpose of the NodeBrain features illustrated in this section is to provide secure and reliable transmission of events from an event collector to a NodeBrain monitor and enable remote administration of the monitor.

If you are performing a task similar to one shown here, but that does not require rules, you should first consider more common utilities like ssh, scp, cron and your favorite scripting language. You also have the option of using NodeBrain in combination with these tools.

Each gray oval in the following diagrams represents nb running in one of the modes described in the previous section.

1.2.1 Interactive Remote Shell Command Execution

When properly configured, NodeBrain can be used to issue host shell commands on a remote system. A listening agent on the remote system accepts a connection and hands it off to a servant that acts as a command proxy. The shell and servant brains authenticate each other and encrypt the transmitted commands and command output.

1.2.2 Batch Remote Shell Command Execution

Scripted commands can be sent to a remote system under the control of a local NodeBrain agent, or another scheduling utility like cron.

1.2.3 Interactive File Transfer

An encrypted file transfer can be done as a push or a pull. A pull is illustrated here.

1.2.4 Batch File Transfer

A file transfer may be done under the control of a NodeBrain agent. A push is illustrated here, although a pull is also possible.

1.2.5 Batch Command Scheduling

NodeBrain can be used as a batch command scheduling utility similar to cron. The cron utility is very good for simple calendar based schedules where you do not need state or event triggered schedules. NodeBrain supports complex calendar based schedules and also supports state and event based schedules.

1.2.6 System or Application Monitoring

NodeBrain can be used to monitor a system or application. System administrators often run shell scripts as cron jobs to monitor their systems. These same scripts could run under the control of NodeBrain to no advantage. However, one can often simplify the scripts by reducing their task to assessing the state of a particular type of monitored element. You can code NodeBrain rules to make decisions about “out of threshold states” and select appropriate actions. NodeBrain can also directly monitor log files looking for matches to regular expressions you specify.

States

System or Application Log Files

1.2.7 Application Message Store-and-Forward

An application requiring a flow of on-demand one-line messages between two systems can use NodeBrain as the plumbing. The loose coupling of the application end points is similar to mail delivery. A message queue on the sending system (system A below) holds messages if the receiving system or NodeBrain agent is unreachable. A message queue on the receiving system (system B below) holds messages awaiting processing by the receiving application component.

1.2.8 NodeBrain Command Store-and-Forward

If we replace the application components above with NodeBrain agents, we have a NodeBrain application. Here the messages are simply NodeBrain commands.

1.2.9 Application Alert Correlation

Suppose we have an application that generates alerts on several different systems and we want to monitor those alerts using event correlation rules. In this case the application is the source, but a NodeBrain monitor is on the receiving end.

1.2.10 System or Application Monitoring with Alert Correlation

The application generating the alerts can also be a NodeBrain monitor application as shown in this diagram. Rules control the monitor agent, defining the conditions under which action is required. The NodeBrain monitors may take local corrective actions or send alerts up to the correlating monitor.

Alert queue

1.2.11 Diagnostic Problem Solving Scripts

Until now we have been talking about applications of NodeBrain where it reacts to events based on a set of rules. It is designed specifically for this “reactive” mode. However, it is also designed to operate in a problem solving mode where it attempts to “resolve” unknown conditions. For example, given a rule set for identifying plants, NodeBrain could help an interactive user identify a specific observable plant. Questions are asked about unknown attributes of the plant that may lead to a solution. Rule conditions may be found to be true or false without asking all possible questions. Answers given may rule out possibilities that NodeBrain will decide not to pursue further. (Why does this diagram look like a hot air balloon?)

1.2.12 Automated Diagnostic Scripts

We can replace the human observer with consultant scripts that reference an information source to answer questions posed by the NodeBrain diagnostic script. In this case, the diagnostic script may be invoked by a NodeBrain agent to “investigate a potential problem.” The conclusion may be reported back to the agent in the form of an event or state assertion. The agent can then take the appropriate action based on its own rules.

1.3 Invoking NodeBrain

The NodeBrain utility, nb, may be invoked directly at a shell command prompt with arguments.

$ nb [arguments]

It may also be invoked indirectly by a NodeBrain script starting with the following shebang (“#!”) line.

#!/usr/local/bin/nb [argument]

… enter commands here – one per line…

If nb is installed in a different directory, a symbolic link should be placed in /usr/local/bin so you can use the standard path name in script headers.

cd /usr/local/bin

ln –s /some/where/else/nb nb

/usr/local/bin/nb --> /some/where/else/nb

When NodeBrain is invoked without arguments the user is prompted for commands.

$ nb

@> enter command here

@> quit

To pass a command as an argument, enclose the command in quotes and prefix the command with a colon “:”. If quotes are needed in the command, you will need to escape them (\”).

$ nb “:command”

To set a switch option, specify the option name following a minus symbol. Switch options may be turned off using an uppercase letter.

$ nb –d [ Turn daemon option on ]

$ nb –D [ Turn daemon option off ]

Switch options may also be specified using words following a double minus.

$ nb –-daemon [ Turn daemon option on ]

$ nb –-noDaemon [ Turn daemon option off ]

To set a control variable option, separate the option name and value with an equal symbol. It should look like an assignment.

$ nb --option=”value”

For a complete list of switch and control variable options refer to the SET command in the NodeBrain Language Reference.

1.4 User Account Configuration

When NodeBrain is invoked, it first checks for an optional private.nb file in a hidden directory in the user’s home directory.

$home/.nb/private.nb

This file should be readable only by the owner. It is intended primarily for statements that control user authentication. It may also contain other statements the user wants to apply to every execution of NodeBrain. This file is not required, but is recommended if you intend to use NodeBrain peer-to-peer communications.

Let’s say we have a user named “myuser” with a home directory of “/home/myuser”. The full path to private.nb is shown here.

/home/myuser/.nb/private.nb

After creating and protecting an empty file by this name, a NodeBrain identity can be generated by invoking NodeBrain and issuing an identify command.

$ nb

@> identify myid

@> quit

The identify command appends an identity declaration statement to the private.nb file. Assuming we started with an empty file, we should have a single line in the file that looks like this.

declare myid identity 3.3574658473647a8b34.3434578934738473.0;

This declaration associates the identity name “myid” with the randomly generated private NodeBrain key. The identity name is not required to match the user name.

Because a given user account may have multiple private identities declared in the private.nb file, you must tell NodeBrain which identity you want to assume at any given time. This is done with a portray command. You can manually add one to your private.nb file.

declare myid identity 3.3574658473647a8b34.3434578934738473.0;

portray myid;

If a listening NodeBrain agent is to run under the myuser host system account on the local system, listening on port 12345, and the myuser account will also be used to communicate with it, then we recommend declaring the agent brain in the private.nb file. This is done by manually editing the file and adding the highlighted line below.

declare myid identity 3.3574658473647a8b34.3434578934738473.0;

portray myid;

declare myagent brain myid@localhost:12345;

The syntax for brain declaration is

declare name brain identity@host:port;

If the myuser host account is also to be used to communicate with an agent running on a remote system under an identity named “buddy”, the declaration for the identity and brain may be included in private.nb for convenience.

declare myid identity 3.3574658473647a8b34.3434578934738473.0;

portray myid;

declare myagent brain myid@localhost:12345;

declare buddy identity 3.453647382637647388.0.0;

declare mybud brain buddy@rosebud:12321;

In this case, the public identity is obtained from the owner of buddy and pasted into the private.nb file. A private identity can be converted to a public identity by replacing the third set of hex digits with a 0. For example, the public identity for myid would be declared as follows. Compare this to the private identity declaration above.

declare myid identity 3.3574658473647a8b34.0.0;

1.5 Agent Configuration

The following script is sufficient to define a listening brain that does nothing until remotely commanded.

#!/usr/local/bin/nb

set log="/home/myuser/log/myagent.log";

set out="/home/myuser/out";

define l1 listener type=”NBP”,port=12345;

This configuration file (script) uses the set command to establish a log file and directory for storing the output of spawned processes.

When one or more listeners are defined, NodeBrain automatically “daemonizes” itself after processing all input and continues running in the background. In this example, we defined a NodeBrain Protocol (NBP) listener.

If we had not issued the portray command in our private.nb, or wanted to run under a different identity, it would be necessary to include a portray command in this script.

portray identity;

To permit a different identity to connect to our agent, we use the RANK command to grant guest, peer, user, or owner permissions. (See RANK command.)

rank identity guest;

1.6 Peer Authentication and Data Encryption

A brain in listener mode (an agent) must restrict access to the knowledge base, user account and host system. Brains initiating or accepting socket connections must be able to authenticate their peers and apply access permission rules. Our notion of a brain, system user, identity and permission are important elements of the security model.

A system user is given a “private identity string” (key) which enables operation of a brain under a specific identity. Identities may be used on multiple systems by multiple system accounts. Brains participating in communication with peer brains use “public identity strings” to authenticate the identity of a peer brain. When a listening brain accepts a connection, the following information is received from the peer to begin the authentication process.

Attribute	Description
Identity	Peer identity name
IP address	Peer IP address
Challenge	A “random” string encrypted with the recipient’s public key.
Signature	The same string encrypted with the sender’s private key.

What we call a “public key” is not actually intended to be public in the sense of a Public Key Infrastructure (PKI). There is no key authority and no key server. NodeBrain public keys should be handled like shared secret keys within an application community.

The challenge string (nonce) is used to generate the secret key for data encryption based on the Rijndal (AES) algorithm.

1.7 Directing Commands to an Agent

To send a command to a NodeBrain agent, we simply prefix the command with the symbol “>” and the brain name.

>brain command

We could type the following to send a command to the brain named myagent.

$ nb

@> >myagent command

The following script could be used as a client terminal to communicate with myagent.

/home/myuser/myagentClient

---------------------------------

#!/usr/local/bin/nb -prompt

address >myagent;

The address command is used to establish the destination of subsequent commands. When this script is invoked, a command prompt is displayed and we can start issuing commands to the agent.

$ cd /home/myuser

$ ./myagentClient

>myagent.@< enter command here

There is no difference in the way we command a local agent and a remote agent. To command the remote agent mybud, we could use the following script.

/home/myuser/mybudClient

---------------------------------

#!/usr/local/bin/nb -prompt

address >mybud;

If you want to send commands from a script or program to a remote brain defined in your private.nb file, you can open up a pipe to nb and write commands to it with a ">brain" prefix. The nb program will handle the connection to the remote brain for you.

#!/usr/bin/perl

open(IN,"/tmp/infoo")||die "unable to open input file";

open(NB,"|nb")||die "unable to open pipe to NodeBrain";

while(<in>){ # you would generate the commands here instead

print(NB ">brain $_");

}

close(NB);

close(IN);

1.8 Issuing Shell Commands

Shell commands are invoked via NodeBrain by prefixing the command with a minus symbol “-“.

$ nb

@> -date

Mon Sep 23 09:50:19 PDT 2002

2002/09/23 09:50:19 NB000I Exit Code (0)

To issue a command via a spawned process without waiting for the output, prefix the shell command with an equal symbol “=”. (Think parallel vs. serial.)

$ nb

@> =date

2002/09/23 09:50:34 NB000I Exec pid[30858]

Shell commands are invoked via a remote agent the same; that is, by prefixing commands with “-“ or “=” at the NodeBrain command prompt.

$ nb

@> address >mybud;

>mybud.@< -date

Mon Sep 23 09:44:00 PDT 2002

>mybud.@<

This same command could be issued as a single shell command in two different ways.

$ echo “>mybud –date” | nb

-or-

$ nb “:>mybud –date”

We could also script the command.

/home/myid/remotedate

---------------------------

#!/usr/local/bin/nb

>mybud -date

$ /home/myid/remotedate

If you want to execute multiple shell commands, just add more lines to the script. However, because NodeBrain is not a general purpose scripting language, you are advised to do your procedural scripting in an appropriate scripting language and use NodeBrain where appropriate to invoke your scripts.

1.9 Encrypted File Transmission

The NodeBrain copy command is used to push or pull files to or from a remote system with help from a listening brain on the remote system. Normal NodeBrain authentication and encryption is performed for the communication between the local brain and the remote agent brain. As with FTP, you must select an ascii or binary file copy, which you specify with “a” or “b”.

copy {a|b} source destination

Both the source and destination must conform to the following syntax.

[@brain:] path

The optional brain name is not specified for local files. A brain is specified on the source for a pull, and on the destination for a push. A brain may be specified on both source and destination to copy a file from one remote system to another.

An interactive file copy can be performed from a command prompt.

$ nb

@> copy a /tmp/foo @mybud:/tmp/foo

-or-

$ nb “:copy a /tmp/foo @mybud:/tmp/foo”

A file copy can also be scripted.

/home/myuser/copyfoo

--------------------------

#!/usr/local/bin/nb

copy a /tmp/foo @mybud:/tmp/foo

$ /home/myuser/copyfoo

1.10 Writing to Command Queues

A NodeBrain command queue is a simple text file where each line is interpreted as a command. Multi-line commands are not supported. Unprocessed commands are identified by “>” in the first byte. Any other symbol in the first byte identifies a processed command. Successfully processed commands are identified by a “#” in the first byte. This is the comment symbol in NodeBrain.

We associate a command queue with a brain, and specify the path to the queue in the brain declaration. We’ll update our declaration of mybud to add a queue directory.

declare myid identity 3.3574658473647a8b34.3434578934738473.0;

portray myid;

declare myagent brain myid@localhost:12345;

declare buddy identity 3.453647382637647388.0.0;

declare mybud brain buddy@rosebud:12321[/home/myuser/queue];

When we want NodeBrain to write a command to a queue we simple prefix it with the store command “\” and the brain name.

\brain command

For example, to queue up a “-date” command for mybud interactively, we could type the following commands.

$ nb

@> \mybud –date

@> quit

We could also use the following command, which we might prefer when coding scripts.

$ echo “\mybud –date” | nb

-or-

$ nb “:\mybud –date”

The queue path defined for a brain is extended by the brain name and currently portrayed identity.

/home/myuser/queue/mybud/myid/filename.q

The target brain declaration does not need to identify a listening agent. It may be defined simply as a queue. The queue may be used as input to another application, a database loader for example.

declare mydb brain [/home/myuser/queue];

The syntax for writing to this queue is simply

\mydb whatever

and the queue file is named appropriately.

/home/myuser/queue/mydb/myid/filename.q

You may notice from this example that NodeBrain does not parse the command before writing it to the queue file, and has no idea if it will be found to be acceptable by the recipient.

1.11 Transmitting Message Queues

In previous sections we showed how to send commands to a remote NodeBrain agent called mybud, and we showed how to write commands to a local queue for this agent. We can also use this queue to store-and-forward commands to the agent by reversing the slash. (Think store down “\” and forward up “/”.)

/brain command

We’ll use our silly “-date” command again in our example because we have yet to introduce any useful command that we might want to send to mybud. Here we illustrate a forward command passed to NodeBrain as an argument.

$ nb “:/mybud –date”

The forward command first performs the same function as the store (“\”) command. However, after appending the command to the queue file, it attempts to forward all unprocessed commands from the queue to the specified brain. As long as we are unable to establish a connection with the target brain, the queue file will grow. Once we establish a connection and successfully forward all previously unprocessed commands to the brain, the queue file is reset to an empty file.

If no command is specified in the body of the forward command, NodeBrain simply attempts to forward any unprocessed commands from the queue.

/brain

NodeBrain only, and always, attempts to forward commands when a forward command is issued. If the commands you are forwarding represent alerts (a topic for later), this could result in too few connection attempts or too many connection attempts, depending on the frequency and pattern of alerts. You can batch alerts up by using the store (“\”) command as alerts occur and schedule a forward to be applied on an appropriate frequency (e.g. every 5 minutes). If alerts occur infrequently, you can use the forward command for every alert and schedule a forward on a regular interval to pick up alerts stranded by an outage of the network, remote host, or remote agent.

A sample rule to schedule a forward command is shown here without explanation. Time expressions are covered in Chapter 5.

Define r1 on(~(5m)):/mybud

1.12 NodeBrain over SSH Tunnel

You can provide an extra layer of security by running NodeBrain peer-to-peer communication over a secure tunnel. The idea here is to establish an encrypted tunnel between two systems using a form of authentication and encryption that is already trusted and approved in your environment. The tunnel can be configured to listen only for local connections on system A and forward these connections to a NodeBrain agent at the other end of the tunnel on system B. On system B, you can configure NodeBrain to only accept connections from the local host. This way, you will not expose the NodeBrain service to attack from systems other than A and B. (Additional tunnels can be established for other trusted hosts.)

A tunnel can be established from portA on hostA to portB on hostB by executing the following command on hostA.

ssh -2 -f -N -L portA:localhost:portB hostB

On host A, you specify the host B agent as follows. Notice we are defining it as the local end of the SSH tunnel.

declare brainB brain identityB@localhost:portA;

On host B, the agent is configured to listen only to the loop back address, which will be used by B’s end of the tunnel.

define ear listener type=”NBP”,address=”localhost”,port=portB;

In a configuration like this, NodeBrain is unaware of the tunnel and authenticates peers as if the tunnel didn’t exist. This is appropriate since the SSH tunnel doesn’t know the identity or permissions of individual user’s on system A that may attempt a connection to the agent on system B.

There are several variations of this scenario that you can explore by studying the ssh documentation. For example, you may want to open portA to remote hosts using the –g option if the purpose of the tunnel is to get through a firewall. Or you could put both ends of the tunnel on a dual homed host to provide a gateway to a NodeBrain agent listening for remote connections on one side of the gateway.