Chapter 2
zWebit Engine Internals

How It Works

The zWebit engine is based on a simple idea. Basically, a string of data (a.k.a. a transaction) is captured from some data source, is logged into a transaction file and the source system is acknowledged. The data is optionally modified as needed and forwarded to one or more destinations. This is basically all any interface engine needs to do.

This diagram depicts how the system works. Data acquisition programs (B) and data delivery programs (D) read configuration files (G) at startup. The sending system (A) sends a data transaction to the data acquisition program (B). When a transaction is received by the acquisition program (B), it pulls the next sequence number (seqno) from the ptrkeys table (E), inserts the transaction into the idata table (F), acknowledges the sending system (A) and sends a wake up message to the data delivery program (D). When the delivery program (D), receives a message from the data acquisition program, it determines which transactions need to be processed and sends them. Data delivery programs (D) determine which transactions need to be sent by comparing the sequence number of its predetermined entry in the ptrkeys table (E), and compares it to the NEXT control record in the ptrkeys table. If the sequence numbers match, the delivery program (D) is current, otherwise, it will process all the records for the appropriate sequence numbers.

zWebit supports multiple simultaneous input sources as well as multiple simultaneous output sources. Delivery programs can process all data submitted to the engine or discern data records based on source, format or record type.

There are two categories of zWebit interface communication programs: data acquisition and data delivery. The communication programs use two database tables: ptrkeys and idata.

Each interface program has its own record in the ptrkeys table, and there is a unique record in the ptrkeys table for the NEXT seqno. If the record is NEXT, it is the last seqno assigned. A significant field in the ptrkeys table is seqno. If the interface is a receiving system, the seqno field is the seqno of the last record received. If the interface is a sending system, the seqno field is the number of the last record successfully sent.

Every inbound data transaction is appended to the idata table with a unique sequence number (seqno) as key. The seqno field is incremented by one for each inbound transaction. The first transaction into the engine is seqno 1, the second transaction is seqno 2, and so on.

All zWebit programs start by reading a configuration file which assigns run time values including settings such as database name, log file parameters, inter-program communication parameters and any custom program settings.

Acquisition programs typically behave as server programs and wait for incoming data. Upon receipt of incoming data, the program retreives the NEXT seqno from the ptrkeys table increments, appends the transaction to the idata table and notifies delivery programs about incoming data.

The current version of zWebit utilizes the Postgres DBMS. Postgres has proven to be steady and reliable. zWebit could be ported to other DBMS tools if needed.

Ptrkeys Table

Each interface program has a tuple (record) in the ptrkeys table.

In the above example, the last transaction received was from interface FROMPTREG. Interface TORAD is caught up. Interface TOLAB is four transactions behind. Interface TORX is turned off and sixty transactions behind.

There is a special instance in the table with an interface key of NEXT. NEXT contains the last seqno assigned to any transaction. Acquisition programs pull the last seqno from the NEXT tuple, increment it by one, use the seqno to append a record to the idata table and update the seqno NEXT to the new seqno.

Idata Table

The idata table is used for storing the actual data transactions received. Acquisition programs insert transactions into the idata tables and delivery programs select data from the idata table to send to destination systems.

Database Names

The default zWebit configuration uses two Postgres databases, test and prod. Each database has two tables, ptrkeys and idata.

The webit/bin directory contains executable programs and script files. Each interface has its own script file that is used for controlling a particular interface link. Script files can be executed standalone or called from the interface web-based monitoring tools. The input parameters to the script file are system (test or prod) and command (start, stop, status or status2).

The name of the script file matches the name of the interface in the ptrkeys table. In the above example, tolab is the name of the interface in the ptrkeys table and also the name of the script file.

Test and prod identify which system.

Command is one of status, stop, or start.

Data acquisition programs are responsible for receiving data, updating the idata and ptrkeys database tables, acknowledging the sending system and signaling appropriate delivery programs. Acquisition programs also remove any framing characters from the inbound transaction before updating the idata table. The included sample program is written in C++.

A data source could be found in any number of places:
• A foreign system sending transactions via TCP/IP Client Server Communications
• An SNA LU6.2 communication program (contact HealthCare Systems and Consulting for examples)
• A data file or report
• Serial communication device

A data acquisition program works as follows:

Data Acquisition Program Pseudocode

1. Open config file and pull initialization variables
2. Verify initialization variables are reasonable
3. Open log files
4. Initialize DBMS connection
5. Enter a process loop
   1. Receive transaction from data source
   2. Remove encapsulation characters
   3. Get next available seq-no from ptrkeys table
   4. Append transaction to idata table
   5. Commit transaction to DBMS
   6. Acknowledge sending system of succesful receipt of transaction
   7. Notify all appropriate delivery programs of incoming data

Data delivery programs are responsible for reliably delivering data and updating the ptrkeys table.

Data Delivery Program Pseudocode

1. Open config file and pull initialization variables
2. Verify initialization variables are reasonable
3. Open logfiles
4. Initialize DBMS connection
5. Pull interface record from ptrkeys file
6. Enter a process loop
   1. For each record needing to be sent, pull record from idata table
      1. Reformat or modify data if necessary
      2. Add framing characters
      3. Send it to destination until receipt of acknowledgement
      4. Update seq-no in ptrkeys table for this interface
   2. Wait for wake up call from any appropriate inbound interface

On Unix computers, cron performs automating functions according to the time and command that are specified by the user. The cron table, crontab, uses a format consisting of six fields in this order: minute, hour, day, month, day of week, and command. Each field must be filled with either a number or letter, or left unspecified with an asterik.

As user zwebit, load crontab with the following information:

• To auto purge any old data every morning at 1:00 A.M.
• Restart any down links every five minutes.
• Check the seqno field

# crontab file
#
# clean old transactions out of webit databases
0  1 * * * /usr/local/webit/bin/zpurge
#
#
# auto start any down links every 5 minutes
0  * * * * /usr/local/zwebit/bin/zstartall prod
5  * * * * /usr/local/zwebit/bin/zstartall prod
10 * * * * /usr/local/zwebit/bin/zstartall prod
15 * * * * /usr/local/zwebit/bin/zstartall prod
20 * * * * /usr/local/zwebit/bin/zstartall prod
25 * * * * /usr/local/zwebit/bin/zstartall prod
30 * * * * /usr/local/zwebit/bin/zstartall prod
35 * * * * /usr/local/zwebit/bin/zstartall prod
40 * * * * /usr/local/zwebit/bin/zstartall prod
45 * * * * /usr/local/zwebit/bin/zstartall prod
50 * * * * /usr/local/zwebit/bin/zstartall prod
55 * * * * /usr/local/zwebit/bin/zstartall prod
#
#
#
# Check monthly for seqno reset
0 3 15 * * /usr/local/zwebit/bin/zresetseqno prod

Communication programs send and receive data in a predefined pattern of data. For example, HL7 (healthcare level 7) transactions are usuallly formatted with the first byte of the transaction as 0x0B and the last two bytes of the transaction as 0x1C and 0x1D. With a predefined format, communication programs have a marker for knowing where a transaction begins and ends. Framing characters are not stored internally in the idata table. Class zString has functionality for handling framing characters.

Configuration files are used for providing configuration information to programs. Each interface has its own configuration profile that is processed at startup. To edit an interface profile, use a text editor. Profiles are standard text files with keyword value pairs. Class zProf is used by zWebit programs to read configuration files and gather needed run time data at startup.

Like many programs operating in a Unix/Linux environment, zWebit uses log files for monitoring ongoing activity. The zWebit logs are configurable, and they roll. For example, a program could be set to have four log files, each 200K in size. When the first log file fills, it rolls to the second log file, then the third, and finally the fourth. When the fourth log file fills, it rolls back around to the first log file. zWebit appends a number between 1 and n to the logfile name and links the base logfile name to the current open file.

All log entries are prefixed with a current time stamp. The default number of logfiles is two, but this number can be changed with zLog::zlSetCount ( ). The default logfile size is 50,000 bytes, but it can be changed using zLog::zlSetLogSize ( ).

There is an internal buffer used by the zLog::zlWriteFmt ( ) method that is 1024 bytes as default. If the program using this class will be logging data longer than 1024 bytes with the zLog::zlWriteFmt ( ) method, the buffer size can be adjusted with zLog::zlGetBufSize ( ) and zLog::zlSetBufSize ( ). If the buffer is not big enough for the data in zLog::writeFmtLog ( ), it can cause a buffer overflow, which can crash the program.

When programs are first started, they may log messages to the system log (/var/log/messages) if the programs encounter initialization errors and are not able to log messages in the zWebit log files.

zWebit uses standard Unix pipe structures for interprocess communication. After a data acquisition program receives a transaction and updates the idata table with the transaction, the program notifies the delivery programs about the data via pipe messages.

zWebit includes and uses the zPipe class to handle interprocess communication. Each data delivery program has a unique pipe assigned to it. When data arrives, data acquisition programs send delivery programs a wake up call via method zPipe::writePipes ( ). After the delivery program performs its task, it returns to sleep and waits until it receives another message.

The general idea is that each data delivery program has one and only one pipe associated with it. Any time a data acquisition program captures data, it will send a message to the appropriate delivery programs. When a delivery program receives a wake-up message via pipe, the program determines which transactions it needs to process and processes them sequentially (according to the order in which they were received). Once its tasks are finished, it returns to sleep and waits for another pipe message.

The zPipe class structure is designed so that many processes can write to the same pipe, and a single process can write to many pipes as well. However, a process only reads from a single pipe.

This class is designed for one zPipe instance in each program. Multiple instances of this class would result in undefined behavior in the program due to multiple copies of interrupt handler logic.

Data acquisition programs may need to notify several delivery programs. Acquisition programs need to make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ) for each delivery program.

Data delivery programs are assigned one pipe each. Delivery programs make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ).

zWebit treats each interface transaction as one string of data. String data is defined as one or more contiguous bytes of data with a Null (0x00) terminating end byte.

zWebit includes a string parsing class called zString. Newer versions of C++ include a string class. When this software was initially developed, there was not a standard portable C++ string class available, so this class was created and has been carried forward.

Healthcare data is frequently passed between systems using a special format called zHl7 (healthcare level 7). If you are using zWebit in an environment other than the Healthcare Industry, then this class will not be needed. If you are using zWebit in a healthcare environment, the zHL7 class can be used for manipulating HL7 formatted data in C++ programs.

The zHL7 routines basically work by treating the entire transaction as one string of data, and applying seek and find or seek and update logic on the string.

The zHL7 class does not check for HL7 validity and is not as powerful as some other HL7 parsing tools available. However, it has proven adequate for zWebit, and it is extensible as well.

zWebit is not exclusive to the Healthcare industry alone. In fact, it can be utilized for most endeavors concerning interface communication and data manipulation. This is because zWebit is capable of using other classes or tools for modifying data.

This directory structure is a sample setup. After you get the idea of how zWebit works, you could use different interface system names (instead of test and prod) and modify the directory structure to include the new names. The directory names (test and prod) match the database names used in Postgres.

Visit the GNU home page.

FSF & GNU inquiries & questions to gnu@gnu.org.

Comments on these web pages to info@zhsac.com.

Copyright (C) 2003 HealthCare Systems and Consulting

Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.

Last updated: 07/21/2003