[ <- Previous Page ] [ ^ Table Of Contents ^ ] [ Next Page -> ]
zWebit Engine Internals

Chapter 2
zWebit Engine Internals

How It Works

The zWebit engine is based on a simple idea. Basically, a string of data (a.k.a. a transaction) is captured from some data source, is logged into a transaction file and the source system is acknowledged. The data is optionally modified as needed and forwarded to one or more destinations. This is basically all any interface engine needs to do.

Detail Design Diagram

Single Interface Configuration

This diagram depicts how the system works. Data acquisition programs (B) and data delivery programs (D) read configuration files (G) at startup. The sending system (A) sends a data transaction to the data acquisition program (B). When a transaction is received by the acquisition program (B), it pulls the next sequence number (seqno) from the ptrkeys table (E), inserts the transaction into the idata table (F), acknowledges the sending system (A) and sends a wake up message to the data delivery program (D). When the delivery program (D), receives a message from the data acquisition program, it determines which transactions need to be processed and sends them. Data delivery programs (D) determine which transactions need to be sent by comparing the sequence number of its predetermined entry in the ptrkeys table (E), and compares it to the NEXT control record in the ptrkeys table. If the sequence numbers match, the delivery program (D) is current, otherwise, it will process all the records for the appropriate sequence numbers.

Multiple Interface Configuraion

zWebit supports multiple simultaneous input sources as well as multiple simultaneous output sources. Delivery programs can process all data submitted to the engine or discern data records based on source, format or record type.

General Program Processing

There are two categories of zWebit interface communication programs: data acquisition and data delivery. The communication programs use two database tables: ptrkeys and idata.

Each interface program has its own record in the ptrkeys table, and there is a unique record in the ptrkeys table for the NEXT seqno. If the record is NEXT, it is the last seqno assigned. A significant field in the ptrkeys table is seqno. If the interface is a receiving system, the seqno field is the seqno of the last record received. If the interface is a sending system, the seqno field is the number of the last record successfully sent.

Every inbound data transaction is appended to the idata table with a unique sequence number (seqno) as key. The seqno field is incremented by one for each inbound transaction. The first transaction into the engine is seqno 1, the second transaction is seqno 2, and so on.

All zWebit programs start by reading a configuration file which assigns run time values including settings such as database name, log file parameters, inter-program communication parameters and any custom program settings.

Acquisition programs typically behave as server programs and wait for incoming data. Upon receipt of incoming data, the program retreives the NEXT seqno from the ptrkeys table increments, appends the transaction to the idata table and notifies delivery programs about incoming data.

Database Table Definitions

The current version of zWebit utilizes the Postgres DBMS. Postgres has proven to be steady and reliable. zWebit could be ported to other DBMS tools if needed.

Ptrkeys Table

Each interface program has a tuple (record) in the ptrkeys table.

INTERFACECHAR (12) This is the primary key to the table. Each interface has an entry with a unique interface name.
SEQNOINTEGER For sending programs, this is the last seqno sent. For receiving programs, this is the last seqno received. For special entry NEXT, this was the last seqno assigned.
  • I for Inbound
  • O for Outbound
  • S for System
  • D for Database update (future use)
    ONOFFSWCHAR (3) ON or OFF, used to turn interface links on and off.
    LASTUPDATETIMESTAMP This is the last interface activity timestamp.
    DESCRIPTIONTEXT This is a description of the interface.
    CMDTEXT The script file associated with this interface is used for starting, stopping, and checking interface status. See the section on script files for more information.

    Sample Ptrkeys Table

    InterfaceTypeStatusSeqno Description
    NEXTSystem1032 Next available sequence number
    FROMPTREGINBOUNDON1032 Data from registration system
    TO LABOUTBOUNDON1028 Data to lab system
    TORADOUTBOUNDON1032 Data to radiology system
    TORXOUTBOUNDOFF972 Data to pharmacy system

    In the above example, the last transaction received was from interface FROMPTREG. Interface TORAD is caught up. Interface TOLAB is four transactions behind. Interface TORX is turned off and sixty transactions behind.

    There is a special instance in the table with an interface key of NEXT. NEXT contains the last seqno assigned to any transaction. Acquisition programs pull the last seqno from the NEXT tuple, increment it by one, use the seqno to append a record to the idata table and update the seqno NEXT to the new seqno.

    Idata Table

    The idata table is used for storing the actual data transactions received. Acquisition programs insert transactions into the idata tables and delivery programs select data from the idata table to send to destination systems.

    SEQNOINTEGER Primary key of the table.
    SOURCECHAR (12) Interface responsible for placing this record here.
    RECTYPECHAR (8) User defined record type (like result).
    FORMATCHAR (8) User defined format (like hl7).
    LASTUPDATETIMESTAMP Timestamp when the record was added.
    DATATEXT Actual transaction data.

    Database Names

    The default zWebit configuration uses two Postgres databases, test and prod. Each database has two tables, ptrkeys and idata.

    Script Files

    The webit/bin directory contains executable programs and script files. Each interface has its own script file that is used for controlling a particular interface link. Script files can be executed standalone or called from the interface web-based monitoring tools. The input parameters to the script file are system (test or prod) and command (start, stop, status or status2).

    Interface Program Script File Example

    tolab test status Display test interface status for tolab
    tolab test stop Stop test interface tolab
    tolab test start Start test interface status for tolab
    tolab prod status Display prod interface status for tolab
    tolab prod stop Stop prod interface tolab
    tolab prod start Start prod interface tolab

    The name of the script file matches the name of the interface in the ptrkeys table. In the above example, tolab is the name of the interface in the ptrkeys table and also the name of the script file.

    Test and prod identify which system.

    Command is one of status, stop, or start.

    System Script Files

    zstartallzstartall test|prod Script file zstartall is used for starting all interfaces that are not currently active and are turned on in the ptrkeys table.
    zstopallzstopall test|prod Script file zstopall is used for stopping active interfaces.
    zpurgezpurge test|prod Script file zpurge is used for clearing processed transactions out of the idata table and reorganizing (vacuum) the database.

    Data Acquisition Interface Overview

    Data acquisition programs are responsible for receiving data, updating the idata and ptrkeys database tables, acknowledging the sending system and signaling appropriate delivery programs. Acquisition programs also remove any framing characters from the inbound transaction before updating the idata table. The included sample program is written in C++.

    A data acquisition program works as follows:

    Data Acquisition Program Pseudocode

    1. Open config file and pull initialization variables
    2. Verify initialization variables are reasonable
    3. Open log files
    4. Initialize DBMS connection
    5. Enter a process loop
      1. Receive transaction from data source
      2. Remove encapsulation characters
      3. Get next available seq-no from ptrkeys table
      4. Append transaction to idata table
      5. Commit transaction to DBMS
      6. Acknowledge sending system of succesful receipt of transaction
      7. Notify all appropriate delivery programs of incoming data

    Data Delivery Interface Overview

    Data delivery programs are responsible for reliably delivering data and updating the ptrkeys table.

    A data delivery program works as follows:

    Data Delivery Program Pseudocode

    1. Open config file and pull initialization variables
    2. Verify initialization variables are reasonable
    3. Open logfiles
    4. Initialize DBMS connection
    5. Pull interface record from ptrkeys file
    6. Enter a process loop
      1. For each record needing to be sent, pull record from idata table
        1. Reformat or modify data if necessary
        2. Add framing characters
        3. Send it to destination until receipt of acknowledgement
        4. Update seq-no in ptrkeys table for this interface
      2. Wait for wake up call from any appropriate inbound interface

    Cron Jobs

    On Unix computers, cron performs automating functions according to the time and command that are specified by the user. The cron table, crontab, uses a format consisting of six fields in this order: minute, hour, day, month, day of week, and command. Each field must be filled with either a number or letter, or left unspecified with an asterik.

    As user zwebit, load crontab with the following information:

    Use the crontab -e command to edit the crontab table.

    # crontab file
    # clean old transactions out of webit databases
    0  1 * * * /usr/local/webit/bin/zpurge
    # auto start any down links every 5 minutes
    0  * * * * /usr/local/zwebit/bin/zstartall prod
    5  * * * * /usr/local/zwebit/bin/zstartall prod
    10 * * * * /usr/local/zwebit/bin/zstartall prod
    15 * * * * /usr/local/zwebit/bin/zstartall prod
    20 * * * * /usr/local/zwebit/bin/zstartall prod
    25 * * * * /usr/local/zwebit/bin/zstartall prod
    30 * * * * /usr/local/zwebit/bin/zstartall prod
    35 * * * * /usr/local/zwebit/bin/zstartall prod
    40 * * * * /usr/local/zwebit/bin/zstartall prod
    45 * * * * /usr/local/zwebit/bin/zstartall prod
    50 * * * * /usr/local/zwebit/bin/zstartall prod
    55 * * * * /usr/local/zwebit/bin/zstartall prod
    # Check monthly for seqno reset
    0 3 15 * * /usr/local/zwebit/bin/zresetseqno prod


    Communication programs send and receive data in a predefined pattern of data. For example, HL7 (healthcare level 7) transactions are usuallly formatted with the first byte of the transaction as 0x0B and the last two bytes of the transaction as 0x1C and 0x1D. With a predefined format, communication programs have a marker for knowing where a transaction begins and ends. Framing characters are not stored internally in the idata table. Class zString has functionality for handling framing characters.

    Configuration Profiles

    Configuration files are used for providing configuration information to programs. Each interface has its own configuration profile that is processed at startup. To edit an interface profile, use a text editor. Profiles are standard text files with keyword value pairs. Class zProf is used by zWebit programs to read configuration files and gather needed run time data at startup.

    Log Files

    Like many programs operating in a Unix/Linux environment, zWebit uses log files for monitoring ongoing activity. The zWebit logs are configurable, and they roll. For example, a program could be set to have four log files, each 200K in size. When the first log file fills, it rolls to the second log file, then the third, and finally the fourth. When the fourth log file fills, it rolls back around to the first log file. zWebit appends a number between 1 and n to the logfile name and links the base logfile name to the current open file.

    All log entries are prefixed with a current time stamp. The default number of logfiles is two, but this number can be changed with zLog::zlSetCount ( ). The default logfile size is 50,000 bytes, but it can be changed using zLog::zlSetLogSize ( ).

    There is an internal buffer used by the zLog::zlWriteFmt ( ) method that is 1024 bytes as default. If the program using this class will be logging data longer than 1024 bytes with the zLog::zlWriteFmt ( ) method, the buffer size can be adjusted with zLog::zlGetBufSize ( ) and zLog::zlSetBufSize ( ). If the buffer is not big enough for the data in zLog::writeFmtLog ( ), it can cause a buffer overflow, which can crash the program.

    When programs are first started, they may log messages to the system log (/var/log/messages) if the programs encounter initialization errors and are not able to log messages in the zWebit log files.

    Interprocess Communication

    zWebit uses standard Unix pipe structures for interprocess communication. After a data acquisition program receives a transaction and updates the idata table with the transaction, the program notifies the delivery programs about the data via pipe messages.

    zWebit includes and uses the zPipe class to handle interprocess communication. Each data delivery program has a unique pipe assigned to it. When data arrives, data acquisition programs send delivery programs a wake up call via method zPipe::writePipes ( ). After the delivery program performs its task, it returns to sleep and waits until it receives another message.

    The general idea is that each data delivery program has one and only one pipe associated with it. Any time a data acquisition program captures data, it will send a message to the appropriate delivery programs. When a delivery program receives a wake-up message via pipe, the program determines which transactions it needs to process and processes them sequentially (according to the order in which they were received). Once its tasks are finished, it returns to sleep and waits for another pipe message.

    The zPipe class structure is designed so that many processes can write to the same pipe, and a single process can write to many pipes as well. However, a process only reads from a single pipe.

    This class is designed for one zPipe instance in each program. Multiple instances of this class would result in undefined behavior in the program due to multiple copies of interrupt handler logic.

    Data acquisition programs may need to notify several delivery programs. Acquisition programs need to make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ) for each delivery program.

    Data delivery programs are assigned one pipe each. Delivery programs make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ).

    String Fields

    zWebit treats each interface transaction as one string of data. String data is defined as one or more contiguous bytes of data with a Null (0x00) terminating end byte.

    zWebit includes a string parsing class called zString. Newer versions of C++ include a string class. When this software was initially developed, there was not a standard portable C++ string class available, so this class was created and has been carried forward.

    Hl7 Transactions

    Healthcare data is frequently passed between systems using a special format called zHl7 (healthcare level 7). If you are using zWebit in an environment other than the Healthcare Industry, then this class will not be needed. If you are using zWebit in a healthcare environment, the zHL7 class can be used for manipulating HL7 formatted data in C++ programs.

    The zHL7 routines basically work by treating the entire transaction as one string of data, and applying seek and find or seek and update logic on the string.

    The zHL7 class does not check for HL7 validity and is not as powerful as some other HL7 parsing tools available. However, it has proven adequate for zWebit, and it is extensible as well.

    zWebit is not exclusive to the Healthcare industry alone. In fact, it can be utilized for most endeavors concerning interface communication and data manipulation. This is because zWebit is capable of using other classes or tools for modifying data.

    Model zWebit Directory Structure

    Directory NameDescription
    webit/binContains all webit shell programs and script files
    webit/docUser directory for string interface documentation specification
    webit/sourceContains subdirectories of source code
    webit/testContains subdirectories for the test system interfaces
    webit/test/configContains test system configuration files
    webit/test/logsContains test system log files
    webit/test/pipesContains interprocess communication pipe files for the test system programs
    webit/test/dataMiscellaneous test files
    webit/prodContains subdirectories needed for the prod system interfaces
    webit/prod/configContains prod system configuration files
    webit/prod/logsContains prod system log files
    webit/prod/pipesContains interprocess communication pipe files for the prod system programs
    webit/htmlContains html and php scripts used for monitoring the interface programs
    webit/sqlContains sql files neede to create and maintain database tables
    webit/zzzIf you need to set up interfaces with a name other than test or prod, create zzz/config, zzz/logs, and zzz/pipes subdirectories
    (zzz = name of your database)

    This directory structure is a sample setup. After you get the idea of how zWebit works, you could use different interface system names (instead of test and prod) and modify the directory structure to include the new names. The directory names (test and prod) match the database names used in Postgres.

    [ <- Previous Page ] [ ^ Table Of Contents ^ ] [ ^ Top Of Page ^ ] [ Next Page -> ]

    Visit the GNU home page.

    FSF & GNU inquiries & questions to gnu@gnu.org.

    Comments on these web pages to info@zhsac.com.

    Copyright (C) 2003 HealthCare Systems and Consulting

    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.

    Last updated: 07/21/2003