|[ <- Previous Page ]||[ ^ Table Of Contents ^ ]||[ Next Page -> ]|
The zWebit engine is based on a simple idea. Basically, a string of data (a.k.a. a transaction) is captured from some data source, is logged into a transaction file and the source system is acknowledged. The data is optionally modified as needed and forwarded to one or more destinations. This is basically all any interface engine needs to do.
This diagram depicts how the system works. Data acquisition programs (B) and data delivery programs (D) read configuration files (G) at startup. The sending system (A) sends a data transaction to the data acquisition program (B). When a transaction is received by the acquisition program (B), it pulls the next sequence number (seqno) from the ptrkeys table (E), inserts the transaction into the idata table (F), acknowledges the sending system (A) and sends a wake up message to the data delivery program (D). When the delivery program (D), receives a message from the data acquisition program, it determines which transactions need to be processed and sends them. Data delivery programs (D) determine which transactions need to be sent by comparing the sequence number of its predetermined entry in the ptrkeys table (E), and compares it to the NEXT control record in the ptrkeys table. If the sequence numbers match, the delivery program (D) is current, otherwise, it will process all the records for the appropriate sequence numbers.
zWebit supports multiple simultaneous input sources as well as multiple simultaneous output sources. Delivery programs can process all data submitted to the engine or discern data records based on source, format or record type.
There are two categories of zWebit interface communication programs: data acquisition and data delivery. The communication programs use two database tables: ptrkeys and idata.
Each interface program has its own record in the ptrkeys table, and there is a unique record in the ptrkeys table for the NEXT seqno. If the record is NEXT, it is the last seqno assigned. A significant field in the ptrkeys table is seqno. If the interface is a receiving system, the seqno field is the seqno of the last record received. If the interface is a sending system, the seqno field is the number of the last record successfully sent.
Every inbound data transaction is appended to the idata table with a unique sequence number (seqno) as key. The seqno field is incremented by one for each inbound transaction. The first transaction into the engine is seqno 1, the second transaction is seqno 2, and so on.
All zWebit programs start by reading a configuration file which assigns run time values including settings such as database name, log file parameters, inter-program communication parameters and any custom program settings.
Acquisition programs typically behave as server programs and wait for incoming data. Upon receipt of incoming data, the program retreives the NEXT seqno from the ptrkeys table increments, appends the transaction to the idata table and notifies delivery programs about incoming data.
The current version of zWebit utilizes the Postgres DBMS. Postgres has proven to be steady and reliable. zWebit could be ported to other DBMS tools if needed.
Each interface program has a tuple (record) in the ptrkeys table.
|INTERFACE||CHAR (12)||This is the primary key to the table. Each interface has an entry with a unique interface name.|
|SEQNO||INTEGER||For sending programs, this is the last seqno sent. For receiving programs, this is the last seqno received. For special entry NEXT, this was the last seqno assigned.|
|ONOFFSW||CHAR (3)||ON or OFF, used to turn interface links on and off.|
|LASTUPDATE||TIMESTAMP||This is the last interface activity timestamp.|
|DESCRIPTION||TEXT||This is a description of the interface.|
|CMD||TEXT||The script file associated with this interface is used for starting, stopping, and checking interface status. See the section on script files for more information.|
|NEXT||System||1032||Next available sequence number|
|FROMPTREG||INBOUND||ON||1032||Data from registration system|
|TO LAB||OUTBOUND||ON||1028||Data to lab system|
|TORAD||OUTBOUND||ON||1032||Data to radiology system|
|TORX||OUTBOUND||OFF||972||Data to pharmacy system|
In the above example, the last transaction received was from interface FROMPTREG. Interface TORAD is caught up. Interface TOLAB is four transactions behind. Interface TORX is turned off and sixty transactions behind.
There is a special instance in the table with an interface key of NEXT. NEXT contains the last seqno assigned to any transaction. Acquisition programs pull the last seqno from the NEXT tuple, increment it by one, use the seqno to append a record to the idata table and update the seqno NEXT to the new seqno.
The idata table is used for storing the actual data transactions received. Acquisition programs insert transactions into the idata tables and delivery programs select data from the idata table to send to destination systems.
|SEQNO||INTEGER||Primary key of the table.|
|SOURCE||CHAR (12)||Interface responsible for placing this record here.|
|RECTYPE||CHAR (8)||User defined record type (like result).|
|FORMAT||CHAR (8)||User defined format (like hl7).|
|LASTUPDATE||TIMESTAMP||Timestamp when the record was added.|
|DATA||TEXT||Actual transaction data.|
The default zWebit configuration uses two Postgres databases, test and prod. Each database has two tables, ptrkeys and idata.
The webit/bin directory contains executable programs and script files. Each interface has its own script file that is used for controlling a particular interface link. Script files can be executed standalone or called from the interface web-based monitoring tools. The input parameters to the script file are system (test or prod) and command (start, stop, status or status2).
|tolab test status||Display test interface status for tolab|
|tolab test stop||Stop test interface tolab|
|tolab test start||Start test interface status for tolab|
|tolab prod status||Display prod interface status for tolab|
|tolab prod stop||Stop prod interface tolab|
|tolab prod start||Start prod interface tolab|
The name of the script file matches the name of the interface in the ptrkeys table. In the above example, tolab is the name of the interface in the ptrkeys table and also the name of the script file.
Test and prod identify which system.
Command is one of status, stop, or start.
|zstartall||zstartall test|prod||Script file zstartall is used for starting all interfaces that are not currently active and are turned on in the ptrkeys table.|
|zstopall||zstopall test|prod||Script file zstopall is used for stopping active interfaces.|
|zpurge||zpurge test|prod||Script file zpurge is used for clearing processed transactions out of the idata table and reorganizing (vacuum) the database.|
Data acquisition programs are responsible for receiving data, updating the idata and ptrkeys database tables, acknowledging the sending system and signaling appropriate delivery programs. Acquisition programs also remove any framing characters from the inbound transaction before updating the idata table. The included sample program is written in C++.
A data acquisition program works as follows:
Data delivery programs are responsible for reliably delivering data and updating the ptrkeys table.A data delivery program works as follows:
On Unix computers, cron performs automating functions according to the time and command that are specified by the user. The cron table, crontab, uses a format consisting of six fields in this order: minute, hour, day, month, day of week, and command. Each field must be filled with either a number or letter, or left unspecified with an asterik.
As user zwebit, load crontab with the following information:
# crontab file # # clean old transactions out of webit databases 0 1 * * * /usr/local/webit/bin/zpurge # # # auto start any down links every 5 minutes 0 * * * * /usr/local/zwebit/bin/zstartall prod 5 * * * * /usr/local/zwebit/bin/zstartall prod 10 * * * * /usr/local/zwebit/bin/zstartall prod 15 * * * * /usr/local/zwebit/bin/zstartall prod 20 * * * * /usr/local/zwebit/bin/zstartall prod 25 * * * * /usr/local/zwebit/bin/zstartall prod 30 * * * * /usr/local/zwebit/bin/zstartall prod 35 * * * * /usr/local/zwebit/bin/zstartall prod 40 * * * * /usr/local/zwebit/bin/zstartall prod 45 * * * * /usr/local/zwebit/bin/zstartall prod 50 * * * * /usr/local/zwebit/bin/zstartall prod 55 * * * * /usr/local/zwebit/bin/zstartall prod # # # # Check monthly for seqno reset 0 3 15 * * /usr/local/zwebit/bin/zresetseqno prod
Communication programs send and receive data in a predefined pattern of data. For example, HL7 (healthcare level 7) transactions are usuallly formatted with the first byte of the transaction as 0x0B and the last two bytes of the transaction as 0x1C and 0x1D. With a predefined format, communication programs have a marker for knowing where a transaction begins and ends. Framing characters are not stored internally in the idata table. Class zString has functionality for handling framing characters.
Configuration files are used for providing configuration information to programs. Each interface has its own configuration profile that is processed at startup. To edit an interface profile, use a text editor. Profiles are standard text files with keyword value pairs. Class zProf is used by zWebit programs to read configuration files and gather needed run time data at startup.
Like many programs operating in a Unix/Linux environment, zWebit uses log files for monitoring ongoing activity. The zWebit logs are configurable, and they roll. For example, a program could be set to have four log files, each 200K in size. When the first log file fills, it rolls to the second log file, then the third, and finally the fourth. When the fourth log file fills, it rolls back around to the first log file. zWebit appends a number between 1 and n to the logfile name and links the base logfile name to the current open file.
All log entries are prefixed with a current time stamp. The default number of logfiles is two, but this number can be changed with zLog::zlSetCount ( ). The default logfile size is 50,000 bytes, but it can be changed using zLog::zlSetLogSize ( ).
There is an internal buffer used by the zLog::zlWriteFmt ( ) method that is 1024 bytes as default. If the program using this class will be logging data longer than 1024 bytes with the zLog::zlWriteFmt ( ) method, the buffer size can be adjusted with zLog::zlGetBufSize ( ) and zLog::zlSetBufSize ( ). If the buffer is not big enough for the data in zLog::writeFmtLog ( ), it can cause a buffer overflow, which can crash the program.
When programs are first started, they may log messages to the system log (/var/log/messages) if the programs encounter initialization errors and are not able to log messages in the zWebit log files.
zWebit uses standard Unix pipe structures for interprocess communication. After a data acquisition program receives a transaction and updates the idata table with the transaction, the program notifies the delivery programs about the data via pipe messages.
zWebit includes and uses the zPipe class to handle interprocess communication. Each data delivery program has a unique pipe assigned to it. When data arrives, data acquisition programs send delivery programs a wake up call via method zPipe::writePipes ( ). After the delivery program performs its task, it returns to sleep and waits until it receives another message.
The general idea is that each data delivery program has one and only one pipe associated with it. Any time a data acquisition program captures data, it will send a message to the appropriate delivery programs. When a delivery program receives a wake-up message via pipe, the program determines which transactions it needs to process and processes them sequentially (according to the order in which they were received). Once its tasks are finished, it returns to sleep and waits for another pipe message.
The zPipe class structure is designed so that many processes can write to the same pipe, and a single process can write to many pipes as well. However, a process only reads from a single pipe.
This class is designed for one zPipe instance in each program. Multiple instances of this class would result in undefined behavior in the program due to multiple copies of interrupt handler logic.
Data acquisition programs may need to notify several delivery programs. Acquisition programs need to make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ) for each delivery program.
Data delivery programs are assigned one pipe each. Delivery programs make one call to zPipe::SetPipePath( ) and one call to zPipe::AddPipeToList( ).
zWebit treats each interface transaction as one string of data. String data is defined as one or more contiguous bytes of data with a Null (0x00) terminating end byte.
zWebit includes a string parsing class called zString. Newer versions of C++ include a string class. When this software was initially developed, there was not a standard portable C++ string class available, so this class was created and has been carried forward.
Healthcare data is frequently passed between systems using a special format called zHl7 (healthcare level 7). If you are using zWebit in an environment other than the Healthcare Industry, then this class will not be needed. If you are using zWebit in a healthcare environment, the zHL7 class can be used for manipulating HL7 formatted data in C++ programs.
The zHL7 routines basically work by treating the entire transaction as one string of data, and applying seek and find or seek and update logic on the string.
The zHL7 class does not check for HL7 validity and is not as powerful as some other HL7 parsing tools available. However, it has proven adequate for zWebit, and it is extensible as well.
zWebit is not exclusive to the Healthcare industry alone. In fact, it can be utilized for most endeavors concerning interface communication and data manipulation. This is because zWebit is capable of using other classes or tools for modifying data.
|webit/bin||Contains all webit shell programs and script files|
|webit/doc||User directory for string interface documentation specification|
|webit/source||Contains subdirectories of source code|
|webit/test||Contains subdirectories for the test system interfaces|
|webit/test/config||Contains test system configuration files|
|webit/test/logs||Contains test system log files|
|webit/test/pipes||Contains interprocess communication pipe files for the test system programs|
|webit/test/data||Miscellaneous test files|
|webit/prod||Contains subdirectories needed for the prod system interfaces|
|webit/prod/config||Contains prod system configuration files|
|webit/prod/logs||Contains prod system log files|
|webit/prod/pipes||Contains interprocess communication pipe files for the prod system programs|
|webit/html||Contains html and php scripts used for monitoring the interface programs|
|webit/sql||Contains sql files neede to create and maintain database tables|
|webit/zzz||If you need to set up interfaces with a name other
than test or prod, create zzz/config, zzz/logs, and
(zzz = name of your database)
This directory structure is a sample setup. After you get the idea of how zWebit works, you could use different interface system names (instead of test and prod) and modify the directory structure to include the new names. The directory names (test and prod) match the database names used in Postgres.
|[ <- Previous Page ]||[ ^ Table Of Contents ^ ]||[ ^ Top Of Page ^ ]||[ Next Page -> ]|
Visit the GNU home page.
FSF & GNU inquiries & questions to email@example.com.
Comments on these web pages to firstname.lastname@example.org.
Copyright (C) 2003 HealthCare Systems and Consulting
Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
Last updated: 07/21/2003