iNTERFACEWARE™ Inc.

Example: Data Scrubbing

One very useful application of scripted transformation is for building a utility to remove confidential data from a set of HL7 messages.

This sample VMD file is a very basic example of how you could begin to remove confidential data from an HL7 message. This VMD file removes Z segments from a stream of HL7 messages and replaces patient name fields with the text string fff.

This VMD file provides a default message containing a Python transformation script. This script looks like this:

current_segment = environment.input_segment_iterator()

current_segment.apply_transform()
output = current_segment.output()

while current_segment.move_one():
   # append each segment which is not a Z segment
   if current_segment.segment_id()[0] <> 'Z': 
      current_segment.apply_transform()  # apply segment scripts
      output = output + current_segment.output()

value = output

Transformation Script

The key things that this script does:

• It uses a segment iterator object to process all the segments, only copying those which are not Z segments, and
• It invokes apply_transform() to invoke segment functions which are used to scrub individual fields in the message.

If you examine the PID-5 outbound script from the PID segment in the GUI, you will see the following script:

# we invoke a global function called scrub_name
# there is a small issue with the first field "value"
# which means we have to get it as the return value of
# function
value = scrub_name(field)

PID-5 Outbound Script

This is where the data scrubbing is occurring. The script is invoking a global function called scrub_name to change the values in the field object. It also assigns the value variable to the return value of this function. The return value of the function is the first sub-field of the field.

The global function scrub_name is defined in the Global Inbound Script window, which you can access by clicking Chameleon > Global Inbound Script.

In this window, you will find the following code:

def scrub_name(Field):
   for FieldIndex in range(0, Field.count_of_subfield()):
      if not Field.subfield(FieldIndex).is_null():
         Field.subfield(FieldIndex).value = 'fff'
   return Field.subfield(0).value

Global Inbound Script

This is the function that takes every populated sub-field and translates it to fff.

Obviously, this is just a start. A well-developed data scrubber would preserve the coherency of the data: always randomizing strings in same standard way by keeping a hash table of random strings, preserving numerical data as numerical data, and so on.

The easiest way to use the data scrubber is to use the msgtransform tool to invoke this scripted transformation on a large batch of messages.

> msgtransform scrub.vmd confidential.txt cleaned.txt

Scrubbing a Data File

The original messages should be in the confidential.txt file. The cleaned data will appear in cleaned.txt.

	To test your message transformation logic, you can use the Message Diff tool to display what fields you have altered.