CLARK v1 User and Technical Documentation

CLARK v1 is a machine-learning classification software created by NC TraCS and CoVar Applied Technologies to enable computable phenotyping in unstructured data. CLARK’s user-friendly interface makes natural language processing (NLP) an accessible option for searching free-text clinical notes. This page includes user instructions and technical documentation for CLARK v1.

For instructions on CLARK v2, go here.

For a conceptual guide to CLARK including research applications and interpretation of results, go here.

Both CLARK v1 and v2 are free and available for download here.

Table of Contents

Getting Started
      System Requirements
      Installation
      CLARK: Basic Steps
      Navigation
      Loading and Saving Progress

Key Concepts
   Clinical Notes
      Formatting
      Metadata
      Free Text

   Regular Expressions
      Basic Regular Expressions
      Section Break
      Clinical Examples

Training Corpus
      Loading the Training Corpus
      Troubleshooting

Features
   Algorithm Setup
      Training Corpus
      Regular Expressions Library
      Active Regular Expressions
      Sectioning
   Notes
      Patients and Notes
      Note with Additional Markup
      Using the Notes Viewer

Algorithm
      Algorithm Steps-Training Corpus
      Algorithm Steps-Evaluation Corpus
      Machine Learning Classifiers
      Cross-Validation

Explore
      Distribution by Labels
      Confidence
      Filtered Records
      Evaluation Corpus Results
      Exporting Results
      Sensitivity and Specificity

Technical Appendix
      Cross-validation
      Algorithms in detail
      General Troubleshooting

Getting started

System requirements

CLARK runs best on Windows machines with 16 GB of RAM, and does not require special infrastructure to operate. Processing may take longer with 8GB of RAM.

Installation

CLARK can be downloaded from tracs.unc.edu after creating a free account with NC TraCS. Follow the instructions under the “Sign In” menu or click here to create an account. The files need to be un-zipped using an application such as winzip or 7zip. Simply double-click the CLARK Installer 1.0.3.exe, and read README.txt and the license agreement. CLARK opens automatically once the installation finishes.

Figure 1 (hover to animate): Load a set and view individual notes in CLARK to get started.

Using the Animal Dataset CLARK comes with a few example files to practice with: AnimalCorpus_V2.json, AnimalCorpus_asMRN.json, AnimalKeywords.json, and AnimalExpressions.json. The AnimalCorpus_V2.json and AnimalCorpus_asMRN.json mimic labeled clinical notes, where each animal represents a patient. In AnimalCorpus_asMRN.json, IDs are represented with medical record numbers (MRNs) instead of animal names, and more metadata fields are included. This documentation uses examples from AnimalCorpus_V2.json. Both files can be used in CLARK and contain the same “notes.” AnimalKeywords.json contains input for the Regular Expressions Library, and AnimalExpressions.json can be uploaded to the Active Regular Expressions tab in CLARK.

CLARK: Basic Steps

Generally, CLARK can be used in the following steps:

1. Form a classification question, then identify groups and group-defining features of interest through a literature review.

2. Select a training (“gold standard”) corpus and evaluation (unlabeled) corpus of clinical notes. Gather, process, and load clinical notes into CLARK on the Training Corpus page.

3. Format features (words, phrases, or values) as regular expressions to match text in the body of clinical notes.

4. Iteratively train and assess algorithms and combinations of features until satisfied with CLARK’s performance on the labeled training data.

5. Transfer algorithm of choice to the unlabeled evaluation corpus.

6. Review, interpret, and use classification results in context.

Navigation

CLARK contains several “pages,” each covering a different step in the analysis process: Training Corpus, Features, Algorithm, and Explore. These pages, their sections, and sub-sections will be described in detail throughout this documentation. When a page’s sections are completed in the CLARK environment, a green checkmark appears next to the page header. Users can (and should) return to previous sections throughout the analysis process to make modifications.

Loading and saving progress

Click More at the top of the CLARK environment to reveal a drop-down menu with options for saving and loading work.

Figure 2: Re-start, load, or save work under “More”.

Saving a session preserves regular expressions in use, clinical notes in use, and current algorithm results. Sessions are saved as .zip files. Previously saved sessions can be loaded from the More menu or on the home screen. There is no need to un-zip a saved session to load it into CLARK again. The contents of the .zip file can be viewed locally, but the corpora and regex files are compressed into a single .json. Regular expressions created/updated in CLARK can be saved individually from the Training Corpus page. To save the results of several algorithms for one set of clinical notes, save the sessions separately.

When using CLARK to explore clinical data, be mindful that the saved session will contain identified protected health information (PHI). The session should be saved in a location approved at your institution for storing PHI.

Key Concepts

Clinical Notes

Clinical notes contain free-text patient data ranging from family history to current symptoms and imaging results. A set of clinical notes is called a corpus. CLARK requires two different corpora for a complete classification: the “gold standard” training set with known patient classifications (labels), and the unlabeled “evaluation” set. Machine learning algorithms are trained first using the gold standard data, then applied to the unlabeled evaluation set for classification. For use in CLARK, these notes must be converted to a specific format that renders a whole set usable and searchable.

Creating Gold Standards
Researchers can build a gold standard corpus through manual chart review for a subset of the patient population, leaving the rest of the population unlabeled for the evaluation corpus. Each patient should have one distinct label, and patients of the same group should have identical labels. For example, CLARK recognizes “diabetic”, “diabetes”, and “T1D” as different groups; however, these terms indicate the same characteristic and should share a common label in the corpus.
Gold standard labels provide CLARK’s classification options for the evaluation corpus, so all possible groups should be represented in the training data. The size of a gold standard set depends on several factors: analysis goals, rarity of condition of interest, population size, and desired confidence level for classifications.

Formatting

When first extracted from a database, clinical notes vary in format across institutions. A corpus must be saved as a .json file

Figure 3: Sample formatting for 1 entry in a corpus.

in which a single note is enclosed in { } brackets, and the whole corpus is enclosed with [ ] brackets and fields are separated by commas. For clarity in this example, each field is on a separate line–this is not necessary in practice, as long as entries are properly separated with punctuation. The name and content of each field is enclosed with quotations, and a delimiting colon populates the field with the content.
All fields other than “note” will populate CLARK as structured metadata. Here is a sample script for formatting clinical notes in R.

Metadata

The Metadata header for notes in CLARK includes limited structured data for each subject in a corpus. Metadata is visible in the Note with additional markup pane. MRN, noteID, noteDate, and label are required. Other metadata fields, such as gender and age, are optional, and some notes may have these extra metadata while others do not. The Animal Corpus example includes the following fields:

Field	Description	Example	Required?
noteDate	The date and time of the clinical note. This distinguishes several patient notes from each other.	“2016-11-13 19:45:00”	Yes
note	The body of a clinical note	free-text formatting	Yes
noteID	Distinct note ID, possibly including several per MRN.	“1”	Yes
label	Classification label for a patient. Each distinct label in a training corpus is a possible group for patients in the evaluation corpus.	“mammal”	Yes*
MRN	Patient medical record number or other identifier	“Aardvark”	Yes
noteCount	Counter of notes within a patient	“1”	No

Figure 4: Metadata section visible in CLARK

• The label* field should always be included in both corpora, but left blank for the evaluation corpus.
• A CLARK-ready corpus includes each note as a separate entry, and notes require correctly populated metadata to be linked together.
• The MRN connects several notes under the same patient so that, during classification, matched features are attributed to the correct subject.
  
• The noteID field should be distinct for each note, especially if there are several notes per patient. Each note should inherently have a different date/time associated, but CLARK uses noteID to separate them.

Free Text

The “note” field is the only non-metadata field in a corpus, and this is where CLARK uses NLP to search for matched features. Text in the note fields may be just that: a block of text. However, sections from clinical notes can be preserved through syntax in .json files. Since notes can be viewed within CLARK, sectioning may be worthwhile for readability. The following note has sections that are distinct in CLARK, and these can be excluded because the headers are specified in the note and then recognized by regular expressions.

Figure 5: Free-text clinical note entry including section breaks in a .json file

Figure 6: The entry from Figure 5 in CLARK.

• Figure 5 is a simplified entry of a “corpus.json” with sections defined in the “note” field. The “\##” expression in the .json file causes a header to display in CLARK. “\##” also allows regular expressions to match a section header for optional exclusion, and “\n” creates a line break after the name of the header.
• The number of “#” only affects the size of the header, and each header is recognized by CLARK and the built-in section break regex as the same. A clinical note can contain any number of sections.
• Figure 6 displays the note “as is” under the “Note with Additional Markup” section of the CLARK environment.
• To see the sections and “#” highlighted, navigate to the Sectioning tab in the note-viewing pane.
• Section headers will be different sizes depending on the number of “#” in the sequence before the header in the clinical notes.

Regular Expressions

Basic Regular Expressions

A regular expression (regex) is a string of letters or numbers that uses additional special characters to define a search pattern in a body of text. The purpose of using regular expressions in CLARK is to robustly identify features (words, values, or phrases) in clinical notes that distinguish patient groups of interest from one another.

• A single regular expression can match several words. For example, “\bfly?(ies|ight|ing|)\b” matches “fly”, “flight”, “flying”, and “flies” by offering alternate suffixes to “fly”, and the “?” qualifier makes “y” optional in the prefix.

Writing regular expressions does not require any text-mining expertise and can be learned through online tutorials. Regex101.com provides a useful interface to practice using regular expressions, and RexEgg.com includes an in-depth tutorial to help develop more complicated expressions. Note that CLARK uses python “flavored” regex. This is generally the same as other regex flavors, and more information can be found here.

General Regular Expression Examples

RegEx	Purpose	Example	Matches
“(?i)”	Searches case-insensitive	“(?i)heart”	“HEART”,“Heart”,“heart”
“\b”	Encloses any word or phrase	“\bhigh BP\b”	“high BP”, will not match “high”
“[abc]”	A single character a,b,c	“arm[sy]”	“arms”,“army”
“\d”	Single digit 0-9	“\d\d\d”	“456” or "333
“\D”	Anything not a digit	“\D\D\d”	“AB1”
“.”	Any character except line break	“…”	“abc” “a c”
“*”	Zero or more times	“a*h*”	“ah” “aaaaahh”
“\”	Some characters have special functions in regex. To search for the literal character as text, precede with a backslash.	“1\.5\+1\?”	“1.5+1?”

More examples of regular expressions can be found in the AnimalKeywords.json and AnimalExpressions.json files that come with the Animal Corpus example. Note that in the .json files, the backslash should be duplicated any time it is used. For example, the word boundary “\b” should be entered as “\\b” in a .json file.

Section Break

Figure 7: Section Break tab within CLARK

Once notes are loaded into CLARK, Sections are identified using the “Section Break Regex”, found under the Section Definitions tab on the Training Corpus page. This regular expression recognizes any word (requiring user input) functioning as a header. Altogether, it matches any number of consecutive #, followed by any number of consecutive characters that are not a space.

• # tells CLARK that a header is present, and #+ allows CLARK to match any number of consecutive #.
• The [] brackets enclose optional matches, in this case ^ . The ^ is translated as “not”, and preceding a space, [^ ] means “not a space”.
• The * qualifier means “matching as many times as possible” what is enclosed in [] brackets.

Instructions for including or excluding specified sections can be found here.

Clinical Examples

There are endless possible regular expressions that can match words and phrases in clinical notes. Below is a sample of features that could be used to classify some common conditions.

Clinical Regular Expression Examples

Name	RegEx	Matches
Obesity	(?i)\b(?<!not )obes(ity|e)\b	“Obese”,“obesity”
Tobacco	(?i)\b(?<!non-)(smok(ing|er|es))|(tobacco)|(cigar[est]{0,5})\b	“tobacco”, “smoking”, “smoker”, “cigars”, “cigarettes”
Coughing	(?i)\b(cough[ings]{0,3})|(wheez[inges]{0,3})\b	“cough”, “coughs”, “wheezing”, “wheezes”
High Blood Pressure	(?i)\b(high\s(bp|blood pressure))|(hypertension)\b	“high BP”, “hypertension”, “high blood pressure”

Training Corpus

A CLARK analysis begins on the Training Corpus page. Here, the user loads data (notes and metadata) to be analyzed along with regular expressions.

What is a training corpus? The training corpus consists of “gold standard” data: a set of subjects’ clinical notes with their true labels included. Labels, or groups, are defined in context of the classification question. This set is used to build and assess an algorithm before attempting to classify patients of unknown groups. Training an algorithm is an iterative process, taking several rounds of feature selection and model performance evaluation. The Animal Corpus example that comes with CLARK includes gold standard data for animals of 5 classes: mammal, bird, fish, insect, and reptile. The features selected for this classification (defined in the example regular expressions files) help to distinguish the 5 groups.

Loading the Training Corpus

The training corpus can be loaded on CLARK’s home screen by selecting “Load a Corpus.” A file explorer pops up, then users may navigate to the location of the training corpus and double-click on a .json file to upload it. A green check appears once the notes are successfully loaded, and then they appear in the “Note with Additional Markup” pane.

Troubleshooting

If the training corpus or regular expressions files are not correctly formatted, CLARK will present this error: “Failed to Load Regular Expression Library”. To fix it, pay close attention to the structure of the example animal corpus materials and check that the following criteria in the .json files are satisfied:

• Each expected field is populated. For example, “expr”: “regex” and “name”: “name” for active regular expressions, and the metadata fields for clinical notes.
• The first character in the corpus or active regular expressions files should be an opening bracket “[", and the last should be a closing bracket "]”. The regular expressions library file can begin with “{”.
• Clinical notes are enclosed with “{ }”.
• Fields within notes, and entire notes, are separated by commas.
• The last entry in a .json file should be followed by the closing bracket, not a comma.
  

Features

The Features page allows users to define/upload regular expressions, define sections, and explore notes.

Figure 8: The features page in CLARK

Algorithm Setup

New Training Corpus

Here, users can upload a different training corpus without restarting a CLARK session from the home screen. Similarly to on the home screen, a file explorer pops up to select a “corpus.json” from.

Regular Expressions Library

The RegEx Library stores regular expressions to be used in different classifications. These may be included or suppressed throughout the feature selection process. Regexes may loaded from a file or added directly from the CLARK interface. CLARK accepts a .json file, and its structure differs slightly from those uploaded to the Active Regular Expressions.

Figure 10: RegEx Library in “Algorithm Setup.”       

Figure 9: RegEx Library input as .json

• Features saved in the RegEx Library are not automatically used in the classification algorithm. They must be called in Active Regular Expressions, either in the .json file uploaded to CLARK or added directly in the program. If specified in the .json file, the example in Figure 9 populates CLARK as in Figure 10. The “+” icon adds new features to the RegEx Library which can be saved locally using the floppy disk icon (default format will be CLARK-compatible and .json). Up and down arrows move a feature in the list, and do not affect whether a feature is included in the algorithm. To delete a feature, click on its row, then use “X” to the right of the “+” icon.
  

Active Regular Expressions

Similarly to entries in the RegEx Library, Active RegEx can be added, deleted, edited, arranged, and exported locally. The “X” icon deletes an active feature, and it will no longer be used by the algorithm or highlighted in clinical notes. If a deleted active feature calls on a feature in the RegEx Library, it still appears in the Library. There are three ways a user can add regular expressions to Active Regular Expressions.

1. Load a .json file containing regular expressions
To load a .json file into Active Regular Expressions, click on the folder icon in the Active Regular Expressions tab under Algorithm Setup. An “Active RegEx.json” file can include new regular expressions or reference the RegEx Library. The regular expression and the name must be on different lines, and each must be specified with “expr” or “name”. The “expr” line includes the regular expression with special characters, and the “name” line includes the label for that feature.

Figure 11: Active RegEx tab in CLARK

   Figure 12: Active RegEx input as .json

2. Enter regex directly into CLARK interface

• Use the “+” button to add a row under the Active Regular Expressions.
• Double-click on the empty field under “NAME” to name a regex, and enter the regular expression syntax under “REG.EXP”.
• The “COMPILED” field will automatically populate if the regex syntax is valid. If not, the row will highlight in red until the regular expression syntax is fixed.

3. Add from Regular Expression Library

• To include a feature from the RegEx Library, it can either be referenced in the “Active Regex.json” file, or a reference can be created in the Active RegEx tab.
• In the example from figures 11-12, the “parent” regular expression is defined in the AnimalKeywords.json file, and called to be used in analysis by “#parent” in the AnimalExpressions.json file. When calling a regex from the Library, do not duplicate the regular expression.
• If other regex are defined in the ‘regex library.json’ file, but not in the ‘active regex.json’ file upon loading into CLARK, create a reference under Active RegEx. The reference should simply copy the “name” from the library, and include “#name” in the Reg.Exp field under Active Regex.
• If active regular expressions created within CLARK are also deleted within CLARK, they are not saved automatically to the .json file, but will be saved as part of a CLARK session.

Differences between Active RegEx and the RegEx Library:
The .json file uploaded to the Library has one line per feature. To include these features in analysis, the Active Regex file must reference the desired features by name. To exclude an active regular expression from analysis, it must be deleted from the Active RegEx pane. To exclude a feature that is defined in the Library, it can remain in the library pane as long as it is not called by “#feature” in the Active RegEx pane.

Sectioning

Entire subjects or standalone clinical notes cannot be excluded once loaded into CLARK. However, sections of a clinical note can be excluded across the corpus. This is done using regular expressions in the Section Definitions tab. Sections, if included, are separated by headers in the free-text clinical notes within the corpus. They should already be defined in the corpus.json file before uploading to CLARK. This portion of the documentation explains how to make CLARK recognize and select sections. Instructions for defining sections in clinical notes can be found here.

By default, all sections of a note are searched for features to be used in classification. To match a section with regex that can be included or excluded, add a row to the Section Definitions tab using the “+” button. Type the exact header of the section under the REG. EXP box between the word boundaries, “\b and \b”.

Figure 13. Features found by regex are highlighted in notes.

The section specified will be highlighted in red if not in use, or blue if in use. To toggle section usage, double-click on the box under IN USE and select from the drop-down list. Sections excluded from the training corpus will also be excluded from the evaluation corpus.

When to Use Sectioning
If users expect certain information is irrelevant to classification, sectioning can come in handy. For example, conservation efforts for animals may not tell us anything new about how they are classified. In clinical notes, general family history might not reveal anything new about a condition among all the other information provided. Excluding sections of clinical notes can also speed up the algorithm when there is a large volume of notes or patients to process.

Notes

Patients and Notes

The pane under Training Corpus includes a list of ID numbers (or MRNs) and clinical notes for each patient by date. Scrolling through and selecting a patient ID and note date changes the note in the next pane, under Note with additional markup. By default, the note associated with the earliest date shows up when a patient is selected.