Regression Testing

The intent of the procedure described below is to find new bugs or unexpected side effects in the code after a release lifecycle is completed.

Goal: Compare two files obtained from RedCAP after two separate imports are performed using the production and the candidate versions of REDI software.

Note: The login credentials for the RedCAP web application on the testing VM at http://localhost:8998/redcap/index.php are: admin/password

Steps

  1. Obtain the candidate version (redi_B) of the software

    $ git clone git@repo_redi redi_new
    $ cd redi_new && git checkout develop
    
  • Clone the config repository at ./config and checkout the tag that matched the REDI version

    git clone git@repo_config config cd config && git checkout develop

  • If needed, revise the settings.ini lines that identify the host and authorize access. For this VM, the lines are

    "redcap_uri": "http://localhost:8998/redcap/api/",
    "token": "121212",
    
    $ sed -i 's/^\s\+"redcap_uri.*/\t"redcap_uri" : "http:\/\/localhost:8998\/redcap\/api\/",/' settings.ini
    $ sed -i 's/^\s\+"token.*/\t"token" : "121212",/' settings.ini
    
  • Go to the vagrant folder and remove the old virtual machine (VM) instance if necessary:

    $ cd vagrant && vagrant destroy
    
  • Start a fresh VM:

    $ vagrant up

The expected output from this command should look like:

redcap.zip content indicates Redcap version: 5.7.4
Setting the connection variables in: /var/www/redcap/database.php
Checking if redcap application is running...
   <b>Welcome to REDCap!</b>
  • If you need to preserve the database state then you can create a backup of the REDCap database before we insert any data by running the redcapdbm.php script on the VM:

    $ vagrant ssh -c ‘cd /vagrant/scripts && php redcapdbm.php -b’

Note: The commands above create a file like backup-redcap-YYYYmmdd-HHMM.sql on the VM

  • Erase the data in the REDCap instance using redcapdbm.php. First, enumerate the REDCap projects

    $ vagrant ssh -c ‘cd /vagrant/scripts && php redcapdbm.php -l’

    The following projects are currently available in the redcap database:

    project_id: 1, project_name: redcap_demo_cda700 project_id: 2, project_name: redcap_demo_f3746b project_id: 3, project_name: redcap_demo_117155 ... project_id: 12, project_name: hcvtarget_20_development ...

Erase the data in the correct project if necessary:

$ cd ../vagrant && vagrant ssh -c 'cd /vagrant/scripts && php redcapdbm.php -d 12'

 Deleting data for project: 12, name: hcvtarget_20_development
 Rows deleted from `redcap_surveys_response`:0
 Rows deleted from `redcap_surveys_participants`:0
 Rows deleted from `redcap_surveys_emails`:0
  • If needed, load a minimal set of data to get research identifiers into REDCap. Use the redcap_records utility to load this data.

    $ redcap_records --token=121212 --url=http://localhost:8998/redcap/api/ -i demographic_test_data.csv
    

On success the following text is returned:

{u'count': 5}

  • Upload raw.txt file to REDCap using redi_B

    Note: You may want to increase the number of requests allowed for processing before proceeding with data upload.

    Open the address "localhost:8998" using your browser
    
    Login to REDCap and then click on "Control Center" tab
    
    Click on the "Security & Authentication Configuration" link on the left menu
    
    Find and adjust the "Rate Limiter" field to something like 60000
    
    
    $ python ../redi/redi.py
    

If the output from the command above produces an exception then check if your IP was not banned due to numerous requests sent (@see related code in Config/init_functions.php: checkBannedIp() and storeHashedIp())

select * from redcap_ip_banned where ip = '10.0.2.2';
+----------+---------------------+
| ip       | time_of_ban         |
+----------+---------------------+
| 10.0.2.2 | 2014-06-06 18:59:25 |
+----------+---------------------+

To fix this, use these SQL commands:

update redcap_config set value = 600000 where field_name = 'page_hit_threshold_per_minute';
delete from redcap_ip_banned;

If the token is invalid the following error is returned:

Cannot connect to project at http://localhost:8998/redcap/api/ with token 121212

  • Download relevant forms from REDCap using a command like:

    $ redcap_records --token=121212 --url=http://localhost:8998/redcap/api/ -f "demgraphics chemistry" > output_B.csv
    

If you have a lot of forms, the output comparison is easier if you export the forms separately like this:

#!/bin/bash

batch=$1
forms="demographics chemistry cbc inr hcv_rna_results"
if [ ! -e $batch ]; then
    mkdir $batch
fi

for form in $forms
    do
      redcap_records --token=121212 --url=http://localhost:8998/redcap/api/ --forms=$form > $batch/$form.csv
    done

Later do the diff like this:

diff -ur a/ b/
  • At this point we have gathered the output from the release candidate software redi_B. If we there is reference output file available than we can just compare the outputs:

    $ diff -u output_A.csv output_B.csv
    
  • If there is no reference output file available than we have to get a previous version redi_A and generate it.

    Erase the REDCap data first:

    cd ../vagrant && vagrant ssh -c 'cd /vagrant/scripts && php redcapdbm.php -d 12'
    
  • Obtain the reference version (redi_A) of redi software.

    $ git clone git@repo_redi redi_old
    $ cd redi_old && git checkout TAG_ID_OLD
    
    $ git clone git@repo_config config
    $ cd ../config && git checkout TAG_ID_OLD_CONFIG
    
  • Repeat steps 8-10 with redi_A software with the only difference being that the output file is changed to output_A.csv on step 10.

  • Compare the files output_A.csv and output_B.csv to insure there are no differences or expected differences are present:

    $ diff -u output_A.csv output_B.csv
    

If no new behavior was introduced the output from the command above should be an empty string.