Architecture

The CSVUploader is located under hd-connect/csvuploader. It uses both hd-connect-csvuploader and hd-connect-proxy modules.

The general architecture of the CSVUploader is explained in the sequence diagram below.

Third-party libraries and frameworks

• Apache Camel: https://camel.apache.org/
• Spring Boot

Testing and working

• The CSVUploader creates a folder at root level (SFTP for end-user, or hd-all for developer) that contains a subfolder per existing organisation.
• The CSVUploader will poll with a delay of 1 min, process the csv file and then create 3 folders:
  • ARCHIVE folder: contains the source csv file.
  • RESULTS folder: contains the results of processing the csv file. This file contains the specified data, and the final status of the processing: Success or Error. If an error occurred, the error message is displayed. For multiple uploads, the result is added to the end of the result file each time.
  • ERROR folder: this folder is created if the csv test file was not processed, due to an I/O error (file corrupted, not found etc.). So for now, only technical errors are caught and the source csv file is moved to that folder instead of the ARCHIVE folder. In principle, this folder should contain any result that is an error. The RESULT folder should only contain results that end with a SUCCESS status.

Formats

Some formats are specific:

• Dates: should be dd/mm/yyyy
• Boolean: true / false
• Codes: the value of the code (not the translation)
• Multi codes: there is only one column per field. So when a select box is set as multiple, values have to be separated by a "|". e.g.: 68452|68453|68454
• Repeatable blocks: in some DCDs, a complete block of fields is repeatable. In that case, value have to be separated by a ";"
  • e.g.: A block is containing 3 fields: A (Lob), B (Type klep) and C (Aantal kleppen)

Examples:

In case of a multiple choice:

Example of a multiple choice as presented in the DCD:

Process to Verify and place the file for CSVUploader processing for DCD's

• Go to the Shared folder(Network Drive within Sciensano) for which csv processing job will get the file(DevOps team will provide this path).
  Generally, path should be :- "/data/csv/"
  

• In this folder we can see the different DCDs that can be used to upload the CSV file(s).
• Go to the relevant DCD where the data has to be processed and double click on it to open the folder.
• In this folder we can see the different subtype of DCDs:
  • For Orthopride DCDs(Hip, Knee and Total Femur) that will be:
    • Primo-implantation
    • Revision
    • Resection
  • For Endobronchial valve (Zephyr) that will be:
    • Primo-implantation
    • Replacement
    • Follow-up
  • For Hand hygiene that will be:
    • PRE campaign
    • POST campaign
    • IN & OUTSIDE campaign
  • For Surgical and percutaneous heart valves that will be:
    • Implantation
    • Follow-up
  • For Spine Tango that will be:
    • Intake
    • Conservative treatment
    • Surgery
    • Patient questionnaire
• Open the relevent DCD subtype:
  (In this example we will take Orthopride Knee)

Uploading the csv file into the correct DCD folder

The main folder structure of the csv upload folder will look as such:

Each folder consists of sub folders which contains the DCD.

Find below an overview of the DCD's that can be uploaded.

• Orthopridetotalfemur-9:
  • Primo-implantation
  • Revision
  • Resection

The csv files will be placed in the folders of the DCD which we want to upload. We double click on the folder of the dcd and the structure will be either empty, whether filled with folders containing the following names:

• ARCHIVE (after a csv file has been processed, the original csv file will be saved in this folder)
• RESULT (when the csv file has been processed, a file will be created or append with the result of the upload of the csv file)
• ERROR (when the csv file contains erroneous formatting, the csv file won't get processed and an error file will be created or append with the errors and reason why the csv file couldn't be processed)

I want to upload an Orthopride Knee Primo-Implantation DCD, so I will place the csv file into the dcd-21-v-1-kneePrimoImplantation folder:

• Open the dcd-21-v-1-kneePrimoImplantation folder

• Put the orthopride knee primo implantation csv file into the folder

• Wait until the file has been processed (the file will disappear from this folder if it has been processed - don't forget to click on the refresh button)

• Go into the RESULT folder and refresh the folder to update the file with the latest changes

• Double click the file to open it and to read the result of the upload process

The status is the most important line that indicates whether the upload was a success or not.

Verification of the upload process is also possible in the application HD4DP2.

After login to HD4DP2 go to relevant DCD and check the processed data.