caArray FAQ2

caArray FAQ: How Do I Work Around The 2 GB File Size Limit When Uploading Experiment Data?
When uploading experiment data from large microarrays into caArray, you may find that size of your data archive is larger than the application's individual file size limit of 2 GB. This article presents a simple workaround which allows you to break down your data into smaller, more manageable chunks that can be individually uploaded without violating the 2 GB limit.
Overview
Your experiment data consists of multiple IDF metadata files, each of which is associated with one or more SDRF metadata files. Each of these SDRF files, in turn, is associated with one or more raw array data and derived array data matrix files. In this tutorial, the array files we use are in the Agilent TXT and TSV formats; the file formats for your data may differ.
We can break down this collection of files into more manageable chunks, each of which consists of an individual IDF file and all its associated SDRF, TXT, and TSV files. Each of these chunks can be packaged into a single ZIP archive and uploaded individually.
Prerequisites
This tutorial assumes that you have past experience and basic familiarity with uploading data into caArray. Specifically, it assumes that you have already created an experiment for your data, uploaded the corresponding array designs, and associated the experiment with those designs. In case you lack a basic background on uploading caArray data, please refer to the official caArray User's Guide on the NCI wiki at https://wiki.nci.nih.gov/x/LBo9Ag.
You must have all your experiment data readily accessible on your computer (i.e., not archived or compressed). The data should preferably be consolidated into a single location (i.e., a folder containing every single IDF, SDRF, TXT, and TSV file from the experiment). You will also need an archive creation utility installed on your computer. In this tutorial, we will use WinZip (www.winzip.com), but any comparable utility with support for the ZIP format will do.
Getting Started – Preparing Data for Upload
In preparing your data for upload, the first step is to find all the files associated with a given IDF file. To so, open any of the IDF files from your experiment in Microsoft Excel or another application suited for viewing tab-limited data. The partial screenshot below shows the first of twelve IDF files from our example experiment as viewed in Excel.

Most of the content of this file is irrelevant to our goal here and beyond the scope of this tutorial. Of interest is the bottom row with the field 'SDRF Files' and its corresponding value, the name of the SDRF file that is associated with this IDF file.
The IDF file in this example references a single SDRF file, but your IDF file may reference many, with each SDRF filename listed in its own row. Whether your IDF references one SDRF or many, make a note of all the SDRF filenames, as we will need to open each of them individually. The two partial screenshots below show the SDRF file that was referenced in our IDF file, as viewed in Excel.

As with the IDF, most of the SDRF content is irrelevant to our goal here and beyond the scope of this tutorial. Of interest are the two columns headed 'Array Data File' (upper screenshot) and 'Derived Array Data Matrix File' (lower screenshot). These columns list the filenames of all the TXT and TSV files, respectively, from the example experiment. (Note that both of these columns are from the same file – they are shown in different screenshots only because the file is too large to be displayed in a single screenshot.)
Make a note of all the TXT and TSV filenames, as we will need to include each of them in the data archive that we upload to caArray. The easiest way to note these filenames is to copy them from the SDRF file and paste them into a plain text file, which you can print for reference.
Creating the Archive
Now that we've identified all the files associated with our IDF, our next step is to creative an archive of them. Launch WinZip, click the 'New' toolbar button, and enter a name for your archive in the 'New Archive' dialog. We'll call ours 'upload.zip', as shown below.

Once we've created the archive, we can now add files to it. We can refer to our previous notes of all the filenames associated with our IDF file. In our example, the archive will consist of a total of 42 files: one IDF, one SDRF, 20 TXT, and 20 TSV files. We can select these files in the 'Add' dialog as shown below. (Hint: Hold down the CTRL key to select multiple files).

(WARNING: After you've created the archive, ensure that the resulting file size is less than 2 GB. If it isn't, you will have to re-create the archive with a higher compression ratio. In our example, the size of the 'upload.zip' archive came out to approximately 900 MB, as shown below, so the file is ready to upload as is.)

Uploading the Archive
To upload the archive, first log in to caArray and navigate to the experiment you will be upload your data into, then select the 'Data' tab, followed by the 'Manage Data' tab beneath it. Now click on the 'Upload New Files' button as shown below.

A new pop-up window entitled 'Experiment Data Upload' will appear in your Web browser, prompting you to upload files. Click on the 'Browse' button, then select the 'upload.zip' archive we created previously from the Open dialog as shown below.

Back on the 'Experiment Data Upload' window, make sure that the box labeled 'Unpack Compressed Archive' is checked, then click on the 'Upload' button to begin uploading the file.

Depending on the size of the archive and the performance of the server your experiment is stored on, it may take anywhere from five to 20 minutes (and possibly longer!) for the archive to upload. Remember to keep the upload window open throughout the entire upload process, even after the blue progress bar has reached 100%.

You'll know when the upload is complete when you see a new window overlaid over the upload window with the message 'Your file upload is complete', as shown below. Click the 'OK' button below this message, then click on the 'Close Window' button behind it to return to the main experiment window.

Verifying the Archive
Back in the main experiment window, the contents of the archive we just uploaded are now listed under the 'Manage Data' tab. The TSV matrix files are considered supplemental, so we will move them to the 'Supplemental Files' tab by checking off all the TSV files in the list, then clicking on the 'Add Supplemental Files' button below.

These TSV files now appear under the 'Supplemental Files' tab, alongside other TSV files from a previous upload to the same experiment.

Back on the 'Manage Data' tab, the remaining files from our upload are one IDF, one SDRF, and 20 TXTs (only the first three of these files is shown below due to space constraints). Note that the status of the TXT file from the screenshot (and of all other TXT files in the list) shows as 'Unknown', which means that caArray did not automatically recognize the file type. As a result, we will have to manually specify the file type by selecting all the TXT files in the list, then clicking the 'Change File Type' button below.

For the particular experiment in this example, the array data files are in the Agilent Raw TXT format. To specify this, in the 'Manage Files' window shown below, select 'Agilent Raw TXT' from the 'Select New File Type' drop-down list, then click on the 'Save' button above it.

Back on the 'Manage Data' window, the status of all the TXT files now shows as 'Agilent Raw TXT', indicating that caArray now correctly recognizes the file type.

Our next step is to validate all the files, which we will do in two passes: the first pass, validating only the TXT files (but not the IDF and SDRF), and the second pass, validating all the files (IDF, SDRF, and TXT). For the first pass, check off all the TXT files in the list, then click the 'Validate' button below.

The page will now refresh with the updated status of the files showing as 'In Queue'. Depending on the size of your array and the performance of your server, the TXT files may take several minutes to validate, so be patient. To further complicate matters, the page will not automatically refresh once the files have finished validating, so you will have to manually refresh the page periodically yourself using your browser's 'Refresh button' until the file status updates again.

You'll know when the first pass validation is successful when the status for TXT files shows as 'Validated, Not Parsed'. (NOTE: The 'Not Parsed' status doesn't indicate any problem with your array data; caArray still has not implemented a parser for the Agilent TXT format and is thus unable to parse these files. The files can still be imported into your experiment without being parsed.)

For the second pass, check off every single file on the 'Manage Data' page (IDF, SDRF, and TXT), then again click the 'Validate' button below.

The page will again refresh with the files' status showing as 'In Queue'. After a few minutes, click the browser's 'Refresh button' until the file status updates again.

You'll know when the second pass validation is successful when the status of the IDF and SDRF files shows as 'Validated' and the status of the TXT files shows as 'Validated, Not Parsed'. Once you've reached this stage, you can import the validated files into the study by checking all the files in the list, then clicking on the 'Import' button below.

The page will again refresh with the files' status showing as 'Importing'. After a few minutes, click the browser's 'Refresh button' until the file status updates again.

You'll know when the importing is successful when the uploaded files no longer appear under the 'Manage Data' tab, with a message stating, 'Nothing Found To Display' in their place, as shown below.

The files now appear under the 'Imported Data' tab, as shown below, with a status of 'Imported'. Note that other, previously uploaded files from the same experiment appear under this tab as well alongside the files we just imported.

Reproducing the Procedure
So far, only one-sixth of the data has been uploaded. You can reproduce the procedure we followed so far to upload the data from your experiment. The procedure, summarized below, is as follows: