Transferring FASTQ files to NCBI using FTP

You can either use desktop app for FTP (The File Transfer Protocol ) which are:

Your FTP address, user ID and password will be provided by NCBI. You can simply login  on to desktop app and transfer (deposit) those files from your local computer to NCBI Dropbox.

Alternatively, which was a challenge for me today you can use Linux or Unix transfer files from your supercomputer server to the NCBI server. I am going to give examples of steps to use to  login and transfer those files from your super computer server to NCBI Dropbox.

In my case, I was informed that 90/400 FASTQ.GZ files were ended different format ( binary?!) in NCBI server. I suspected the FTP software that I used in desktop corrupted those 90 files since they look good on super computer server.

You can check those with

zcat example.fastq.gz | head -50

you can also check the file fingerprint ( unique ID) using

md5sum  example.fastq.gz and see if your number are matching with theirs

Anyways, today’s task today I had 180 fastq.gz files ( they gave me the list of files that I need to reupload).

First I needed to copy those files from my folder where I have all fastq.gz files ( almost 800) here is the loop for it:

for the files in $( cat subset.txt); do cp $file /scratch/gizemlevent/FASTQfiles/subsetFASTQ ; done

So what we do here we basically ask the computer to go to subset.txt files ( lists of 90 fast.gz) and copy them into a new folder called “subsetFASTQ” 

After you created this fastq file folder with your desired FASTQ files you login to ftp in our case we use

module load lftp/4.8.4-GCCcore-6.4.0

lftp sra@ftp-private.ncbi.nlm.nih.gov

After this command you will be asked to enter your password and you can reach the Dropbox of NCBI. Make sure before you login their server when you are at the “subsetFASTQ”.And type:
mput *.fastq.gz

This ill be transfer every fastq.gz file you have in that folder into their server.

 

May the luck be with you!

 

Gizem

 

Leave a Reply