Home Forums .NET libraries Xceed SFTP/FTP for .NET FTP of MASSIVE files fails across firewall

Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • User (Old forums)
    Member
    Post count: 23064
    #21182 |

    Hello, my company has multiple domains, where some are inside a dmz firewall, and some are outside.  I am trying to FTP files from outside our dmz firewall, to our local domain.  The problem, I am trying to FTP our database backup files, where one of those 330gb, and a total of 415gb between 8 *.bak files.

    Here’s a little more info, I am using Xceed.Ftp.SendFile().  I have the KeepAliveInterval set at 60 (seconds), and no timeout period set.  Two nights ago, my process failed after 18 hours and 21 minutes.  Last night, it failed after 12 hours 45 minutes.  I am able to do a manual FTP and in total it takes around 2.5 days to do the whole file transfer.

    I have a couple questions about this transfer, first, why does it keep timing out when I don’t set a timeout period and I also have the KeepAliveInterval set?  I posed the question to my network team asking about a Firewall timeout setting, but they have not responded yet.

     The second question, I write to a log file after each of the files is transferred, and the files that DO copy ok before the fatal timeout occurs, the log file still shows that they timed out also, evern though the file is there (Im more confused because that means it’s coming from my catch statement…

     Any help on this would be great.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by ganders (had 6242 views)

    User (Old forums)
    Member
    Post count: 23064

    The default Timeout value is 30 seconds if you do not set it.  So it is possible that it fails between a KeepAlive and a Timeout, though it would be strange.

    Can you paste the code snippet you use to set up the component and do the transfer, along with the communication with the server you can obtain through the TraceWriter property?

    e.g.:

    FtpClient.TraceWriter = new StreamWriter( @”D:\ftp.log”, true );

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by André (had 532 views)

    User (Old forums)
    Member
    Post count: 23064

    Sure, here’s the method that is used to actually do the FTP process…

     

    #region Make sure we are still connected

    if (!ftpClient.Connected)

    {

    ftpClient = CreateClient(ftpSite, ftpUser, ftpPass, writer);

    ftpClient.ChangeCurrentFolder(ftpPath);

    }

    #endregion

    #region

    FTP the file

    WriteThis(writer,

    “FTP’ing the file “ + fInfo.Name.ToString());

    DateTime start = System.DateTime.Now;

    try

    {

    ftpClient.SendFile(file);

    TimeSpan span = new TimeSpan();

    span = System.

    DateTime.Now – start;

    WriteThis(writer,

    ” FTP took “ + span.Minutes.ToString() + ” minutes.”);

    }

    #region

    FTP Exceptions

    catch (FtpAbortException fae)

    {

    WriteThis(writer,

    “FTPAbortException: “ + fae.Message.ToString());

    }

    catch (FtpException fex)

    {

    WriteThis(writer,

    “FTPException: “ + fex.Message.ToString());

    }

    catch (Exception ex)

    {

    WriteThis(writer,

    “Exception: “ + ex.Message.ToString());

    }

    And here’s my CreateClient method…

    private

    static FtpClient CreateClient(string site, string user, string pass, StreamWriter writer)

    {

    FtpClient ftp = new FtpClient();

    try

    {

    http://ftp.Connect(site);

    http://ftp.Login(user, pass);

    string logString = string.Format(“Connecting to: {0} as ‘{1}'”, site, user);

    WriteThis(writer, logString);

    }

    catch (FtpException fex)

    {

    WriteThis(writer, fex.Message.ToString());

    ftp =

    new FtpClient();

    ftp =

    null;

    }

    catch (Exception ex)

    {

    WriteThis(writer, ex.Message.ToString());

    ftp =

    new FtpClient();

    ftp =

    null;

    }

    // This will send a command across the data channel letting the firewall know that we are still

    // in the process of transferring data via the data channel. Without this, the program will timeout while trying to copy

    // the IARTS backup because it is so huge.

    http://ftp.KeepAliveInterval = 60;

    //ftp.Timeout = int.MaxValue – 1;

    return ftp;

    }

     

    I’ll get you the FTP logs after awhile.  I need to start the job so that it will fail.  (I won’t be able to post the logs for at least 12 hours….or tomorrow.

     

    Thanks Andre!

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by ganders (had 3169 views)

    User (Old forums)
    Member
    Post count: 23064

    Andre, I have the log file from yesterday’s “try”…

     > CWD SqlBackups/DailySQLBackups
    < 250 CWD command successful.
    > TYPE A
    < 200 Type set to A.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,4,94).
    > LIST
    Data connection established with 172.28.132.71:1118 on 4/12/2009 @ 2:04:47 AM
    < 125 Data connection already open; Transfer starting.
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,4,95).
    > STOR ACAM_backup_200904112017.bak
    Data connection established with 172.28.132.71:1119 on 4/12/2009 @ 2:04:47 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,4,96).
    > STOR Admin_backup_200904112014.bak
    Data connection established with 172.28.132.71:1120 on 4/12/2009 @ 2:07:52 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    …..
    > NOOP
    > NOOP
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,4,118).
    > STOR APS_backup_200904112016.bak
    Data connection established with 172.28.132.71:1142 on 4/12/2009 @ 2:22:16 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    …..
    > NOOP
    > NOOP
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,4,138).
    > STOR Archive_backup_200904112008.bak
    Data connection established with 172.28.132.71:1162 on 4/12/2009 @ 2:31:38 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    ……
    > NOOP
    > NOOP
    > NOOP
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,5,44).
    > STOR IADS_backup_200904111959.bak
    Data connection established with 172.28.132.71:1324 on 4/12/2009 @ 4:24:27 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    …..
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    < 226 Transfer complete.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,6,27).
    > STOR IARTS_backup_200904111851.bak
    Data connection established with 172.28.132.71:1563 on 4/12/2009 @ 7:10:43 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    ….
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > NOOP
    > TYPE I
    < 426 Connection closed; transfer aborted.
    < 200 Type set to I.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,16,95).
    > STOR Reporting_backup_200904112015.bak
    Data connection established with 172.28.132.71:4191 on 4/13/2009 @ 3:11:05 AM
    < 550 Reporting_backup_200904112015.bak: The specified network name is no longer available.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by ganders (had 3212 views)

    User (Old forums)
    Member
    Post count: 23064

    We notice that you are still setting the KeepAliveInterval to 60 seconds without changing the Timeout value (30 seconds by default), which could cause a problem.  Note that when setting the Timeout value, it must be done before calling the Connect() method, since this value is set on the Socket used for the connection and cannot be changed once the connection is established.

    So we suggest you do :

    FtpClient client = new FtpClient();
    client.Timeout = 120;
    client.KeepAliveInterval = 60;
    … the rest of your connection code.

    And see if it resolves the issue.

    Do you receive FtpException or FtpTimeoutException?  If so, the complete stack trace of the exception could be helpful identifying the issue.

     

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by André (had 3025 views)

    User (Old forums)
    Member
    Post count: 23064

    Here’s the latest attempt after changing the parameters you suggested:

     > CWD SqlBackups/DailySQLBackups
    < 250 CWD command successful.
    > TYPE A
    < 200 Type set to A.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,7,218).
    > LIST
    Data connection established with 172.28.132.71:2010 on 4/26/2009 @ 2:06:46 AM
    < 125 Data connection already open; Transfer starting.
    < 226 Transfer complete.
    > DELE ACAM_backup_200904182018.bak
    < 250 DELE command successful.
    > DELE Admin_backup_200904182015.bak
    < 250 DELE command successful.
    > DELE APS_backup_200904182017.bak
    < 250 DELE command successful.
    > DELE Archive_backup_200904182009.bak
    < 250 DELE command successful.
    > DELE IADS_backup_200904181959.bak
    < 250 DELE command successful.
    > DELE IARTS_backup_200904181851.bak
    < 250 DELE command successful.
    > DELE Notes_backup_200904182017.bak
    < 250 DELE command successful.
    > DELE Reporting_backup_200904182016.bak
    < 250 DELE command successful.
    > TYPE I
    < 200 Type set to I.
    > PASV
    < 227 Entering Passive Mode (172,28,132,71,7,219).
    > STOR ACAM_backup_200904252016.bak
    Data connection established with 172.28.132.71:2011 on 4/26/2009 @ 2:06:50 AM
    < 125 Data connection already open; Transfer starting.
    > NOOP
    > NOOP
    < 226 Transfer complete.
    < 421 Timeout (120 seconds): closing control connection.
    < 421 Terminating connection.
    Disconnected from 172.28.132.71:21 on 4/26/2009 @ 2:11:34 AM
    Disconnected from 172.28.132.71:21 on 4/26/2009 @ 2:11:34 AM
    > TYPE I
    Error sending command: The FtpCommandChannel is not connected.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by ganders (had 622 views)

    User (Old forums)
    Member
    Post count: 23064

    The only option left is for us to have access to your server, so we can try to reproduce and investigate what is the source of the issue.

    Send us an email to support@xceed.com, referring this thread in the email.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by André (had 2762 views)

    User (Old forums)
    Member
    Post count: 23064

    Unfortunately, my company won’t allow that; and since the time that I started this thread, they have gone on to tell me that they are going to no longer allow me to do that large of a file transfer anymore.  They want me to use an external USB drive now…there goes automation.

     

    Thanks anyway for the time Andre, I appreciate your help.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by ganders (had 4748 views)

Viewing 8 posts - 1 through 8 (of 8 total)
  • You must be logged in to reply to this topic.