Home Forums .NET libraries Xceed SFTP/FTP for .NET Over 30,000 files transfer.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • User (Old forums)
    Member
    Post count: 23064
    #21232 |

    Hi,

    I have version 4 of Xceeed for FTP. For my task, i transfer around 20.000-30.000 files from the server to local machine for further log parsing.

    I filter the directory before hand to get files based on date: FtpItemInfoList list = http://ftp.GetFolderContents(fileNamePrefix + “*”);

    and then use that list to compare to files that were already downloaded to local disk.

    I know about http://ftp.ReceiveMultipleFiles method, and that is takes a filter, but i need to download only files that are on the server and not on my local computer already. Therefor i am using foreach http://ftp.ReceiveFile(); to go through each file i need to download.

    My question is i guess, what is the best way to download 20,000 > files that are being pulled from a List<string> for example?

    Thanks

     

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by too (had 1964 views)

    User (Old forums)
    Member
    Post count: 23064

    Hi,

    there are a couple of things to consider. 

    First of all, are you going to have the same folder structure on both machines?  If so, you could try to use synchronization.  You should refer to section 6.4 of the Xceed FileSystem Snippet Explorer.  For your information, here is a copy of this code entitled “One-way synchronization”.  (This snippet can be modified from the example without the need to have Visual Studio installed.)  This snippet is installed when the latest “general package” is installed. 

    using System;
    using System.IO;
    using System.Windows.Forms;
    using Xceed.Ftp;
    using Xceed.FileSystem;
    using Xceed.Synchronization;

    namespace Xceed.FileSystem.Samples.SnippetExplorer
    {
      public class Snippet
      {
        public void Example()
        {
          // We will connect to the following FTP server, and login anonymously.
          // For the purpose of this example, the FTP server must allow
          // listing, readind and writing.
          string hostname = “http://ftp.xceed.com“;
          string username = “anonymous”;
          string password = “guest”;

          // We will synchronize some files in the specified local folder.
          string localPath = System.IO.Path.GetTempPath() + @”SnippetExplorer\”;

          // Get a folder object from our local path.
          AbstractFolder localFolder = new DiskFolder( localPath );
         
          // If the local folder exists
          if( localFolder.Exists )
          {
            Console.WriteLine( “Deleting {0}”, localFolder.FullName );
           
            // Delete it
            localFolder.Delete();
          }
         
          // Get a new file in the local folder
          AbstractFile newFile = localFolder.GetFile( “NewFile.txt” );
           
          // If it doesn’t already exist
          if( !newFile.Exists )
          {
            // Create it
            newFile.Create();
          }

          // Always start with an FtpConnection using the parameters we set up above.
          using( FtpConnection connection = new FtpConnection( hostname, username, password ) )
          {
            // Let’s access the starting folder on that FTP server.
            AbstractFolder remoteFolder = new FtpFolder( connection );

            // We will limit ourselves to files smaller than 25k to save time since long
            // transfers are not the point of this example.
            Filter[] filters = Filter.GetFiltersFromParams( new SizeFilter( 0, 25 * 1024 ) );
           
            // For the purpose of this example, we’ll monitor the file comparison and analysis
            // of the synchronizations we perform in order to output what’s going on.
            SynchronizationEvents synchronizationEvents = new SynchronizationEvents();
            synchronizationEvents.Analysis += new AnalysisEventHandler( synchronizationEvents_Analysis );
           
            // Create a synchronization options object. It will be set to the default values.
            SynchronizationOptions options = new SynchronizationOptions();
           
            // For the purpose of this example,
            // allow the synchronizatio to solve any conflict by itself so the synchronization
            // is never stopped because of a possible conflict.
            options.AutoConflictResolution = true;
           
            Console.WriteLine( “First one-way synchronization…” );
           
            // Synchronize the remote and local folder together,
            // selecting only files that are smaller than 25k,
            // one-way from the remote folder (index 1 in the list) to the local folder.
            Synchronizer.EasySynchronize(
              synchronizationEvents,
              filters,
              options,
              1, localFolder, remoteFolder );
             
            /* The synchronization wanted to delete newFile because it is not present in the
               master folder.
              
               The results show however that deletion of newFile was not allowed. This is the
               default behavior of the synchronization to prevent accidental loss of data.
              
               We’ll assume we know what we’re doing and enable deletions. */

            // Allow file deletions to be made when appropriate.
            options.AllowDeletions = true;
           
            Console.WriteLine( “Second synchronization (deletions allowed)…” );
           
            /* In this case, ‘newFile’ will need to be deleted since it is not part of the remote
               folder, which we have selected as master. */
           
            // Synchronize the remote and local folder together,
            // selecting only files that are smaller than 50k,
            // one-way from the remote folder (index 1 in the list) to the local folder.
            Synchronizer.EasySynchronize(
              synchronizationEvents,
              filters,
              options,
              1, localFolder, remoteFolder );
          }      
        }

        void synchronizationEvents_Analysis( object sender, SynchronizationAnalysisEventArgs e )
        {
          int index, count;
          string text;
          AbstractFile masterFile;
         
          // Get the master file of this synchronization.
          masterFile = e.MasterFile;

          // Go through all the files involved in the synchronization.
          for( index = 0, count = e.Files.Length; index < count; index++ )
          {
            // Convert the action to a string
            text = this.ActionToString(
              e.Actions[ index ],
              index,
              masterFile,
              e.Files[ index ] );

            // If we have a string.
            if( text.Length > 0 )
            {
              // Print it.
              Console.WriteLine( text );
            }
          }
         
          Console.WriteLine( ”  —” );
        }

        string ActionToString(
          SynchronizationAction action,
          int index,
          FileSystemItem master,
          FileSystemItem target )
        {
          string text;

          text = String.Empty;

          // Build an information string for each applicable action
          switch( action )
          {
            case SynchronizationAction.None:
              text = String.Format( “No action on {0}”, target.FullName );
              break;

            case SynchronizationAction.CreateOrOverwriteWithMaster:
              text = String.Format( “Copying {0} to {1}”,
                master.FullName,
                target.FullName );
              break;

            case SynchronizationAction.OverwriteAttributesOnly:
              text = String.Format( “Applying attributes of {0} to {1}”,
                master.FullName,
                target.FullName );
              break;

            case SynchronizationAction.Delete:
              text = String.Format( “Deleting {0}”,
                target.FullName );
              break;

            case SynchronizationAction.SuspendedCreateOrOverwrite:
              text = String.Format( “Create/Copy to {0} is not allowed”,
                target.FullName );
              break;

            case SynchronizationAction.SuspendedDelete:
              text = String.Format( “Deletion of {0} is not allowed”,
                target.FullName );
              break;
          }

          return text;
        }
       
        public Snippet( Xceed.Samples.SnippetExplorer.MainForm resultsForm )
        {
          m_resultsForm = resultsForm;
        }
       
        private Xceed.Samples.SnippetExplorer.MainForm m_resultsForm = null;
      }
    }

    You could also consider the

    C:\Xceed Component Samples\Xceed FTP for .NET 4.0\Samples\Synchronize\CSharp\SynchronizeUISample\SynchronizeUISampleCS.csproj

    project, which should let you do what you want without having to change too many lines.

    Applies to Xceed FTP for .NET. Imported from legacy forums. Posted by Ghislain (had 1881 views)

Viewing 2 posts - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.