Home Forums .NET libraries Xceed Zip & Real-Time Zip for .NET Time to add a file in a zip grows with Zip size

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • User (Old forums)
    Member
    Post count: 23064
    #19622 |

    Hi,

    I’m adding multiple files in a zip, and I don’t want to use BeginUpdate and EndUpdate, because I want the files to be directly added to the physical file.

    The problem is that it takes more and more time to add a file to the zip when its size is growing.

    I logged the time it takes to execute the CopyTo method :

    File n° 1 to zip, time elapsed = 62 ms
    File n° 2 to zip, time elapsed = 16 ms
    File n° 3 to zip, time elapsed = 16 ms
    File n° 4 to zip, time elapsed = 31 ms
    File n° 5 to zip, time elapsed = 16 ms
    File n° 6 to zip, time elapsed = 31 ms
    File n° 7 to zip, time elapsed = 31 ms
    File n° 8 to zip, time elapsed = 47 ms
    File n° 9 to zip, time elapsed = 47 ms
    File n° 10 to zip, time elapsed = 47 ms
    File n° 11 to zip, time elapsed = 62 ms
    File n° 12 to zip, time elapsed = 78 ms
    File n° 13 to zip, time elapsed = 78 ms
    File n° 14 to zip, time elapsed = 62 ms
    File n° 15 to zip, time elapsed = 109 ms
    File n° 16 to zip, time elapsed = 78 ms
    File n° 17 to zip, time elapsed = 94 ms
    File n° 18 to zip, time elapsed = 78 ms
    File n° 19 to zip, time elapsed = 94 ms
    File n° 20 to zip, time elapsed = 109 ms
    File n° 21 to zip, time elapsed = 94 ms
    File n° 22 to zip, time elapsed = 109 ms
    File n° 23 to zip, time elapsed = 109 ms
    File n° 24 to zip, time elapsed = 172 ms
    File n° 25 to zip, time elapsed = 125 ms
    File n° 26 to zip, time elapsed = 125 ms
    File n° 27 to zip, time elapsed = 125 ms
    File n° 28 to zip, time elapsed = 125 ms
    File n° 29 to zip, time elapsed = 125 ms
    File n° 30 to zip, time elapsed = 141 ms
    File n° 31 to zip, time elapsed = 203 ms
    File n° 32 to zip, time elapsed = 156 ms
    File n° 33 to zip, time elapsed = 172 ms
    File n° 34 to zip, time elapsed = 156 ms
    File n° 35 to zip, time elapsed = 172 ms
    File n° 36 to zip, time elapsed = 188 ms
    File n° 37 to zip, time elapsed = 250 ms
    File n° 38 to zip, time elapsed = 188 ms
    File n° 39 to zip, time elapsed = 188 ms
    File n° 40 to zip, time elapsed = 188 ms
    File n° 41 to zip, time elapsed = 203 ms
    File n° 42 to zip, time elapsed = 203 ms
    File n° 43 to zip, time elapsed = 203 ms
    File n° 44 to zip, time elapsed = 203 ms
    File n° 45 to zip, time elapsed = 219 ms
    File n° 46 to zip, time elapsed = 297 ms
    File n° 47 to zip, time elapsed = 219 ms
    File n° 48 to zip, time elapsed = 234 ms
    File n° 49 to zip, time elapsed = 219 ms
    File n° 50 to zip, time elapsed = 281 ms
    File n° 51 to zip, time elapsed = 234 ms
    File n° 52 to zip, time elapsed = 250 ms
    File n° 53 to zip, time elapsed = 250 ms

    All the files I’m adding are blank text files (0 Ko).

    Here is a part of the sample code I wrote to reproduce this issue :

    private void button4_Click(object sender, System.EventArgs e)
    {
    ZipArchive zip = new ZipArchive(new DiskFile(textDestination.Text.Trim()));
    System.IO.DirectoryInfo srcDir = new System.IO.DirectoryInfo(textSourceFolder.Text.Trim());
    fileNumber=1;

    object[] param = {zip,srcDir};

    System.Threading.WaitCallback mdelegate = new System.Threading.WaitCallback(AddFolderToZip);

    System.Threading.ThreadPool.QueueUserWorkItem(mdelegate,param);
    }

    private void AddFolderToZip(object stateInfo)
    {
    object[] param = (object[]) stateInfo;
    ZipArchive myZip =(ZipArchive) param[0];
    System.IO.DirectoryInfo source =(System.IO.DirectoryInfo) param[1];

    foreach (System.IO.FileInfo file in source.GetFiles())
    {
    DiskFile mFile = new DiskFile(file.FullName);

    //Copy disk file to memory file
    MemoryFile src = new MemoryFile();

    mFile.CopyTo(src,true);

    DateTime start = DateTime.Now;
    //Rename memory file
    src.Name = Guid.NewGuid().ToString() + “.src”;

    src.CopyTo(myZip,true);

    TimeSpan ts = DateTime.Now.Subtract(start);
    System.Diagnostics.Debug.WriteLine((“File n° ” + fileNumber++ + ” to zip, time elapsed = “).PadRight(50) + Math.Round(ts.TotalMilliseconds) + ” ms”);
    }

    foreach (System.IO.DirectoryInfo dir in source.GetDirectories())
    {
    object[] parameters = {myZip,dir};
    AddFolderToZip(parameters);
    }
    }

    I didn’t used an AbstractFolder to add the files because I need to manipulate each file before adding it to the zip. I must add them, one at a time.

    Does anyone have an idea ? Has anybody already had the same problem ?
    In fact I don’t know if it’s a bug or if I’m doing something wrong.

    Olivier

    Imported from legacy forums. Posted by littleb (had 4773 views)

    Odi [Xceed]
    Spectator
    Post count: 426

    Hi Olivier,

    Your findings are correct. A fair amount of new development is required in order to improve the speed in this situation – we have done this work for the TAR+GZip support, which performs well and thus proves the concept, but to do this in the Zip algorithms is another ballgame. We have inserted this into the Xceed Zip for .NET development pipeline (no projected date at the moment). Thus for the time being, it seems only the BeginUpdate/EndUpdate method can help, if you are able to use it.

    Imported from legacy forums. Posted by Odi [Xceed] (had 338 views)

    User (Old forums)
    Member
    Post count: 23064

    Thanks for your reply.
    I will have to find a way around 🙁

    Olivier

    Imported from legacy forums. Posted by littleb (had 401 views)

    User (Old forums)
    Member
    Post count: 23064

    If I understand you correctly, the current version (3.2) does’nt have this problem with TAR / G-Zip compression.

    We are using version 2.2, so we used Zip compression because it was the only one available. Anyway we may use an other compression format. So if with an upgrade we may be able to do it with TAR compression format, it would be fine for us.

    Could you confirm, that I won’t have this behavior with TAR compression with version 3.2 ?

    Olivier

    Imported from legacy forums. Posted by littleb (had 389 views)

    Odi [Xceed]
    Spectator
    Post count: 426

    Actually, what I said does apply to the TAR code in version 3.2, but won’t work if you are using a .TAR.GZ file. Let me explain: If you were not compressing the TAR file, everything would be great, you would be able to add files to the TAR file very quickly, without the time taking longer and longer for each new file. However, if the TAR file itself is compressed and becomes a .TAR.GZ, that wouldn’t be the case, as the *entire* TAR file has to be uncompressed in order to add a new file, and then recompressed. Back to square one.

    On the other hand, if each file is first GZipped, and *then* added to the TAR file, that scenario would work fine. The .TAR would contain compressed files just like a Zip file, but the header data (512 bytes per file in the case of TAR, less in the case of Zip files) would not be compressed (just as it is not in Zip files). I suspect that won’t be a problem.

    So to conclude, I suggest trying out v3.2 with a trial key (I can email you one, send an email to me at kosmatoso@xceedsoft.com) and using the following archive and compression combination: first compress the file to add to the archive with GZip, then add it to the master TAR archive, and do not compress the TAR archive itself. This should work well.

    Odi

    Imported from legacy forums. Posted by Odi [Xceed] (had 332 views)

    Xceed Support
    Member
    Post count: 5658

    I just have one more thing to add to Odi’s response. If you use the BeginUpdate/EndUpdate when using TAR.GZ, the GZ archive will only be recreated once. (when the EndUpdate is called)

    We have included an example in our Xceed FileSystem Snippet Explorer that demonstrate the use of BeginUpdate/EndUpdate and that shows how much faster it can be using this approach. The example number is 5.8.1 (optimize performance).

    Hope this help!

    Imported from legacy forums. Posted by Jacques [Xceed] (had 5167 views)

Viewing 6 posts - 1 through 6 (of 6 total)
  • You must be logged in to reply to this topic.