User (Old forums)MemberOctober 30, 2008 at 10:13 pmPost count: 23064
I am writing an archive for my company and have decided to use the zip file format as the archive format. The production databases will probably be nearly 1 TB in size but just for some preliminary tests I built downloaded and built a few 1 – 2 GB zip files using Xceed Zip for .NET. Everything seemed to work great, the performance (from a quick visual inspection) was around 1 MB per second. I did need to use the Begin/End update structure but after implementing this everything seems fine.
The problem I have is that when I would like to update these archived files, even adding a single file takes quite a long time (~10 minutes). I did some quick debugging and also watched the file size and realized it is copying the contents of the zip file to some temporary location one file at a time, emptying the zip archive and then completely rewriting it with the new file in it.
After reading most of the posts in this forum I found a post that was somewhat similar but the responses indicated that gzip/tar were a bit faster. The problem is, the post was talking about a file that is compressed, while the archives I am writing are neither compressed nor encrypted (that comes next).
So, putting it simply, is there any way to write a file this large (up to 1 TB), only once, read it, and append files as necessary without Xceed rewriting the entire archive?
Mideo Systems Inc
Imported from legacy forums. Posted by Andrew (had 2852 views)User (Old forums)MemberNovember 3, 2008 at 2:18 pmPost count: 23064
Unfortunately, no. Our control will rewrite the archive every time you append a file, unless you append all files within a Begin/EndUpdate block, but it will still do it once for all the files, at the end of the block.
Imported from legacy forums. Posted by André (had 218 views)User (Old forums)MemberNovember 3, 2008 at 2:50 pmPost count: 23064
Only one more question then: I am sure Xceed must have considered trying to improve the algorithm for update. Would you have any idea how long this algorithm change might take starting from the code available with the Blueprint edition?
Imported from legacy forums. Posted by Andrew (had 311 views)User (Old forums)MemberNovember 4, 2008 at 10:42 amPost count: 23064
It depends on what features you want to support.
If you simply want to add the ability to append a file at the end of an existing archive (which seems to be what you are looking for), it could take you a week or two. This includes time to familiarize yourself with the code and the zip specification. This is the simplest case.
However, if you want to support replacing an existing file in the archive, rename or delete files, support cases where the zip file is split over several files and all the other zip scenarios, we are talking about a few months (at least 3). Several internal structures would have to be modified and a smart storage mechanism implemented that can detect when a temporary file isn’t needed and when it’s unavoidable. Extensive testing to verify that nothing was broken would take a significant portion of that time.
Imported from legacy forums. Posted by André (had 3698 views)
- You must be logged in to reply to this topic.