Uploading 500 GB to encrypted storage at once, which resulted in corrupted images and videos

I recently tried to backup some of my photos and videos,
And I was encountered with some file corruption issues,

Some context:
503 GB of data - 29,622 files, 1819 folders
Internet speeds ~ Download 932.13 Mbps, Upload 250.1 Mbps
Browser - Brave (tried it with chrome too)

After uploading, I noticed that some of the files were corrupted,
And in addition there was no way to distinguish between corrupted files from regular ones, (the thumbnail is the same)
Due to the nature of the encrypted storage in icedrive, you need to manually go over thousands of files to verify whether they are corrupt or not,
Which is just madness…

The goal is to be able to upload the 503 GB of data, in one go, wait the amount of time required,
And be sure that all the files were uploaded with no corruption issues.

Or to be a able to distinguish corrupted files from regular files in the encrypted storage

What should I do in order to solve this?

If you’re using the web app you should be able to click the status circle in the bottom right when uploading and this will list all ongoing transfers. It will also indicate if there any particular files that have failed to upload. Uploading to the encrypted section does not provide thumbnails as we use zero-knowledge encryption which means that the files are encrypted locally on your device before uploading to the cloud. The reason there are no thumbnails is that not even our software can access those files to produce a thumbnail.

Thank you Chris for your response,
Is there any way to avoid corruption in files,
When uploading large amount of data to the encrypted storage?

The files you upload shouldn’t get corrupted unless for example you lose connection to the internet and then the particular files that were uploading at the time may not have finished their transfer. If you do get any that fail to upload you will see them in the list. We would always recommend uploading in smaller chunks to avoid the risk of for example losing connection to the internet etc.

1 Like

Thanks again Chris for your response.

What data size would you recommend to upload in one go?
Considering my internet speeds

In addition, I found some power & sleep settings that people suggest when uploading big files,
Like PC sleep time and hard disk turning off after certain time,

I will tweak those settings, and see if it helps.

If you’re using a laptop definitely leave it plugged in. Potentially change your power & sleep settings to prevent it from going into sleep mode during the transfer, thus potentially cutting off your connection to the internet and causing upload failures. Any advice on specific chunk sizes would be fairly arbitrary however given your upload speeds perhaps if you upload in 100GB chunks and let us know how you get on.

Using a desktop :muscle:

Going to change the sleep timer to “Never” and “Turn off hard disk after” setting to a very high number,
Just for when I am trying to upload \ backup my files.

Will do!