Sync only picks up if app is running when files changed

@pvdb64 @lawgiver_ai

Sorry to toot my own horn here, butā€¦

Totally called it. Arctic Adaptation: Icedriveā€™s Response to UK Privacy Laws - #20 by tambo

1 Like

@FrankS

So weā€™re out here figuring out ways to make IceDrive work the way it should, with not a peep from the IceDrive team. :roll_eyes:

Absolutely right. Yā€™know what, F it. Iā€™m tired of this situation where we all get frustrated at the obvious regressions and lack of transparency, talk amongst ourselves on these forums, and the people who can actually do something about it give placating but noncommital answers at best. Letā€™s use electronic accountability to make it harder for them to ignore, or more obvious if they do. @Chris @Tom @JimmyB - where we at on this?

@Tom , your message

ā€œWe hear you, I will forward this on to our development team to consider.ā€

was on Oct 1. That was almost 2 months ago. What was communicated to the development team, how has their ā€œconsiderationā€ gone, and what was the outcome or proposed solution, if any? What is the timeline to a functional sync, given the fact that the team has apparently been aware of this issue since the inception of v3 on September 3rd, and likely before? Three months is a long time for a glaring flaw in a sync feature for a paid product/service that advertises sync capabilities. And since youā€™re releasing new products, I would assume that everything in your core offering is already on the up and up, or being worked on around-the-clock to ensure that it gets to a functional state post-haste. Please help restore my faith.

3 Likes

@FrankS you beat me to it
seeing your assistance to another user here asking for some help on syncing with 2.75 I was wondering how long it still may take to get a robuster update in 3.23 as has been a while

For the past few weeks, I havenā€™t really thought about computer stuff and just kept Icedrive running in the background, being careful not to shut it down. Yesterday I ran a comparison with FreeFileSync, and found several problems:

  • 4 new files were not uploaded to the cloud.
  • 1 edited file was not updated in the cloud.
  • 1 deleted folder (with 3 files in it) was not removed from the cloud (even though ā€˜deletion policy remoteā€™ is set to ā€˜deleteā€™).
  • 35 moved files were correctly moved in the cloud, but also remained in their original location (i.e. they were duplicated in the cloud).

So it seems to me that Icedrive is not working properly even when running in the background. :thinking: Can anyone confirm this?

2 Likes

@Guido - confirmed. Very similar results to yours.

Iā€™ve actually deleted all my sync pairs and am now experimenting with FreeFileSync. I also tried out Syncback, but abandoned that effort after it wanted to transfer too many files at once and Icedrive crashed!

Now Iā€™m waiting for the backlog to transfer before looking at FreeFileSync again.

Itā€™s all a bit of a mess over here.

But it is Black Fridayā€¦ Iā€™ll just leave that comment there. Make of it what you will.

I see what you mean. :joy: Actually, I had almost decided to look for a competitor, until I realized that the combination of Icedriveā€™s mounted drive and FreeFileSync is not just a patch, but is in fact gold. Buggy as Icedriveā€™s syncpairs may be, the fact that FreeFileSync can check and correct everything gives me huge peace of mind. After reading this comment by @Abe, I am not sure competitors can provide the same level of assuranceā€¦

@Guido Iā€™ll show my cards. I actually did buy a deal with a competitor.

So right now, Iā€™m actually comparing two side by side. Thereā€™s a school of thought I stumbled across that says you donā€™t actually need the baked-in encryption. You can achieve the same effect with Cryptomator and a regular mounted cloud drive. This nullifies the selling point of zero-knowledge encryption.

And that, in turn, means, for Icedrive to be truly useful is to get those sync pairs working properly, because for me, right now, the competition is between Icedrive syncing reliably and Cryptomator + competitor.

3 Likes

@Guido @lawgiver_ai Just want to add my two cents here. I use my own encryption before uploading to ANY cloud. And I use FreeFileSync to sync files at home. The problem with ID occurs during upload and/or (re-)download. If the file being uploaded to ID cloud OR being downloaded from ID cloud becomes corrupted, thereā€™s nothing for FreeFileSync to find when it mattersā€“cloud file fidelity. FreeFileSync doesnā€™t compare whatā€™s in IDā€™s cloud with what we expect to upload. Maybe that works differently with a locally synced folder, but I donā€™t have luck with sync pairs. (I used FIVE privacy clouds and all have sync folders constantly getting screwed up. If it matters, Sync.com is most reliable and has a Linux work-around, but is based in Canada, a Five-Eyes country. Buyer Beware.). Amazon Cloud & Microsoft (cloud) employ a cloud checksum (from what Iā€™ve read of their services) so customers can feel confident whatā€™s in these companiesā€™ clouds is what customers expect. But that is probably not compatible with zero-knowledge E2E-encryption.

Over the years, Iā€™ve found using my own encryption before uploading to clouds complicates syncs tremendously. And no privacy-friendly cloud offers a reliable way to double-check what was uploaded to the cloud matches my local copy. Could do Rclone, but I donā€™t have the time to set that up. I wanted an out-of-the-box solution. About half of the files I download from IDā€™s cloud over about 1 GB either never arrive on iOS or are corrupted on Windows or Mac OS despite high-speed, stable internet. I want to love ID. But I just canā€™t trust ID with my documents. The only way ID works for me is to manually upload tiny filesā€“under 1GB or so. Not bashing the company. I support ID because privacy is super-important to me. But corrupt files are useless to me and a massive waste of time since I donā€™t know WHICH files in the cloud are corrupt. Which is why now I rely on my own home backups only.

Agree with @lawgiver_ai:

3 Likes

@lawgiver_ai That sounds interesting. Let us know about your experiences with Cryptomator! I am especially curious about how this works out in everyday life. I often need to move files around in the file explorer and I cannot afford for my backup system to slow me down.

@Abe Thanks for weighing in. I hadnā€™t given much thought to file corruption yet. Another complicationā€¦ :face_with_diagonal_mouth: Perhaps one should have two remote backups to account for corrupted files in either one of them (and more than two if you deal with large files).

This has been really helpful. Iā€™ve had endless issues with Icedrive syncing and not trusting results since I bought their plan several years ago. Can you clarify: you use FFS to ā€˜syncā€™ to the mounted Icedrive?

And you (so far, based on your experience) trust the mounted Icedrive to upload reliably?

Thanks in advance.

@Guido - to answer your Cryptomator question. Iā€™ve set it up like this:

  • Mounted drive to cloud folder (X:)
  • Cryptomator sets up another mounted drive (Z:)
  • But Z: is actually a ā€œdecrypted viewā€ of X:\MyFolder
  • I copy files into Z:
  • Cryptomator encrypts before writing to Z: (= X:\MyFolder)
  • Cloud software uploads X:\MyFolder to remote and knows nothing of Z:

What do I see:

  • If I open Z: I see all my files as I copied them, whether theyā€™re actually uploaded or not.
  • If I open X:\MyFolder, I see a load of incomprehensible stuff.
  • If I open MyFolder on the cloud web interface, I see the same incomprehensible stuff.

It seems the copy stage can be done any way you like. Iā€™ve tested out FFS and Syncback and both are fine as far as I can tell. The copying to Z: does seem to take time, as Iā€™d expect, since thatā€™s when the encryption takes place. The upload to the cloud just runs at my network speed and seems perfectly fine.

If youā€™re wondering about disk usage - it seems to depend on whatā€™s been uploaded to the cloud. If I overwhelm Z: with a ton of files, I see it in my disk usage. But that slowly claws back down as the encrypted files are uploaded.

1 Like

It is certainly a possibility to completely rely on FFS for the syncing process. I think that @tambo is exploring this route. However, I still use Icedriveā€™s syncpairs, and use FFS to ā€œcheck and correctā€. So every now and then I have FFS compare my local drive with my mounted drive (which in fact represents the cloud), like this:

The settings for the comparison should be ā€œdate and sizeā€:

If FFS finds any discrepancies, I hit the synchronize button. Make sure that the settings for synchronization are set to ā€œmirrorā€, so that FFS only syncs in one way, and leaves your local drive undisturbed:

Good question. I think it does, because I assume that the mounted drive reflects exactly what lives in the cloud. It doesnā€™t contain files itself, but is more like a portal to the cloud. (Can anyone confirm this view?) So if FFS finds no problems, the local drive and the cloud should be exactly the same. Having said that, there is still the problem of file corruption (see the comment by @Abe above), which FFS probably wonā€™t detect.

2 Likes

Why wouldnā€™t it detect corruption? Is it because a corrupted file could still have the same date and size? And, if so, whatā€™s the issue with a content comparison?

Yes, if the corruption means that one bit is changed (from 0 to 1, or vice versa), the file size is unaffected.

The issue with the content comparison is that FFS will try to access the files on the mounted drive, which in turn triggers the mounted drive to download them into the temp-folder and decrypt them. That is not a problem per se, but it takes a looong time and your temp-folder will explode.

However, now that I think about it, it may be worthwhile to do this anyway, once in a while (once a year or so). :thinking: (tagging @Abe here)

1 Like

According to the changelog, the app should now track changes made when it is not running:

ā€œSync: Background monitor implemented to track changes in files and folders when the app is not running, and sync when app is reopened. Windows and macOS supported.ā€

I upgraded to version 3.31 and did some tests. However, it still doesnā€™t work for me. :slightly_frowning_face: I have waited for at least 12 hours, but the app doesnā€™t seem to scan my folders for changes. Are others having more success?

same issue for me doesnā€™t update a file when app not active

1 Like

I just tried it on 3.32 and seems to be working now (after a reboot to start the monitor service)

2 Likes

I thought we were also waiting on a return of the scheduled sync functionality. It seems theyā€™ve doubled-down on detecting running changes.

Maybe itā€™s me, but I think thereā€™s something messy about having two apps running to provide one service. Version 2.x was a single app that did everything and did it well.

If this new background service were to crash, would we have visibility? Or, at least, visibility at a time when it mattered? At least if the main app crashes, we know about it - and a scheduled sync is guaranteed to pick up any missed changes.

Itā€™s already been discussed there was recognised demand for a scheduled sync, but weā€™ve had silence since.

1 Like

completely agree, a reconciliation full sync after the client startup would seem to me to be a more robust solution. that seems to be the way other cloud providers work but as long as this way does work iā€™m not too bothered. and i think we would probably have visibility if it did fail. when i upgraded to 3.32 yesterday, the upgrade didnā€™t ask for a reboot (it probably should have) as when i started the client, it did alert that the monitor service wasnā€™t running so the client must do some checking, which is why i rebooted and that defined & started the monitor service.

1 Like

I can confirm: it works now! Our prayers have been heard. :partying_face: Very happy with this upgrade. I didnā€™t need to reboot by the way.

EDIT: While version 3.32 is certainly an improvement (it protects against accidentally shutting down the app), I must agree with @lawgiver_ai that it is still not watertight. The best solution remains to scan all folders from time to time. Even more so because the syncing process seems to be unreliable even when the app is running (see my previous post).

3 Likes