I use rclone to sync my files with icedrive once every night. Lately I’m getting an error: “You have been temporarily rate limited.: 429 Too Many Requests”. Rclone keeps trying it seems. I guess I need to adjust some settings for rclone. What is the limit I need to account for?
3 Likes
Hey @koen welcome to the community! We would not recommend using WebDAV to backup your data to the Icedrive cloud in bulk.
Our implementation of the WebDAV service is not designed for this. You may run into “Too many requests” error, which means that you are temporarily blocked from using the WebDAV service due to excessive network requests.
Use our desktop apps instead of WebDAV service (e.g. the ‘sync’ feature available in our Windows mounted app):
Hello Chris and team I am having a similar issue that affects my rclone jobs from working properly.
Though I understand defaulting to advising us to use the Windows Desktop app and Windows being a popular operating system a chunk of the customer base no doubt uses Linux or Mac based operating systems where there isnt the option of doing file sync without Webdav.
Koen asked:
“I guess I need to adjust some settings for rclone. What is the limit I need to account for?”
I am asking this as well, would you please be able to provide safe settings/limits so that we can adjust this accordingly so that this works better and hopefully puts less load on your network
When Webdav is working its impressive fast, we just need to tweak our settings abit to make it play nice with the IceDrive network
I didn’t have the time yet to look at the relevant rclone settings. For my modest backup (handful of new and changed files every day) it shouldn’t be a problem to find something that will work. Lowering speed isn’t a problem for me.
As I use a headless ubuntu server fir my backup using the desktop application isn’t an option.
I am no rclone expert but I believe the settings that might be affecting this are:
–transfers 10 --checkers 50
I have no idea what this should be limited to but if there was guidance on this sync might then properly work
Maybe the speed needs to be limited as well? (im uploading at 250 megs)
–transfers 1 --checkers 1 isn’t enough. Just tried it. What is the speed limiting option?
Interesting…
When IceDrive setup/configured their new and improved Webdav they should have a solid idea on what limits to recommend based on its setup, I am hoping that they provide this information on this thread to give direction for its non-Windows customers.
I am testing the following rclone command to limit traffic to 10 megabytes and will see how that goes
–bwlimit 10M
But again an official recommendation vs stabs in the dark will be helpful @Chris @Tom
i got the same error “You have been temporarily rate limited.: 429 Too Many Requests” using rclone to sync data from pcloud to icedrive. i tried adding --bwlimit 10M but it seems useless. any idea?
Hi all, I had a good chat with @Tom who indicated that this thread might not have been actively monitored and to log a ticket which I have now done. In the ticket I did reference this thread to let IceDrive know that it is not just myself affected by this.
Currently, I am still getting:
“You have been temporarily rate limited: 429 Too Many Requests”
and that is with the very low settings:
–verbose --progress --size-only --transfers 1 --checkers 1 --bwlimit 1M --tpslimit 1 --min-age 16m
Once I know more i’ll be happy to share on here
1 Like
I got some swift support (thanks @Tom)…
The low down is as follows:
“we are indeed using server side rate limiting - if your application makes more than 300 requests per minute - it will be blocked for a few minutes (HTTP error 429 – too many requests).
Also, we allow a maximum of 10 connections from a single IP address.”
I have 3 rclone jobs that sometimes run simultaneously. When combined they would go above the threshold so IceDrive would then automatically limit me.
I would suggest contacting support so that they can reset this and then you set your different rclone jobs to combined not go over the limits.
Even though this is now working quite nicely I will switch on the 3 cron jobs that automatically run several times an hour to see how this works in practice vs the manual test rclone.
Thanks again IceDrive peoples!
1 Like
thank you very much, according to this rclone flag --tpslimit 5 should solve the error 429. i tried twice and it seems to work!
@pcalosso Sadly this isnt working for me, can you please confirm that it works after a sustained upload (roughly 10 min)? I am finding with quite conservative settings I am getting the 429 error
@smartieee
yeah, it is working for me at the moment: rclone is syncing about 250gb from pcloud to icedrive. i set this as crontab activity every 8 hours: the first one took 17m35s, the next ones about 11m20s. for now i have no more errors and the only flag i used is the tpslimit.
I have some git repositories in my backup containing many small files. I just tried the --tpslimit 5 setting and got rate limited after 2 hours.
I am waiting for the rate-limiting to go away then want to test:
–transfers 1 --checkers 3 --bwlimit 20M --tpslimit 3
But from the sounds of it I should just try --tpslimit 5 by itself even though itll default to several transfers?
I am finding that it takes hours of non-use for the speeds to come back again i.e. trying a different setting every morning so am convinced that the rate-limiting is too strict
I have also installed Wine and am using the Windows app in Manjaro, which is working but for some reason is limited to an upload speed of 385kbs which is too slow but at least its chugging away…
today i got error 429 again after a couple of days working flawlessly. got rate limited after 45 mins.
This is unusable for me and I’m on the minimal settings now: --transfers 1 --checkers 1 --bwlimit 20M --tpslimit 1
@smartieee --transfers 1 --checkers 1 and --tpslimit 1 probably work without error 429 but, you are absolutely right, is unusable for almost everyone imho
2 Likes
Agreed,
I did talk with support again and they apparently are working on changes to the strictness with rate limiting and will also be making changes to add WebDav traffic details to the account → statistics section. To make it clearer to users where they are at with their bandwidth limits.
While an ETA was not given I did get a sense that this was a higher priority and will be ready in the near future
1 Like
