The real reason your iCloud Drive isn't syncing

9 Jul 2019 18:34 | apple | macOS

I recently had the logic board replaced in my 2017 Macbook Pro. I use the awesome Carbon Copy Cloner to keep an image of my system as a bootable backup which can then be easily restored when the machine is returned to me.

This time however I had some issues with iCloud Drive. After restoring the backup I found it wasn’t syncing. I tried the usual troubleshooting steps: turn it off, reboot the machine, turn it back on again. Still wasn’t syncing, just sitting there with all my files and folders seemingly stuck in the state of “waiting to download”. I scoured the internet, lots of people with the same issue and no obvious solution. Curiously though some of the threads had people reporting that their stuff would just suddenly start syncing out of nowhere without them having done anything to prompt it.

Well I have AppleCare so with a due sense of exhaustion and dread I gave them a call. We started a screen share session so they could see the problem. Their first suggestion was to try creating a new user account and seeing if that would sync. I took exception to this - I don’t need this to work on another account, I need it to work on this account. I hung up the call and decided to investigate myself.

I knew from previous messing around with macOS that the main process involved in iCloud Drive syncing was bird. I decided to see if it was logging anything:

$ log stream |egrep -i bird 2019-07-09 18:11:09.546067+0100 0x5afcd Activity 0x6c4a4 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down 2019-07-09 18:11:10.672324+0100 0x5afcd Activity 0x6c4a5 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down 2019-07-09 18:11:11.626342+0100 0x5afcd Activity 0x6c4a6 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down 2019-07-09 18:11:12.629151+0100 0x5afcd Activity 0x6c4a7 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down 2019-07-09 18:11:13.496400+0100 0x64538 Activity 0x6c4a8 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down 2019-07-09 18:11:14.429404+0100 0x5afcd Activity 0x6c4a9 1609 0 bird: (CloudDocsDaemon) server-zone/did-sync-down

Interesting. These log messages seem to indicate that it’s syncing something down, but what? My files don’t appear to be syncing at all. There’s no obvious activity in the Finder. Curious I had a look at what files the bird process had open:

lsof -p 1609

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME bird 1609 admin cwd DIR 1,4 832 2 / bird 1609 admin txt REG 1,4 26656 1400386 /System/Library/PrivateFrameworks/CloudDocsDaemon.framework/Versions/A/Support/bird bird 1609 admin txt REG 1,4 21024 3013734 /Library/Preferences/Logging/.plist-cache.hjohswfh bird 1609 admin txt REG 1,4 27154336 2775567 /usr/share/icu/icudt62l.dat bird 1609 admin txt REG 1,4 32768 3015531 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-shm bird 1609 admin txt REG 1,4 32768 3082860 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-shm bird 1609 admin txt REG 1,4 32768 3015536 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-shm bird 1609 admin txt REG 1,4 32768 3082865 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-shm bird 1609 admin txt REG 1,4 65536 3083096 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-shm bird 1609 admin txt REG 1,4 288952 897141 /Library/Application Support/CrashReporter/SubmitDiagInfo.domains bird 1609 admin txt REG 1,4 238448 2720071 /private/var/db/timezone/tz/2019a.1.0/icutz/icutz44l.dat bird 1609 admin txt REG 1,4 6270976 3079606 /private/var/folders/3b/xf71n9xx2jz2spvzcy9wc4fh0000gn/0/com.apple.LaunchServices-231-v2.csstore bird 1609 admin txt REG 1,4 973824 2745534 /usr/lib/dyld bird 1609 admin 0r CHR 3,2 0t0 311 /dev/null bird 1609 admin 1u CHR 3,2 0t0 311 /dev/null bird 1609 admin 2u CHR 3,2 0t0 311 /dev/null bird 1609 admin 3r DIR 1,4 192 3002924 /Users/admin/Library/Application Support/CloudDocs bird 1609 admin 4u REG 1,4 4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db bird 1609 admin 5u REG 1,4 4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db bird 1609 admin 6u REG 1,4 32768 3015531 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-shm bird 1609 admin 7u REG 1,4 487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db bird 1609 admin 8u REG 1,4 4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal bird 1609 admin 9u REG 1,4 32768 3015536 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-shm bird 1609 admin 10u REG 1,4 4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db bird 1609 admin 11u REG 1,4 32768 3082860 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-shm bird 1609 admin 12u REG 1,4 4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db bird 1609 admin 13u REG 1,4 4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal bird 1609 admin 14u REG 1,4 32768 3082865 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-shm bird 1609 admin 15u REG 1,4 4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db bird 1609 admin 16 NPOLICY bird 1609 admin 17r REG 1,4 487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db bird 1609 admin 18r REG 1,4 4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db bird 1609 admin 19u REG 1,4 4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal bird 1609 admin 20r REG 1,4 4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db bird 1609 admin 21r REG 1,4 4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db bird 1609 admin 22u systm 0x455f0d0e6f39ca23 0t0 [ctl com.apple.netsrc id 8 unit 8] bird 1609 admin 23r REG 1,4 487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db bird 1609 admin 24r REG 1,4 4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db bird 1609 admin 25r REG 1,4 535632 3015535 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-wal bird 1609 admin 26r REG 1,4 2488512 3015530 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-wal bird 1609 admin 27r REG 1,4 4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db bird 1609 admin 28r REG 1,4 4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal bird 1609 admin 29r DIR 1,4 832 2 / bird 1609 admin 30r REG 1,4 4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal bird 1609 admin 31r REG 1,4 4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db bird 1609 admin 32r REG 1,4 4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db bird 1609 admin 33r REG 1,4 4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal bird 1609 admin 34r REG 1,4 4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal bird 1609 admin 35u REG 1,4 1658200064 3083092 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db bird 1609 admin 36u REG 1,4 4321912 3083095 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-wal bird 1609 admin 37u REG 1,4 65536 3083096 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-shm bird 1609 admin 38r REG 1,4 288952 897141 /Library/Application Support/CrashReporter/SubmitDiagInfo.domains

Interesting. Lots of files that end in .db - databases perhaps? The one most obviously associated with cloud docs seemed to be:

/Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db

So I went to that directory and had a look. Sure enough this file is a SQLite database. What’s more it was slowing growing in size. So the obvious inference is that this is some kind of metadata database that is being actively synced by the system. Presumably when this is done the files themselves could be synced.

And that turned out to be the case. Eventually it finished syncing its metadata and the files themselves started appearing. So there wasn’t actually any problem, but within the first minute of my call with Apple Support they agreed with me that there was a problem and would no doubt have wasted hours of my time trying to fix something that wasn’t actually broken.

I haven’t dug too deeply into how the syncing process works but I have a theory about why it stalled for so long in my case. I had some git repos in my iCloud Drive that had a lot of files in them. One of them had over 77k files. The way the SQLite database was growing and the corresponding log messages being reported by the bird process lead me to think that it was syncing records in the database one by one or in batches, rather than simply downloading a snapshot of it. This is presumably slow because the number of API requests is a factor of the number of files for which it has to hold metadata. Somebody with a huge number of files in their iCloud Drive would presumably experience this taking even longer than I did, especially if they didn’t have a particularly fast internet connection.

Why Apple chose to make this process invisible in the UI is a mystery. Their discussion forums are full of people baffled by this. They must have had endless support calls and genius bar visits in relation to it, and in many of those cases there may have been nothing actually wrong. The only issue was that the UI gives no indication of what it’s doing so the user thinks it isn’t doing anything

This is bad design.