The real reason your iCloud Drive isn't syncing

9 Jul 2019 18:34 | apple | macOS

I recently had the logic board replaced in my 2017 Macbook Pro. I use the
awesome Carbon Copy Cloner to keep an image of my system as a bootable backup
which can then be easily restored when the machine is returned to me.

This time however I had some issues with iCloud Drive. After restoring the
backup I found it wasn't syncing. I tried the usual troubleshooting steps: turn
it off, reboot the machine, turn it back on again. Still wasn't syncing, just
sitting there with all my files and folders seemingly stuck in the state of
"waiting to download". I scoured the internet, lots of people with the same
issue and no obvious solution. Curiously though some of the threads had people
reporting that their stuff would just suddenly start syncing out of nowhere
without them having done anything to prompt it.

Well I have AppleCare so with a due sense of exhaustion and dread I gave them a
call. We started a screen share session so they could see the problem. Their
first suggestion was to try creating a new user account and seeing if that would
sync. I took exception to this - I don't need this to work on another account, I
need it to work on *this* account. I hung up the call and decided to investigate
myself.

I knew from previous messing around with macOS that the main process involved in
iCloud Drive syncing was bird. I decided to see if it was logging anything:








$ log stream |egrep -i bird
2019-07-09 18:11:09.546067+0100 0x5afcd    Activity    0x6c4a4              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down
2019-07-09 18:11:10.672324+0100 0x5afcd    Activity    0x6c4a5              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down
2019-07-09 18:11:11.626342+0100 0x5afcd    Activity    0x6c4a6              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down
2019-07-09 18:11:12.629151+0100 0x5afcd    Activity    0x6c4a7              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down
2019-07-09 18:11:13.496400+0100 0x64538    Activity    0x6c4a8              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down
2019-07-09 18:11:14.429404+0100 0x5afcd    Activity    0x6c4a9              1609   0    bird: (CloudDocsDaemon) server-zone/did-sync-down

Interesting. These log messages seem to indicate that it's syncing something
down, but what? My files don't appear to be syncing at all. There's no obvious
activity in the Finder. Curious I had a look at what files the bird process had
open:

# lsof -p 1609
COMMAND  PID  USER   FD      TYPE             DEVICE   SIZE/OFF    NODE NAME
bird    1609 admin  cwd       DIR                1,4        832       2 /
bird    1609 admin  txt       REG                1,4      26656 1400386 /System/Library/PrivateFrameworks/CloudDocsDaemon.framework/Versions/A/Support/bird
bird    1609 admin  txt       REG                1,4      21024 3013734 /Library/Preferences/Logging/.plist-cache.hjohswfh
bird    1609 admin  txt       REG                1,4   27154336 2775567 /usr/share/icu/icudt62l.dat
bird    1609 admin  txt       REG                1,4      32768 3015531 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-shm
bird    1609 admin  txt       REG                1,4      32768 3082860 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-shm
bird    1609 admin  txt       REG                1,4      32768 3015536 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-shm
bird    1609 admin  txt       REG                1,4      32768 3082865 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-shm
bird    1609 admin  txt       REG                1,4      65536 3083096 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-shm
bird    1609 admin  txt       REG                1,4     288952  897141 /Library/Application Support/CrashReporter/SubmitDiagInfo.domains
bird    1609 admin  txt       REG                1,4     238448 2720071 /private/var/db/timezone/tz/2019a.1.0/icutz/icutz44l.dat
bird    1609 admin  txt       REG                1,4    6270976 3079606 /private/var/folders/3b/xf71n9xx2jz2spvzcy9wc4fh0000gn/0/com.apple.LaunchServices-231-v2.csstore
bird    1609 admin  txt       REG                1,4     973824 2745534 /usr/lib/dyld
bird    1609 admin    0r      CHR                3,2        0t0     311 /dev/null
bird    1609 admin    1u      CHR                3,2        0t0     311 /dev/null
bird    1609 admin    2u      CHR                3,2        0t0     311 /dev/null
bird    1609 admin    3r      DIR                1,4        192 3002924 /Users/admin/Library/Application Support/CloudDocs
bird    1609 admin    4u      REG                1,4       4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db
bird    1609 admin    5u      REG                1,4       4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db
bird    1609 admin    6u      REG                1,4      32768 3015531 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-shm
bird    1609 admin    7u      REG                1,4     487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db
bird    1609 admin    8u      REG                1,4    4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal
bird    1609 admin    9u      REG                1,4      32768 3015536 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-shm
bird    1609 admin   10u      REG                1,4       4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db
bird    1609 admin   11u      REG                1,4      32768 3082860 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-shm
bird    1609 admin   12u      REG                1,4       4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db
bird    1609 admin   13u      REG                1,4    4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal
bird    1609 admin   14u      REG                1,4      32768 3082865 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-shm
bird    1609 admin   15u      REG                1,4       4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db
bird    1609 admin   16   NPOLICY
bird    1609 admin   17r      REG                1,4     487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db
bird    1609 admin   18r      REG                1,4       4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db
bird    1609 admin   19u      REG                1,4    4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal
bird    1609 admin   20r      REG                1,4       4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db
bird    1609 admin   21r      REG                1,4       4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db
bird    1609 admin   22u    systm 0x455f0d0e6f39ca23        0t0         [ctl com.apple.netsrc id 8 unit 8]
bird    1609 admin   23r      REG                1,4     487424 3015532 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db
bird    1609 admin   24r      REG                1,4       4096 3015527 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db
bird    1609 admin   25r      REG                1,4     535632 3015535 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/client.db-wal
bird    1609 admin   26r      REG                1,4    2488512 3015530 /Users/admin/Library/Caches/com.apple.bird/unlink/211B4D60-AB4C-416B-B0B2-0951D64A4CB7/db/server.db-wal
bird    1609 admin   27r      REG                1,4       4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db
bird    1609 admin   28r      REG                1,4    4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal
bird    1609 admin   29r      DIR                1,4        832       2 /
bird    1609 admin   30r      REG                1,4    4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal
bird    1609 admin   31r      REG                1,4       4096 3082861 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db
bird    1609 admin   32r      REG                1,4       4096 3082856 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db
bird    1609 admin   33r      REG                1,4    4853392 3082864 /Users/admin/Library/Application Support/CloudDocs/session/db/client.db-wal
bird    1609 admin   34r      REG                1,4    4095312 3082859 /Users/admin/Library/Application Support/CloudDocs/session/db/server.db-wal
bird    1609 admin   35u      REG                1,4 1658200064 3083092 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db
bird    1609 admin   36u      REG                1,4    4321912 3083095 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-wal
bird    1609 admin   37u      REG                1,4      65536 3083096 /Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db-shm
bird    1609 admin   38r      REG                1,4     288952  897141 /Library/Application Support/CrashReporter/SubmitDiagInfo.domains

Interesting. Lots of files that end in .db - databases perhaps? The one most
obviously associated with cloud docs seemed to be:

/Users/admin/Library/Application Support/CloudDocs/session/s/com~apple~CloudDocs:0/db.db

So I went to that directory and had a look. Sure enough this file is a SQLite
database. What's more it was slowing growing in size. So the obvious inference
is that this is some kind of metadata database that is being actively synced by
the system. Presumably when this is done the files themselves could be synced.

And that turned out to be the case. Eventually it finished syncing its metadata
and the files themselves started appearing. So there wasn't actually any
problem, but within the first minute of my call with Apple Support they agreed
with me that there *was* a problem and would no doubt have wasted hours of my
time trying to fix something that wasn't actually broken.

I haven't dug too deeply into how the syncing process works but I have a theory
about why it stalled for so long in my case. I had some git repos in my iCloud
Drive that had a *lot* of files in them. One of them had over 77k files. The way
the SQLite database was growing and the corresponding log messages being
reported by the bird process lead me to think that it was syncing records in the
database one by one or in batches, rather than simply downloading a snapshot of
it. This is presumably slow because the number of API requests is a factor of
the number of files for which it has to hold metadata. Somebody with a huge
number of files in their iCloud Drive would presumably experience this taking
even longer than I did, especially if they didn't have a particularly fast
internet connection.

Why Apple chose to make this process invisible in the UI is a mystery. Their
discussion forums are full of people baffled by this. They must have had endless
support calls and genius bar visits in relation to it, and in many of those
cases there may have been nothing actually wrong. The only issue was that the UI
gives no indication of what it's doing so the user thinks it isn't doing
anything

This is bad design.