A multi-level cache setup like that may work very well, but it would require you to stay very conscious of all caching mechanisms when you want pages refreshed. X3 page-caching, X3 site Object, Apache varnish, Cloudflare, are basically all doing some form of cache, all layered on top of each other. The X3 page-cache is automated, as it will output new page if something changed ... The Site Object (if you are using preload feature), is NOT automatic, and you need to re-create it from panel after something changed. Then you have Varnish and Cloudflare, which will cache outputs of BOTH pages and the site object in front of X3 ....
You can pretty much be certain it was related to either Cloudflare or Varnish. I use Cloudflare, but I am not familiar with Varnish. Sounds like a labyrinth of caching though if you had both. Even if you cleared cloudflare, it would then pull a new page, and if that page was cached by varnish, cloudflare would have a new outdated page. Then if you refreshed your varnish, that would still mean cloudflare had an old page. You would literally need to purge caches simultaneously, in the right order ...