That's more than just a nice collection. It would be hard to move away from those lenses, but I certainly hear you with regards to carrying many of them around with you at the same time. For me it's when I'm hiking or I'm planning on traveling by air that the size/weight/cost of the dSLR equipment becomes a pain.
Have you ever considered instead of moving away from your great Canon equipment altogether supplementing it with a second, smaller and lighter system like the Olympus EM10 III? It's lighter than the Fuji and much less expensive. But perhaps most importantly because it uses a slightly smaller sensor the lenses can be smaller/lighter so that if you take the camera and two or three lenses in your bag you get a big savings in terms of size and weight of the equipment. Because the EM10 III has 5-axis-sensor stabilization you can get by with some fast/light primes and still have stabilization. With the 2x crop factor of the sensor you can save a ton of weight on just a couple of telephoto lenses.
While I think the Oly EM10 line is the best value on the M4/3 side of things the biggest drawback would be their maximum shutter speed is 1/4000th of a second. If that's a deal breaker I'd probably look at the Panasonic GX8. It's a heavier camera(but still lighter than the Fuji) but it's still less expensive than the Fuji. It also has the added benefit in sensor-stabilization that you can use in conjunction with some stabilized lenses to get even better hand held stabilization. And of course if you are into video Panasonic as a brand does some of the best video out there with their M4/3rd cameras. Again, because it has a 2x cropped sensor the lens collection will be smaller and lighter.
That way you could use both systems, each when it's the right tool for the job. I'd just hate to see you stop using that wonderful collection of Canon lenses.
