So you've basically got an object identity disconnect. You don't have a proper distinction between what is a by-value object and what is an object reference (pointer). And you just found out that this just doesn't work in C++. You always need to distinguish between having a value (copied when passed), a C++ reference (cannot be reassigned), and a pointer (can be reassigned).
The solution I recommend is making the by-reference holding explicit.
ex += ref(foo); // Array element references foo
ex += foo; // Array element doesn't reference foo