What I mean by subjective experience is what you might refer to as what reality looks like from a specific viewpoint or what it appears like when observed. I'm not sure whether you're assuming a physicalist or idealist position when you say "what we observe is the physical world". My issue with this is that observation usually implies the existence of something which is being observed, the appearance upon observation, and possibly also an observer.
If you claim that the physical world doesn't exist independently of observation, and is thus nothing beyond the totality of observed appearances (seems to me like a form of idealism), then what is being observed? If there is no object being observed, and the fact it it apparent from multiple perspectives is simply a consequence of the coherence of observation, where do the qualities of those appearances originate from? How come things don't cease to exist when they're not being observed?
If you claim that the appearances don't exist independently of the physical world being observed (the physicalist interpretation), why does the world appear different from different perspectives? How do you explain things like hallucinations (there is no physical object being observed, but still some appearance is present)?
The reason I brought up that example is because physicalists usually deny the existence of qualia and claim they're nothing beyond the brain processes correlated with them.
I agree with this idea that reality without a viewpoint doesn't make much sense (maybe it's not logically impossible, but our reality surely isn't like that), but I don't think an unconscious viewpoint can exist. Really, I would say having/being a viewpoint is precisely what consciousness is about.
It's easy to think of reality as some space you can just freely float around (like your unity example), but that's not how we experience it. The only viewpoints we can be absolutely sure actually exist, are our own. Let's say we extrapolate to other conscious beings to avoid solipsism. This still severely constrains the pool of all known viewpoints, but what they have in common is this; their movement is always constrained to some body, which others percieve as matter. In my opinion this hints at the fact that matter is probably not merely some symmetry within how reality is observed. Since it correlates so well with where other viewpoints are (viewpoints are always located where matter appears to be), it makes sense to say that at least a subset of viewpoints appear as matter when viewed from the outside. I think this dissolves the idea that there is no object being observed.
The reason I'm calling reality subjective rather than relative is because I think the fact we can perceive it rather accurately and that human viewpoints are mostly coherent is more the exception than the rule. Take the hallucination example; when you hallucinate an object, what is being observed? I think the only possible answer is that the "viewpoint" in your head is observing some other stuff in your head. Since brain activity during visual hallucinations is very similar to brain activity when viewing a "real" object, this is likely always the case! What our brain is actually doing is collecting massive amounts of information from the environment and constructing integrated experience based on it, which represents the macroscopic features of reality accurately, because that was evolutionarily favourable. This means that the accurate and coherent perception we experience is likely only inherent to sufficiently complex evolved systems. If other viewpoints exist, they probably perceive reality in a completely different way than we do, and for all we know, they could be completely incoherent.
In short, my metaphysical stance is something like this:
The only ontic thing is experience, which is concentrated into minds
Reality is a plurality of interacting minds
Observation is when one mind affects the experience of another
Matter is what minds appear like from the outside
Space isn't some backdrop, but instead emerges from the relationships between minds, specifically the strength of interaction between them