Learning Interactions Across Spaces: a Framework for Contextualised Multimodal Observations