Video annotation provides a suitable way to describe, organize, and index stored videos. On the other hand,transcoding aims at adapting content to the usedclientcapabilities and requirements. Both cues are now mandatory, given the tremendous demand of multimediaaccess from remote clients, in particular nowadays thatnew terminals with limited resources (PDAs, HCCs, Smartphones) have access to the network. In this paper wepropose an unified framework to define event-based andobject-based semantic extraction from video to provideboth semantic video annotation for video stored andsemantic on-line transcoding from live cameras. Two casestudies (highlights’ extraction from soccer videos for theannotation and people behavior detection in domoticapplication for transcoding) and corresponding experimental results are reported.