Modern handheld devices provided with localization capabilities could be used to automatically create a diary of user's whereabouts, and use it as a complement of the user profile in many applications. In this paper we present the Whereabouts diary, an application/service to log the places visited by the user and to label them, in an automatic way, with descriptive semantic information. In particular, Web-retrieved data and the temporal patterns in which places are visited can be used to define such meaningful semantic labels. In this paper, we describe the general idea at the basis of our service and discuss our implementation and the associated experimental results. In addition, we illustrate an application that can fruitfully exploit the whereabouts diary as a supporting service, and discuss areas for future work.