Smart Environments

 

A manual-procedural activities (MPA) involves following the steps of a given workflow for manipulating the physical world. Examples include manual assembly, repair and maintenance, different crafts, cooking, etc. To learn an MPA the trainee needs to master both the steps of the procedure and the hand skills required for manipulating physical objects and the using the tools. TUMA: An Intelligent Tutoring System for Manual-Procedural Activities supports trainees in learning an MPA. 

Multiple topics available in context of the TUMA project. For details of the topics please see my chair web page. 

According to current studies intuitive ways of controlling are for instance gestures, the use of voice-recognition or mobile phone applications. In this thesis the focus is set to the use of voice-recognition and the combination with existing controls. A possible way of integrating voice-recognition in the current system is using Apple’s HomeKit. This framework provides the possibility of using Siri, Apple’s voice assistance, to control accessories. Furthermore, users can exchange the application with any other HomeKit supporting application in the App store according to their personal preferences.

The central component of this research is an editor for the definition and customization of multimodal interactions and controls in smart buildings.

The master thesis studies how Radio-frequency Identification (RFID) along with Cloud-based computing can be used to improve the user experience of household appliances. Bringing RFID and cloud computing together provides a modern solution enabling the users ubiquitous real time control and management of a wide range of household appliances. This thesis describes the development of a scalable platform named "Cloud oriented Push Platform" (CoPP). The main purpose of CoPP is to connect and control various smart devices. These functions are enabled by storing and retrieving user information, and are supported by push notification services. Furthermore, CoPP may facilitate the development of a low-cost home automation system that is easy to deploy and maintain. Finally, this thesis displays a development and implementation process of a cloud based RFID solution for a content-aware smart fridge. Additionally, in order to determine the best solution for the implementation of the smart fridge, this thesis also evaluates available cloud computing services, RFID technologies, and different micro-controllers from several perspectives, keeping economic, legal, and technical restrictions in mind.

In our research we propose a portfolio of serious games for home-based stroke rehabilitation. The game approach enriches a patient’s training experience and thus establishes a higher level of compliance to prescribed exercises, while maintaining a supportive training environment as found in common therapy sessions. Our system monitors the patient’s performance while exercising and provides clinicians with an interface to personalize the training for their patients. In order to achieve an increased compliance to home-based exercising, we provide a collection of mini games based on rehabilitation exercises used in conventional physical therapy. All games are part of a joint storyline that encourages a patient to accomplish tasks on a treasure island. Playable games change over time to adapt to the patient’s state of rehabilitation and drive diversification. This research was inspired by a previous project at Carnegie Mellon University that provides a non-gamified way of self-governed exercising. While this prototype was a major breakthrough in the domain of computer-aided rehabilitation, patients experienced difficulties using the complex interface resulting in a lack of compliance and motivation to use the system. However, to offer a non-gamified alternative of exercising to our solution, the previous system was integrated as an independent component, giving patients the choice between the two systems.

This thesis introduces a way of combining any number of location providers into one centralized location provider. By using the input of multiple location sources the accuracy of every single source can be improved and the availability of location information can be increased. Simple scenarios where two sources’ location information overlaps will be addressed as well as more complex ways of combining the information. The system introduced is flexible and extensible in terms of fusion algorithms or sources used. It’s ability to provide indoor location information more accurately and with higher availability makes it a better alternative to existing location based services.

In this thesis, a flexible and extensible framework for communication with fixtures, using a variety of communication protocols is proposed. The solution is based on the REST architectural style and the broker pattern that is used for decoupling the clients from fixtures. The broker provides a uniform protocol for controlling all fixtures in the environment, while hiding the complexity of different fixture communication protocols being used. The protocol, provided by the broker, is a HTTP based protocol which can be easily implemented in any programming language that supports the HTTP protocol. Using this feature, a variety of mobile or web based smart environment applications can be developed.

The current thesis proposes a gamified approach that encourages the office workers to start saving energy in a collaborative way. They can define sequences of actions to be performed immediately or at a later time and send tailored recommendations to their colleagues, in the form of action sequences. By actively using the application, controlling their workplace in an energy-efficient way and encouraging their colleagues to have a pro-environmental behavior, the users enter a competition. The gamification strategy used in the current application makes use of game mechanics elements such as points, badges, levels and high scores. The pro-environmental behavior of the users is rewarded with points and/or stars. By accumulating points, they can climb in the high scores. Each player starts the game with five empty American flags, without stars on the canton. By accumulating stars, they can fill in the flags one by one and advance to the next level.

By making use of flow-based programming paradigms and visual programming, we developed an automation system that is easy to learn and use by people without prior programming knowledge. Automation rules created with our system are represented graphically and modifying rules is as simple as dragging-and-dropping and connecting basic building blocks provided by the system. The system also features a plug-in infrastructure that makes it independent from appliance-vendor-specic protocols used to gather data from sensors or control appliances. Plug-ins can be written by third-party developers to make the system adaptable to every environment that allows access and control via a computer network. It is not tied to a specic appliance manufacturer and can be deployed on all major operating systems and hardware platforms. An informal usability evaluation conducted with the prototype we implemented over the course of the project shows, that our approach to graphical programming is easy to learn by people newly introduced to the concept. People can use the system to easily implement automated behaviour for their smart environments to increase their comfort and safe energy through intelligent, on-demand appliance control.

A solution is proposed that aims at leveraging smart meters and delivers insight into consumption as well as control to office occupants using a mobile application. Occupants receive smart meters for tracking their appliances and are personally in control of registering these. The application gives them access to current, monthly and weekly data as well as the possibility to switch appliances on and off. Additionally, consumption comparisons are presented which express Wh (watt hours) in terms of another appliance with the goal to increase the occupant’s understanding of the consumption of their appliances. This prototype of an iPhone application was built at the Intelligent Workplace at the Carnegie Mellon University. The Intelligent Workplace is a living and lived-in laboratory providing an instrumented environment with sensors and controls to developers and occupants who can test applications in their daily use. The application was created in a user-centered and scenario-based approach. Continuous delivery ensured a highly iterative process with weekly deployments to the occupants. The design of the application was reviewed weekly by experts in building controls of the Intelligent Workplace. At the end of the project a survey was conducted, in which occupants were asked about whether the application helped to increase their understanding of the consumption of the appliances and reduce the consumption. The results show that, among the implemented features, presenting the current consumption, split by appliances and giving occupants the possibility to control them has most impact when this functionality can be accessed fast and conveniently.

In this thesis, I propose NICE (Natural Intuitive Camera-based Environment) control which allows occupants to use freehand-gestures, speech commands and a multimodal combination of both in order to interact with individual devices in a smart building. The usage of gestures and speech is a natural user interface that allows for an easy and intuitive interaction. Backed by a 3D camera, users can interact from various positions in a room without any additional personal device. The development of NICE control was complemented by two dedicated user studies as well as a continuous quasi-experiment in a user-centered approach. The first, a wizard-of-oz study, provided essential findings for the design of gestures and speech commands, whereas the second user study evaluated the real-world performance of NICE control. Participants stated they enjoyed natural user interfaces for controlling a smart room and were able to complete about 90% of the given tasks. Speech showed the fastest overall completion times for the tasks. Notably, participants ranked gesture commands as fastest and most fun.

Building occupants have a considerable interest in the current state of their environment. Studies have shown that making building data transparent helps occupants to understand their energy consumption. In addition, transparent data can help to increase the Indoor Environmental Quality (IEQ) and therefore increase the occupants’ health. However, such data is often not accessible, intransparent and mostly not easy to understand for a typical user. Modern buildings equipped with various sensors tend to collect massive amounts of data, but often fail to provide information in an accessible, user-friendly and meaningful way. To give an example, a level of particulates at 170.000 ppcm might not be a helpful answer to understand the air quality. I pose the research hypothesis that an effective visualization of IEQ-related data in a smart building helps its occupants to develop an awareness about their IEQ. Furthermore, many smart environments offer diverse proprietary and therefore incompatible controlling systems. In order to interact with these or to gather information, its occupants have to switch between multiple interfaces. As a solution, I combine comprehensive and meaningful IEQ information with the possibility to control and improve the environment using a mobile application. The system’s usability is evaluated in iterative cycles with application domain experts and by means of two user studies. The first study serves the purpose of illustrating the user’s comprehension of the design concept whereas the second study evaluates the resulting system’s performance in terms of usability. The studies show that the mobile application enables users to get a better understanding for IEQ-relevant data.

In this thesis, a flexible and extensible framework for maintaining supermarket content and recommending products to supermarket customers, has been proposed. The system is composed of three distinct subsystems namely: Client Applications, Supermarket Content Management System (SCMS) and a Product Recommendation Engine (PRE).

The aim of the thesis is the develop a framework for context-aware recipe recommendation systems. In order to address user-specific recommendations, the system is using three different information sources. First, it analyzes ratings from the community that is using the application (that is developed as a part of the thesis). Second, it extracts the current context of the user by using measurements of the user’s smartphone sensors. The context information includes location, season and activity. Finally, the system uses the data of an activity tracker information to draw inferences about the current mood of the user, which is also part of the current context.