Cloud Platform for Research Crowdsourcing in Mobile Testing

Abstract

Mobile application testing and testing over a cloud are two highly topical fields nowadays. Mobile testing presents specific test activities, including verification of an application against a variety of heterogeneous smartphone models and versions of operating systems (OS), build distribution and test team management, monitoring and user experience analytics of an application in production, etc. Cloud benefits are widely used to support all these activities. This study conducts in-depth analyses of existing cloud services for mobile testing and addresses their weaknesses regarding research purposes and testing needs of the critical and business-critical mobile applications. During this study, a Cloud Testing of Mobile Systems (CTOMS) framework for effective research crowdsourcing in mobile testing was developed. The framework is presented as a lightweight and easily scalable distributed system that provides a cloud service to run tests on a variety of remote mobile devices. CTOMS provides implementation of two novel functionalities that are demanded by advanced investigations in mobile testing. First, it allows full multidirectional testing, which provides the opportunities to test an application on different devices and/or OS versions, and new device models or OS versions for their compatibility with the most popular applications in the market, or just legacy critical apps, etc. Second, CTOMS demonstrates the effective integration of the appropriate testing techniques for mobile development within such a service. In particular, it provides a user with suggestions about coverage of configurations to test on using combinatorial approaches like a base choice, pair-wise, and t-way. The current CTOMS version supports automated functional testing of Android applications and detection of defects in the user interface (UI). This has a great value because requirements for UI and user experience are high for any modern mobile application. The fundamental analysis of possible test types and techniques using a system like CTOMS was conducted, and ways of possible enhancements and extensions of functionality for possible research are listed. The first case studies prove the work of implemented novel concepts, their usefulness, and their convenience for experiments in mobile testing. The overall work proves that a study of cloud mobile testing is feasible even with small research resources.