Real Device Cloud For Native App Testing

According to Statista, the global number of app downloads in 2020 was 218 Billion. This means more testing and optimization for mobile app testers. Testers are continuously looking out for services that offer real device cloud or virtual testing platforms to test their apps on different devices.

Source

Mobile app testing presents a number of challenges to the testers and the team as a whole. The decisions that you make on every step will shape the output you get in the form of an application. But what influences such decisions? Well, it depends on a lot of things. The location of the CTA button can be influenced by the user’s behaviour and past analysis. With so many things to consider, real device testing helps you ensure that your app is compatible and responsive across different devices.

That said, the question is should you build your own real device testing labs or go for already prepared and ready-to-use infrastructure provided by other organisations? The answer to all these questions is much more complex than a simple yes or no. This post aims to give some clarity on this dilemma. The blog is divided into two separate sections. The first one is dedicated to answering the virtual device vs real device cloud testing while the second one focuses on real-device cloud and real-device in-house setup.

Virtual Testing Platform vs. Real Device Cloud

Virtual testing platforms are software that emulates a mobile devices. You can test your mobile apps using an emulator or simulator directly from your desktop.

Also read – Getting Started With Mobile App Testing Using Emulators And Simulators

The need for virtual testing platforms arose when computers and other high-tech systems were costly and hard to obtain. In the 1980s, emulators and simulators were designed to test software without installing it on the physical device to verify if it’s working correctly or not. One such example is HP printers. Since hardware development was costly, emulators and simulators could give a brief idea about the correctness of the system. This was confined to high-tech companies such as HP though.

Windows XP running an emulator

The decade 1990s is the most well known era of emulators and simulators. The need for virtual platforms grew as gamers wanted to play the outdated and forgotten arcade games of the 1980s in their new systems.

Virtual testing platforms were extensively used for mobile application testing and development to carefully test if our application is as stable as it should be or not. On the other hand, the technologies involved in mobile applications and device development have evolved too over the years.

This is where the question arises. Should we continue using virtual testing platforms as we did ten to fifteen years back? Or do we test on real devices? But maintaining an on-premise device lab is extremely costly and difficult to maintain considering the device matrix.

To analyze this question and come to a conclusion, I have drafted a few points with respect to today’s dynamic and challenging mobile app testing environment. We can compare both emulators/simulators and real device testing against these points and see which one weighs more than the other.

Also read – 11 Key Challenges & Solutions Of Mobile App Testing

Point by Point Analysis

The following parameters can be considered while choosing a virtual testing platform and real device farm on cloud for testing your mobile application.

User Interface

The user interface is an important aspect of any web application or a mobile application as in our case. 85% of adults think that a company’s mobile website should be as good or better than their desktop website. So what can be the best choice for testing the user interface of the application?

The user interface does not need any type of hardware to be tested. If you have an application that can run on minimum configurations and the only thing it focuses on is how well it looks, a virtual testing platform is the best choice for you and you can just go ahead with that.

Network Conditions

Network conditions are one of the major focus areas in mobile app testing because of the regular mobile usage by the user. The different bandwidths on which different users operate cannot be anticipated. Even if we analyze the data and consider that so many people are today on 4G and above, it does not alter the fact that network conditions are unreliable when the device is on the move. Some experts say that mobile app testing (android app testing and iOS app testing) should also focus on offline data management to avoid bluntly showing the user about the poor networks.

The focus has been so much that almost all of our mobile app testing options now come equipped with network bandwidth throttle. Even chrome dev tools can help you select networks. Virtual testing platforms have started giving options for network throttle and there are software like ‘Clumsy’ that can help you lose or tighten the bandwidth number for testing. But all of these options are not enough to test your application’s performance in real conditions.

LambdaTest Emulator Network Throttling

When we talk about bandwidth, we not only talk about the reception but also the hardware issues and latency in the network reception due to other reasons. The perfect choice for this is to perform real device testing with network options.

Sensors

As far as only Android devices are concerned, the following are the sensors:

• Accelerometer – measuring acceleration.
• Gyroscope – measuring the rotation angle.
• Magnetometer – for magnetic intensity (compass related app etc.)
• Barometer – for GPS and air pressure.
• Light – to measure light intensity.
• Proximity – distance from the sensor to the closest visible surface.

Apart from the aforementioned popular sensors, there are other sensors such as temperature measuring sensors and relative humidity measuring sensors. If your application uses any of the sensors, the accurate reading from them is important to generate an action on the application. For example in some mobile gaming applications gyroscope is used when a user tilts the screen. Tilting can relate to one specified action of moving forward or jumping above etc. The speed of moving forward or how high he jumps depends on the exact tilt value (as seen in the below image). It cannot be kept constant and users are extremely cautious (and sensitive) when it comes to sensor-related operations.

Source

The sensor issue is too sensitive to be left on the emulators. They sure do support these sensors and should definitely be used in the initial stages of testing. However, when you are done with development and initial testing, it is important to test on a real device to be sure that the application is reacting exactly accurately to each sensor value. This is one of the things that should never be left on chance or on emulators.

Performance

The next thing we focus on in mobile application testing is the performance metrics. This is a very wide area and to understand this completely would probably need a guide of itself. To brief about it, the performance of the application showcases how well our application performs technically (not through market engagement). It gives us a scope of improvement and helps us realise where the application is lagging behind. The following areas can be considered as highlighted benchmarks for the analysis of mobile app performance:

• Load time in different devices.
• Threshold traffic bandwidth – how the application behaves on high traffic.
• Parameters in different network conditions.

Apart from the network conditions, the load time and threshold can be worked around in virtual testing too. Load time in virtual testing and in the Mobile testing lab may not exceed more than 1 second generally. But to generate a final performance report, you may need network conditions data and offline application working data. Both of these may require real device testing in the end.

Price of Testing

Price is not a parameter in testing but is a matter of consideration for the team and the organisation. It is a fact that even though you might want to use real devices, you may have to refrain from it because it costs a lot more than emulators. A lot of the virtual testing platforms are even free to use such as Android Studio and other open-source ones. But they do lack functionalities required to a tester and just focus on simple app rendering.

If you want good emulators with a lot of functionalities, you can get them through online tools at a very little price. You can explore LambdaTest in the same aspect.

Maintenance

The next point to judge virtual platform vs. real device cloud is the maintenance department. The maintenance of the devices is a major challenge in mobile app testing and there is no doubt that a virtual testing platform will win this point by a large margin.

Virtual testing platforms are fairly easy to maintain. All that you have to do is access the platform from the cloud or download them onto your system. On the other hand, when you want to test on real device, you have to actually buy that new device. Not only this, the real device comes with high operational and maintenance costs. All this has to be taken care of by the app tester if they are going for the real device.

Battery

As the maintenance is won by virtual platform by a large margin, “battery” goes on the side of real devices. Battery drainage is a real issue today. The mobiles are far more advanced and a user sticks to the mobile for long hours in a day. Mobile device manufacturers try their best to keep notorious applications restricted from sucking the battery dry. Not only this, mobile manufacturers make sure that they highlight their battery usage (such as how long a charge will last) in the release events.

Graph showing steep battery drain

So why does something become so important that even manufacturers showcase it like an extraordinary feature? Because people are concerned about it while purchasing a mobile device. Similarly, people are concerned about it while installing mobile applications on their devices. If your mobile application stands out from other apps in the rate by which they drain the battery, people will uninstall it.

This needs to be tested regularly, religiously and accurately. Virtual testing platforms are never the way to go forward when it comes to battery usage. There is no way that a virtual platform can tell you precisely (or even in the ballpark) how a battery will drain with your mobile app usage. Always choose to test on real devices (maybe fewer if cost is an issue) when it comes to battery.

CPU Usage

Similar to the battery of the device, the CPU usage on the virtual platform can also mislead you into a lower number when the story could be something else entirely. The two segments i.e. battery and CPU usage are even related to each other as a higher CPU usage generates more heat and consumes more battery. Since there are hundreds of processes going on in a real device, it is the only way to get an accurate idea of the CPU usage. While you are at it, don’t miss the memory usage and the GPU usage metrics too.

It is clear from our point by point analysis that real device testing is the winner in testing the applications accurately. They do cost more but a cost-efficiency tradeoff always exists when it comes to electronic devices. The next question is, should we build our own labs or look out for any other convenient option? Like before, let’s not rush and analyse which method is more helpful to us than the other.

Also read – The Ultimate Mobile App Testing Checklist

Why Is Real Device Cloud Testing A Better Option?

Real device cloud is a mobile testing environment that allows users to access real devices to test their apps on a wide range of Android and iOS devices directly from the desktop browser. Real device cloud testing allows you to test your native apps for functionality, compatibility and reliability.

Why should you go for a real device cloud testing when you can build your own device lab? If you can buy devices and you can hire people to build the device lab, you should. The foremost concern in implementing this is the cost factor and is definitely one of the factors you should consider while choosing to build your own device lab. The more device coverage you are willing to achieve (the higher the better), the more it is going to cost you.

Also read – Getting Your Mobile App Testing Strategy Right The First Time

However, we may definitely not jump straight to the conclusion with this one answer only. As time goes on, you may find a number of reasons rising up. We have listed a few below.

Just of highlight, you can use LambdaTest’s testing cloud to perform manual and automated end-to-end testing at scale. Signup now to increase your browser coverage and drastically cut down your test execution times.

Device Management

A mobile application test matrix can invite 100 to 300 (or 500) unique devices. Acquiring so many devices can incur costs but a lot more can be spent on their maintenance. Newer devices keep on coming into the market and older ones are phased out. One part of maintaining your device lab is to keep purchasing new ones. On the other hand, you need to keep each device updated to their latest firmware version, operating system etc. All this multiplied to 300-500 costs a lot of time and hence a lot of money.

Real device cloud testing can solve your problem as you don’t need to worry about the updation of devices. All the maintenance work is done by the service providers. With LambdaTest you can access over 3000+ real devices instantly. View the list of devices supported by LambdaTest.

Now with LambdaTest, you can perform real-time live interacting testing of your website or webapp starting from Firefox 63 Beta to the latest Firefox version available.

LambdaTest real device cloud screen

Infrastructure Management

Infrastructure management may be seen as the most troublesome thing when you are maintaining a real device cloud. Once you have devices in your hand, everything else related to your desktop screen to these devices comes under infrastructure. You won’t plug-in each device to test the application. You have a connection, a grid (like selenium grids), mobile app testing frameworks attached to this network, a method to pass the automation data so that it propagates through each device, a network that maintains the health of devices and much more.

Also read – Best Mobile App Testing Framework for Android and iOS Application

It becomes very hard to maintain and set up the infrastructure when we are personally managing the device lab. It is not that it is a one-time thing. Infrastructures, real devices or any other, constantly require repair work, upgradation and other such tasks. When you are on a real device cloud, you don’t need to worry about it. You don’t even need to worry about the servers that are keeping everything alive. All you need to do is log in to your account and select the device. How the infrastructure is maintained or setup does not concern you in any way.

Workforce

All of the above points require a good number of man-hours to keep everything running. Organisations reduce this load by increasing the number of engineers working just on the device lab maintenance and everything related to it. Needless to say, this costs extra money as a part of their salary which can be avoided if there was no device lab to maintain such as when we opt for real device cloud labs.

Scaling

Scaling in mobile app testing through the real devices is keeping the latest device in hand which are released very frequently today. These devices do not phase out at the same rate as they are released into the market. With real device cloud testing scaling becomes easier as the new devices are added on the cloud as soon as they are released. This also boosts up our compatibility criteria while releasing the application.

Software Delivery

The consequence of all the points discussed above comes down to the software delivery time. When we are spending so much of our time on non-testing related things, software delivery gets delayed which affects our credibility and business. Maintaining a real device ourselves poses this challenge and therefore it is advisable to go for real device cloud testing as it keeps you aligned to the timelines especially when agile makes things even more time-related.

Another factor that influences software delivery is when multiple teams are using your device lab. Real device cloud is made for multiple users accessing the devices simultaneously. This does not happen in the personal real device lab. If one team is operating on the device lab (be it the development team for alpha testing or another testing team) you just have to wait for your turn. This can further delay the timelines.

Reports

The final point we may consider before deciding to go for a manual real device lab setup or cloud-based is the reporting. Reports are much more important than they sound like at the first stage. They don’t tell you what tests you ran, what functions you tested or even what language you used. They tell you the overall analysis of how everything went so that anyone can give it a glance and understand the current situation. But how it is represented makes a lot of difference too.

Practically, your in-house real device lab and real device cloud lab both can provide a good reporting system. There are only two differences here. First is that you have to invest some time in aggregating the results, building a mechanism that is robust and visually appealing. This is a one-time job unless you have the same infrastructure and in every sense, manageable. The real device cloud will have all this ready for you and there are a few engineers who are regularly involved in improving these reports. This is challenging in the in-house setup as real device cloud reports will regularly improve.

Second, cloud-based reports give a lot more options for the team that makes the collaboration quite easy. For example, real device cloud based reports are saved automatically on the account. Any team member can have a look at the reports from any location at any time easily. Moreover, when a report is generated, cloud-based real device labs have a lot of collaborations with companies that make report sharing easier. For example, you can directly share the file to the team via email with one click, or you can attach it to a JIRA ticket etc. If you can invest your time and costs in developing such a system by yourself, it generally becomes a personal choice and therefore there is no winner here.

Why Choose LambdaTest?

LambdaTest is a cross browser testing platform that allows you to test web and mobile apps using cloud. It enables app testers to perform cross browser testing, mobile app testing, responsiveness test, on different browsers and mobile devices.

LambdaTest’s mobile device testing feature allows you to perform both manual and automated device testing of your native apps.

Here’s a short glimpse of LambdaTest mobile app testing features:

1. Test your websites and native applications on a wide range of iOS and Android devices.
2. Perform responsiveness test, and compatibility test on any device from their real device cloud.
3. Completely eliminate the need for an on-premise device lab.
4. Test instantly on real devices using our real device testing cloud with zero setup or maintenance charges.
5. Run high volumes of tests on real devices and reduce defect rate.
6. Integrate directly with popular CI/CD tools.
7. Identify and debug apps in real-time using device logs, network logs, app logs, screenshots and videos. Identify UI anomaly using inspect element.
8. Upload your .apk, .aab or .ipa files to test new apps or install production apps by downloading them directly from the App Store or Play Store.
9. Test native device features such as camera, touch screen, gestures, etc using real device cloud as if the mobile were in your hand.
10. Reduce false positives while running automation and ship with confidence.
11. Configure your automation frameworks tests easily by using our real device capabilities generator.

Just of highlight, you can use LambdaTest’s testing cloud to perform manual and automated end-to-end testing at scale. Signup now to increase your browser coverage and drastically cut down your test execution times.

What Are Your Thoughts?

In this post, we drew a line between two sets – virtual platform and real device cloud. Generally, when I meet people I never get a clear answer on any of the questions you would have asked yourself when you landed on this page. Should I use an emulator or go for a real device? Why should I choose cloud over in-house development? Sometimes, people are sure that they can build their own setup as they have money. A lot of them fail to do so but a lot of them succeed too. Now perform free cross browser & live testing.

Here is a quick video tutorial on the real-time browser testing.

In my opinion, if you are using real devices, the better option is to use real device cloud testing because they make life easier. They are not like renting a data centre from AWS whose bills keep increasing to millions of USD with time. Real device cloud keeps their prices affordable with customized enterprise plans and other variety of plans for different teams (individual, small team etc.). They are free of any overheads and let your teams focus on only one thing – testing. The end result is the increased development cycles and faster and timely software delivery which is what each organisation strives for.