One of the many struggles of developing applications for a mobile audience is testing your app—with so many different devices, you’re usually limited to how many you have on hand. AWS DeviceFarm makes this process easier.
What Is DeviceFarm?
The concept of DeviceFarm is fairly simple. AWS spends the money to buy a bunch of devices (over 2,500 are in their library, currently), and rents them out whenever someone needs to use them. You can connect to the device directly and interact with it, just like if you were using a device emulator, but with real hardware on the other end.
While you can connect to individual devices and test on devices you don’t have, the real benefit of DeviceFarm is the automation. It enables you to run automated UI tests with the help of the Appium testing framework to test your apps on multiple devices simultaneously. DeviceFarm records the screen and inputs of each automated test, so you can go in and review what went wrong should any bugs occur.
DeviceFarm charges based on the number of minutes used, at a rate of $0.17 per device minute. This number is multiplied by the number of devices you’re testing on, if you’re running automated tests.
If you’re doing a lot of tests and want to take advantage of DeviceFarm’s automated testing tools, you can purchase unlimited usage at $250 per month, per device. This is only worthwhile if you’re doing more than 24 hours of testing per month on any single device, which is a lot of hours to be running tests, but it may be worthwhile for some enterprise customers.
Do I Need On-Device Testing?
DeviceFarm supports native apps, hybrid apps, and full web apps. Benefits vary, depending on what kind of application you’re developing.
Native mobile apps definitely need constant on-device testing. While XCode provides iOS emulators, they won’t always be perfect, and you’ll catch a lot of bugs testing on actual hardware.
Hybrid apps are a bit different. This category includes apps built with frameworks like React Native; most of the app’s code is written in JavaScript, which makes building a web app and a corresponding mobile app much smoother as a lot of code can be shared. However, rather than rendering in a browser, the hybrid framework enables you to render native components in your JSX, making hybrid apps more performant.
With hybrid apps, you won’t need to as much on-device testing, as almost all native code is handled in the library. However, you may still run into bugs between emulated and real devices, and you should still do on-device testing.
With web apps, you may be able to forego on-device testing for the most part; Safari on desktop uses same WebKit-rendering engine as iOS browsers, so you won’t find many differences between the two. Both Chrome and Safari both have tools for testing responsive design. If you open up the dev tools, you can choose between a few different device resolutions to preview how your site works.
While this doesn’t emulate the device, only the resolution, most of the rendering bugs are related to tight mobile constraints anyway.
While it’s not as necessary with web apps, we would still recommend you double check your site on real devices to make sure the experience is smooth and bug free. You may not need DeviceFarm to do it, but it’s good if you don’t have a wide variety of devices on hand.
How to Get Started with DeviceFarm
Head over to the DeviceFarm console and enter a new project name. Manual access is easier to set up, so start with that. Click on the “Remote Access” tab and create a new session.
From here, you can choose a device to connect to. The availability of these devices may depend on who is using them at any given moment, but there seems to be a wide array of device types from which to choose.
Create a new session, and after a minute or so you’ll be greeted with the home screen. The connection isn’t perfect (about equivalent to a standard remote desktop view), but it’s enough to do basic testing. Your keyboard should connect to the device if you’re focused on it, making typing easier.
From here, if you’re debugging a web app, you can open it up directly in the browser. If you’re debugging an app that needs to be installed, you can sideload the .apk or .ipa through the “Install Applications” dropdown in the sidebar. This installs your app to the device for testing. If you’re repeatedly testing multiple devices, you can choose from a dropdown of your recent uploads rather than uploading again.
For automated testing, the process is a bit different. You need to head back to the console and instead create a new run under the “Automated Testing” tab:
From here, select which kind of app you’re developing, and upload your .apk or .ipa if you’re developing a native or hybrid app.
Next, upload your tests. DeviceFarm supports the Appium automation framework for running UI tests on both web and native apps. For native apps, a built-in “Fuzz” tester randomly can click around and test for bugs.
After that, select all the devices on which you want to perform testing. By default, DeviceFarm picks the five most popular devices, but you can create your own device pool.
Now, you are asked to specify the maximum number of minutes you want the devices to run for, and AWS can show you a total based on the number of devices you’re testing in parallel. This will be the maximum cost you’ll pay, but if your test finishes early, you won’t be charged past then.
