In the far-flung olden days (say, the year 2000 and before), many testing groups struggled to justify automated testing. If you dig into the textbooks and case studies, such as they are, you’ll find advice that payback for automated test scripts hinges largely on the number of test cases. The now-outdated canon said when you get somewhere between 500 and 5000 test cases, you begin to get payback. One of the generally-accepted, main assumptions, however, was that database managers, transaction managers, web servers, and the like gave us two important levers of control: 1) updates weren’t that frequent (maybe once or twice a year), and 2) the enterprise could decide when, or if, to upgrade the platforms.
Mobility has upended this assumption. The new canon is, 1) updates come at the rate of dozens in a calendar year; 2) the enterprise has no control over when to update the platforms (mobile OS and devices).
Even ISV’s like Mobile Labs are affected by the continuous stream of new hardware and software that flow from Apple and Google. To bring the picture into sharp focus, I thought it might be useful to pull together a comprehensive look at Apple’s releases, hardware and software, for two of the platforms that we support with deviceConnect, iOS and Mac OS X. We continually test our products to ensure compatibility with Apple’s evolving software during beta cycles.
For each of these releases, we tested and then updated products. The path may not be straightforward or obvious, however, because we’re mostly dealing with Beta releases, the whole point of which is to find and fix bugs. When we find a problem with a feature or function, we have to decide whether the problem is a Beta release bug in iOS or Mac OS X, or whether the problem requires an update to our product set.
The numbers are pretty interesting; in the second half of the year, we downloaded Apple software or tested Apple devices 34 times, with the intense period from mid-September to mid-December seeing 27 releases or devices. While the Android updates were not as numerous, a new release of Android and several new devices were added to the mix during the same period.
This level of activity reinforces the idea that it’s a brand new day for automated testing, and certainly, mobility has rewritten the canon. Continuous updates and continuous tests are the new necessity.