Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems - PowerPoint PPT Presentation

Loading...

PPT – Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems PowerPoint presentation | free to download - id: 4c46a9-ZGM1N



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems

Description:

Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems Student: Shu-Wei Chou Advisor: Dr. Ying-Dar Lin – PowerPoint PPT presentation

Number of Views:178
Avg rating:3.0/5.0
Slides: 52
Provided by: Char1161
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Designing and Experimenting Conformance and Compatibility Test Suites on Android-Based Systems


1
Designing and Experimenting Conformance and
Compatibility Test Suites on Android-Based Systems
  • Student Shu-Wei Chou
  • Advisor Dr. Ying-Dar Lin
  • Degree Program of ECE and CS Colleges,
  • National Chiao Tung University

2
Outline
  • Android software architecture
  • Dalvik Virtual Machine
  • Android SDK
  • Motivation
  • Conformance and compatibility problem of
    platforms
  • Survey
  • Java(TM) Compatibility Kit (JCK)
  • TAHI IPv6 Conformance Test Suite
  • Problem statement
  • Design of CTS
  • Experiment
  • Contributions
  • Schedule
  • References

High Speed Network Laboratory _at_ NCTU
2
3
Android software architecture
High Speed Network Laboratory _at_ NCTU
3
4
Dalvik Virtual Machine
.java
.class
JVM
javac
Normal Java program
.class
.dex
DVM
dx
Android program
5
DVM vs JVM
DVM JVM
Format .dex .class
Architecture Register-based Stack-based
JIT compiler No Yes
App-to-VM One-to-one Many-to-one
6
Android SDK
SDK version 1.5_r3
Packages Classes Methods
128 ?2480 ?49600
Packages
Classes
Methods



closeContextMenu() createPendingResult (int
requestCode, Intent data, int flags) dismissDialo
g(int id) dispatchKeyEvent (KeyEvent event)
dispatchTouchEvent (MotionEvent ev)
findViewById(int id) .
. .
Activity ActivityGroup ActivityManager AlarmMana
ger AlertDialog AliasActivity Application Date
PickerDialog Dialog . .
.
android android.app android.appwidget
android.content android.content.pm
android.content.res android.database
android.database.sqlite android.graphics
. . .
7
Motivation
  • Conformance problem of platforms
  • System vendors implement different platforms with
    different packages, classes or methods
  • Compatibility problem of applications

High Speed Network Laboratory _at_ NCTU
7
8
Vendor A
A App of vendor A
A
1
2
3
4
1
2
1
2
1
2
6
4
6
8
Android SDK Android SDK
A package
B package
C package
Vendor package
1..9 Classes
Vendor B
B App of vendor B
B
2
3
4
4
2
3
1
2
3
1
2
5
4
5
6
Vendor C
C App of vendor C
C
1
2
3
4
2
3
4
2
3
4
3
4
5
2
3
4
5
9
Compatibility table between vendor and application
A B C
A Y N N
B N Y N
C N N Y
Vendor
Application
10
Survey
  • Java(TM) Compatibility Kit (JCK)

Test Suites Number of Test Cases Summary
JCK-compiler-6a 19119 Check the condition that .java file is compiled to .class file.
JCK-runtime-6a 58903 Check the condition that .class file is executed.
JCK-devtools-6a 18692 Focus on java and xml schema.
  • ANVL(Automated Network Validation Library)
  • TAHI IPv6 Conformance Test Suite

11
Windows CE Test Kit (1/3)
Device
CETK
ActiveSync
Test suites
12
Windows CE Test Kit (2/3)
13
Windows CE Test Kit (3/3)
14
JCK-runtime-6a (1/4)
api - Java API test suite lang - The Java
Language Specification Tests vm - The Java
Virtual Machine Specification Tests xml_schema -
W3C XML Schema Test Suite
  • java_applet - API tests for the java.applet
    package
  • java_awt - AWT tests
  • java_beans - API tests for the java.beans package
  • java_io - API tests for the java.io package
  • java_lang - API Tests for java.lang
  • java_math - API tests for the java.math package
  • java_net - API tests for the java.net package
  • java_nio - API Tests for java.nio
  • java_rmi - API Tests for Remote Method Invocation
    (RMI)
  • java_security - index.html
  • java_sql - API tests for the java.sql package
  • java_text - API Tests for java.text

15
JCK-runtime-6a (2/4)
API Tests for the java.lang Package
  • AbstractMethodError - Tests for
    AbstractMethodError
  • AbstractMethodError serial - AbstractMethodError
    Serialized Compatibility Tests
  • ArithmeticException - Tests for
    ArithmeticException
  • ArithmeticException serial - ArithmeticException
    Serialized Compatibility Tests
  • ArrayIndexOutOfBoundsException - Tests for
    ArrayIndexOutOfBoundsException
  • ArrayIndexOutOfBoundsException serial
    ArrayIndexOutOfBoundsException
    Serialized Compatibility Tests
  • ArrayStoreException - Tests for
    ArrayStoreException
  • ArrayStoreException serial - ArrayStoreException
    Serialized Compatibility Tests
  • AssertionError - Tests for AssertionError
  • AssertionError serial - Test Specifications and
    Descriptions for AssertionError.serial
  • Boolean - Test Specifications and Descriptions
    for Boolean
  • Boolean serial - Boolean Serialized Compatibility
    Tests
  • Byte - Test Specifications and Descriptions for
    Byte

16
JCK-runtime-6a (3/4)
Test Specifications and Descriptions for Boolean
  • public static final Boolean FALSE
  • public static final Boolean TRUE
  • public static final Class TYPE
  • public Boolean(String s)
  • public Boolean(boolean value)
  • public boolean booleanValue()
  • public int compareTo(Boolean b)
  • public boolean equals(Object obj)
  • public static boolean getBoolean(String name)
  • public int hashCode()
  • public static boolean parseBoolean(String s)
  • public String toString()
  • public static String toString(boolean b)
  • public static Boolean valueOf(String s)
  • public static Boolean valueOf(boolean b)

17
JCK-runtime-6a (4/4)
  • public Boolean(String s)
  • Description
  • Domain testing of input and output conditions,
    and external pre-conditions for class Boolean,
    constructor public Boolean(String s).
  • Assertion testing
  • Assertion
  • Allocates a Boolean object representing the value
    true if the string argument is not null and is
    equal, ignoring case, to the string "true".
  • Test Case ID
  • Boolean2005
  • Test Descriptions
  • SeeTest descriptions for CtorTests.java
    (Boolean2005)

18
(No Transcript)
19
(No Transcript)
20
Classification (java.lang)
  • Constructor
  • Serialization
  • Constant
  • Parameters
  • String, int, Object, boolean, char, double,
    float, long, byte
  • Class, char, byte, long, Thread,
    AccessibleObject, Object, StackTraceElement
  • ClassltTgt, ListltStringgt
  • java.lang.Class, java.lang.String
  • ClassLoader, Annotation, URL, Throwable,
    PrintStream, PrintWriter, Thread, ThreadGroup,
    InputStream, InetAddress, Permission,
    FileDescriptor, StringBuffer, StringBuilder,
    CharSequence, Properties, SecurityManager,
    Runable, ReferenceQueue, InvocationHandler

21
public ArithmeticException()
public ArrayIndexOutOfBoundsException()
22
public ArithmeticException() - Test
  • public Status Ctor001()
  • ArithmeticException e new
    ArithmeticException()
  • if( e.getMessage() null)
  • return Status.passed("OKAY")
  • else
  • return Status.failed("FAILED")

public Status Ctor003() int zero 0
try int x 5 / zero
return Status.failed("FAILED")
catch(ArithmeticException e) return
Status.passed("OKAY")
23
public ArrayIndexOutOfBoundsException() - Test
public Status Ctor001()
ArrayIndexOutOfBoundsException e new
ArrayIndexOutOfBoundsException() if(
e.getMessage() null) return
Status.passed("OKAY") else
return Status.failed("FAILED")
public Status Ctor004() int array
0, 1, 2, 3, 4 int testedValues
-5, 5 boolean isPassed true
for(int i 0 i lt testedValues.length i)
try int x
arraytestedValuesi
ref.println("Error for " testedValuesi)
isPassed false
catch(ArrayIndexOutOfBoundsException e)
if(isPassed)
return Status.passed("OKAY") else
return Status.failed("FAILED")

24
public AbstractMethodError(String message)
public ArithmeticException(String message)
25
public AbstractMethodError(String message) - Test
public ArithmeticException(String message) - Test
26
public static Class forName(String name, boolean
initialize, ClassLoader loader) throws
ClassNotFoundException
27
protected Package definePackage(String name,
String specTitle, String specVersion, String
specVendor, String implTitle, String implVersion,
String implVendor, URL sealBase) throws
IllegalArgumentException
protected final Class defineClass(String name,
byte b, int off, int len, ProtectionDomain
protectionDomain) throws ClassFormatError
28
ANVL Test Suite Development
TAL
RFC
Analyze
Negative Tests
Test Method
Test Suite Code
C language
29
ANVL Example
RFC
RFC 1723, section 3, page 3, Protocol
Extensions. RFC 1058, section 3.1, pages 1920,
Message formats.
TAL
The portion of the datagram from the address
family identifier through the metric may appear
up to 25 times.
Test Method
1. ANVL sends a datagram with 26 tuples to
DUT. 2. DUT should not accept the packet. 3. ANVL
sends a RIP request to the DUT. 4. DUT should
return an unchanged routing table.
Test Suite Code
C Code
30
TAHI IPv6 Conformance Test Suite
  • Conformance Test Specification
  • for host
  • IPv6 Specification
  • ICMPv6 for IPv6 Specification
  • Neighbor Discovery
  • IPv6 Stateless Address Autoconfiguration
  • IPv6 Path MTU Discovery
  • Default Router Preferences, More-Specific Routes,
    and Load Sharing
  • IPSec AH and ESP for IPv6
  • IPSec ESP for IPv6(UDP)
  • IPSec AH and ESP for IPv4
  • IPSec AH and ESP for IPv4(UDP)
  • Mobility Support in IPv6 (Link to v6PC MIPv6
    page)
  • Default Address Selection for Internet Protocol
    version 6 (IPv6)
  • DNS discovery
  • IPv6 over IPv4 Tunnel
  • Robustness
  • Address Architecture
  • Multicast Listener Discovery for IPv6 (v1)

31
IPv6 Specification (for host)
No. Title
1 Initialization
2 Source and Destination Address (ping)
3 Unrecognized Next Header in IPv6 Header
4 Payload Length is Odd value
5 Traffic Class Non-Zero
6 Flow Label Non-Zero
7 Hop Limit 0
8 Unrecognized Next Header in Extension Header
9 Hop-by-Hop Options Header Position
10 Destination Options Header Pad1 Option
11 Destination Options Header PadN Option
12 Destination Options Header Unrecognized Option (00)
13 Destination Options Header Unrecognized Option (01)
14 Destination Options Header Unrecognized Option (10)

32
Problem statement
  • To verify the conformance of platforms with
    android SDK and reduce the compatibility issue of
    applications on android-based systems
  • Sub problems
  • Providing test cases to verify the conformance of
    platforms
  • Designing automatic framework test tool

33
Conformance Test Suite (CTS)
  • To provide one method
  • To verify if device conforms android SDK
  • To avoid incompatible application
  • Automatic framework
  • Selective modules
  • Test suites with methodology

High Speed Network Laboratory _at_ NCTU
33
34
Design of CTS
  • Based on Android SDK
  • Process
  • Loading to device under test
  • Run CTS
  • Display test report
  • Design test cases
  • Packages, Classes and methods

High Speed Network Laboratory _at_ NCTU
34
35
CTS architecture
User Interface
Test Cases
Android SDK
36
JCK vs JUnit vs Android
JCK
JUnit
Android
JCKTest Framework
JUnitTest Framework
Android Test Framework
My Java Program
My Android Program
Java packages
37
JCK example
  • package javasoft.sqe.tests.api.java.lang.AbstractM
    ethodError
  • import java.io.PrintWriter
  • import javasoft.sqe.javatest.Status
  • import javasoft.sqe.javatest.lib.MultiTest
  • public class CtorTests extends MultiTest
  • / standalone interface /
  • public static void main(String argv)
  • CtorTests test new CtorTests()
  • test.run(argv, System.err,
    System.out).exit()

38
JUnit example
  • import junit.framework.Test
  • import junit.framework.TestSuite
  • public class TestAll
  • public static Test suite()
  • TestSuite suite new TestSuite(TestSuite
    Test)
  • suite.addTestSuite(TestSample.class)
  • return suite

39
Android example (1/3)
  • //MathTest.java
  • import android.test.AndroidTestCase
  • import android.util.Log
  • public class MathTest extends AndroidTestCase
  • protected double fValue1
  • protected double fValue2
  • protected double fRe
  • static final String LOG_TAG MathTest
  • protected void setUp()
  • fValue1 2.0
  • fValue2 3.0
  • fRe 5.0
  • public void testAdd()
  • Log.d( LOG_TAG, testAdd )
  • assertTrue( LOG_TAG1?, ( (fValue1
    fValue2 ) fRe ) )

40
Android example (2/3)
  • //ExampleSuite.java
  • import junit.framework.TestSuite
  • public class ExampleSuite extends TestSuite
  • public ExampleSuite()
  • addTestSuite( MathTest.class )

41
Android example (3/3)
42
Android package architecture
Package
Interfaces
Classes
Exceptions
Enums
Errors
Public Methods
Protected Methods
Inherited Methods
43
Android.app Package
Number
Class 37
Method 493
Parameter Count Number
0 157
1 196
2 82
3 36
4 11
5 7
6 2
7 1
10 1
Parameter combination 123
44
Conformance Test Suite
public class actsTestSuite extends TestSuite
public class ACTS extends Activity implements
TestListener
AndroidTestRunner testRunner new
AndroidTestRunner()
public class TestCase1 extends AndroidTestCase
testRunner.setTest( new actsTestSuite() )
public void addError public void
addFailure public void endTest public void
startTest
public class TestCase2 extends AndroidTestCase
45
Test Case Design
public class Activity
public final void dismissDialog (int id)
int -2147483647 2147483648
public void setUp() public void
testMethodName() public void testAndroidTestCase
SetupProperly()
isPass true
Activity a new Activity() try
a.dismissDialog(TestValue) isPass
false catch (IllegalArgumentException e)

46
Coverage verification
Debug.startMethodTracing("test") Debug.stopMethod
Tracing()
Generate HTML report
Tcl script
Execute Applicatoin
Parsing class and method
Generate test.trace file
dmtracedump
47
Compatibility test study
1.Modify Android source code 2.Build new
image 3.Load to android handheld
Design 3 conditions 1.Method disappear 2.Methods
parameter inconsistency 3.Version issue
Load App
Observe Result
48
Experiment
  • Market place
  • Android handheld

49
Contributions
  • Using CTS tool we can
  • Check if device conforms android SDK
  • Avoid incompatible condition in applications

High Speed Network Laboratory _at_ NCTU
49
50
Schedule
  • Study android SDK - before 9/30
  • Design CTS tool framework - before 10/31
  • Test cases design - before 11/30
  • Analysis the result of test - before 12/31
  • Writing and revising paper - after 99/1/1

High Speed Network Laboratory _at_ NCTU
50
51
References
  • The Java Compatibility Kit (JCK) read-only
    source bundles (https//jck.dev.java.net/)
  • JT Harness Project (https//jtharness.dev.java.net
    /)
  • JCK6b(http//www.openjdk.org/groups/conformance/do
    cs/JCK6bUsersGuide/html/p10.html)
  • Android SDK(http//developer.android.com/guide/ind
    ex.html)

High Speed Network Laboratory _at_ NCTU
51
About PowerShow.com