Rotor Shared Source CLI 1.0 A short introduction

1
Rotor / Shared Source CLI 1.0A short introduction

• David EMSELLEM - Michel RIVEILL
• Laboratoire I3S - Univ. Nice Sophia Antipolis
• emsellem_at_essi.fr - riveill_at_unice.fr

2

• Lécole ICAR03 est soutenue par

3
Agenda

• Rotor vs .Net
• others project
• Inside Rotor
• the component model
• Thanks
• Peter Drayton, Program Manager, CLR, Microsoft
  Corporation, pdrayton_at_microsoft.com

4
Main reference

• Rotor book
• Shared Source CLI Essentials, David Stutz, Ted New
  ard, Geoff Shilling, March 2003, ISBN
  0-596-00351-x, 336 pages
• Rotor web site
• http//www.sscli.net
• http//www.ondotnet.com (see CLI and rotor entry)
• Mailing list and newsgroup
• http//dotnet.di.unipi.it (Università di Pisa)
• http//discuss.develop.com (Develop Mentor)
• nntp//microsoft.public.shared_source.cli

5
The Shared Source CLI / Rotor

• Example implementation of full ECMA standards
• 334 C language
• 335 Common Language Infrastructure (CLI)
• Non-commercial, source code distribution
• Modifiable (and doced for modification)
• Easy to re-distribute and share changes
• Cross-platform implementation
• Code builds/runs on FreeBSD, Windows XP and
  MacOS X
• Code designed to easily port to other platforms

6
Why MS built Rotor

• Objective Foster a vibrant research and teaching
  community around our most important technology
• This drove the definition of Rotor
• Complete example to enable research and support
  teaching of modern programming languages
  infrastructure
• Also tied closely to the whole of .NET
• Support .NET developer learning
• Support our standards efforts
• Provide a laboratory for future work

7
C/CLI Standardization

• ECMA standards work began Summer 2000
• Participants included Intel, HP, IBM, Netscape
• C CLI standards ratified December 2001
• ECMA forwarded the work to ISO
• Successfully completed ballot October 2002
• ISO/IEC 23270 (C), 23271 (CLI), 23272
  (libraries)
• Structure of the CLI standard (ECMA-335)
• Partition I - Concepts and Architecture
• Partition II - Metadata
• Partition III - Common Intermediate Language
• Partition IV - The Libraries and Profiles
• Partition V - Annexes

8
Overview of the ECMA CLI

• A common type system
• and a specification for language integration
  (CLS)
• Execution engine with garbage collector and
  exception handling
• Integral security system with verification
• A file format
• PE/COFF format, with extensions
• An extensible metadata system
• An intermediate language
• CIL Common Intermediate Language
• A factored class library
• A modern equivalent to the C runtime
• Access to the underlying platform

9
Rotor Commercial CLR

• Both CLI and CLR are complete ECMA
• CLR Virtual Machine
• CLI CLR base classes
• But other use CLI Rotor CLR .Net
• Rotor is superset of ECMA, subset of commercial
  .NET Frameworks
• Rotor derived from the main product tree
• Subsequently re-integrated with main line
• Shared source CLI implementation differs
• JIT and garbage collector replaced with more
  portable, more approachable, implementations
• Many Windows-specific features not included COM
  interop, WinForms, and other integration
• Commercial features not included, such as
  ADO.NET, enterprise services, NGEN (JIT-ahead),
  and ASP.NET

10
C/CLI Standards
VS.NET
System.Web
System.WinForms
ECMA-334
UI
SessionState
C
Design
ComponentModel
HtmlControls
Caching
JScript
Security
WebControls
System.Drawing
Configuration
VB
System.Web.Services
VC/MC
Drawing2D
Printing
Protocols
Imaging
Text
Debugger
Description
Discovery
Designers
System.Data (ADO.NET)
System.Xml
ADO
SQL
SDK Tools
Design
Adapters
CorDBG
System
ECMA-335
ILAsm
Collections
IO
Security
Runtime
InteropServices
Configuration
Net
ServiceProcess
ILDbDump
Remoting
Diagnostics
Reflection
Text
SN
Serialization
Globalization
Resources
Threading
ILDAsm
Common Language Runtime
MetaInfo
JIT
GC
App Domain Loader
PEVerify
MSIL
Common Type System
Class Loader
Platform Abstraction Layer
Networking
Boot Loader
Sync
Threads
Timers
Filesystem
11
Shared Source CLI Rotor
ECMA-334
VS.NET
System.Web
System.WinForms
UI
SessionState
C
Design
ComponentModel
HtmlControls
Caching
JScript
ECMA-242
Security
WebControls
System.Drawing
Configuration
VB
System.Web.Services
VC/MC
Drawing2D
Printing
Protocols
Imaging
Text
Debugger
Description
Discovery
Designers
System.Data (ADO.NET)
System.Xml
ADO
SQL
SDK Tools
Design
Adapters
CorDBG
System
ECMA-335
ILAsm
Collections
IO
Security
Runtime
InteropServices
Configuration
Net
ServiceProcess
ILDbDump
Remoting
Diagnostics
Reflection
Text
SN
Serialization
Globalization
Resources
Threading
ILDAsm
Common Language Runtime
MetaInfo
JIT
GC
App Domain Loader
PEVerify
MSIL
Common Type System
Class Loader
Platform Abstraction Layer
Networking
Boot Loader
Sync
Threads
Timers
Filesystem
12
Distribution Stats for SSCLI

• Packaged as 15.75MB .tgz archive
• Over 3.6 million lines of code
• 10721 Files (8577 source)
• 3239 C
• 2657 C and C
• 1734 Jscript
• 911 CIL
• 19 perl
• 17 assembler
• 2 lisp !

• Additional 5.1MB html docs archive
• Additional html doc files 1169
• Over 2 million lines of test code
• Framework tests (in C)
• Integration and utility tests
• CLI standard tests (in CIL)
• PAL tests
• GC stress tests
• JScript tests
• Build output expands to over 1G with tests

13
A Map of the Archive (Unpacked)

• The Lay of the Land
• Four major areas in source code
• Runtime execution engine
• Frameworks / Librairies
• Compilers and tools
• Portability layer, tests, and build
  infrastructure
• Other important points of interest
• License
• Documentation
• Samples

14
Root Dir License Doc

• Shared source license
• Allows for non-commercial redistribution of both
  binary and source, including modified versions
• Simple restrictions (primarily non-commercial
  use)
• READFIRST.html
• Points to relnotes.html with late breaking items
• Points to docs/index.html
• Also important, but not included
• ECMA specifications
• Microsoft .NET Framework SDK

15
Area 1 Execution Engine

• Heart of component-oriented infrastructure
• Converts metadata, resources, and CIL (on disk as
  PE/COFF image) into running code
• JIT compilation and IL verification
• Cross-language exception handling
• Language-agnostic, object-capable, type system
• Automatic heap and stack management
• Dynamic code loading
• Evidence-based security (code access security)

16
Area 2 Libraries

• A minimal toolkit for the 21st century
  programmer
• Base class library
• Collections, arrays, strings, and other compound
  datatypes
• Globalization and formatting
• System services (threads, I/O, synchronization,
  etc.)
• Security
• Floating point and extended arrays libraries
• Networking, regular expressions, and XML
  libraries
• Includes access to runtime infrastructure
• Reflection and custom attributes
• Remoting, unmanaged interop, serialization, and
  marshaling
• AppDomains, Assemblies, GC, other execution
  engine features

17
Area 3 Compilers and Tools

• Full-featured C compiler (also used in build)
• Commercial-grade code base
• JScript compiler written entirely in C
• Illustrates dynamic techniques
• Additional developer tools
• clix, the shared source CLI program launcher
• Assembly tools resource compiler, assembly
  linker, metainfo metadata viewer, assembler,
  disassembler
• Debuggers cordbg managed debugger, plus a debug
  extension for working on managed code from C/C

18
Area 4 Build, Port, Test

• Platform Adaptation Layer (PAL) in pal directory
• System resources all consumed via PAL
• Less than 250 partially complete Win32 APIs
• Missing system features layered into PAL on any
  given platform
• PAL runtime contains shared code for upper
  level tools
• Good specification can be found in pal_guide.html
• Loading and teardown of context
• Threading model, scheduling, and timers
• Synchronous file I/O, asynchronous network I/O
• Synchronization (critsec, mutex, semaphore,
  event)
• Debugging and tool support
• Additional work for port
• JIT (very portable 3 layer macro emitter design)
• Small amount of assembly code in execution engine

19
Documentation

• Large set of simple HTML documentation
• Tool man pages
• Reference for class libraries
• Technical specifics on Rotor
• Build process
• SOS, logging, and debugging
• PAL
• Calling conventions, formats, and encodings

20
Tests

• Incredibly handy when modifying code!
• Two major areas managed code and PAL
• PAL suites (perl driver in pvtun.pl)
• Designed for use on new PAL implementations
• Has no dependencies on CLI only native build
  tools
• Extensive logging support in the PAL to assist
  debugging both parameter and return values
• Happens to be a great way to figure out what your
  program CLI are doing.
• Quality suites (perl driver in rrun.pl)
• IL verification
• BVT smoke tests many small programs
• Tests for the BCL
• GC tests
• Tests for JScript compiler

21
Samples

• Sample compilers
• In addition to JScript and CSharp
• clisp and myc small sample compilers
• Utilities
• Sources for XSD XML schema tool
• Various managed conversion utilities
• Sample profiler
• Additional assortment of sample programs

22
Build Infrastructure Utilities

• Build scripts and programs
• Scripts for environment setup and build
• build, binplace, nmake, and resource compiler
  utilities
• Bootstrap process happens for you.
• Utility configuration programs
• gacutil manage the global assembly cache
• sn manage assembly signing
• peverify verify CIL in an assembly
• caspol modify policy for code access security
• storeadm manage isolated storage

23
Building Rotor

• Building Rotor is simple
• Set up build environment with env.batshcsh
• Issue buildall at source root for first full
  build
• Run build in subdirectories for later partial
  builds
• Three flavors checked, fastchecked, free

Optimize code?
Diagnostics?
Build Flavor
no
yes
checked
yes
yes
fastchecked
yes
no
free
24
Build Bootstrap Process

• Initial phase
• Build PAL using native toolchain, build build
  tools against PAL
• Build resource compiler PAL runtime with newly
  compiled tools
• Main phase has complex interdependencies
• Unmanaged libraries, compilers, and tools, are
  built
• Managed portions of runtime and libraries are
  built
• Assemblies and runtime are configured
• Managed portions used at runtime are JIT compiled
• Results can be found under sscli/build directory

\sscli\build \v1.x86chk.Rotor \assembly
\config \rotor \sdk\bin \symbols
Global assembly cache Runtime config
directory Rotor.ini Utilities (ildasm, sn,
etc) Debug symbols
25
Running Debugging Code

• Rotor ships with a clix.exe program launcher
• Debugging tools
• Rotor has SOS debugger extension for examining
  JIT-compiled code and execution engine runtime
  structures
• VS.NET, ntsd (SOS), WinDBG (SOS), cordbg on
  Windows
• Gdb (SOS), cordbg on FreeBSD, Mac
• debughelp.cpp includes handy debugger functions
• Excellent profiler support
• Many items can be monitored in detail, given
  permissions
• CIL for methods can be changed on-the-fly
• Note No current debuggers simultaneously debug
  managed and unmanaged code!

26
Tracing Diagnostics

• Extensive tracing and dumping infrastructure
• Switches can be set using either environment
  variables or config file

Using rotor\rotor.ini
Using environment variables
Rotor LogEnable1 LogToConsole1 LogLevel1 JitH
altHelloCallback
set COMPLUS_LogEnable1 set COMPLUS_LogToConsole1
set COMPLUS_LogLevel1 set COMPLUS_JitHaltHello
Callback
27
Diagnostic Aids
28
Common SOS Commands
29
Supports Implementers

• CLI and C standards were written to enable
  third-party implementations
• Shared source CLI is an example implementation
• License allows for safe examination of code
• For implementers of CLI alternatives
• Augments large, complex spec with code
• Proves that multiple platform support is possible
• Demonstrates interoperability
• For compiler writers who want to target CLI
• JScript compiler shows dynamic techniques
• C compiler shows nearly all runtime features
• IL Assembler shows low-level API implementation
  and use
• A laboratory for future work
• Relationship to commercial code base facilitates
  transfer

30
Supports Teaching and Learning

• Many areas of interest (3.6 million lines of
  code)
• Advanced virtual execution environment for
  components
• Flexible, powerful, access to underlying platform
• Typesafe, yet language-agnostic, services (think
  C runtime)
• Contains useful component-based frameworks
• Standard programming constructs
• Web-service oriented networking and remoting
• Contains compilers, tools, and utilities
• C, JScript, and IL JIT compilers
• Assembler, disassembler, linker, debugger
• Configuration utilities
• Extensive tests and build harness
• Internationalization localization

31
Example Teaching Topics

• Component-based programming
• Comparative languages/infrastructure
• Language integration
• Type systems
• Compiler internals, runtime internals
• Language topics verification, security
• Asynchronous web services
• Industrial software engineering
• Internationalization
• Code structure and maintenance
• Build and test harness
• More? You tell MS!

32
Potential Research Areas

• Proof-carrying code, reflective environments
• JIT-supported pre/post conditions
• Runtime layout, compaction, or rewriting of
  code/data
• High-level support for mobile/distributed
  computing
• Garbage collection and other memory management
  techniques
• Distributed computing
• Concurrency, memory models, locking
• More? You tell MS!
• Software interaction (http//rainbow.essi.fr ?)

33
Existing Research Projects

• 40 Active known research projects
• Gyro The Generics Project at MSR
• Staged Computations within the CLI
• Aspect.NET A Framework for AOP
• Runtime Assembly Instrumentation Library
• JIT Objects and Dynamic Typing Investigation
• Extending Rotor with Distributed GC
• Adding Reference Counting to Rotor
• Rotor-based Computer Science Curriculum
• and more - for the complete list
  visithttp//research.microsoft.com/programs/euro
  pe/rotor

34
Community Reaction

• Approx. 65k downloads to date of Rotor sources
• Active Rotor-focused mailing lists, newsgroup
• University of Pisa, DevelopMentor, Microsoft
• Rotor conferences organized by community
• C .NET Technologies 2003 (Plzen)
• Rotor Workshop (Pisa)
• MS Rotor events (Workshop, Crash Courses)
• Rotor-related talks at conferences
• Web Services Edge, DevConnections, .NET ONE, etc.
• OSCON , ETAPS, PLDI, OOPSLA, etc.
• Community site to be launched shortly
• Hosted Collab.Net, organized by Cornell

35
Timeline Futures

• Work on the SSCLI ongoing since Fall 2000
• Originally derived from main source tree
• Beta beta code, Windows XP/FreeBSD (Mar '02)
• Beta Refresh bug fixes, docs, etc. (Jun '02)
• SSCLI 1.0 Mac OS X , fit finish (Nov '02)
• Source changes merged back into main tree
• Team is currently working to
• Foster community uptake
• Solicit input on futures
• Identify potential SemiWorks projects
• Hire additional Dev PM resources!
• Academics are our core audience
• Talk to us about any all of these!

36
Additional Resources

• Shared Source CLI home
• http//msdn.microsoft.com/net/sscli
• SSCLI Essentials (Stutz, Shilling, Neward)
• http//www.oreilly.com
• ECMA specifications
• http//msdn.microsoft.com/net/ecma
• Microsoft .NET Framework SDK
• http//msdn.microsoft.com/net
• Shared Source program
• http//www.microsoft.com/sharedsource

37
Others implementations

• ROTOR was a Microsoft implementation of the .Net
  technologie
• http//msdn.microsoft.com/sscli
• full implementation on the ISO/ECMA standard
• run under FreeBSD, Mac OS X and Windows XP
  (license BSD style)
• 100 of the source code is available
• No correct implementation of ROTOR under Linux
• Ximian MONO was an independent projext
• http//www.go-mono.org
• full compatibility .Net product
• run under sous Linux (licence GPL)
• Actually C compiler, Efficient JIT, no tools
• DotGNU Portable .Net
• http//www.southern-storm.com.au/portable_net.html
  
• implementation of the ISO/ECMA standard
• run under GNU/Linux
• but also under Windows, Solaris, NetBSD, FreeBSD,
  and MacOS X.
• The runtime engine has been tested on the x86,
  PowerPC, ARM, Sparc, PARISC, s309, Alpha, and
  IA-64 processors.
• DotGNU Portable.NET is part of the DotGNU
  project, built in accordance with the
  requirements of the GNU Project.
• Actually C compiler, no JIT, a lot of tools

38
Inside Rotor

39
Agenda

• Debugging Code
• Execution Model
• Key Structures
• Web Services
• SSCLI Extensions

40
Debugging, Tracing Diagnostics

• Rotor ships with a clix.exe program launcher
• Debugging tools
• Rotor has SOS debugger extension for examining
  JIT-compiled code and execution engine runtime
  structures
• VS.NET, ntsd (SOS), WinDBG (SOS), cordbg on
  Windows
• Gdb (SOS), cordbg on FreeBSD, Mac
• No current debuggers simultaneously debug managed
  and unmanaged code!
• Extensive tracing and dumping infrastructure
• Switches set using either environment variables
  or config file

Using environment variables
Using rotor\rotor.ini
Rotor LogEnable1 LogToConsole1 LogLevel1 JitH
altHelloCallback
set COMPLUS_LogEnable1 set COMPLUS_LogToConsole1
set COMPLUS_LogLevel1 set COMPLUS_JitHaltHello
Callback
41
Diagnostic Aids
42
Common SOS Commands
43
Execution Model
44
Object Instances
All instances begin with a syncblock index and a
pointer to a MethodTable
Syncblk stores info that is logically part of all
objects but that rarely needs to be allocated.
Instance of A
Instance of B
Instance of A
Instance Data
Instance Data
Instance Data
Objects in GC heap
Private execution engine memory
Sync Block Table
Method Table A
Method Table B
MethodTable is dispatch table for methods, as
well as access to type structure
45
Type Data Structures
Object header
Instance data
index
Sync record
SyncBlock table
Reference
SyncBlock
Interface map
Module
Compiled code
ClassLoader
Prestub
Assembly
Application Domain
46
MethodTable Interface Map
Each entry points to a mini vtable for the
methods that implement that interface for this
class
GCDesc
Interface Map is indexed by an interface number
unique to the application domain
MethodTableheader
sizeof(MethodTable)
Interface map
m_Vtable
Inheritedvirtuals
Introducedvirtuals
Proxyimplementation
Instance andstatic methods
47
Method Layout Example
Type1.Method1, Iface1.Method1
Type1
Type1.Method2, Iface1.Method2
Type2
interface Iface1 void Method1() void
Method2() interface Iface2 void
Method2() void Method3() class Type1
Iface1 void Method1() void
Method2() class Type2 Type1, Iface2 void
Method3() class Type3 Type2 void
Iface2.Method2()
Type1.Method1, Iface1.Method1
Type1.Method2, Iface1.Method2, Iface2.Method2
Type2.Method3, Iface2.Method3
Type3
Type1.Method1, Iface1.Method1
Type1.Method2, Iface1.Method2
Type2.Method3
Iface2.Method2
Iface2.Method3
Class1
Iface2
Iface1
Class2
Class3
Multiple slots pointing to the same method
implementation
48
Important Domain Structures
Application Domain
chainof modules
Module
Assembly
Logical Class
ClassLoader
EEClass(reflect info)
MethodTable(runtime info)
MethodDesc
FieldDesc
49
Field layout (source)

• class B
• private byte a 1
• protected long b 2
• public short c 3
• protected string d "Hello"
• private int e 4
• protected short f 5
• static private byte sa 6
• static protected long sb 7
• static public short sc 8
• static protected string sd "sHello"
• static private int se 9
• static protected short sf 10

50
Field layout (runtime)
object b
MethTab

• 008F 51AC

000C 0000
Flags (0x00)
0000 0020
0000 0002
BaseSize (0x04)
long b 2
02E3 110C
0000 0000
EEClass
...
00C2 18F8
string d "Hello"
0000 0006
NumSlots (0x24)
0000 0004
int e 4
...
0005 0003
short f 5, short c 3
Vtable (0x28)
01
byte a 1
01C2 0864
string sd "sHello"
0000 0007
long sb 7
0091 4028
0000 0000
0000 0006
arrayLen
06
byte sa 6
0000 0005
strLen
0008
short sc 8
0065 0048
'e' 'H'
int se 9
0000 0009
006C 006C
'l' 'l'
000A
short sf 10
006F
'o'
51
Threads AppDomains
B
A
C
C
Thread
OS Thread X
X
A
Thread
Thread
Thread
OS Thread Y
Y
Y
Y
A
B
C
Domain
Domain
Domain
A
B
C
AppDomain A
AppDomain B
AppDomain C
SSCLI
OS Process
52
Web Services Workbench

• Industry is defining a new distributed computing
  fabric in terms of open protocols, formats,
  standards
• Lots of interesting questions about almost
  everything
• Transport, framing, encoding, reporting,
  asynchrony, eventing, authentication,
  authorization, privacy, integrity,
  non-repudiation, transactions, state management,
  scalability, efficiency, programming model
  integration, and much more
• Massive hype needs to be balanced with sound
  research
• Researchers need a factored, extensible workbench
• Remoting under the SSCLI CLR provides this
  platform

53
Methods as Messages

• Proxies convert stack frames into messages
• Formatters convert messages into byte streams
• Channels transport byte streams across remoting
  boundaries
• Sinks extend pipeline to process messages and
  byte streams

54
Extending Remoting

• Proxies convert stack frames into messages
• TransparentProxy RemotingProxy work in concert
• Can build your own proxy (RealProxy,
  ProxyAttribute)
• Formatters convert messages into byte streams
• Two formatters provided (SoapFormatter,
  BinaryFormatter)
• Can build your own formatters (IFormatter)
• Channels transport byte streams across remoting
  boundaries
• Two channels provided (HttpChannel, TcpChannel)
• Can build your own channels (IChannel,
  IChannelSender, IChannelReceiver)
• Sinks extend pipeline to process messages and
  byte streams
• Formatters transports implemented as
  message/channel sinks
• Can build your own channel sinks
  (IClientChannelSink, IServerChannelSink)
• Can build your own message sinks (IMessageSink)

55
Rotor as a starting point

• Rotor is a starting point for doing interesting
  things
• Learning from Rotor extrapolating to the
  desktop CLR
• Building new non-commercial applications
• Building new compilers, class libraries tools
• Extending the existing compilers, class libraries
  tools
• Exploring new JIT optimizations GC algorithms
• Exploring new assembly and class loader
  approaches
• Adding new data types and IL instructions
• Porting Rotor to other OSes processors
• and much, much more

This requires familiarity with the code!
56
Example adding a new instruction

• Goal add exponentiation support to the Rotor VM
• New opcode (exp) should
• Pop 2 operands (x, y) of r8 type off the stack
• Compute xy
• Push r8 result back onto the stack
• New opcode should be
• Verifiable
• Safe
• Efficient
• Usable

57
Learning Rotor from the inside out

• Start from a known point with a working test case
  (mul)
• Step through the entire program in the debugger

.assembly extern mscorlib .assembly simple
.method static void main() .entrypoint
ldc.i4 7 ldc.i4 6 mul call void
mscorlibSystem.ConsoleWriteLine(int32) ret
Simple.il
58
JIT happens (sorry)

• Heart of the JITter is FJitjitCompile in
  fjit.cpp
• The preprocessor can be used to expand the macros

FJitResult FJitjitCompile(...) // Prepare
for JIT while (!FinishedJitting) //
Pre-opcode-JIT housekeeping switch (opcode)
case CEE_XXX // Verify
validate opcode/stack // Emit native code
via fns macros break //
Return pointer to JITted code
set ROTOR_X861 nmake filename.pp
59
Decoding the JIT macros for ldc.i4

• Switch for CEE_LDC_I4 uses emit_LDC_I4 macro

case CEE_LDC_I4 GET(i4, signed int, false)
emit_LDC_I4(i4) pushOp(OpType(typeI4))
break
Note many opcodes use intermediate
compileCEE_xxx() functions
60
Decoding the JIT macros for ldc.i4

• Expanding emit_LDC_I4(i4) gives

case CEE_LDC_I4 GET(i4, signed int, false)
emit_pushconstant_4(i4) pushOp(OpType(typeI4))
break
61
Decoding the JIT macros for ldc.i4

• Expanding emit_pushconstant_4(i4) gives

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS mov_constant(TOS_REG_1, i4)
inRegTOS true pushOp(OpType(typeI4))
break
Note this presumes no x86-specific macro
overrides
62
Decoding the JIT macros for ldc.i4

• Expanding TOS_REG_1, ToS enregistration

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS // If ToS in EAX, save
it mov_constant(X86_EAX, i4) // EAX now
contains result inRegTOS true //
Flag ToS as held in EAX pushOp(OpType(typeI4))
break
63
Decoding the JIT macros for ldc.i4

• Expanding mov_constant(X86_EAX, i4) gives

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS x86_mov_reg_imm(x86Big, X86_EAX,
(unsigned int)i4) inRegTOS true
pushOp(OpType(typeI4)) break
64
Decoding the JIT macros for ldc.i4

• Expanding x86_mov_reg_imm() gives

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS (cmdBlock2(
cmdByte(expOr2(expNum(0xB0 (x86Big ltlt 3)),
X86_EAX)), x86Big ? cmdDWord((unsigned
int)i4) cmdByte((unsigned int)i4)
)) inRegTOS true pushOp(OpType(typeI4))
break
65
Decoding the JIT macros for ldc.i4

• Expanding expOr2, expNum, X86_EAX, x86Big gives

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS (cmdBlock2( cmdByte( ( (
(0xB0 (1 ltlt 3) ) ) (0) ) ), cmdDWord(
(unsigned int)i4) ) ) inRegTOS true
pushOp(OpType(typeI4)) break
66
Decoding the JIT macros for ldc.i4

• Expanding cmdBlock2, doing the calculations gives

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS cmdByte( 0xB8 ) cmdDWord(
(unsigned int)i4 ) ) inRegTOS true
pushOp(OpType(typeI4)) break
67
Decoding the JIT macros for ldc.i4

• This expands to the actual emitter code, as
  follows

case CEE_LDC_I4 GET(i4, signed int, false)
deregisterTOS (((unsigned char)(outPtr))
(unsigned char)(0xB8)) (((unsigned int
)(outPtr)) ((unsigned int)i4) inRegTOS
true pushOp(OpType(typeI4)) break
68
Decoding the JIT macros for mul

• The switch for CEE_MUL delegates to a helper
  function
• Compilers typically inline the function call

case CEE_MUL JitResult compileCEE_MUL()
break
69
Decoding the JIT macros for mul

• Helper function uses TYPE_SWITCH_ARITH macro
• For I4 data types, this expands to the
  emit_MUL_I4 macro

FJitResult FJitcompileCEE_MUL() OpType
result_mul BINARY_NUMERIC_RESULT(topOp(),
topOp(1), CEE_MUL,
result_mul) TYPE_SWITCH_ARITH(topOp(),
emit_MUL, ()) POP_STACK(2)
pushOp(result_mul) return FJIT_OK
70
Balancing portability and speed

• Portable emitter macros defined in fjitdef.h
• Relies on compiler to provide efficient
  implementation

ifndef emit_MUL_I4 define emit_MUL_I4()
\ callInfo.reset()
\ emit_tos_arg( 1,
INTERNAL_CALL ) \ emit_tos_arg( 2,
INTERNAL_CALL ) \ emit_callhelper_I4I
4_I4(MUL_I4_helper) \ emit_pushresult_I4()
\ ifdef
DECLARE_HELPERS int HELPER_CALL
MUL_I4_helper(int i, int j) return j i
endif endif
71
Balancing portability and speed

• Macros are re-defined more efficiently in
  x86fjit.h
• Take advantage of platform FPU or other native
  facilities
• This source organization lets us balance porting
  effort against runtime performance

undef emit_MUL_I4 define emit_MUL_I4()
\ enregisterTOS
\ x86_pop(X86_ECX)
\ x86_uarith(x86OpIMul, x86Big, X86_ECX) \
inRegTOS true
72
Modifying FJit.cpp/FJit.h

• Add validation, verification emit code to
  jitCompile

FJitResult FJitjitCompile() case
CEE_EXP JitResult compileCEE_EXP()
break
FJit.cpp
class FJit FJitResult compileCEE_EXP()

FJit.h
73
Modifying FJit.cpp/FJit.h

• Add validation, verification emit code to
  jitCompile

class FJit FJitResult FJitcompileCEE_EXP
() CHECK_STACK(2) // Ensure
stack gt2 deep VALIDITY_CHECK(topOpE()typeR8
// Check top 2 slots
topOpE(1)typeR8) // items are both R8
emit_EXP_R8() // Emit the JITted
code POP_STACK(2) // Pop
operands off stack OpType result_exp(typeR8)
pushOp(result_exp) // Track the
resulting type return FJIT_OK
FJit.cpp
74
Modifying FJitDef.h

• Add the portable emit_EXP_R8 macro to FJitDef.h

ifndef emit_EXP_R8 define emit_EXP_R8() \
emit_callhelper_R8R8_R8(EXP_R8_helper) \
emit_pushresult_R8() ifdef DECLARE_HELPERS
double HELPER_CALL EXP_R8_helper(double i, double
j) double result pow(i, j) return
result endif endif

• Ideally we'd add an x86-specific emitter to
  x86fjit.h

75
Opcodes defined centrally, used globally

• Opcode data stored in OpCode.def (included
  liberally)
• OPDEF is redefined where OpCode.def is included

OPDEF(cname, sname, pop, push, args, type, l ,
s1, s2, ctrl)
typedef enum opcode_t define
OPDEF(c,s,pop,push,args,type,l,s1,s2,ctrl)
c, include "opcode.def" CEE_COUNT, // of
instructions OPCODE undef OPDEF
OpEnum.h
opcodeinfo_t OpcodeInfo define
OPDEF(c,s,pop,push,args,type,l,s1,s2,ctrl) \
s,c,args,l,s1,s2, include "opcode.def" unde
f OPDEF
DasmEnum.hpp
76
Modifying OpCode.def

• Add the CEE_EXP opcode's data to OpCode.def
• Beware the PERLy jabberwock, my son!
• opcodegen.pl understands strings, not C (no
  comments!)
• Run opcodegen.pl and copy output to
  rotor_dir\clr\src\bcl\system\ reflection\emit

OPDEF(CEE_EXP, // Canonical name of opcode
"exp", // String representation of
opcode PopR8PopR8, // Pops 2 R operands
off stack PushR8, // Pops 1 R operand
onto stack InlineNone, // Operand params
IPrimitive, // Opcode kind 1,
// Length of opcode 0xFF, //
Reference encoding prefix 0xA3, //
Bytecode representation of opcode NEXT)
// Control flow behavior
77
A tale of two verifiers

• FJit verifies the IL stream at JIT time
• Logic intertwined with emit code in
  FJitjitCompile
• PEVerify.exe verifies PE file, IL stream MD
  tables
• Uses IMetaDataValidate to verify metadata
• Uses IValidator to validate IL stream PE file
• IL verification table-driven, rules live in
  vertable.h
• Interpreted by VerifierVerify in verifier.cpp

ltpop stackgt ltoperand checksgt ltpush stackgt
ltbranchesgt lt!gt
VEROPCODE(CEE_LDLEN, "4")
VEROPCODE(CEE_LDC_I4, "4")
VEROPCODE(CEE_CALL, "!")
VEROPCODE(CEE_MUL, "N-")
78
Modifying VerTable.h

• Add CEE_EXP opcode verification rules to
  VerTable.h

VEROPCODE(CEE_EXP, "ddd") // Pops 2 doubles,
pushes 1
79
That's all?!

• In total, the modifications needed for exp are
• OpCode.def - centralized definition of opcodes
• FJit.cpp/FJit.h - JIT compilation verification
  code
• FJitDef.h - platform-neutral JIT helper code
• VerTable.h - table-driven verification rules
• Tools automatically affected by changes
• Rotor's JIT compilation and verification
• ILAsm.exe/ILDasm.exe
• PEVerify.exe
• Reflection.Emit (with a little bodging)
• Outstanding work items
• Compiler support, implicit conversions,
  x86-specific JIT

80
Summary

• Rotor is an unique platform for research
  teaching
• Commercial-quality VM, tools, compilers, tests
• Surprisingly approachable if tackled
  incrementally
• Laboratory for interesting research
  experimentation
• SemiWorks provides technology transfer
  opportunities
• The CLR team is hardcore about
• Helping you adopt Rotor for research teaching
• Soliciting your input on Rotor futures
• Identifying interesting SemiWorks projects
• Hiring! additional Dev PM resources

81
Thank you Q A