Ahead of Time Dynamic Translation

1
Ahead of Time Dynamic Translation

• PreJit/NGEN by any other name
• George Bosworth
• Microsoft
• MRE04
• March 21, 2004

2
Dynamic Translation

• What is it? a compiler (not)
• What does it do? produce good fast code
  (hopefully)
• When does it run? Before the code is executed.
  i.e ahead of time.
• How much before is the subject of this talk

3
Outline

• Code Generation and execution in the Common
  Language Runtime (CLR)
• PreJit
• What is it, why do it, at what cost
• What is it really, and what is hard vs easy
• Simple example
• What the resultant code could look like

4
Common Language Runtime
Frameworks
Common Language Runtime
Base Classes
Security
ExecutionSupport
IL to native code compilers
GC, stack walk, code manager
Class loader and layout
5
Runtime Control FlowJust in Time
Assembly
ClassLoader
First reference to type
IL to nativecode compiler
ExecutionSupport
ManagedNativeCode
First call to method
CodeManagers
CPU
SecuritySystem
6
Execution Model
VB
VC
...
Script
IL
NativeCode
PreJit timeCode Gen
Common Language Runtime
Econo-JITCompiler
Standard JITCompiler
NativeCode
7
Compiling IL To Native

• Econo JIT
• Generates unoptimized native code
• Code can be discarded and regenerated
• Standard JIT
• Generates optimized native code
• Includes verification of IL code
• PreJit code generation
• Done at install time
• Reduces start-up time
• Native code has version checks and reverts to
  runtime JIT if they fail

8
Assemblies

• Unit of deployment
• One or more files, independent of packaging
• Self-describing via metadata (manifest)
• Versioning
• Captured by compiler
• Policy per-application as well as per-machine
• Security boundary
• Assemblies are granted permissions
• Methods can demand proof that a permission has
  been granted to entire call chain
• Mediate type import and export
• Types named relative to assembly

9
What Happens at Compile Time?

• Compiler reads any imported metadata and builds
  internal symbol table
• Compiler resolves method overloading using
  language-specific type matching rules
• Compiler adds any required coercions or casts
• Compiler selects and records a precise member
  reference
• Compiler provides object layout information
  through its choice of
• Layout technique
• Non-static fields
• Virtual methods, with new slot vs use existing
  slot
• Compiler emits metadata by merging incoming
  metadata with metadata it has generated
• Compiler emits IL with metadata tokens

10
What Happens at Class Load Time?, I

• A type reference occurs and is resolved either to
  an existing type or to an assembly reference.
• The assembly is loaded and some validation checks
  occur.
• The assembly is asked to resolve the type
  reference and a module is located
• It is at this point that the security system runs
  inheritance permission checks

11
What Happens at Class Load Time?, II

• If no managed native code is available, the IL
  module is loaded and preliminary validation
  checks occur.
• The particular class is then validated and the
  CLR creates in-memory data structures
  representing the class, including thunks for use
  when a method is called the first time.
• When a method on the class is called the first
  time, the code is JITted (and verified if needed).

12
What Happens at JIT Time?

• If a member reference appears in the code, the
  type is first loaded (see below), which will also
  load classes from which it inherits.
• If the type does not itself have the member, the
  inheritance chain is searched to see if it can be
  resolved higher up in the class hierarchy.
• The member access check, including a possible
  security check, occurs at this time.
• The JITter may insert calls to security code for
  declarative security checks in addition to those
  specified using the imperative security
  mechanism.
• The code runs.

13
Unmanaged (Traditional) Code
EXECUTION
14
Managed Code Execution
15
PreJit What Is It?An MSIL-to-Native Compiler
(Not)

• Pre-compile, pre-load, pre-layout
• Validate at execution time
• May factor in processor type, other installed
  assemblies, etc.
• If invalid, use the normal execution path
• Goal faster start-up
• Because of smaller start-up working set
• Usually as part of assembly installation
• ngen.exe, ships in Visual Studio .Net

16
What Happens at PreJit Time?

• Record version information about any dependents
• Classes are laid out
• Internal runtime data structures allocated
• Generate managed native code, verifying as needed
• Emit a specially formed PE file (.exe, .dll) in
  the code cache and associate it with the input IL
  assembly

17
NGen Code Creation
DEPLOYMENT
PEVerify
NGEN
AssemblyLoader
GAC, app. directory, download cache
18
NGen Code Execution
DEPLOYMENT
EXECUTION
GAC, app. directory, download cache
JIT verification
Fixups / Relocs (or JIT)
19
Deployment Model

• Shipping and installation
• Compile IL in build lab
• Ship IL
• PreJit during installation (optional)
• Possible JIT during execution
• Patching
• Deliver new IL image
• Repair native images (all depending assemblies)

20
Why Add Complexity of PreJit?

• Performance
• Remove expensive post-deployment work from app
  startup
• Signature validation
• Cross-component validity checks
• Type safety verification
• Optimization (distribution -gt execution)
• Code generation
• Data structures
• Packing
• Disk-based images allow sharing data/code across
  processes and across machine runs

21
Performance Gains

• Warm startup time
• Up to 6x faster
• Private pages
• Reduced by as much as 2/3
• Startup working set
• Reduced by as much as 1/2
• Basis
• Whidbey (the upcoming release of the CLR)
• Reference WinForm app with 100s of controls
• Caveats your mileage may very and these are very
  preliminary based on pre-release builds and hence
  subject to change

22
Costs

• Disk space
• Typical prejit image is 2-2.5x IL size
• Stored in Native Image Cache (alongside GAC)
• Installation time
• Typically 3s per MB of IL
• 40s to prejit Framework assemblies
• Steady state performance
• Slowdown due to indirections in code
• ASP.net
• V1 PreJit 7-12 slower
• Whidbey PreJit 2-5 slower
• Bytemark
• Whidbey same performance
• Future PreJit will have better performance
• Recompilation

23
Scenarios

• Critical libraries
• Loaded into multiple processes
• Warm startup is the norm
• Maximize shared pages
• Used by OS like components
• Important large apps
• Large percentage of code is app code
• Large impact on machine
• Benefit is primarily startup time
• Important shared apps
• Terminal server
• Shared pages are key

24
Scenarios

• Questionable small single instance apps and
  addins
• 1MB working set improvement
• 15 startup improvement
• Relatively small amount of compilation overhead
• Likely to be numerous
• Recommendation do not ngen unless heavily used
• Not helpful transient code
• Reflection emit
• Run and (probably) throw away code
• Frequently updated apps (relative to execution)
• Code under development (edit-compile-debug)
• Assumption some processes will be jitting some
  assemblies

25
PreJit Architectural Background

• IL was designed as CLR deployment format
• Verifiable type safety
• Small
• Self describing
• Portable
• PreJit image was designed as internal format
• Designed for execution speed
• Working set rather than disk size
• Data structures paged off of disk used directly
• Highly dependent on CLR implementation
• Data structure snapshots
• Complex codegen requirements
• High rate of churn over time
• Specialized for particular scenario

26
Specialization and Recompilation

• PreJit images are specialized translations of an
  IL assembly for a particular environment
• Processor
• Version of CLR
• Versions of assembly dependencies
• Development scenario (debug, profile, IBC
  instrument)
• Other factors
• Processor features
• Security policy
• Hosting hooks (SQL, IE,???)
• AppDomain shareable (state access)
• Profiling data
• Different environments require different prejit
  images
• Different machines may have different
  environments
• Servicing can change CLR or assembly dependencies
• Environmental changes can invalidate prejit
  images
• App-level configuration can change requirements
  across apps
• Recompilation is needed to adjust available
  prejit images over time to adapt to a changing
  environment

27
Client code generation motivation

• IL Distribution
• Verifiable (limited trust)
• Small
• Self describing
• Specialized native code
• Reach
• Processor
• CLR Implementation
• CLR Version
• Platform (assemblies) Version
• Recompilation

Brittleness okay
Rebuilt as needed
28
Why is specialization important?

• Innovation
• Managed APIs and implementation
• CLR implementation
• Compatibility
• Application needs to deploy on multiple OSs
• OS update must retain application compatibility
• Plus performance (of course)

29
Framework evolution recompilation

• We want to be able to recompile application
  native code as frameworks (platform assemblies)
  evolve
• Flexible compatible update rules
• Add public stuff
• Change any non-public stuff
• Rules based on semantics no implementation
  driven constraints
• Rearrange and add fields to public structures
  classes
• Rearrange and add virtual methods
• Insert into inheritance heirarchy
• Non-compatible changes are possible and handled
  relatively gracefully
• Versionability is a fundamental assumption when
  designing managed APIs
• Implementation inheritance
• Each layer up the hierarchy may add fields,
  methods, virtuals
• All are rolled into single flat structure
• All can version independently from each other and
  from client
• Typesafe extensibility
• Public fields on a structure or class
• Set can be extended as needed
• Example delegates on form, properties on window
• Fine grained, version-resilient component
  interaction

30
CLR evolution recompilation

• We want to be able to recompile native code as
  CLR evolves
• Much of runtime implementation is baked into
  code
• Code sequences
• Helper calls, Dispatch sequences, Global variable
  access
• Runtime data structures
• Highly specialized, Slow to build from IL, Likely
  to change over time
• Invariants
• Garbage collection, Lazy initialization
• Recent Changes
• Interface/Virtual dispatch
• Working set tuning
• Single file loading
• Generics
• And on and on
• Future potential
• More data structure tuning
• Phoenix
• The CLR is a great vehicle for delivering
  improvements into Windows

31
Future Directions

• Phoenix
• Changes prejit equation
• Higher compile cost
• Better perf for prejit vs JIT
• More cross-assembly analysis
• Client compilation optimizations
• Profile data feedback
• Active base address management
• Enable some additional specialization
  conditionally
• Allow developer to dial conservative behavior vs.
  aggressive optimizations

32
Why its Not So Easylet me count the ways

33
Issue Compilation Overhead

• Installation cost
• Recompilation cost
• Cascaded low level changes have very high cost
• Image churn can have secondary costs (OS
  optimizations, virus scan)
• Goals
• Ensure compilation performance is well tuned
• Minimize recompilation
• Minimize factors which cause recompilation
• Allow limited assembly servicing without
  cascading recompilation
• Ensure that most low-level servicing falls in
  this limited category
• Be smart about deferring compilation work
• However, some compilation cost is fundamental

34
Issue Managed Watson Dumps

• Native code and CLR data structures are generated
  on user machine, not available in any shipped
  dlls
• Much of this data is required to do a stack trace
• ILlt-gtNative mapping info is typically not
  available unless debug code is in use
• Source mapping is described in terms of IL
• Goals
• Leverage repeatability of codegen to optimize
  ngen image contents out of process dump
• Result is faster dump collection and smaller
  dumps
• Also leverage repeatability to regenerate
  ILlt-gtnative mapping info post mortem

35
Cross-assembly optimizations

• These optimizations expose internal
  implementation details to other assemblies code
  generation
• Examples
• Method inlining
• Constant value inlining
• Persisted instance data created at compile time
• Class constructors
• Struct enregistration
• Likely to be much more in Phoenix

36
NGen Fixup Blobs

• Descriptive reference to something in another
  module
• Blobs are grouped by referenced module?
• Module level table describes ranges of blobs
• Each table entry has an assembly ref token file
  ref token
• Could roughly follow metadata signature byte
  encoding?
• Byte sized tags with fixed following values
• Variable length encoded numbers tokens
• Treat blob is a type signature?
• However, scope is in destination assembly
• This avoids symbolic lookups, however introduces
  fragility if token numbers change
• Other possible blob formats
• MODULE (no args)
• TYPE type signature
• METHOD ltvtable indexgt containing type
  signature
• FIELD ltfield indexgt containing type signature
• STRING ltstring tokengt
• SIGNATURE ltsignature tokengt
• Blobs are read-only? interned within a module?
• Typically described with an RVA?

Details Details Details
37
NGen Fixup Tables

• Set of tables which provide fixup cells for code
  and data to reference
• Header describes ranges of tables?
• Table entry initially contains tagged RVA of
  fixup blob?
• Table index implies type of fixup
• Handle (runtime data structure)
• Method call address
• Static field address
• Varargs signature
• Others?
• Side effects? Which ones? (e.g. type
  initializations?)
• Entries are fixed up lazily?
• All fixups are idempotent?, tag allows atomic
  update
• Eager update is too expensive?
• Fixup can fail?

And they never end
38
But when it works

• it can work well
• An example
• (take with a grain of salt)

39
Example
Assembly B (b.cs)   public class InheritFromA
A     static public int a1()         
...    public class B     public static
void b()             A a new A()        
A.a()                 InheritFromA.a1()    
 
Assembly A (a.cs)   public class A        
static public int a()         ...     
40
Example Naive
b() has 4 fixups requires fixup guarantee
before execution A a new A() 20002552
mov     ecx,20006180       // Fixup on A's
method table pointer 20002558 call   
CLRStub_at_200040e0     // Call to new 2000255d
mov     ecx,eax 2000255f mov    
eax,20006190       // Fixup on A.ctor 20002565
call    dword ptr eax      // Call to
A.ctor A.a() 20002567 mov    
eax,20006194       // fixup on A.a 2000256d
call    dword ptr eax      // Call to
A.a InheritFromA.a1() 2000256f call    dword
ptr 20005cb8 // Indirect call to
InheritFromA.a1 // (InheritFromA may
need one time init) 20002575 ret    
41
Example In the Limit
b() has no fixups and can be called directly A
a new A() 2000254a mov     ecx,0x6ab95b98      
// Note no fixup on A's method table
pointer 2000254f call    CLRStub_at_200040e0    //
call to new 20002554 mov     ecx,eax 20002556
call    a!A..ctor()         // Note direct call
to A.ctor A.a() 2000255b call   
a!A.a()             // Note direct call to
A.a         InheritFromA.a1() 20002560 call   
b!InheritFromA.a1() // Note direct call to
// InheritFromA.a1  (InheritFromA
// is pre-inited) 20002565 ret
42
In Conclusion

• Ahead of Time Dynamic Translation (PreJit)
• What is it? a compiler (not)
• What does it do? produce good fast code
  (hopefully)
• When does it run? Before the code is executed.
  i.e ahead of time, (just a bit earlier than a
  JIT).

43
Bottom Line

• All that is left is compiler style optimizations
• Prejit makes sense in some scenarios, but not all