Academic Days 2005 Windows OS Course Supplement Kit: What should be there

1
Academic Days 2005 Windows OS Course Supplement
Kit What should be there?

• Arkady Retik
• Microsoft

2
Presentation Plan

• Core OS education is there a problem?
• What should our goals be?
• Is our current thinking correct?
• Whats next?

3
What is the problem?

• Windows specific
• Use of Windows OS in Core OS courses is low
• Is this a Microsoft only problem?
• Why isnt Linux a solution?
• --------------------------------------------------
  ------------------------
• Operating System courses in general
• CS intake overall is shrinking (USA, UK, ...)
• In particular, OS courses are not attractive

4
Why is Windows underutilized?

• No Windows source code (Almost Windows Source
  Code is available for 150 Universities WW under
  Shared Source )
• Current license discourages teaching and
  publication
• Insufficient environment to run software out of
  the box
• No Windows based instructional system (such as
  Nachos, MINIX)
• Few Windows OS oriented labs exercise materials
  
• Inadequate and outdated textbook coverage of
  Windows
• No academic oriented reference books of Windows
• Limited academic research publications using
  Windows
• Windows source footprint is large ( 50 large
  than Linux)
• ABM

5
Are students interested in OS classes?

• Most of the CS programs offer a core OS class
  during 2nd or 3rd year of study
• About 90 will not take further OS classes

6
Program Goal, Objectives and Scope

• Our goal is to develop and deliver a program for
  computer science and engineering undergraduate
  and graduate courses that provides modern and up
  to date OS knowledge by leveraging Microsoft
  software and products
• The programs top level objectives are
• Offer Windows based teaching and learning, tools
  and materials
• Promote presence of Windows in OS courses
  world-wide
• Increase knowledge of Windows Internals among
  students and faculty
• Demonstrate the innovations underlying the
  Windows solutions
• Provide infrastructure for the future academic
  research
• Support reproducibility requirements for OS
  research publications e.g. OSDI, SOSP
• Complement other MS Academic initiatives (Shared
  Premium Source, Rotor, WinCE)

Scope CS OS courses a. Undergrad courses b.
Graduate courses c. Research and Publications
7
Our current thinking

• Program components
• Limited Source Alternative to existing Shared
  Source Premium
• Simplify current license
• Add tools out of DDK that academics can use to
  rebuild the kernel
• Textbook Internals book acceptable to CS faculty
• OS Project Course (Project OZ) New offering
• Undergrad projects based on NT Kernel APIs
• Uses NT subsystem model, with licensed wrapper
  for NTAPIs (no Windows sources)
• C/C, C, and maybe Java support
• Companion lab project book will supplement OS
  textbook
• Curriculum Development Kit (CDK) New offering
• Aimed at teaching OS courses
• Provides ACM/IEEE Core OS curriculum content

8
Background
Connecting NT and Shared Source Brief History
and Architecture overview
9
OS Evolution
Windows Server 2003
Windows NT3.1
Windows NT4.0
Windows XP
VMS v1.0
1970
1980
1990
2000
Unix
Unix v6
Linux v1.0
Linux v2.0
Linux v2.6
Linux v2.4
Unix Public
10
History of NT

• Team forms November 1988
• Developers from DEC and Microsoft
• Build from the ground up
• Advanced PC Operating System
• Designed for for desktops and servers
• Secure, scalable SMP design
• All new code
• Initial effort targeted at Intel i860 code-named
  N10, hence the name NT which doubled as N-Ten and
  New Technology

11
NT Timeline first 15 years

• 2/1989 Coding Begins
• 7/1993 NT 3.1
• 9/1994 NT 3.5
• 5/1995 NT 3.51
• 7/1996 NT 4.0
• 12/1999 NT 5.0 Windows 2000
• 8/2001 NT 5.1 Windows XP
• 3/2003 NT 5.2 Server 2003
• 4/2004 NT 5.2 Windows XP 64 Bit Edition

12
Goal Setting

• High level goals
• Portability Ability to target more than one
  processor, avoid assembler, abstract away machine
  dependencies.
• Reliability Nothing should be able to crash the
  OS. Anything that crashes the OS is a bug.
• Extensibility Ability to extend the OS over
  time
• Compatibility With DOS, OS/2, POSIX, or other
  popular runtimes.
• Performance All of the above are more important
  than raw speed!

13
NT Architecture
14
Overview of Windows Architecture

• NT is not a microkernel, but does support
  user-mode OS personalities (i.e. for posix, OS/2,
  Win32)
• Primary supported programming interface Win32
• Win32 and other subsystems built on native NT
  APIs
• NT APIs generally not documented (not intended as
  the supported programming model) but specific
  APIs are documented in the DDK
• Kernel implementation organized around the object
  manager
• NT APIs are rich (many parameters) and need
  re-factoring and simplification for student use

15
Windows Architecture
16
Windows Kernel Organization

• Kernel-mode organized into
• NTOS (kernel-mode services)
• Run-time Library, Scheduling, Executive services,
  object manager, services for I/O, memory,
  processes,
• HAL (hardware-adaptation layer)
• Insulates NTOS drivers from hardware
  dependencies
• Providers facilities, such as device access,
  timers, interrupt servicing, clocks, spinlocks
• Drivers
• kernel extensions (primarily for device access)

17
Major Kernel Services

• Process management
• Process/thread creation
• Security reference monitor
• Access checks, token management
• Memory manager
• Pagefaults, virtual address, physical frame, and
  pagefile management
• Services for sharing, copy-on-write, mapped
  files, GC support, large apps
• Lightweight Procedure Call (LPC)
• Native transport for RPC and user-mode system
  services.
• I/O manager ( plug-and-play power)
• Maps user requests into IRP requests,
  configures/manages I/O devices, implements
  services for drivers
• Cache manager
• Provides file-based caching for buffer file
  system I/O
• Built over the memory manager
• Scheduler (aka kernel)
• Schedules thread execution on each processor

18
OS Project Course(code name Project OZ)(see
examples of the projects in the Appendix)
19
The Project OZ Course

• Objectives
• Provide an environment to build a rich set of
  projects that explore OS principles by leveraging
  the NT subsystem model for implementing OS
  personalities
• Use real OS features rather than a toy
  simulation
• Reduce the complexity required to learn / build
  experiments
• A simple development environment, using standard
  tools for building, debugging, and
  instrumentation
• Encourage out-of-the-box thinking by students

20
The Project OZ Course

• OZ Overview
• Library of functions that wrap the native NT APIs
  to provide access to low-level primitives to
  provide address spaces, threads, exceptions, and
  IPC
• Languages C/C, C, Java
• A runtime of support functions that simplify
  student projects
• Documentation for the OZ functions/runtime
• A rich set of projects, with many variations,
  that allow students to explore qualitatively (
  quantitatively) a large assortment of OS
  principles
• Tools for instrumentation and measurement

21
CDK
22
What will CDK cover?

• CDK modules
• cover all OS topics (based on Windows XP/Server
  2003)
• scaleable to multiple levels
• modular (can be used in whole / in part).
• Basic Module will provide materials to
  incorporate into a complete basic level OS course
  of one semester in length. The module will cover
  the Windows OS specific topics in the core and
  elective units of the OS BOK of Computing
  Curricula 2001.
• Advanced Module will provide materials to
  incorporate into an advanced level OS course of
  one semester in length. The module will cover
  the Windows OS specific topics in the core and
  elective units of the CC2001 OS BOK as well
  as relevant Networking and other units.
• All curriculum materials (syllabi, course
  outlines, lecture notes, labs, exercises, etc.)
  will be available in the MSR Curriculum
  Repository.

23
What OS Body of Knowledge topics will CDK cover?
a. Core topics OS1. Overview of operating
systems OS2. Operating system principles OS3.
Concurrency OS4. Scheduling and dispatch OS5.
Memory management b. Elective topics OS6.
Device management OS7. Security and
protection OS8. File systems OS9. Real-time and
embedded systems OS10. Fault tolerance OS11.
System performance evaluation OS12. Scripting c.
Advanced topics A13. Windows networking
A14. Comparing the Linux and Windows Kernels
A15. Windows Unix Interoperability d. Labs and
Exercises to reinforce the topics
Anything else?
24
What will CDK consist of?

• Instructors material
• An exemplar course syllabus and outline.
• Classroom materials (i.e. lecture slides, notes
  and other supporting materials).
• Lab exercises, assignments and testing materials
  with notes, manuals and instructions.
• Software tools referenced in lecture materials or
  used in labs or exercises.
• Copy of Windows Internals 4th edition MS Press
  book.
• Student material
• Any material that should be delivered to student
  by professor (i.e. lab assignments, lab set up
  and descriptions, tools use instructions) will be
  part of Instructors material as stand alone
  redist packages. Windows Internals 4th edition
  will be a recommended textbook for the course.

25
How do we achieve best content and top quality?

• Engage the best available experts
• Tailor to the academic style and requirements
• Conform to the latest ACM/IEEE Computing
  Curricula de facto educational standard 
• Capitalize  on 4th Edition Windows Internals book
  - best Windows reference material
• Build on rich experience of Microsoft OS training
  and experienced OS faculty
• Validate by independent and randomly selected
  academic reviewers WW pilot.   

26
Whats next

• Our current plans to provide initial input for
  next academic year (see Appendix for specific
  topics)
• Core topics will be available early July 2005
• Elective topics will be available in Fall 2005
• Will be looking for participants in pilots and
  trials
• If you are interested - let your academic
  contact and me know
• arkadyr_at_microsoft.com
• Leave your business card /contact details
• More information watch this space in June
• Shared Source
• http//www.microsoft.com/resources/sharedsour
  ce
• Curriculum Repository on MSDN AA
• http//www.msdnaa.net/curriculum

27
Appendix Support Information
28
Teaching Core OS Classes
50
15-20
30-35
teaching resources
Theoretical Courses
Mixed Courses
Practical Courses
few labs
labs
many labs
lab resources
System program
OS design
Sysadmin/programming
Kernel modify
System program
Textbooks/Project Books
Windows CE/XPe
Shared Source
Project OZ
CDK
products
Academic papers
Advanced Courses
Research
29
Virtual PC

• http//www.microsoft.com/windowsxp/virtualpc/
• Virtual PC 2004 as a tool for running several
• versions of Windows simultaneously on one
  machine.
• It could also be used to run one or more
  independent Unix (or Linux, Natchos, etc)
  sessions as hosted operating systems under
  Windows. You dont have to reboot, an OS can
  crash without taking Windows down, and each
  session runs real Unix (or Windows, if you
  choose).
• The virtualized file system allows you to wipe
  out changes made during a session, so you can
  experiment without rendering the OS unbootable.
  And every Virtual PC-hosted OS automatically
  inherits (through virtualization) all of the
  devices and networking youve set up for Windows.
  
• It gives us exactly what we need and also allow
  students and teachers utilize their Windows
  machines.

30
What is our approach?
to provide content in three areas 1.
undergraduate, 2. graduate, 3. research and
publications

• Faculty requirements
• Comprehensive, modular and up-to-date curricula
  content
• Continuation of subjects / Spiral learning /
  Non-prescriptive
• Flexible in levels (institutions courses) and
  delivery (web-based teaching)
• Cool topics and technology to attract and retain
  students (e.g. Gaming Technology, Mobility,
  Embedded, Web development, Robotics, etc. )
• Multidisciplinary topics (engineering business)
• Reference material and textbooks

• University Requirements
• Meet existing curriculum accreditation
  industry recognition guide lines
• USA ACM/IEEE Body of Knowledge
• WW varies by country (incl. government/industry
  requirements)

• Motivators
• Incentive to Faculty
• 1. Modern, best teaching OS, easy to introduce
  modular offerings saves work (No special lab
  equipment - use Virtual PC)
• 2. Software Engineering tools (Dev. Analysis
  and Test Tools)
• 3. Research/publication potential
• Why should I change my material?
• Incentive to Students
• Employability 2. Engaging
• 3. Up to date knowledge
• Will this class help me to get a job or a place
  in Graduate School?

31
CDK Core Topics available in early July 2005

• Overview of Operating Systems (Core) OS1
• Windows Operating System Internals Course
  Overview (Core)
• The Evolution of Operating Systems (Core)
• Windows Operating System Family Concepts
  Tools (Core)
• Operating System Principles (Core) OS2
• Structuring of the Windows Operating System
  (Core)
• The Windows API Naming Conventions, Types
  (Core)
• History of the Windows NT/2000/XP/2003 operating
  system (Core)
• OS Principles labs, quizzes, and assignments
• Concurrency (Core) OS3
• Critical Sections, Semaphores and Monitors (Core)
  
• Windows Object Manager, Trap Dispatching,
  Synchronization (Core)
• Windows Inter-process Communication
  (Core/Advanced)
• Concurrency labs, quizzes, and assignments
• Scheduling and Dispatch (Core) OS4
• The Concept of Processes and Threads (Core)
• Windows Processes and Threads (Core)
• Windows Process and Thread Internals
  (Core/Advanced)
• Windows Thread Scheduling (Core)

32
CDK elective supplementary topics available
in Fall 2005

• Device Management - The Input/Output System
  (Elective) OS6
• Principles of I/O Systems (Elective)
• The Windows I/O System Components (Elective)
• Windows I/O Processing (Elective/Advanced)
• Device Management labs, quizzes, and assignments
• File System (Elective) OS8
• Background Unix File Systems (Elective)
• The Windows File System (NTFS) (Elective)
• Encrypting File System Security in Windows OS
  (Elective/Advanced)
• NTFS Recovery Support (Elective/Advanced)
• Windows File and Directory Management (Elective)
• File System labs, quizzes, and assignments
• Real-time and Embedded Systems (Elective) OS9
• Introduction and Vocabulary (Elective)
• Real-Time Systems with Windows (Elective)
• Embedded Systems with Windows XP Embedded
  (Elective)
• Fault-tolerance (Elective) OS10
• System Performance Evaluation and Troubleshooting
  (Elective) OS11
• Scripting (Elective) OS12

33
Project OZ examples by Dr. Dave Probert(in no
particular order)
34
Loading Program Images

• Take the sections in a file and load into an
  address space
• dynamic relocation
• shared libraries
• creating stacks and initial thread contexts
• create environment and other parental state
• Initialize IO descriptors from parent
• Using
• NtCreateProcess to create an NT process container
• NT VM to manage/modify process contents
• NT library functions to take apart images and set
  environment
• NT thread APIs to execute thread in new process
• NT handle duplication to set IO descriptors

35
System Calls

• Communicate requests for system services
• implement basic system calls
• access data cross domains
• parameter validation and penetration testing
• asynchronous operations
• servicing models
• Using
• NT LPC to perform RPC
• NT VM to access/modify child process state
• NT threads to support flow-of-control

36
Manage address spaces, physical memory ptes

• Build data structures for managing VM-related
  resources
• basic data structures and algorithms
• support various page-table models
• Using
• NT VM to implement actual mappings (and shared
  memory) for address spaces
• NT VM to implement dirty bits
• NT VM shared memory provides page sharing and
  simulates physical pages with virtual pages
• NT threads provide DMA (for IO)

37
Implementing Virtual Memory

• Build various kinds of virtual memory systems
• use previous resource management for addresses,
  mem, page tables
• page file organization
• replacement algorithms
• shared memory, object/file-backed memory regions
• IO/DMA simulation
• performance measurement of algorithms
• Using
• NT LPC/exceptions to handle page faults
• NT VM to control actual mappings
• NT IO for access to page file

38
Create Processes

• Implement processes
• create/delete processes in the Oz world
• use loader project for loading programs
• experiment with different models for process
  creation and program execution
• implement handle tables for referencing objects
• manage run-down and synchronization issues
• Using
• NT processes to create real VM state (but for
  little else)

39
Create Threads / Scheduling

• Implement threads
• create/delete threads in an Oz process
• build appropriate data structures to represent
  thread
• tie into Oz process data structures
• create stacks and manage thread execution context
  (PCB)
• experiment with ideas like scheduler activations
• implement a scheduler, using different policies,
  priorities, etc
• add multi-processor support
• Using
• NT threads to represent processors and execute Oz
  threads on NT threads (Oz threads are essentially
  user-mode threads)
• NT APCs (Asynchronous Procedure Calls) to deliver
  timer interrupts to running Oz threads

40
Synchronization

• Implement various synchronization primitives
• build user-mode, kernel-mode, and hybrid
  synchronization primitives of various kinds
  (events, rw locks)
• extend Oz scheduler to support blocking of
  threads
• experiment with deadlock issues, priority
  inversion
• Using
• NT compareswap primitives

41
Other project areas

• IO Architecture
• File Systems
• Networking
• System bootstrap
• Performance measurement instrumentation (e.g.
  tracing)
• Error handling
• Exceptions and traps, stack unwinding
• Management of physical resources
• Management of resource sharing policies
• NUMA multiprocessors
• Security, authentication, ACLs
• Object support
• Namespaces
• Virtual machines