Memory Chapter 7

1
MemoryChapter 7

• Cache Memories

2
Memory Challenges

• Ideally one desires a huge amount of very fast
  memory for little cost, but
• Fast memory is expensive
• Cheap memory is slow
• The solution on a fixed budget for memory is a
  hierarchy
• A small amount of very fast memory (Think SRAM)
• A medium amount of slower memory (Think DRAM)
• A large amount of slower yet memory (Think Disk)
• Comparing
• Technology Access Time Cost/GB
• SRAM 0.5 5 ns 4,000 -
  10,000
• DRAM 50 70 ns
  100 - 200
• Disk 5 20 ms
  0.50 - 2

Recall We used 200ps or 0.2 ns in our pipeline
study. Why the difference?
3
The Memory Wall

• Logic vs DRAM speed gap continues to grow

Clocks per DRAM access
Clocks per instruction
4
Philosophically

• How does one UTILIZE the very fast memory
  effectively?
• Think The Principal of Locality
• Temporal Locality (Close in Time)
• Memory that has been accessed recently is most
  likely to be accessed sooner
• Spatial Locality (Close in location)
• Memory that is close to memory that has been
  accessed recently is most likely to be accessed
  sooner
• Organize memory in blocks
• Keep blocks likely to be used soon in the very
  fast memory
• Keep the next most likely blocks in medium fast
  memory
• Keep those not likely to be used soon in slower
  memory

5
Hierarchical Memory Organization
6
Cache memory

• What is cache?
• A small amount of very high speed memory between
  the main memory and the CPU
• How is it organized?
• Organized in a number of uniform sized blocks of
  memory that have a high likelihood of being used.
• How is kept current?
• When a block in main memory is more likely to be
  needed, that block replaces a block in the cache.
• How do we know it is needed?
• An access fails to find the word in the cache
• Where does it get placed in the cache?
• Likely in place of the last used block
• How do we rate the performance of the cache?
• Based upon Hit rates and Miss rates
• Should there be Instruction Caches and Data
  Caches?

7
Hierarchical Memory Organization
Registers are the fastest
Cache is the fastest Memory - SRAM
DRAM makes good main memory
Disk is best for the rest (majority)
8
The Memory Hierarchy

• Take advantage of the principle of locality to
  present the user with as much memory as is
  available in the cheapest technology at the speed
  offered by the fastest technology

Processor
Increasing distance from the processor in access
time
L1
L2
Main Memory
Secondary Memory
(Relative) size of the memory at each level
9
The Memory Hierarchy Pictorially

• Temporal Locality (Locality in Time)
• Keep most recently accessed data items closer to
  the processor
• Spatial Locality (Locality in Space)
• ? Move blocks consisting of contiguous words to
  the upper levels

Lower Level Memory
Upper Level Memory
To Processor
Blk X
From Processor
Blk Y
10
The Memory Hierarchy Terminology

• Hit data is in some block in the upper level
  (Blk X)
• Hit Rate the fraction of memory accesses found
  in the upper level
• Hit Time Time to access the upper level which
  consists of
• RAM access time Time to determine
  hit/miss
• Miss data is not in the upper level so needs to
  be retrieve from a block in the lower level (Blk
  Y)
• Miss Rate 1 - (Hit Rate)
• Miss Penalty Time to replace a block in the
  upper level Time to
  deliver the block the processor
• Hit Time ltlt Miss Penalty

11
How is the Hierarchy Managed?

• registers ? memory
• by compiler (or programmer?)
• cache ? main memory
• by the cache controller hardware
• main memory ? disks
• by the operating system (virtual memory)
• virtual to physical address mapping assisted by
  the hardware (TLB)
• by the programmer (files)

12
Cache

• Two questions to answer (in hardware)
• Q1 How do we know if a data item is in the
  cache?
• Q2 If it is, how do we find it?
• Direct Mapped Caching
• For each item of data at the lower level, there
  is exactly one location in the cache where it
  might be - so lots of items at the lower level
  must share locations in the upper level
• Address mapping
• (block address) modulo ( of blocks in the cache)
• First consider block sizes of one word

13
Caching A Simple First Example
Main Memory
0000xx 0001xx 0010xx 0011xx 0100xx 0101xx 0110xx 0
111xx 1000xx 1001xx 1010xx 1011xx 1100xx 1101xx 11
10xx 1111xx
Two low order bits define the byte in the word
(32-b words)
Cache
Tag
Data
Valid
Index
00
01
10
11
Q2 How do we find it? Use next 2 low order
memory address bits the index to determine
which cache block (i.e., modulo the number of
blocks in the cache)
Q1 Is it there? Compare the cache tag to the
high order 2 memory address bits to tell if the
memory block is in the cache
(block address) modulo ( of blocks in the cache)
14
Caching A Simple First Example
Main Memory
0000xx 0001xx 0010xx 0011xx 0100xx 0101xx 0110xx 0
111xx 1000xx 1001xx 1010xx 1011xx 1100xx 1101xx 11
10xx 1111xx
Two low order bits define the byte in the word
(32b words)
Cache
Tag
Data
Valid
Index
00
01
10
11
Q2 How do we find it? Use next 2 low order
memory address bits the index to determine
which cache block (i.e., modulo the number of
blocks in the cache)
Q1 Is it there? Compare the cache tag to the
high order 2 memory address bits to tell if the
memory block is in the cache
(block address) modulo ( of blocks in the cache)
15
Direct Mapped Cache

• Consider the main memory word reference string
• 0 1 2 3 4 3
  4 14

Start with an empty cache - all blocks initially
marked as not valid
0
1
2
3
3
4
15
4
16
Direct Mapped Cache

• Consider the main memory word reference string
• 0 1 2 3 4 3 4
  15

Start with an empty cache - all blocks initially
marked as not valid
0
1
2
3
miss
miss
miss
miss
00 Mem(0) 00 Mem(1)
00 Mem(0)
00 Mem(0) 00 Mem(1) 00 Mem(2)
00 Mem(0)
00 Mem(1)
00 Mem(2)
00 Mem(3)
3
4
15
miss
miss
hit
hit
4
00 Mem(0) 00 Mem(1) 00 Mem(2) 00
Mem(3)
01 Mem(4) 00 Mem(1) 00 Mem(2) 00
Mem(3)
01 Mem(4) 00 Mem(1) 00 Mem(2) 00
Mem(3)
01 Mem(4) 00 Mem(1) 00 Mem(2) 00
Mem(3)

• 8 requests, 6 misses

17
MIPS Direct Mapped Cache Example

• One word/block, cache size 1K words

What kind of locality are we taking advantage of?
18
Handling Cache Hits

• Read hits (I and D)
• this is what we want no challenges
• Write hits (D only)
• What is the problem here?
• Strategies
• allow cache and memory to be inconsistent
• write the data only into the cache block
  (write-back the cache contents to the next level
  in the memory hierarchy when that cache block is
  evicted)
• need a dirty bit for each data cache block to
  tell if it needs to be written back to memory
  when it is evicted
• require the cache and memory to be consistent
• always write the data into both the cache block
  and the next level in the memory hierarchy
  (write-through) so dont need a dirty bit
• writes run at the speed of the next level in the
  memory hierarchy so slow! or can use a write
  buffer, so only have to stall if the write buffer
  is full

19
Read / Write Strategies
Read Through Word read from memory No Read
Through Word word from cache after block is read
from memory Write Through Word written to both
Cache and Memory Write Back Word written only
to Cache Write Allocate Block is loaded on a
write miss, followed by a write hit Write No
Allocate Block is modified on a write miss but
not loaded

• Write Hit Policy Write Miss
  Policy
• Write Through
  Write Allocate
• Write Through Write No
  Allocate
• Write Back Write
  Allocate
• Write Back
  No Write Allocate

20
Write Buffer for Write-Through Caching
Cache
Processor
DRAM
write buffer

• Write buffer between the cache and main memory
• Processor writes data into the cache and the
  write buffer
• Memory controller writes contents of the write
  buffer to memory
• The write buffer is just a FIFO
• Typical number of entries 4
• Works fine if store frequency (w.r.t. time) ltlt 1
  / DRAM write cycle
• Memory system designers nightmare
• When the store frequency (w.r.t. time) ? 1 /
  DRAM write cycle leading to write buffer
  saturation
• One solution is to use a write-back cache
  another is to use an L2 cache

21
Another Reference String Mapping

• Consider the main memory word reference string
• 0 4 0 4 0 4 0 4

Start with an empty cache - all blocks initially
marked as not valid
0
4
0
4
4
0
4
0
22
Another Reference String Mapping

• Consider the main memory word reference string
• 0 4 0 4 0 4 0 4

Start with an empty cache - all blocks initially
marked as not valid
miss
miss
miss
miss
0
4
0
4
00 Mem(0)
00 Mem(0)
01 Mem(4)
00 Mem(0)
4
0
4
0
miss
miss
miss
miss
00 Mem(0)
00 Mem(0)
01 Mem(4)
01 Mem(4)

• 8 requests, 8 misses

• Ping pong effect due to conflict misses - two
  memory locations that map into the same cache
  block

23
Sources of Cache Misses

• Compulsory (cold start or process migration,
  first reference)
• First access to a block, cold fact of life, not
  a whole lot you can do about it
• If you are going to run millions of
  instruction, compulsory misses are insignificant
• Conflict (collision)
• Multiple memory locations mapped to the same
  cache location
• Solution 1 increase cache size or block length
• Solution 2 increase associativity
• Capacity
• Cache cannot contain all blocks accessed by the
  program
• Solution increase cache size
• What about the relationship between cache size
  and block length?

24
Handling Cache Misses

• Read misses (I and D)
• stall the entire pipeline, fetch the block from
  the next level in the memory hierarchy, install
  it in the cache and send the requested word to
  the processor, then let the pipeline resume
• Write misses (D only)
• stall the pipeline, fetch the block from next
  level in the memory hierarchy, install it in the
  cache (which may involve having to evict a dirty
  block if using a write-back cache), write the
  word from the processor to the cache, then let
  the pipeline resume or (normally used in
  write-back caches)
• Write allocate just write the word into the
  cache updating both the tag and data, no need to
  check for cache hit, no need to stall or
  (normally used in write-through caches with a
  write buffer)
• No-write allocate skip the cache write and just
  write the word to the write buffer (and
  eventually to the next memory level), no need to
  stall if the write buffer isnt full must
  invalidate the cache block since it will be
  inconsistent (now holding stale data)

25
Multiword Block Direct Mapped Cache

• Four words/block, cache size 1K words

What kind of locality are we taking advantage of?
26
Taking Advantage of Spatial Locality

• Let cache block hold more than one word
• 0 1 2 3 4 3
  4 15

Start with an empty cache - all blocks initially
marked as not valid
27
Taking Advantage of Spatial Locality

• Let cache block hold more than one word
• 0 1 2 3 4 3
  4 15

Start with an empty cache - all blocks initially
marked as not valid
miss
hit
miss
00 Mem(1) Mem(0)
00 Mem(1) Mem(0)
00 Mem(1) Mem(0)
00 Mem(3) Mem(2)
hit
miss
hit
00 Mem(1) Mem(0)
00 Mem(1) Mem(0)
01 Mem(5) Mem(4)
00 Mem(3) Mem(2)
00 Mem(3) Mem(2)
00 Mem(3) Mem(2)
hit
miss
01 Mem(5) Mem(4)
01 Mem(5) Mem(4)
00 Mem(3) Mem(2)
00 Mem(3) Mem(2)

• 8 requests, 4 misses

28
Miss Rate vs Block Size vs Cache Size

• Miss rate goes up if the block size becomes a
  significant fraction of the cache size because
  the number of blocks that can be held in the same
  size cache is smaller (increasing capacity misses)

29
Block Size Tradeoff

• Larger block sizes take advantage of spatial
  locality but
• If the block size is too big relative to the
  cache size, the miss rate will go up

• Larger block size means larger miss penalty
• Latency to first word in block transfer time
  for remaining words

• In general, Average Memory Access Time
• Hit Time Miss Penalty
  x Miss Rate

30
Multiword Block Considerations

• Read misses (I and D)
• Processed the same as for single word blocks a
  miss returns the entire block from memory
• Miss penalty grows as block size grows
• Early restart datapath resumes execution as
  soon as the requested word of the block is
  returned
• Requested word first requested word is
  transferred from the memory to the cache (and
  datapath) first
• Nonblocking cache allows the datapath to
  continue to access the cache while the cache is
  handling an earlier miss
• Write misses (D)
• Cant use write allocate or will end up with a
  garbled block in the cache (e.g., for 4 word
  blocks, a new tag, one word of data from the new
  block, and three words of data from the old
  block), so must fetch the block from memory first
  and pay the stall time

31
Cache Summary

• The Principle of Locality
• Program likely to access a relatively small
  portion of the address space at any instant of
  time
• Temporal Locality Locality in Time
• Spatial Locality Locality in Space
• Three major categories of cache misses
• Compulsory misses sad facts of life. Example
  cold start misses
• Conflict misses increase cache size and/or
  associativity Nightmare Scenario ping pong
  effect!
• Capacity misses increase cache size
• Cache design space
• total size, block size, associativity
  (replacement policy)
• write-hit policy (write-through, write-back)
• write-miss policy (write allocate, write buffers)

32
Measuring Cache Performance

• Assuming cache hit costs are included as part of
  the normal CPU execution cycle, then
• CPU time IC CPI CC
• IC (CPIideal Memory-stall cycles) CC

• Memory-stall cycles come from cache misses (a sum
  of read-stalls and write-stalls)
• Read-stall cycles reads/program read miss
  rate read miss penalty
• Write-stall cycles (writes/program write
  miss rate write miss penalty)
• write buffer stalls
• For write-through caches, we can simplify this to
• Memory-stall cycles miss rate miss penalty

33
Impacts of Cache Performance

• Relative cache penalty increases as processor
  performance improves (faster clock rate and/or
  lower CPI)
• The memory speed is unlikely to improve as fast
  as processor cycle time. When calculating
  CPIstall, the cache miss penalty is measured in
  processor clock cycles needed to handle a miss
• The lower the CPIideal, the more pronounced the
  impact of stalls
• A processor with a CPIideal of 2, a 100 cycle
  miss penalty, 36 load/store instrs, and 2 I
  and 4 D miss rates
• Memory-stall cycles 2 100 36 4 100
  3.44
• So CPIstalls 2 3.44 5.44
• What if the CPIideal is reduced to 1? 0.5?
  0.25?
• What if the processor clock rate is doubled
  (doubling the miss penalty)?

34
Reducing Cache Miss Rates 1

• Allow more flexible block placement
• In a direct mapped cache a memory block maps to
  exactly one cache block
• At the other extreme, could allow a memory block
  to be mapped to any cache block fully
  associative cache
• A compromise is to divide the cache into sets
  each of which consists of n ways (n-way set
  associative). A memory block maps to a unique
  set (specified by the index field) and can be
  placed in any way of that set (so there are n
  choices)
• (block address) modulo ( sets in the cache)