Rapid%20Development%20of%20Radio%20Astronomy%20Instrumentation%20using%20general%20purpose%20FPGA%20boards,%20tools%20

1
Rapid Development of Radio Astronomy
Instrumentation using general purpose FPGA
boards, tools libraries(how to build eight
radio astronomy instruments in two years)
Dan Werthimer and Casper Group, University of
California, Berkeley
http//casper.berkeley.edu
2
The Problem with the CurrentHardware Development
Model

• Takes 5 to 10 years
• Cost Dominated by NRE because of custom Boards,
  Backplanes, Protocols
• Antiquated by the time its released.
• How to buy the hardware at the last minute?
• Each observatory designs from scratch

3
(No Transcript)
4
(No Transcript)
5
CASPER (mostly from Arecibo work) Center for
Radio Astronomy Signal Processing and Electronics
Research Henry Chen, Daniel Chapman, Terry
Filiba, Griffin Foster, Bill Hodge, Jason
Manley, Peter McMahon, Vinayak Nagpal, Aaron
Parsons, Andrew Siemion, Dan Werthimer Radio
Astronomy Lab Don Backer, Matt Dexter, Joeri
van Leeuwen, David MacMahon, Oren Milgrome, Mel
Wright, Lynn Urry Berkeley Wireless Research
Center Bob Broderson, Chen Chang, Kevin Chao,
Pierre Droz, Borivoje Nikolic, Brian Richards,
, John Wawrzynek Collaborators Xilinx,
Fujitsu, HP, Sun Microsystems, NSF, NASA, NRAO,
NAIC, Chris Dick, Jeff Mock, CFA, Haystack,
Caltech, Cornell, CSIRO/ATNF, JPL/DSN, South
Africa KAT, Manchester/Jodrell Bank,
GMRT, Bologna (SKA), Metsahovi Observatory/Helsink
i University, Chalmers (Sweden), Seti Institute
6
CASPER Real-time Signal Processing
Instrumentation (NSF ATI, MRI)

• Low NRE, shared by the community
• Rapid development (8 instruments / 2 years)
• Open-source, collaborative
• Reusable, platform-independent gateware
• Modular, upgradeable hardware
• Industry standard communication protocols
• Low Cost

7
Solution

• Modular Hardware
• Low number of board designs
• Can be upgraded piecemeal or all together
• Reusable
• Standard signal processing model which
• is consistent between upgrades.

8
Solution use FPGAs1 FPGA 100 Pentium,
1/500 the power per op
Moores Law for FGPAs
3X improvement per year!
9
FPGA Field Programmable Gate Arrayreconfigurab
le computing - 1 minute100 times faster than
CPU, 5 times less powerinteger arithmetic, not
good at F. Point highly parallel (500
multipliers per chip)harder to program (mathlab
simulink)tools to abstract the hardware
awaysignal processing libraries

10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Compute Module Diagram
15
Board Interconnect

• Problem Backplanes are short lived
• (S100, Multibus, VME, ISA, EISA, PCI, PCIx,
  PCIe, compactPCI, compactPCIe,
  ATCA)
• Solution Use 10Gbit Ethernet
• (10Gbe, Infiniband, Myrinet, Xaui, Aurora)
• Copper CX4 (40 meters max) or Optical

16
Global Interconnects

• Commercial 10GBe switch from HP, Fujitsu,
  Foundry, Extreme Networks, Force 10, Cisco
• Packet switched, non-blocking
• lt 224 ports (4X) per chassis
• Up to 10,000 ports in a system
• 2001000 ns switch latency
• 4001200 ns FPGA to FPGA latency
• 2.88Tbps full duplex constant cross section
  bandwidth
• 600 per port

17
(No Transcript)
18
(No Transcript)
19
Platform-Independent, Parameterized Gateware

• What is Gateware?
• Design logic of FPGAs
• (between hardware and software)
• Need libraries for signal processing which dont
  have to be rewritten every hardware generation.
• Matlab Simulink!

20
Simulink-based Design Tool Flow

• Simulink Xilinx System Generator Library
• Custom BEE2 Library Blocksets
• Software programmable registers
• BEE Platform Studio

21
(No Transcript)
22
Biplex Pipelined FFT (Lynn Urry, Aaron Parsons,
David MacMahon, Jeff Mock, JPL)

• Uses 1/6 the resources of the Xilinx module.

23

• FFT controls
• Simulink Library Aaron Parsons, David MacMahon
  Verilog Library
  Jeff Mock
• Transform length
• Bandwidth
• Complex or Real
• Number of Polarizations
• Input bit width and output bit width
• twiddle coefficient bit width
• Run-time programmable down-shifting
• Decimate option

24
Filter Response PFB vs. FFT
25
PFB vs. FFT
26

• Additional PFB controls
• (Aaron Parsons, Jeff Mock)
• Filter overlap
• Width of filter coefficients
• Window function for filter (hamming, hanning,
  etc.)
• Import filter coefficients for custom filter
  performance

27
Digital Down-Converter

• Selectable of FIR taps
• On-the-fly programmable mix frequency
• Selectable FIR coeff
• Agile sub-band selection.

28
X-Engine Correlation Architecture (Lynn Urry,
Aaron Parsons)
29
X-Engine Architectureapplied to an arbitrary
sized antenna array
30
Hardware and Software Librarieslegend
31
Applications
32
(No Transcript)
33
Beowulf Cluster Like General Purpose
Architechture Dynamic Allocation of Resources,
need not be FPGA based
34
Applications

• VLBI Mark 5B data recorder Haystack, NRAO
  500 MHz
• Beamforming ATA, SMA
• Vinayak Nagpal, Jonathan
  Weintroub, Lynn Urry, Oren Milgrome
• SETI Arecibo (UCB)
• JPL/UCB DSN (Preston, Gulkis,
  Levin, Jones)
• Correlators and Imagers
• ATA (Aaron Parsons, Mel
  Wright)
• Reionization Experiment
  (Backer, Bradley, Parsons)
• Carma Next Gen (Dave
  Hawkins, Caltech)
• MeerKAT/SKA South Africa
  (Justin Jonas, Alan Langman)
• GMRT next gen correlator ??
• Pulsar Timing and Searching
• NRAO, Berkeley, Swinburne,
  Jodrell

35

• SETI Spectrometers
• Parkes Southern SERENDIP
• ALFA SETI Sky Survey (300 MHz x 7 beams)
• JPL DSN Sky Survey (eventually 20 GHz bandwidth)
• Radio Astronomy Spectrometers
• GALFA Spectrometer Arecibo Multibeam Hydrogen
  Survey
• Astronomy Signal Processor ASP Don Backer,
  Ingrid Stairs, et al(pulsars)
• Antenna Holography, ATNF, China
• Gavert (DSN education, outreach)
• CMB Bolometer Readout Caltech, UCB

36
128 Million Channel SETI Spectrometer

• 200 MHz Bandwidth, 2 Hz resolution

37
(No Transcript)
38
1 GHz bandwidth Pocket Spectrometer

• Using ATMEL ADCs at 2 Gsamples/sec
• Performing 4 real FFTs in 1 (complex) biplex
  pipelined FFT module.
• 2048 channels
• Uses just 1 ADC, 1 IBOB, and your laptop.

39
(No Transcript)
40
NASA MARVEL Mission Mars Volcanic Emission and
Life Scoutsub-mm spectroscopy water vents,
methane, ??
41
4096-Point Spectrometer
PolyphaseFIR
Real ? Complex8K FFT
Powerr2 i2
Vector Accumulator(64-bit x 2048)
CPU

TVG
ASIC
FPGA (Emulation)

• Stream-based 2K Packets
• Continuous operation

42
4096 channel Mars spectrometer Chip in a
day FPGA to ASIC
43
Portable VLBI backend Henry Chen

• Interfaces to MARK 5B data recorder
• 500 MHz spectrum recorder.
• (This makes 4 instruments in 1 year!)

44
VLBI Mark 5B Front End 500 MHz BW, 32
channel filter bank
45
Pulsar Searching and TimingNRAO, MeerKAT,
Swinburne, Effelsburg

• 1 GHz bandwidth polyphase filter banks
• 1024 channels, 30 uS readout , full
  stokes
• (Parkes 300 MHz 7 beams)
• 1 GHz coherent dedispersion (PFB, FFT, FFT-1)

46
lsar
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
Correlators and Beamformers

• Globally Asynchronous (like a computer cluster)
• Data is time stamped with 1 PPS at ADC
• Locally Synchronous, Globally Asyncronous
• Use Fifos to align data
• before correlation or beamforming

52
1960 First Radio Astronomy Digital Correlator
21 lags 300kHz clock discrete
transistors 19,000
Sandy Weinreb
53
Correlator processing power
107
SKA
EVLA
109
106
.
ALMA
LOFAR
105
GFlops
SMA
EVN/WSRT
104
106
103
VLA
DXB
DAS
102
DLB
DCB
10
1
103
70
75
90
85
80
95
2000
05
10
2015
source Arnold van Ardenne
54
Selected correlator quotes
Sandy Weinreb In 1960 there were no chips just
discrete transistors! The 19,000 was the cost
of the samplers, shift registers, and counter. It
did not include the cost of the 21 accumulators
which I made myself in a few months getting paid
240/month.
Ray Escoffier With correlator performance having
gone up by a factor of 922,000 over the last 30
years, its only fair that correlator design
engineers' salaries should have gone up by a
similar factor!!
Sergei Pogrebenko

It is desirable that the output data rate
from a data processor is less than the input data
rate.
55
Ray Escoffier With correlator performance having
gone up by a factor of 922,000 over the last 30
years, its only fair that correlator design
engineers' salaries should have gone up by a
similar factor!!
56
Moores Law Instruments using FPGAs 2X per
year (1,000,000 over 20 years)
57
Future Spectrometers
58
Upcoming Boards

• MiniRoach (V5 test board, spectrometer)
• Roach (MeerKat, NRAO) (V5, CX4, DRAM,
  SRAM)
• Bee3 (4V5, CX4, DRAM, Microsoft)
• Quad ADC (200 Msps, 12 bit)
• E2V ADC (5 Gsps, 2 2.5 Gsps, 4 1.25 Gsps)

59
CASPER the Friendly...

• Group Helping Open-source Signal-processing
  Technology (GHOST?)
• Goal to help develop signal processing
  instrumenation and libraries for the community.
• Open source hardware, gateware, and software.
• Provide training and tutorials
• Not so much delivering turn-key instruments
• Promote Collaboration

60
Collaboration (not turn key instruments)

• Share Open Source Libraries
• Workshops (Tamara)
• Videos and Docs on Tool Flow, Libraries
• Wiki, Mailing List
• Open Source Boards (available from vendors)

61
Where to get tested boards

• Build and test them yourself (cheapest for large
  Q))
• iBOB, iADC, 5 volt cables, Digicom Electronics
• Mo Ohady, mo_at_digicom.org
• Bee2, enclosure and power supply, SAE Materials
• Mario Salazar, marios_at_saemtl.com
• coordinate with Ken Lutz, BWRC,
  lutz_at_eecs.berkeley.edu
• iBOB mounting plates, front panels make or
  buy NRAO?
• Other Vendors?? Rakesh Mehta, MTE India?
  

62
(No Transcript)
63
(No Transcript)
64
Multi-Purpose Spectrometer Low Bandwidth
Aaron Parsons
200 Aux. I/O
I
200 Mhz ADC

Pol. 1
Q
200 Mhz ADC
I
200 Mhz ADC

Pol. 2
Q
200 Mhz ADC
65

• SETI Applications
• JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth)
• Parkes Southern SERENDIP
• ALFA Sky Survey (300 MHz x 7 beams)
• SETI Italia (Bologna)
• SETI_at_home
• Astronomy Applications
• GALFA Spectrometer Arecibo Multibeam Hydrogen
  Survey
• Astronomy Signal Processor ASP Don Backer,
  Ingrid Stairs, et al(pulsars)
• ATA4 Correlator F Engine
• Reionization Experiments (Don Backer, Rich
  Bradley, Chippendale, Ekers)
• Antenna Holography, ATNF, China
• GMRT correlator

66
Astronomy Signal Processor Don Backer, Jeff
Mock, Paul Demorest
SERENDIP V Polyphase Filter Bank
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
128 MHz
Pol. 1
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
128 MHz
Pol. 2
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
67
(No Transcript)
68
GALFA Spectrometer at Arecibo Jeff Mock,
Henry Chen, Aaron Parsons
69
GALFA Spectrometer
Multipurpose Spectrometer Board
Quadrature Downconverter Board
-50 to 50 MHz
LPF
?
IF Pol. 1
100 MHz
sin
LPF
?
Decimate by 16
cos
FIR LPF
LPF
?
IF Pol. 2
e-i?t
Decimate by 16
100 MHz
sin
FIR LPF
LPF
?
e-i?t
cos
-50 to 50 MHz
12.5 Mhz Digital
70
GALFA Lowpass Filter, Jeff Mock
71
GALFA Lowpass Filter
72
Caveats

• Simulink new, not open source
• verilog, vhdl old, slow, painful, low risk