Rapid Development of Correlators, Spectrometers, Beam Formers, and Pulsar Engines using general purp

1
Rapid Development of Correlators, Spectrometers,
Beam Formers, and Pulsar Engines using general
purpose FPGA boards, tools libraries(how to
build eight radio astronomy instruments in two
years)
Dan Werthimer University of California,
Berkeley
http//seti.berkeley.edu/casper
2
Why you might not want our group to help build
instruments for the GBT

• WARNING

3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
Our research group is really 3 groups

• SETI (plus primordial black holes, HI mapping)
• Public Participation Scientific Computing
• CASPER Center for Astronomy Signal Processing
  and Electronics Research

9
UC Berkeley SETI Programs
10
The SETI_at_home Client
11
SETI_at_home Statistics TOTAL
RATE
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Public Participation Supercomputing Group

• David Anderson, Rom Walton, SETI Group
• aka Distributed Computing
• aka edge resource aggregation)

16
BOINC NSF

• Berkeley Open Infrastructure for Network
  Computing
• General-purpose distributed computing framework.
• Open source.
• Will make distributed computing accessible to
  those who need it. (Starting from scratch is
  hard!)

17
Projects

• Astronomy
• SETI_at_home (Berkeley)
• Astropulse (Berkeley)
• Einstein_at_home gravitational pulsar search
  (Caltech,)
• PlanetQuest (SETI Institute)
• Stardust_at_home (Berkeley, Univ. Washinton,)
• Earth science
• Climateprediction.net (Oxford)
• Biology/Medicine
• Folding_at_home, Predictor_at_home (Stanford,
  Scripts)
• FightAIDSathome virtual drug discovery
• Physics
• LHC_at_home (Cern)
• Other
• Web indexing/search
• Internet Resource mapping (UC Berkeley)

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
Where's the computing power?
your computers
home PCs
academic
business

• 2010 1 billion Internet-connected PCs
• 55 privately owned
• If 100M participate
• 100 PetaFLOPs, 1 Exabyte (1018) storage

22
CASPER Center for Radio Astronomy Signal
Processing and Electronics Research Henry Chen,
Daniel Chapman, Pat Crescini, Christina DeJesus,
Pierre Droz Kirsten Meder, Arash Parsa, Aaron
Parsons, Andrew Siemion, Dan Werthimer Radio
Astronomy Lab Don Backer, Paul Demorest, Matt
Dexter, Carl Heiles, David McMahon, Mel Wright,
Lynn Urry Berkeley Wireless Research Center Bob
Broderson, Chen Chang, John Wawrzynek SETI
Institute Dave Deboer, Gerry
Harp Collaborators Jeff Mock, NAIC, NRAO,
ATNF, JPL/DSN, Harvard/Smithsonian/CFA,
MIT/Haystack, GMRT, Caltech, South Africa KAT
23
(No Transcript)
24
Casper Origins

• NSF proposal to build SETI spectrometer (2003)
• (added one paragraph BTW, this can be used
  for other astronomy instrumentation, potential
  spin offs are .)
• Reviewers comments (paraphased)
• SETI is bullshit, SETI will never find
  anything,
• But these instruments are useful for the
  community, strongly recommend funding

25
CASPER Real-time Signal Processing
Instrumentation (NSF ATI, MRI)

• Low NRE, shared by the community
• Rapid development (8 instruments / 2 years)
• Open-source, collaborative
• Reusable, platform-independent gateware
• Modular, upgradeable hardware
• Industry standard communication protocols
• Low Cost

26
MOTIVATIONATA, SKA, Focal Plane Arrays,
SETI,need gtgt PetaOp/sec Instruments take a
long time to build, very high NRE
27
The Radio Revolution
MWA XNTD PAPER FAST PAST LAR LWA
28
Allen Telescope Array

• 6.1-meter offset Gregorian (2.4-meter secondary)
• rim-supported, hydroformed dishes

29
(No Transcript)
30
ATA-42 Operational This Summer
31
(No Transcript)
32
(No Transcript)
33
The Problem with the CurrentHardware Development
Model

• Takes 5 years
• Cost Dominated by NRE because of custom Boards,
  Backplanes, Protocols
• Antiquated by the time its released.

34
Solution

• Modular Hardware
• Low number of board designs
• Can be upgraded piecemeal or all together
• Reusable
• Standard signal processing model which
• is consistent between upgrades.

35
Solution use FPGAs1 FPGA 100 Pentium,
1/500 the power per op
Moores Law for FGPAs
3X improvement per year!
36
FPGA Field Programmable Gate Arrayreconfigurab
le computing - 1 minute100 times faster than
CPU, 5 times less powerinteger arithmetic, not
good at F. Point highly parallel (500
multipliers per chip)harder to program (mathlab
simulink)tools to abstract the hardware
awaysignal processing libraries

37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
Compute Module Diagram
41
Platform-Independent, Parameterized Gateware

• What is Gateware?
• Design logic of FPGAs
• (between hardware and software)
• Need libraries for signal processing which dont
  have to be rewritten every hardware generation.
• Matlab Simulink!

42
(No Transcript)
43
Biplex Pipelined FFT

• Uses 1/6 the resources of the Xilinx module.

44

• FFT controls Simulink Library Aaron Parsons
• Verilog Library Jeff
  Mock
• Transform length
• Bandwidth
• Complex or Real
• Number of Polarizations
• Input bit width and output bit width
• twiddle coefficient bit width
• Run-time programmable down-shifting
• Decimate option

45
Filter Response PFB vs. FFT
46
PFB vs. FFT
47

• Additional PFB controls
• (Aaron Parsons, Jeff Mock)
• Filter overlap
• Width of filter coefficients
• Window function for filter (hamming, hanning,
  etc.)
• Import filter coefficients for custom filter
  performance

48
Digital Down-Converter

• Selectable of FIR taps
• On-the-fly programmable mix frequency
• Selectable FIR coeff
• Agile sub-band selection.

49
X-Engine Correlation Architecture (Lynn Urry,
Aaron Parsons)
50
X-Engine Architectureapplied to an arbitrary
sized antenna array
51
Hardware and Software Librarieslegend
52
Applications
53
(No Transcript)
54
(No Transcript)
55
Global Interconnects

• Commercial Infiniband switch from Mellanox,
  Voltaire, etc.
• Packet switched, non-blocking
• 24 144 ports (4X) per chassis
• Up to 10,000 ports in a system
• 2001000 ns switch latency
• 4001200 ns FPGA to FPGA latency
• 480Gbps 2.88Tbps full duplex constant cross
  section bandwidth
• lt400 per port

56
(No Transcript)
57
Beowulf Cluster Like General Purpose
Architechture Dynamic Allocation of Resources,
need not be FPGA based
58
Targeted Applications

• Moderate to high-bandwidth problems
• For low bandwidths, just use CPUs
• Lower to mid-scale computation
• For very large applications (SKA), may be more
  cost effective to design ASICs
• Rapid Development

59
Applications

• VLBI Mark 5B data recorder - Haystack 500 MHz
• Beamforming SMA Vinayak Nagpal, Jonathan
  Weintroub
• SETI Arecibo (UCB)
• JPL/UCB DSN (Preston, Gulkis,
  Levin, Jones)
• Correlators and Imagers
• ATA (Mel Wright)
• Reionization Experiment
  (Backer, Bradley)
• Carma Next Gen (Dave
  Hawkins, Caltech)
• SKA demonstrator South
  Africa (Justin Jonas)

60
128 Million Channel SETI Spectrometer

• 200 MHz Bandwidth, 2 Hz resolution

61
(No Transcript)
62
1 GHz bandwidth Pocket Spectrometer

• Using ATMEL ADCs at 2 Gsamples/sec
• Performing 4 real FFTs in 1 (complex) biplex
  pipelined FFT module.
• 2048 channels
• Uses just 1 ADC, 1 IBOB, and your laptop.

63
(No Transcript)
64
Portable VLBI backend

• Interfaces to MARK 5B data recorder
• 500 MHz spectrum recorder.
• (This makes 4 instruments in 1 year!)

65
VLBI Mark 5B Front End 500 MHz BW, 32
channel filter bank
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
Multi-Purpose Spectrometer Low Bandwidth
Aaron Parsons
200 Aux. I/O
I
200 Mhz ADC

Pol. 1
Q
200 Mhz ADC
I
200 Mhz ADC

Pol. 2
Q
200 Mhz ADC
70
SERENDIP V Spectrometer
71

• SETI Applications
• JPL/UCB/SI DSN Sky Survey (20 GHz Bandwidth)
• Parkes Southern SERENDIP
• ALFA Sky Survey (300 MHz x 7 beams)
• SETI Italia (Bologna)
• SETI_at_home
• Astronomy Applications
• GALFA Spectrometer Arecibo Multibeam Hydrogen
  Survey
• Astronomy Signal Processor ASP Don Backer,
  Ingrid Stairs, et al(pulsars)
• ATA4 Correlator F Engine
• Reionization Experiments (Don Backer, Rich
  Bradley, Chippendale, Ekers)
• Antenna Holography, ATNF, China
• GMRT correlator

72
Astronomy Signal Processor Don Backer, Jeff
Mock, Paul Demorest
SERENDIP V Polyphase Filter Bank
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
128 MHz
Pol. 1
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
128 MHz
Pol. 2
Server w/ EDT card
GbE Switch
PC
PC
PC
PC
PC
PC
PC
PC
73
(No Transcript)
74
GALFA Spectrometer at Arecibo Jeff Mock,
Henry Chen, Aaron Parsons
75
GALFA Spectrometer
Multipurpose Spectrometer Board
Quadrature Downconverter Board
-50 to 50 MHz
LPF
?
IF Pol. 1
100 MHz
sin
LPF
?
Decimate by 16
cos
FIR LPF
LPF
?
IF Pol. 2
e-i?t
Decimate by 16
100 MHz
sin
FIR LPF
LPF
?
e-i?t
cos
-50 to 50 MHz
12.5 Mhz Digital
76
GALFA Lowpass Filter, Jeff Mock
77
GALFA Lowpass Filter
78
Mars Orbiter mm Spectrometer
79
ASIC based spectrometer (mars)

• 2W/ADC 2W/ASIC 4 Watts
• Use UCBs Chip in a Day software
• (compiles FGPA code into ASIC)
• Use rad hard libraries from LBL

80
1960 First Radio Astronomy Digital Correlator
21 lags 300kHz clock discrete
transistors 19,000
Sandy Weinreb
81
Correlator processing power
107
SKA
EVLA
109
106
.
ALMA
LOFAR
105
GFlops
SMA
EVN/WSRT
104
106
103
VLA
DXB
DAS
102
DLB
DCB
10
1
103
70
75
90
85
80
95
2000
05
10
2015
source Arnold van Ardenne
82
Moores Law Instruments using FPGAs 2X per
year (1,000,000 over 20 years)
83
Future Spectrometers
84
Caveats

• Risky
• Simulink new, buggy, not open source
• (verilog, vhdl old)
• just a bunch of clever students,
• Weve built the easy instruments so far,
• (Not the hard ones), yet to demonstrate
  packetized
• Correlator and compute cluster

85
CASPER the Friendly...

• Group Helping Open-source Signal-processing
  Technology (GHOST?)
• Goal to help develop signal processing
  instrumenation and libraries for the community.
• Open source hardware, gateware, and software.
• Provide training and tutorials
• Not so much delivering turn-key instruments.

86
Selected correlator quotes
Sandy Weinreb In 1960 there were no chips just
discrete transistors! The 19,000 was the cost
of the samplers, shift registers, and counter. It
did not include the cost of the 21 accumulators
which I made myself in a few months getting paid
240/month.
Ray Escoffier With correlator performance having
gone up by a factor of 922,000 over the last 30
years, its only fair that correlator design
engineers' salaries should have gone up by a
similar factor!!
Sergei Pogrebenko

It is desirable that the output data rate
from a data processor is less than the input data
rate.
87

• http//seti.berkeley.edu/casper