NeXus in DANSE

1
NeXus in DANSE
Tim KelleyDANSE workshopCaltech, Sept. 4, 2003
2
NeXus is

• File format to store neutron and x-ray data
• Stores meta-data
• Application Program Interface (API)
• NeXus is subset of HDF
• (Hierarchical Data Format)
• only Groups, Datasets, and Links
• Analogy file system
• Nexus File system
• Groups??Directories
• Datasets??Files
• Links??Links

NeXus API
HDF API
3
NeXus is

• Hierarchies for neutron x-ray data
• Classes for Groups
• Sample, Instrument, Detector
• Contents of classes
• Naming rules
• In other words A Markup Language!

lt NXentry name"entry" gt lt NXsample
name"sample" gt ltDataset name"ch_data"
rank"1" dimension1"12" datatype"4" /gt
lt/NXsamplegt
4
Markup language?

• Examples HTML (webpages), LaTex (printed docs)
• Markup indicates structure
• HTML ltpgt Its a paragraph! lt/pgt
• LaTex \beginitemize \item List! \enditemize
• Easy cross-platform communication (ASCII, not
  binary)
• Human readable (ASCII, not binary)

5
eXtensible Markup Language (XML)

• international standard for markup languages
• (loosely) any markup language that complies with
  the standard
• neither HTML nor LaTex!
• many tools available in many languages/platforms
• parse XML documents
• validate documents

6
NeXus in DANSE possible users view
DANSE component
data.nxs
Selector array1
DANSE component
Reader Widget
Selector array2
DANSE component
7
NeXus in Python
client.py
Python
disk
Data.nxs
8
Using the NeXus API

• nxopen(...)
• nxopengroup(...)
• nxopendata(...)
• nxgetdata(...)
• nxclosedata(...)
• nxclosegroup(...)
• nxopengroup(...)
• ...

9
NeXus in DANSE
UI
ReaderWidget
SelectorWidget
NexusReader
NexusFile
NexusFile Repository
nexus.py
10
NeXus in DANSE
UI
ReaderWidget
SelectorWidget
Reader
NexusFile
3ColASCIIFile
SPEFile
nexus.py
File Repository
11
User Interface view
DANSE component
data.nxs
Selector array1
DANSE component
Reader Widget
Selector array2
DANSE component
ISIS .spe
3-col ASCII
12
NeXus in DANSE
UI
ReaderWidget
SelectorWidget
nexml
NexusReader
search
Renderer
NexusFile
NexusFile Repository
nexus.py
13
class NexusFile

• Medium between NeXus API and higher-level DANSE
  components
• Build representation of NeXus Groupss contents
• Graph of Python objects
• Serve representation
• Python graphs root object
• XML representation
• Insert data sets into streams
• Future
• extract data sets from streams

14
NexusFile.graph()

• Create graph of Python objects
• represent hierarchy of NeXus entities (groups and
  datasets)
• Python classes
• NexusElement (Base class)
• Group (NexusElement)
• Dataset (NexusElement)
• Support Visitor pattern

15
Nexus elements
Group
id
expand
children
subgroups
className
16
NexusFile.graph() operation
NexusFile myNXF
Group root
def graph(self, group None) if group is
None group self._root if group is not
self._root nexus.nxopengroup()
group.expand(self)
def expand(self, nexusFile) subroups,
children nexusFile.listGroup()
self._children.update( ) for subgroup in
subgroups if subgroup not in
self._subgroups self._subgroups.appen
d(subgroup) nexusFile.graph( subgroup)
17
NexusFile XML representation

• Formed by Renderer class
• Simplified set of tags
• only convey structural information
• ltGroupgt
• name, class name, all group attributes appended
  to element as XML attributes
• ltDatasetgt
• name, rank, dimensions, datatype become XML
  element attributes
• ltDataset name"ch_data" rank"1" dimension1"12"
  datatype"4" /gt
• Renderer uses Visitor Pattern
• (cf. Design Patterns, Gamma et al.)

18
Visitor pattern

• Separates iterating a graph and actions at each
  node of the graph
• Good when
• more than one thing to do with a graph
• multiple node types
• double-dispatch system

19
Visitor pattern

• Two participants
• Host (graph of Python Node objects,
• Node ? Dataset, Group, etc.)
• Each Node object has id( ) method
• id( ) receives reference to visitor
• id( ) invokes Visitors onNode( ) method
• someGroup.id( yourVisitor) calls
  yourVisitor.onGroup( self)
• Visitor (e.g. Renderer)
• For each Node class, implements onNode()
• For container nodes, onNode() iterates over
  children
• calls child.id( ) on each

20
Renderer
weaver
Mill
pickle

• Inherits Visitor methods from base class
• Inherits XML writing methods from Mill
• Implements _renderDocument method expected by Mill

_begin
_end

21
NexusFile.insert()
from nexus import nexus class NexusFile

• def insert( self, dataset, stream)

self._navigate( dataset) nexus.nxgetdata(
self._handle, stream) return
22
class NexusReader

• Provide simple search service
• searchSelect
• user specifies name of datatset, attributes,
  parent info
• returns Dataset object
• Insert dataset into stream
• forwards NexusFile.insert()

NexusReader
searchSelect
insert
XMLstring
XMLstream
NexusFile
23
NexusReader.searchSelect()

• Search for dataset
• dataset attributes
• optionally parent name class name
• Search implemented via XML SAX parser
• class Searcher (ContentHandler)
• function nexml.search() manages search
• coordinates parser, content handler
• filters results
• makes Dataset object for each match

24
Parsing with SAX

• Parser provided by Python standard library
• (several, actually)
• Event-driven
• reaction to events determined by Content
  Handler
• Examples
• when parser reads a start tag ltGroup nameentry
  classNameNXentrygt, calls contentHandler.startEl
  ement(Group, attributes)
• when parser reads lt/Groupgt, calls
  contentHandler.endElement(Group)
• We override ContentHandler methods to determine
  response to content

25
ContentHandler
xml.sax.ContentHandler
def startElement(self, name, attributes)
push current node on stack if name
self._targetType Check if attributes
match target attributes If
so, copy stack to results list
startElement
endElement
characters

def endElement(self, name) current node
pop stack return
nexml.Searcher
startElement
endElement
results
26
In conclusion

• NeXus reading software constructed in several
  layers
• Each layer develops new interfaces
• add new functions
• adapt underlying layers for new purposes
• Light-weight layers
• more reusable
• more maintainable