Researching Fedora, Fedora Meeting Copenhagen 09/28/05

1
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
Researching Fedora To Serve As Central Repository
For Denmarks State And University Library
Building An Use Case Implementation With
Denmarks Broadcasting Archive Of Radio And
Television (BART)
Stephan Drescher, std_at_statsbiblioteket.dk
2
picture 1 Analog Antenna Set
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
Thats where it all started In an attic
3
Options for System
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• complete outsourcing of task
• from shelf
• taylormade system
• own development
• around black box professional equipment
• high-end consumer Linux

4
Nature of the data harvest process
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• 24/7/365
• single shot
• large quantities of data (220 GB a day)
• several independent input sources
• no unique identifier in or between sources
• different MIME types
• data needs to be evaluated and eventually
  corrected after 48 hours
• automatically ingested into repository

5
figure 1 Where will be Fedora within Barts
resource workflow?
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
6
figure 2 Setup Capture
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
PC 1
DR1
Hauppauge WinTV
lokal disk
DR2
Hauppauge WinTV
TV2
Hauppauge WinTV
EMC mass- storage
PC 1
DR1
Hauppauge WinTV
lokal disk
DR2
Hauppauge WinTV
TV2
Hauppauge WinTV
PC 2
DR P1
Hauppauge WinTV
lokal disk
DR P2
Hauppauge WinTV
DR P3
Hauppauge WinTV
PC 2
DR P1
Hauppauge WinTV
lokal disk
DR P2
Hauppauge WinTV
DR P3
Hauppauge WinTV
7
Encoder setup
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• 3 x MPEG1/2 hardware encoder control quality
  and amount of data
• PCM output lossless audio, but down-sampled
• both FM Radio and TV frequency
• tuning on a single card flexible fallback
  configuration
• parallel setup
• backup from original signal source

8
picture 2 Encoder And Sampler
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
digital resources factory basement
9
Mass storage specs
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• 80 - 100 TB a year (average 220 GB a day)
• Backup on tape

10
figure 3 Mass Storage
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
The cooled place
11
figure 3 Signal coverage
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
12
Create recordings order
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• Failover definitions
• Channel definitions
• Virtual encoder definition
• Compression definitions
• Source definitions
• Physical encoder definition
• Valid Compression per physical encoder type
• Encoder mappings
• Virtual encoder
• Physical encoder
• Abstract schedule definition
• Query definitions (where to get start/stop info)

13
Metadata
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• Descriptive metadata
• program information
• creator, author --gt Dublin Core
• Technical metadata
• encoding format
• create/modified dates --gt METS
• Logistical data
• addresses
• Path to footage e.g. --gt METS

14
Collect metadata
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• Ritzau Media Service, (MSSQL database)
• DR 1, (XML stream)
• Logfiles recording, (filesystem)
• Logfiles storage, migration (filesystem)

15
Automatic harvest and ingest system
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
16
Scheduler application
Researching Fedora, Fedora Meeting Copenhagen
09/28/05

• Modules
• Bouncer
• Harvester
• LogSniffer
• Recorder
• Transformer
• Inspector
• Ingester
• Migrator
• Syncronizer

17
e.g. Virtual encoder definition in Recorder module
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
18
figure 4 Main Class Table
Researching Fedora, Fedora Meeting Copenhagen
09/28/05
Bouncer Access to system and modules Login, registration Dispatcher
Harvester Collects metadata Harvest Definition Syncronizer, LogSniffer
LogSniffer Knows how to parse log files List of name-value Recorder
Recorder Schedules encoder Recording Definition Syncronizer
Transformer Arrange data Into different formats, locations e.g. From SQLResultSets to XML, METS, HTML METSWriter, Connector
Inspector Data quality control Checksum, Scene detection LogSniffer
Ingester Feed repository Upload file XML stream Transformer, Fedora
Migrator Move repository Download file XML Stream Fedora, Transformer
Syncronizer Tune a task Mirror Databases Correct Data Transformer
19
Thank you!