-
Notifications
You must be signed in to change notification settings - Fork 1
daqinterface v3 02 00 instructions
DAQInterface is a set of Python and Bash scripts which make it easy to start and run an existing artdaq-based DAQ system. Specifically, this means the ability to select how many of each type of artdaq process (BoardReaders, EventBuilders, DataLoggers and Dispatchers) to use in a running DAQ system, as well as what hosts they should run on. Additionally, DAQInterface provides built-in support for sending transitions via XML-RPC to artdaq processes at the command line as well as the saving of metadata related to a run (such as the version of artdaq used, the location of the artdaq logfiles, etc.).
Getting DAQInterface to perform runs takes about 5-10 minutes, assuming
you already have it available as a UPS product (e.g., this would be the
case if you have access to the products area
/cvmfs/fermilab.opensciencegrid.org/products/artdaq/) and
you have an installation of the artdaq-demo package which is based on
artdaq v3. If you don’t already have such an installation, you can just
create a new directory, cd into it, and execute the following:
wget https://github.com/art-daq/artdaq-demo/repository/revisions/da3dbe9cf38923581a07544831442b1ad0f3df1d/raw/tools/quick-mrb-start.sh
chmod +x quick-mrb-start.sh
./quick-mrb-start.sh --tag=v3_02_00
For using other artdaq-based packages besides artdaq-demo with DAQInterface, after reading this wiki take a look at the links at the bottom of this page. If you’re stuck after reading the documentation, please contact John Freeman at jcfree@fnal.gov.
To set up the DAQInterface product, do the following:
source <products directory>/setup # Replace <products directory> with the name of the actual products directory
setup artdaq_daqinterface v3_02_00
Now, let’s create a DAQInterface work area:
mkdir $HOME/DAQInterface # You can, of course, make a directory with another name if you like
cd $HOME/DAQInterface
cp $ARTDAQ_DAQINTERFACE_DIR/docs/user_sourcefile_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/settings_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/known_boardreaders_list_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/boot.txt .
Since DAQInterface doesn’t know the experiment-specific requirements of its user, we’ll want to create a file to source which sets up an environment appropriate for our needs. In our area, this file is “user_sourcefile_example”. Let’s open it up, and look at the top line:
export DAQINTERFACE_SETTINGS=$ARTDAQ_DAQINTERFACE_DIR/docs/settings_example
What you’ll want to do is change this top line so that the DAQINTERFACE_SETTINGS variable refers to fully-qualified filename of the settings_example file we copied into our area. Once you’ve done that, let’s open the settings_example file itself, and modify the line
productsdir_for_bash_scripts: $HOME/path_to_product_dir_not_yet_defined/products
so that the value of “productsdir_for_bash_scripts” points to a products directory which contains xmlrpc_c (for sending messages to DAQInterface) and root (so that the events in output rootfiles can be counted by DAQInterface for crosschecking purposes). Then modify the lines
log_directory: /tmp
record_directory: $HOME/run_records
so they refer to the directories in which you’d like artdaq to output
its logfiles (log_directory) and to save records of what
happened during each run (record_directory). Of course,
make sure these directories actually exist.
The last file we’ll edit is the boot.txt file. This file will be passed to DAQInterface on the boot transition (i.e., the transition in which DAQInterface launches the artdaq processes). Take its top line:
DAQ setup script: name_of_DAQ_setup_script_not_yet_defined
…and replace name_of_DAQ_setup_script_not_yet_defined with the fully-qualified filename of the script you want DAQInterface to source before it tries to launch artdaq processes. E.g., if you installed artdaq-demo with the quick-mrb-start.sh script in $HOME/artdaq-demo-installation, you’d want to replace “name_of_DAQ_setup_script_not_yet_defined” with “$HOME/artdaq-demo-installation/setupARTDAQDEMO”.
You’ll also see that hosts (all localhost) and port numbers
have been defined for two EventBuilders, a DataLogger and a Dispatcher;
we’ll leave them like that for now.
Now that we’ve made the needed modifications to the files in our area, it’s time to set up the environment for DAQInterface to run. In the terminal, type
export DAQINTERFACE_USER_SOURCEFILE=$PWD/user_sourcefile_example
source $ARTDAQ_DAQINTERFACE_DIR/source_me
and you’ll see something like the following:
* The command was successful
* To launch, just type "DAQInterface &" (excluding quotes, ampersand optional)
* Output will be logged in /tmp/daqinterface_<your username>/DAQInterface_port<N>.log, where <N> is the port DAQInterface is listening on
* Help is available at https://github.com/art-daq/artdaq_utilities/wiki/Artdaq-daqinterface
so then we type
DAQInterface &
and you’ll fire up DAQInterface, seeing the following:
DAQInterface launched and now in "stopped" state, listening on port 5570
…where the port may be something other than 5570 if a previously-launched DAQInterface is already listening on that port. Hit Enter. DAQInterface is now ready to receive transition commands. Now type the following:
just_do_it.sh $PWD/boot.txt 0 --config "demo" --comps "component01 component02"
This command can be understood to mean, “Give DAQInterface the file $PWD/boot.txt tell it how to launch the artdaq processes, initialize the processes after they’re launched with the FHiCL documents found in the configuration called demo, and in particular, initialize two BoardReaders with FHiCL documents called component01_hw_cfg.fcl and component02_hw_cfg.fcl”. After executing the command, give it a few moments, and you should eventually see something like the following:
Mon Sep 18 14:29:13 CDT 2017: START transition complete for run 959
Will acquire data until Ctrl-C is hit
DAQInterface is now in the “running” state. We’ll step through the
commands executed automatically by the script in a moment, but
essentially what we’ve done is launched a system running two
ToySimulator fragment generators in two BoardReader processes, where
events are assembled by two EventBuilders and sent to a DataLogger which
saves them to disk and sends them to a Dispatcher. To stop datataking,
hit Ctrl-c from the terminal running
just_do_it.sh. Momentarily, you should return to the
command line, and in the terminal running DAQInterface, you should see
something like:
Wed Dec 21 14:34:53 CST 2016: TERMINATE transition complete
To see logfile(s), on localhost run "ls -ltr /tmp/pmt/pmt-55811.*.log"
Whereas just_do_it.sh strung multiple transitions together
in order to make life simpler for the user, for educational purposes
it’s useful to step through running DAQInterface one transition at a
time. In this section, we’ll do just that, as well as provide a fuller
explanation for the output produced during a run. While not required,
you may want to execute the commands from another terminal as it makes
the interface a bit cleaner; in order to do this, open a second
terminal, cd into the “DAQInterface” working directory you created
earlier, and do the following:
source <existing products directory>/setup
setup artdaq_daqinterface v3_02_00
cd $HOME/DAQInterface # Or whatever your DAQInterface working directory is
export DAQINTERFACE_USER_SOURCEFILE=$PWD/user_sourcefile_example
source $ARTDAQ_DAQINTERFACE_DIR/source_me
…where for the future, you may want to consider putting all those commands in a single file to source. Now that you’ve set up the environment, assuming you launched DAQInterface in partition 0, run the following:
status.sh
and you should see this:
Result:
String: 'stopped'
Essentially, status.sh queries the DAQInterface state
machine and prints DAQInterface’s state to the screen.
just_do_it.sh uses this script to ensure that the
transitions it sends DAQInterface went through successfully. We’re now
in the “stopped” state of DAQInterface - this is the ground state, and
means that no artdaq processes exist. In order to launch processes, we
send the “boot” transition. To send individual transitions, we use the
send_transition.sh script. However, before we boot the
system, we need to specify which components (fragment generators) we
want to use. The allowed fragment generators will be listed by
DAQInterface if you issue it the following command:
listdaqcomps.sh
where the output, in the terminal in which DAQInterface was launched, will look like this:
# of components found in listdaqcomps call: 2
component01 (runs on localhost)
component02 (runs on localhost)
..etc. These lines refer to ToySimulator instances designed to run on
the current host; in a real experiment you’d expect a more complex
listing. They are defined in the
known_boardreaders_list_example file. Let’s select two
components (although we could run with just one, or all of them, if we
wanted):
setdaqcomps.sh component01 component02
…at the end of the output when you execute this is “Result: Nil”; this is expected, and occurs whenever you send DAQInterface this command, or any of the upcoming transitions. Now we can send the boot transition:
send_transition.sh boot boot.txt
In the terminal running DAQInterface, you’ll see something like this:
Tue May 22 14:24:41 CDT 2018: BOOT transition underway
Finished call to launch_procs(); will now confirm that artdaq processes are up...
Checking that processes are up (check 1 of a max of 5)...
Checking that processes are up (check 2 of a max of 5)...
All processes appear to be up
artdaq_mfextensions v1_03_01, e15:prof:s64, appears to be available; if
windowing is supported on your host you should see the messageviewer
window pop up momentarily
Tue May 22 14:25:01 CDT 2018: BOOT transition complete
If you’re sending transitions in the same terminal as DAQInterface is
running, hit Enter to return to the command line. What the above output
tells us is that we’ve successfully launched the artdaq processes, and
that since the version of artdaq_mfextensions needed by the
MessageViewer application is available, assuming your terminal’s display
settings are set correctly, a window will pop up which will print
messages sent by the artdaq processes. The boot.txt file which you
edited earlier is used by DAQInterface during the boot transition to
determine where to find the script to source before it tries to launch
the processes, and then how many EventBuilder, DataLogger and Dispatcher
processes to launch and where to launch them. The BoardReader processes
which get launched are defined when setdaqcomps.sh is
called, in concert with the information provided by
known_boardreaders_list_example.
Now that the processes are alive, we’ll configure them with FHiCL documents. Simply execute the following:
send_transition.sh config demo
“demo” is the name of the configuration which defines the FHiCL
documents used to initialize the artdaq processes. Technically speaking,
it refers to the subdirectory which can be found in the
$ARTDAQ_DAQINTERFACE_DIR/simple_test_config directory, and
which contains the needed FHiCL documents. It’s possible to change the
way DAQInterface interprets the configuration name by editing the user
source file, either by switching it to look at another directory on the
filesystem or getting it to use an instance of artdaq-database,
discussed in the advanced links at the bottom of this wiki. If you wish
to see the list of available configurations, just type
listconfigs.sh
and you’ll see something like the following:
Available configurations:
demo
demo_artdaq_v1
demo_routingmaster_2x2
no_aggregators
pull_mode
request_dataflow_windowmode
See file "/tmp/listconfigs_$USER.txt" for saved record of the above configurations
Most of these configurations are intended for developer use; we’ll be working with “demo”. Once you’ve sent the config transition, you’ll see the return values of the individual artdaq processes (should be “Success”) and then something like this:
To see logfile(s), on localhost run "ls -ltr /tmp/pmt/pmt-32989.*.log"
Wed Dec 21 15:02:48 CST 2016: CONFIG transition complete
Basically, as a courtesy to users - a particularly useful one, if
MessageViewer isn’t available - the name of the logfiles for the run are
provided in wildcard form. Please be aware that there’s a possibility
that logfiles from much older runs may be encompassed by that wildcard
if artdaq’s Process Management Tool had the same process ID for the
older run as for the current run; this is typically only an issue if
there are logfiles for hundreds or thousands of runs in the same
directory. It’s possible to watch the logfile update in real time if, in
a separate terminal, you type tail -f .
Along with the new logfiles, you’ll also find in the “/tmp” directory a directory called “run_record_attempted_”, where “” is your username, stored in the environment variable $USER. In here, what you’ll find are the following:
-The FHiCL documents which are used to initialize the artdaq processes
-The DAQInterface boot file, boot.txt
-The metadata file, metadata.txt
-The script which DAQInterface sourced before launching the artdaq
processes, setup.txt
If something goes wrong during configuration,
/tmp/run_record_attempted_ can potentially be useful
for diagnostic purposes. Otherwise, its contents get moved to the run
records directory on the start transition, covered next. Click on “Show”
for details on the metadata file:
{{collapse()
The metadata file which gets saved in the run records directory (defined
in the settings file as described earlier in this wiki) is formatted
like the following example. The meaning of parameters will be explained
below:
Config name: demo
Component #0: component01
Component #1: component02
DAQInterface directory: /scratch/jcfree/DAQInterface_for_instructions
DAQInterface commit: v3_02_00
artdaq commit: e27f3f3a7dac3bf3e85a649f19848b93011e24db
artdaq-core-demo commit: 0023a890bc3bf77b14cfeddfddb2c16248e8a276
artdaq-demo commit: dbe2f4e22c8c1494c3d0cf0db9426cf5670a0879
pmt logfile(s): woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/pmt/pmt-25781.*.log
boardreader logfiles:
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/BoardReader-woof-5305/BoardReader-woof-5305-20180521144555-25841.log
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/BoardReader-woof-5306/BoardReader-woof-5306-20180521144555-25843.log
eventbuilder logfiles:
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/EventBuilder1-woof-5235/EventBuilder1-woof-5235-20180521144555-25844.log
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/EventBuilder2-woof-5236/EventBuilder2-woof-5236-20180521144555-25846.log
aggregator logfiles:
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/DataLogger1-woof-5265/DataLogger1-woof-5265-20180521144555-25847.log
woof.fnal.gov:~/scratch/DAQInterface_for_instructions/log/Dispatcher1-woof-5269/Dispatcher1-woof-5269-20180521144555-25848.log
-
Config name: This is the label of the configuration which got passed during the config transition -
Component #: The Nth component (BoardReader w/ fragment generator defined) selected for the run -
commit: The git commit hash of the package used in the artdaq-based DAQ system operated via DAQInterface. These packages can be selected in the settings file. -
pmt logfile(s): The wildcard of artdaq’s Process Management Tools logfiles for the run, prefaced by the host they were written to. These are the files typically of greatest interest for users (as opposed to developers). -
logfiles: The names of every individual process type’s output logfile (one per process), each one prefaced by the host the process in question ran on. For historical reasons “aggregator” covers both DataLoggers and Dispatchers.
The above variables are the ones which appear in the metadata file during the config transition. However, other variables related to the actual running will appear as you go through the start and stop transitions described below; by the time the metadata file has been finished being written to, something like the following will have been added:
Start time: Fri Sep 8 18:09:17 UTC 2017
Stop time: Fri Sep 8 18:09:45 UTC 2017
}}
Now let’s issue the start transition:
send_transition.sh start
and then DAQInterface will report
Wed Dec 21 16:22:29 CST 2016: START transition complete for run 2
Now we’re in the “running” state, as you can confirm if you issue the
status.sh command. Note that the run # is deduced from
the run records directory - it’s a value one higher than the most
recently saved run in the run records directory. If you look in the run
records directory you’ll now see that the contents of
/tmp/run_record_attempted have been copied into the
subdirectory called “2/”.
Stopping datataking and putting DAQInterface in the “ready” state is quite simple:
send_transition.sh stop
at which point, you can either start a new run, or you can send the artdaq processes the “shutdown” command and then kill them, via the terminate transition:
send_transition.sh terminate
You can confirm that you’ve created a *.root file from the run by
looking in the /tmp directory.
The DAQInterface package provides a couple of convenient tools for rapidly examining the results of your run after it’s complete:
show_logfile_for_run.sh <run number> <open file (0 or 1)>
…will show the logfile for the run number given in the first argument, and will open the file for you (or not) depending on whether the second argument is 1 or 0.
show_warnings_for_run.sh <run number>
…will show all MessageFacility messages of severity level Warning or Error in the logfile.
If you’ve reached this point in the documentation, you’re now ready to prepare DAQInterface to work not with artdaq-demo, but with your own experiment’s artdaq-based software. You’ll want to make copies of the settings file and boot file you’ve been working with and make the appropriate edits; how to do so is described in [[daqinterface_v3_02_00_settings_file_reference | The settings file reference]] and [[daqinterface_v3_02_00_boot_file_reference | The boot file reference]]. You’ll also want to switch to using the artdaq-database as the source of your configurations rather than the local directory in the DAQInterface package; making this switch is described in [[artdaq_v3_02_00_changing_the_source_of_the_configurations | Changing the source of the configurations]].
Finally, when you call the setdaqcomps.sh command, DAQInterface will look in the file referred to by the DAQINTERFACE_KNOWN_BOARDREADERS_LIST environment variable and use the file’s contents to determine the host and port on which to launch the BoardReaders; the corresponding information for the other artdaq process types is provided in the boot file. You’ll want to make a copy of the $DAQINTERFACE_KNOWN_BOARDREADERS_LIST file we’ve been using, and edit it, since you’ll want to give your experiment’s fragment generators more descriptive names than “component01”, “component02”, etc. E.g. if you want to use the name tpcreceiver02, then you’ll want to both make sure that a file called tpcreceiver02_hw_cfg.fcl exists in the configuration you use, and also that there’s a corresponding row in the file $DAQINTERFACE_KNOWN_BOARDREADERS_LIST which looks like this:
tpcreceiver02 madeuphost.fnal.gov 9445
where of course you’ll want to use a host which is appropriate for your purposes and a port which isn’t already listed on another row. Also make sure to edit the user_sourcefile_example to set DAQINTERFACE_KNOWN_BOARDREADERS_LIST correctly.