- CPV
- EMC
- [QC on FLP - plot integrated over the run] Payload Size/events
- [QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL
- [QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV
- [QC on EPN - plot integrated over the run] Cell Amplitude
- [QC on EPN - plot integrated over the run] Digit Time
- FDD
- FT0
- FV0
- HMP
- ITS
- MCH
- MFT
- MID
- PHS
- TOF
- TPC
- TRD
CPV#
Synthetic runs#
Monitor if plots are not empty and timestamps are updated.
If you see "Number of entries has not changed in the past cycle" but run is still ongoing inform the oncall.
[QC on EPN] Digit Map in M2,3,4#
The plots represent number of digits seen in each channel.
Note that in SYNTHETIC runs the Digit Maps can be rather sparse because of little statistics replayed from MC simulated data.
Ignore any red messages except "Number of entries has not changed in the past cycle".
[QC on EPN] Errors occurance#
The plot shows decoder error occurance per event.
There are 2 types of synthetic runs:
- Data replay as input for synthetic loop. Please inform oncall in case when plot is not OK.
- MC events as input for synthetic loop. For such runs no decoder errors are expected. If plot is not empty please inform the on-call. Bad example: errors present but message is "Ok"
Please ask SL or RM if run type is data replay or MC simulted.
Minimal duration after SOR before taking any action required by these instructions: 5 min
EMC#
Don't call EMC expert during the night if run is not physics
[QC on FLP - plot integrated over the run] Payload Size/events#
| good quality | bad quality | empty |
|---|---|---|
 |
 |
 |
- If "Data OK" is shown, everything is fine.
- If some of the DDL presents entries that are larger than the others, a red message will appears: please call EMCAL oncall.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
[QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL#
| good quality | bad quality | empty |
|---|---|---|
 |
 |
 |
- One peak should be visibile if EMCAL is in data taking.
- If a second peak appears around "Min raw amplitude (ADC)" in the range 20-50, call EMCAL oncall
- If the plot is empty, and EMCAL is included in the data taking: please call EMCAL oncall
[QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV#
| good quality | good quality | bad quality |
|---|---|---|
 |
 |
 |
- The occupancy plots should be uniformly filled.
- if something noisy appears, please call the EMCAL oncall.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
[QC on EPN - plot integrated over the run] Cell Amplitude#
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
[QC on EPN - plot integrated over the run] Digit Time#
- if the peaks are not present, call EMCAL oncall.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
FDD#
FDD: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 5 min. from the start of the RUN to judge about the quality of collected data.
FDD: Out of bunch collisions#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings.
Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FT0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.
- Currently in simulation all the triggers are absent, so this figure should be empty in such case
- Please ignore at the moment if the figure is empty
FDD: Fraction of events with CFD in ADC gate#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Efficiency for getting charge information in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.
Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.
FT0#
FT0: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.
FT0: Out of bunch collisions#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. For the Vertex trigger should be almost ZERO counts.
Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FV0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.
FT0: CFD efficiency#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Efficiency for getting charge information in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.
Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.
FV0#
FV0: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.
FV0: Out of bunch collisions#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the TrgNchan trigger.
Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FT0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.
FV0: Fraction of events with CFD in ADC gate#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Efficiency for getting charge information in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.
Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.
FV0: Fraction of events with the CFD in time gate#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Fraction of events with CFD in gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.
Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.
HMP#
Busy time#
[QC on FLP]
The plot shows the busy time of the detector.
Event size#
[QC on FLP]
The plot shows the event size of the detector.
ITS#
General considerations#
If any of the plots listed on this page remains empty during a run or its timestamp and run number are not up to date, please call the ITS on-call.
Error count vs Error id#
[QC on FLP]
The total number of decoding errors (the error ID is on the x-axis). The left figure provides an example of a good case, while the right corresponds to run with BAD quality;
The BAD quality flag will be triggered when a number of decoding errors exceed some limit; MEDIUM quality corresponds to the case when the number of errors is non-zero, but not significant for the BAD run. The following quality messages can be printed on the plot:
- Quality::GOOD
- Quality:: BAD: "definition of error ID"
- Quality:: MEDIUM: "definition of error ID"
If the number of error with a certain ID show a BAD quality message, call the ITS on-call. In case of the MEDIUM quality, create a log entry linked to run and with tag = ITS.
Please ignore the last bin of this plot. The increase of counts there should be neglected.
Trigger count vs TriggerID and FeeID#
[QC on FLP]
Plot summarizing trigger flags. The X-axis corresponds to the ID of Front-End Electronic; the Y-axis shows the list of all possible triggers that can be received by FEE. The coloured scale represents the counts in each bin. During physical run, we expect that all FEEs receive HB, ORBIT, SOC and TF trigger signals. The GOOD run is shown on the left panel, while the problematic one is depicted on the right. QC can plot the following error messages:
- Quality::GOOD
- Quality:: BAD: Trigger flag (TRIGGER_NAME) of bad quality
call the ITS on-call in the case of BAD quality messages and also in the following cases:
- there are x-bins without entries (i.e., vertical white lines on the plot)
- entries are present in the empty lines in the example plot above.
- the plot is completely empty
Ignore the quality message about the SOC trigger for the SYTH run type
Cluster Occupancy overview#
[QC on EPN]
Overview of the cluster occupancy, i.e. clusters per event, for each stave (1 bin in the plot). Good example for current synthetic pp (left) and synthetic Pb-Pb (right) are provided. Note the different order of magniture for the z axis. Check the general trend, i.e. occupancy decreasing when going from the innermost to the outermost layers. QC provides the following quality messages for this plot:
- Quality:: MEDIUM: Layer_Stave has large cluster occupancy
- Quality:: BAD: Layer_Stave has empty stave
In particular, in synthetic runs you will see messages about larger cluster occupancy in different layers.
call the ITS on-call if the plot is different from example.
Angular Distribution#
[QC on EPN]
Angular distribution: phi vs eta (2D plot) of online reconstructed tracks. Example of GOOD plots for synthetic pp (left) and synthetic Pb-Pb (right), no action needed in this case.
Possible Quality messages that can appear on the plot during a run:
- Quality::GOOD
- BAD: Asymmetric Phi distribution (OK if there are disabled ITS sectors)
- BAD: Asymmetric Eta distribution (OK if there are disabled ITS sectors)
- BAD: NO ITS TRACKS
Please call the ITS on-call if plots look much different from example.
Number of clusters per track#
[QC on EPN]
Distribution of the number of clusters per track. The plot shows a GOOD and BAD example of a synthetic run for both pp and Pb-Pb collisions. The following messages can appear:
- Quality::GOOD
- Quality::Medium Mean is outside 5.2-6.2, ignore for COSMICS and TECHNICALS
- Quality::BAD: 0 tracks with 4/5/6/7 clusters (OK if it's synthetic run)
- Quality::BAD: NO ITS TRACKS
call the ITS on-call if a completely different plot is obtained, if the quality is BAD, or if the plot stays empty after 5 min of data taking.
MCH#
Quality Summary#
[QC on EPN]
The left panel shows a summary of the automated checked on the MCH data, in a human-readable format. The top line describes the aggregated quality status, followed by a message suggesting the appropriate action according to the quality level:
-
Bad: immediately inform the MCH on-call
-
Medium: write a logbook entry, tagging MCH
-
Null: the plots are completely empty. Check in DCS if MCH is in STANBY. If not, inform the MCH on-call.
The right panel shows a trending plot of the aggregated quality. The message in the left panel always corresponds to the most recent point in the trending plot.
If the quality in the trend plot is Bad for the whole duration of a run, MCH should be marked as Bad in the Bookkeeping flags for the run.
Quality Plots#
The following plots show the distribution of various estimators of the MCH data quality. Each horizontal bin shows the value of the monitored quantity, averaged over one Detection Element(DE). The vertical dashed lines show the boundaries between each of the 10 MCH chambers. An horizontal dashed line shows the threshold used by the checker to decide if a given detection element is considered good or bad.
The checker assigns an overall Good (green), Medium (orange) or Bad (red) quality flag to the plot, depending on the number and pattern of bad DEs. In general, the quality is still considered Good if only few DEs are bad. The quality is set to Medium if several DEs are Bad, but no significant impact on the detector acceptance is expected. If the number and pattern of bad DEs is such that the acceptance will be degraded, the quality is set to Bad.
The overall aggregated MCH quality is the logic AND of the qualities of the individual plots.
Fraction of Synchronized Boards#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that are properly synchronized. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Fraction of Boards not in Error#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that do not have decoding errors. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Fraction of Boards with Good Rate#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that have a correct hit rate. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Average Hit Rate#
[QC on EPN]
The plot shows the average hit rate (in kHz) for each detection element. A given DE is coinsidered bad if the corresponding rate is below the horizontal dashed line.
Average Pseudo-efficiency#
[QC on EPN]
The plot shows the average pseudo-efficiency for each detection element. The detection efficiency is estimated from the correlation between the pre-clusters reconstructed in either cathode of each DE. A given DE is coinsidered bad if the corresponding efficiency is below the horizontal dashed line.
MFT#
Quality summary#
The left panel summarises all MFT QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter. * Quality: BAD: contact MFT on-call expert * Quality: Medium: create a log entry * Quality: NULL: the plots are empty.
The bottom line shows an example of BAD status in Chips in error, Digit occupancy and Cluster occupancy plots. The right panel provides a time trend of the MFT QC summary. If MFT quality is BAD for the whole run duration, MFT should be tagged as BAD in the logbook.
Distribution of the #clusters per ROF#
[QC on EPN - plot integrated over the run]
Plot description
This plot is created on the EPNs. The histogram shows the ROF size for clusters (left for synthetic pp, right for synthetic PbPb).
Checks to be done
The histogram should look similar to the reference in this documentation.
Actions to be taken
If the plot looks significantly different from the reference call the MFT on-call.
Cluster Occupancy Summary#
[QC on EPN - plot integrated over the run]
Plot description
This plot is created on the EPNs. The histogram shows the number of clusters per MFT zone.
Checks to be done
There is an automatic checker for empty zones.
Actions to be taken
If several MFT zones are empty, the quality is bad (message in red), call the MFT on-call immediately.
MID#
Local boards occupancy map (DigitsQC)#
[QC on EPN]
The plot shows fired local Boards.
Check just that plots are there and that they are filled.
** Notify the expert via mattermost and/or email ONLY in case there are no plots or completely empty plots **
Hits multiplicity (DigitsQC)#
[QC on EPN]
These plots show hits multiplicity by plane for bending and non-bending
-
if mean value > 100.
call expert
PHS#
Monitor if plots are not empty and timestamps are updated.
Ignore any red messages except "Number of entries has not changed in the past cycle".
If you see "Number of entries has not changed in the past cycle" but run is still ongoing inform the oncall.
[QC on EPN] Cell occupancy#
- The plot shows cell occupancy map in all 4 modules
- Note that because of a very little statistics replayed in synthetic runs, PHS occupancy is rather sparse.
- In case if plots are empty call the PHS on-call
[QC on EPN] Errors occurance#
- The plot shows decoder error occurance per event.
- For synthetic runs no decoder errors are expected. If plot is not empty please inform the PHS on-call.
- Please note that "Ok" message is false in case when there are enrties.
Bad example: errors present
Bad example: errors present but message is "Ok"
Minimal duration after SOR before taking any action required by these instructions: 5 min
TOF#
Ignore alarms if TOF is not READY
Readout map (Slot Participating)#
[QC on FLP - plot integrated over the run]
| Green panel: good quality | Red panel: bad quality |
|---|---|
 |
 |
The plot shows a map of TOF readout slots per crate. The checker controls if enough crates are in the readout, if it detects lower than expected the quality is set to BAD. In case of red allarm please call TOF oncall.
Important: This plot will not update during synthethic replay data runs.
Hit Multiplicity#
[QC on EPN - plot integrated over the run]
| Green panel: good quality | Yellow panel: medium quality |
|---|---|
 |
 |
The plot shows the number of hits detected by TOF, a checker provides instructions for the shifter based on measured counts. ~~In case of yellow allarm please contact TOF on-call via email, in case of red alarm call TOF expert.~~
TPC#
TPC SYNTHETIC#
Quality Observer / Number of Clusters / Quality Trending#
[QC on EPN]
To be checked:
-
The time stamp at the bottom should update every two 2 minutes during running.
-
When run is ongoing all Qualities in the list should be GOOD after 3 update cycles (6 minutes). In case of a BAD quality put log entry.
-
If in the quality trending, the quality is constantly BAD for multiple cycles (>10 minutes) call On-Call.
-
In case the Cluster task quality is BAD you can check if there are holes in the N cluster plots and inform OnCall.
-
If Only PID quality is BAD check if dEdx calculation is activated. If this is not the case the PID quality can be ignored. For the dEdx following parameter has to be in the EPN Extra ENV WORKFLOW_EXTRA_PROCESSING_STEPS=\"TPC_DEDX\" . This is typically diabled for PbPb running
- For 1 MHz replay data the holes shown in the plot below are expected:
TRD#
Layout for synthetic runs#
Note:
- all QC tasks for TRD are running on the EPNs
Data size per sector#
The TRD has 18 sectors which should all produce a similar data size per TF. In case on sector is deviating a lot (its mean is more/less than factor 3 from the rest) please write a bookkeeping entry tagging TRD.
Tracklet distribution in half-chambers#
This plot shows the number of tracklets per half-chamber. The x-axis is the sector number. If you see one column completely empty please notify the TRD oncall.
The crosses on the plot are from a static half-chamber status map which needs to be replaced by a dynamic one to correctly cross out half-chambers where no data is expected because of hardware issues.
Eta-phi distribution of ITS-TPC-TRD tracks#
This plot will be empty if either ITS or TPC is missing in the run. This is expected and not a problem.
Eta-phi distribution for ITS-TPC tracks matched to at least 3 TRD tracklets. The PHOS-hole from abs(eta) < 0.2 and phi ~ 5 leads to almost no tracks in that region. For synthetic runs this plot is expected to be filled very sparsly. No need to contact TRD expert in that case.
Number of tracklets per event and TF#
We would like to be notified via bookkeeping entry in case either more than one distinct peaks are appearing in the distributions or in case there are entries in the underflow bin in either of the two histograms.