- CPV
- EMC
- [QC on FLP - plot integrated over the run] Error Type for SM (Super Module)
- [QC on FLP] Payload Size/events
- [QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL
- [QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV and Cell Occupancy plots (CALIB) for E>0.5 GeV
- [QC on EPN - plot integrated over the run] Cell Amplitude
- [QC on EPN - plot integrated over the run] Cell Time
- FDD
- FT0
- FV0
- HMP
- ITS
- MCH
- MFT
- MID
- PHS
- TOF
- TPC
- TRD
CPV#
Cosmics runs#
[QC on EPN] Digit Map in M2, M3, M4#
If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if he is not informed yet. If PHS is not busy but you see this message then inform the on-call.
Message "Cold (hot) 3G cards (N/X)" is tolerated if N < 100.
Good plots:
The plots represent number of digits seen in each channel. It should be more or less populated. In case when it is very different from reference please inform CPV on-call.
[QC on EPN] Error occurance#
Good plot:
If plot is not OK then put a log entry and call the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.
Minimal duration after SOR before taking any action required by these instructions: 10 min
EMC#
Don't call EMC expert during the night if run is not physics
[QC on FLP - plot integrated over the run] Error Type for SM (Super Module)#
| Green: good quality | Red: bad quality |
|---|---|
 |
 |
- If no entries, a green message "No Error: OK" inform you that everything is working properly.
- In case of errors, a red message will appear: call EMCAL oncall if EMCAL is included in global runs. Take note of the error-type in the y axis.
[QC on FLP] Payload Size/events#
| good quality | bad quality | empty |
|---|---|---|
 |
 |
 |
- If "Data OK" is shown, everything is fine.
- If some of the DDL presents entries that are larger than the others, a red message will appears: please call EMCAL oncall.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
[QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL#
| good quality | bad quality | empty |
|---|---|---|
 |
 |
 |
- One peak should be visibile if EMCAL is in data taking.
- If a second peak appears around "Min raw amplitude (ADC)" in the range 20-50, call EMCAL oncall
- If the plot is empty, and EMCAL is included in the data taking: please call EMCAL oncall
[QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV and Cell Occupancy plots (CALIB) for E>0.5 GeV#
| good quality | bad quality (superMod not sending data) | bad quaity (superMod noisy) |
|---|---|---|
 |
 |
 |
Cell Occ. with E>0.2 GeV: the occupancy plots should be uniformly filled during Cosmics runs.
- if something noisy appears, please call the EMCAL oncall.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
Cell Occupancy plots with E>0.5 GeV (CAL)
- if some large empty areas (expect for the PHOS hole, present in the example for Cell Occ. with E<0.2 GeV) appear, please call EMCAL oncall.
[QC on EPN - plot integrated over the run] Cell Amplitude#
- no instructions for the moment.
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
- If peaks appear but the text in histogram is okay, That's a problem due to bad channel. send a message to EMCAL expert via mattermost.
[QC on EPN - plot integrated over the run] Cell Time#
- If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall. If the errors appear during nigh shift:
- add a note in the logbook, don't call the on-call.
- in case LHC is preparing for collisions the on-call should be notified at "Prepare Ramp" (even during the night in this case) in case of such errors occur in a technical run before, so that the cards can be recovered before taking physics data.
FDD#
FDD: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 5 min. from the start of the RUN to judge about the quality of collected data.
FDD: Amplitude vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Charge amplitude distribution vs channel ID [0-15]. It should be non-empty.
Units: Charge is expressed in ADC channels.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
FDD: Time vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
The time distribution vs channel ID [0-15].
It should be non-empty.
Units: Time is expressed in TDC channels, channel = 13 ps.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
FT0#
FT0: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.
There will be many empty channels in FT0 plots for COSMICS. This can be safely ignored
FT0: Amplitude vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Charge amplitude distribution vs channel ID [0-207]. It should be non-empty.
Units: Charge is expressed in ADC channels.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
FT0: Time vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
The time distribution vs channel ID [0-207].
It should be non-empty.
Units: Time is expressed in TDC channels, channel = 13 ps.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
FV0#
FV0: General#
All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.
FV0: Amplitude vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
Charge amplitude distribution vs channel ID [0-47]. It should be non-empty.
Units: Charge is expressed in ADC channels.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
FV0: Time vs channel#
[QC on EPN/QC nodes - plot integrated over the QC cycle]
The time distribution vs channel ID [0-47].
It should be non-empty.
Units: Time is expressed in TDC channels, channel = 13 ps.
Actions:
- if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.
HMP#
Busy time#
[QC on FLP] 
The plot shows the busy time of the detector. In case more than three equipments exceeds 120 microsec or the plot is empty call HMP on-call
Event size#
[QC on FLP] 
The plot shows the event size of the detector. In case more than three equipments exceeds 13 kB or the plot is empty call HMP on-call
ITS#
General considerations#
The ITS QC plots during the cosmic runs can suffer from the lack of statistics, therefore, report about QC error messages only after at least 15 minutes of data-taking; the exception is the (Lane Status Summary and Decoding Errors plots). If any other plot remains empty after 15 minutes, please call the ITS on-call.
Quality summary#
The left panel summarises all ITS QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter.
- Quality: BAD: contact ITS on-call expert
- Quality: Medium: create a log entry
- Quality: NULL: the plots are empty. Check in DCS if ITS is in STANBY. If not, inform the ITS on-call.
In BAD or MEDIUM cases, this canvas will duplicate the error message from the QC plot in the form "Flag: Unknown: ERROR MESSAGE". The bottom line shows an example of BAD status in Track Angular distribution and MEDIUM in nClusterPerTrack plots.
The right panel provides a time trend of the ITS QC summary. If ITS quality is BAD for the whole run duration, ITS should be tagged as BAD in the logbook.
Error count vs Error id#
[QC on FLP]
The total number of decoding errors (the error ID is on the x-axis). The left figure provides an example of a good case, while the right corresponds to run with BAD quality;
The BAD quality flag will be triggered when a number of decoding errors exceed some limit; MEDIUM quality corresponds to the case when the number of errors is non-zero, but not significant for the BAD run. The following quality messages can be printed on the plot:
- Quality::GOOD
- Quality:: BAD: "definition of error ID"
- Quality:: MEDIUM: "definition of error ID"
If the number of error with a certain ID show a BAD quality message, call the ITS on-call. In case of the MEDIUM quality, create a log entry linked to run and with tag = ITS. Please note that entries in the last bin are not an issue, because this bin is not related to any detector problems.
Please ignore the last bin of this plot. The increase of counts there should be neglected.
Fake-hit rate overview#
[QC on FLP]
Overview of the averaged fake-hit rate value for each ITS stave (given by triangles). The fake-hit rate value is displayed with the colored scale in hits/events/pixels. The left plot provides an example of a GOOD plot, and the right figure is an example of a BAD quality plot.
The following quality messages can be printed on the plot:
- Quality::GOOD
- Quality:: Medium: Max occupancy over 10^{-6}
- Quality:: BAD: Max occupancy over 10^{-5}
- Quality:: BAD: There are staves without hits
In case of BAD quality messages or empty triangles (some might stay empty for the first 25-30 mins of a run, do not call in this case), please call the ITS on-call.
Trigger count vs TriggerID and FeeID#
[QC on FLP]
Plot summarizing trigger flags. The X-axis corresponds to the ID of Front-End Electronic; the Y-axis shows the list of all possible triggers that can be received by FEE. The coloured scale represents the counts in each bin. During physical run, we expect that all FEEs receive HB, ORBIT, SOC and TF trigger signals. The GOOD run is shown on the left panel, while the problematic one is depicted on the right. QC can plot the following error messages:
- Quality::GOOD
- Quality:: BAD: Trigger flag (TRIGGER_NAME) of bad quality
Please, note that the PHYSICS trigger is missing in cosmic runs. This results in a BAD quality message because of this trigger, which should be ignored.
call the ITS on-call in the case of BAD quality messages and also in the following cases: - there are x-bins without entries (i.e., vertical white lines on the plot) - entries are present in the empty lines in the example plot above. - the plot is completely empty
Lane Status Flag: ERROR/FAULT/WARNING#
[QC on FLP]
This plot indicates the faction of lanes (colored axis) in ERROR/FAULT/WARNING status for each ITS stave. The empty plot corresponds to the GOOD run quality (left plot); in other cases (right plot), the following quality messages can appear on the plot:
- Quality:: MEDIUM: ML/OL have staves in ERROR
- Quality:: MEDIUM: Inner Barrel has stave with >2 chips in ERROR/FAULT/WARNING
- Quality:: BAD: Layer 0/1/2/3/4/5/6 has >25% staves with lanes/chips in ERROR/FAULT/WARNING
If the BAD quality message is printed, call the ITS on-call.
Lane Status Global#
[QC on FLP]
This plot shows the fraction of lanes into ERROR, FAULT, and WARNING statuses. The TOTAL bin gives the total fraction of lanes in any not OK status. The BAD quality will be triggered when the bin value exceeds the 10% threshold.
The following Quality messages can appear:
- Quality::GOOD
- Quality::BAD: >10% of the lanes are bad.
In case of BAD quality messages call the ITS on-call.
ITS Misconfiguration plot#
[QC on FLP]
This plot shows the estimated readout rate for each FEE component of the ITS. Note that each FEE should have the same estimated frequency, as it is shown in the example figure.
The following Quality messages can appear:
- Quality::GOOD
- Quality::BAD: MISCONFIGURATION. CALL EXPERTS.
In case of BAD quality messages call the ITS on-call. Additionally, compare the estimated ITS readout rate with the ITS readout rate from the DCS shifter: these numbers should be the same, otherwise, call the ITS on-call.
Cluster Occupancy overview#
[QC on EPN]
Overview of the cluster occupancy, i.e., number of clusters per event, for each stave (1 bin in the plot). The left figure gives example of the GOOD run, while on the right is the problematic distribution. Cluster occupancy should be uniform for comic runs.
- Quality:: MEDIUM: Layer_Stave has large cluster occupancy
- Quality:: BAD: Layer_Stave has empty stave
call the ITS on-call in case of anomalies in the plot or if the BAD quality message will appear.
Fraction of empty lanes#
[QC on EPN]
This figure provides the fraction of empty lanes (without clusters) per IB/OL/ML/Total. When any of the bins exceeds the 10% limit, you will be notified with the Quality:: BAD: message. Please, call the ITS on-call in this case.
Number of clusters per track#
[QC on EPN]
Distribution of the number of clusters per track. The plot shows a GOOD example of a cosmic run. The following messages can appear:
- Quality::GOOD
- Quality::Medium Mean is outside 5.2-6.2, ignore for COSMICS and TECHNICALS
- Quality::BAD: 0 tracks with 4/5/6/7 clusters (OK if it's synthetic run)
- Quality::BAD: NO ITS TRACKS
call the ITS on-call in case of BAD quality messages, if a completely different plot is obtained, or if the plot stays empty after 5 min of data taking. Ignore MEDIUM quality message for COSMIC data.
MCH#
Quality Summary#
[QC on EPN]
The left panel shows a summary of the automated checked on the MCH data, in a human-readable format. The top line describes the aggregated quality status, followed by a message suggesting the appropriate action according to the quality level:
-
Bad: immediately inform the MCH on-call
-
Medium: write a logbook entry, tagging MCH
-
Null: the plots are completely empty. Check in DCS if MCH is in STANBY. If not, inform the MCH on-call.
The right panel shows a trending plot of the aggregated quality. The message in the left panel always corresponds to the most recent point in the trending plot.
If the quality in the trend plot is Bad for the whole duration of a run, MCH should be marked as Bad in the Bookkeeping flags for the run.
Quality Plots#
The following plots show the distribution of various estimators of the MCH data quality. Each horizontal bin shows the value of the monitored quantity, averaged over one Detection Element(DE). The vertical dashed lines show the boundaries between each of the 10 MCH chambers. An horizontal dashed line shows the threshold used by the checker to decide if a given detection element is considered good or bad.
The checker assigns an overall Good (green), Medium (orange) or Bad (red) quality flag to the plot, depending on the number and pattern of bad DEs. In general, the quality is still considered Good if only few DEs are bad. The quality is set to Medium if several DEs are Bad, but no significant impact on the detector acceptance is expected. If the number and pattern of bad DEs is such that the acceptance will be degraded, the quality is set to Bad.
The overall aggregated MCH quality is the logic AND of the qualities of the individual plots.
Fraction of Synchronized Boards#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that are properly synchronized. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Fraction of Boards not in Error#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that do not have decoding errors. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Fraction of Boards with Good Rate#
[QC on EPN]
The plot shows, for each Detection Element, the fraction of FEC boards that have a correct hit rate. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.
Average Hit Rate#
[QC on EPN]
The plot shows the average hit rate (in kHz) for each detection element. A given DE is coinsidered bad if the corresponding rate is below the horizontal dashed line.
Average Pseudo-efficiency#
[QC on EPN]
The plot shows the average pseudo-efficiency for each detection element. The detection efficiency is estimated from the correlation between the pre-clusters reconstructed in either cathode of each DE. A given DE is coinsidered bad if the corresponding efficiency is below the horizontal dashed line.
MFT#
Quality summary#
The left panel summarises all MFT QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter. * Quality: BAD: contact MFT on-call expert * Quality: Medium: create a log entry * Quality: NULL: the plots are empty.
The bottom line shows an example of BAD status in Chips in error, Digit occupancy and Cluster occupancy plots.
The right panel provides a time trend of the MFT QC summary. In cosmics, it usually takes ~5 minutes after the start of run for the occupancy plots to be files. If the quality stays BAD after ~5 minutes from the start of run, call the MFT on-call.
If MFT quality is BAD for the whole run duration, MFT should be tagged as BAD in the logbook.
Chips in Error/Fault/Warning#
[QC on FLP - plot reset at each cycle]
Plot description
This plot is created on the FLPs. The histogram shows the number of MFT chips in Error/Fault/Warning and provides their list up to 20 entries.
Checks to be done
Check the message on the histogram.
Actions to be taken
Follow the instructions on the histogram message.
Digit Occupancy Summary#
[QC on FLP - plot integrated over the run]
Plot description
This plot is created on the FLPs. The histogram shows the number of digits per ROF (Readout Frame) per MFT zone.
Checks to be done
Wait for ~5 minutes after the start of run for the histogram to be filled. Then check the message in the automatic checker for empty zones.
Actions to be taken
If several MFT zones are empty after ~5 minutes from the start of run, the quality is bad (message in red), call the MFT on-call immediately.
Cluster Occupancy Summary#
[QC on EPN - plot integrated over the run]
Plot description
This plot is created on the EPNs. The histogram shows the number of clusters per MFT zone.
Checks to be done
Wait for ~5 minutes after the start of run for the histogram to be filled. Then check the message in the automatic checker for empty zones.
Actions to be taken
If several MFT zones are empty after ~5 minutes from the start of run, the quality is bad (message in red), call the MFT on-call immediately.
Distribution of the #clusters per ROF#
[QC on EPN - plot integrated over the run]
Plot description
This plot is created on the EPNs. The histogram shows the ROF size for clusters.
Checks to be done
The histogram should look similar to the good quality reference in this documentation.
Actions to be taken
If the plot looks significantly different from the good quality reference, call the MFT on-call.
MID#
Local boards occupancy map (DigitsQC)#
[QC on EPN]
The plot shows fired local Boards.
In case of :
-
empty column call On Call
-
empty or very high rate on several neighboring boards of the detector call On Call
Hits multiplicity (DigitsQC)#
[QC on EPN]
These plots show hits multiplicity by plane for bending and non-bending
-
if mean value > 100.
call On Call
PHS#
Cosmics runs#
Monitor if plots are not empty and timestamps are updated.
Ignore any red messages except "Number of entries has not changed in the past cycle".
If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if he is not informed yet. If PHS is not busy but you see this message then inform the on-call.
[QC on EPN] Cell HG occupancy in M1, M2, M3, M4#
Good plots:
The plots represent number of cells seen in each channel. It should be more or less populated. In case when it is very different from reference please inform PHS on-call.
[QC on EPN] Error occurance#
Good plot:
If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.
Minimal duration after SOR before taking any action required by these instructions: 10 min
TOF#
Ignore alarms if TOF is not READY
Readout map (Slot Participating)#
[QC on FLP - plot integrated over the run]
| Green panel: good quality | Red panel: bad quality |
|---|---|
 |
 |
The plot shows a map of TOF readout slots per crate. The checker controls if enough crates are in the readout, if it detects lower than expected the quality is set to BAD. In case of red allarm please call TOF oncall.
Hit Multiplicity#
[QC on EPN - plot integrated over the run]
IMPORTANT: please ignore warnings for this histogram untill further notice, see also TOF known issues paragraph.
| Green panel: good quality | Yellow panel: medium quality |
|---|---|
 |
 |
The plot shows the number of hits detected by TOF, a checker provides instructions for the shifter based on measured counts. ~~In case of yellow allarm please contact TOF on-call via email, in case of red alarm call TOF expert.~~
TPC#
TPC_Physics#
Quality Observer / Number of Clusters / Quality Trending#
[QC on EPN]
To be checked:
-
The time stamp at the bottom should update every two 2 minutes during running.
-
When run is ongoing all Qualities in the list should be GOOD after 3 update cycles (6 minutes).
-
If in the quality trending, the quality is constantly BAD for multiple cycles (>10 minutes) call On-Call.
-
If in the quality trending, the quality is constantly MEDIUM for multiple cycles (>10 minutes) make logbog entry, adding also - if available - a reason from the Quality Observer.
-
In case the Cluster task quality is bad you can check if there are holes in the N cluser plots and call On-Call.
TRD#
Layout for cosmic runs#
Note:
- all QC tasks for TRD are running on the EPNs
Data size per sector#
The TRD has 18 sectors which should all produce a similar data size per TF. In case on sector is deviating a lot (its mean is more/less than factor 3 from the rest) please write a bookkeeping entry tagging TRD.
Tracklet distribution in half-chambers#
This plot shows the number of tracklets per half-chamber. The x-axis is the sector number. If you see one column completely empty please notify the TRD oncall.
The crosses on the plot are from a static half-chamber status map which needs to be replaced by a dynamic one to correctly cross out half-chambers where no data is expected because of hardware issues.
Eta-phi distribution of ITS-TPC-TRD tracks#
This plot will be empty if either ITS or TPC is missing in the run. This is expected and not a problem.
Eta-phi distribution for ITS-TPC tracks matched to at least 3 TRD tracklets. The PHOS-hole from abs(eta) < 0.2 and phi ~ 5 leads to almost no tracks in that region. No need to call TRD expert if plot does not look as example above. We are currently adding automatic checks.
Pulse height plot based on TRD digit data only#
A peak should be visible between timebin 0 and 6. If that is not the case there might be an issue with the trigger settings. Please notify the TRD oncall in that case.
Number of tracklets per event and TF#
We would like to be notified via bookkeeping entry in case either more than one distinct peaks are appearing in the distributions or in case there are entries in the underflow bin in either of the two histograms.