CPV
EMC
FDD
FT0
FV0
HMP
- Busy time
- Event size
ITS
MCH
- Quality Summary
- Quality Plots
MFT
MID
- Local boards occupancy map (DigitsQC)
- Hits multiplicity (DigitsQC)
PHS
TOF
- Readout map (Slot Participating)
- Hit Multiplicity
TPC
- TPC_Physics
  - Quality Observer / Number of Clusters / Quality Trending
TRD
- Layout for physics runs in Pb-Pb
ZDC
VTX
- Vertex distributions from matched central barrel tracks

CPV#

Physics HI runs#

Monitor if plots are not empty and timestamps are updated.

If plots are empty or not updated please inform the on-call.

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if expert is not informed yet. If PHS is not busy but you see this message then inform the on-call.

If you see other red messages please check known issues. If issue is not known please inform the on-call.

[QC on QC merger node] Global CPV quality#

Please check known issues

Good plot:

alt text

The plot summarize CPV quality. If global quality is not good then please check below the details.

Number of digits increases assures that digits are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle".
Number of clusters increases assures that clusters are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle".
Digit occupancy check assures that digit occupancy is good. If bad then check messages on Digit Map in M2,3,4 plots and follow the instructions.
Cluster size check assures that mean cluster size is within allowed limits. If bad please inform the on-call. If medium - put a log entry and inform the oncall during morning and afternoon shift. No need to call during night.
CalibDigit amplitude check assures that observed amplitude spectra are good. If bad please inform the oncall. Medium means lack of statistics: please check later.
Errors presence check that occured number of errors is within limits. If not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues.

[QC on QC merger node] Global quality trend#

Good plot:

alt text

The trend shows evolution of global CPV quality. It can become not good for small portion of cycles due to fluctuations. In this case run to be considered as good.

If the trend is not good for significant part of the run then run quality must be setted to bad. Put bad flag even if issue is known. Expert will have a look and adjust the qaulity later if needed.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

[QC on EPN] Digit Map in M2, 3, 4#

Please check known issues

Good plots: alt text The plots represent number of digits seen in each channel. It should be more or less uniform.

If red messages tell Hot 3G Cards (N1/N2) or Cold 3G Cards (N1/N2) then put log entry and ask the oncall to check the plots.

Bad example: HV Trip in module M2

Bad example: Number of entries has not changed in the past cycle

Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz

Actions to be taken by QC shifters#

Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz.
Inform on-call when quality is bad and it is not a known issue.

EMC#

[QC on FLP - plot integrated over the run] Raw Data Error#

Green: good quality	Red: bad quality

If no entries, a green message "No Error: OK" inform you that everything is working properly.
In case of errors, a red message will appear: call EMCAL oncall if EMCAL is included in global runs. Take note of the error-type in the y axis.

[QC on FLP - plot integrated over the run] Payload Size/events#

good quality	bad quality	empty

If "Data OK" is shown, everything is fine.
If some of the DDL presents entries that are larger than the others, a red message will appears: please call EMCAL oncall.
If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

[QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL#

good quality	bad quality	empty

One peak should be visibile if EMCAL is in data taking.
If a second peak appears around "Min raw amplitude (ADC)" in the range 20-50, call EMCAL oncall
If the plot is empty, and EMCAL is included in the data taking: please call EMCAL oncall

[QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV and Cell Occupancy plots (CALIB) for E>0.5 GeV#

good quality (high-E)	good quality (high_E)	bad quaity

IF PHYSICS RUN - The occupancy plots should be uniformly filled during Physics runs.
- if something noisy appears, please call the EMCAL oncall. - If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

Occupancy plots: Acceptance losses#

Missing acceptance (low-E)	Missing acceptance (high-E)

In case counts in a certain detector segment stop increasing the corresponding segment stops sending data. The corresponding area must be visible in both plots. - call EMCAL on-call

[QC on EPN - plot integrated over the run] Cell Amplitude#

DigitAmplitude

no instructions for the moment.
If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

[QC on EPN - plot integrated over the run] Cell Time#

good quality	bad quality

We expect a gaussian peak roughly centred at 0 and a structure of single lines with a spacing of 100 ns from noisy channels. Depending on the filling scheme smaller gaussian like peaks from from pileup could appear. In such cases the data is OK.
In case a second peak appears which is either of the same magnitude as the main peak or of magnitude 1/3 as the main peak and the peaks are separated by 100 ns call the EMCAL on-call
If the main peak deviates from 0 by more than 100 ns call the EMCAL on-call
If no gaussian like peak is present call the EMCAL on-call
If the plot is empty, and EMCAL is included in the data taking, call EMCAL on-call.

Number of triggers#

good quality	bad quality

The number of physics triggers must increase while the run is ongoing. In case the number of trigger stays constant no data is received any more from the detector (most likely detector busy) - call the EMCAL on-call.

FDD#

FDD: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 5 min. from the start of the RUN to judge about the quality of collected data.

FDD: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fdd_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the Vertex trigger.

Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FT0 and FV0 BC-trigger plots; if they also show WARNING - call FIT-on-call.

FDD: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fdd_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action. - If ERROR - call FIT-on-call.

FDD: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fdd_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FDD: Fraction of charge in ADC range#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fdd_cfd_gate

Fraction of charge in ADC range in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FDD: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_cfd_gate

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FT0#

FT0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

FT0: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the Vertex trigger.

Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FV0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.

FT0: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FT0: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FT0: Fraction of channels out of colliding BCs#

[QC on EPN/QC nodes - plot integrated over the QC cycle ft0_channel_out_of_bunch

Fraction of channles fired out of colliding BCs. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FT0: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_trg_validation

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FV0#

FV0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle.
Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

FV0: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the TrgNchan trigger.

Actions:
- If ERROR - call FIT-on-call.
- If WARNING - compare to FT0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.

FV0: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FV0: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

FV0: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_cfd_gate

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:
- If WARNING - no action.
- If ERROR - call FIT-on-call.

HMP#

Busy time#

[QC on FLP] busy_time

The plot shows the busy time of the detector. In case more than three equipments exceeds 120 microsec or the plot is empty call HMP on-call

Event size#

[QC on FLP] event_size

The plot shows the event size of the detector. In case more than three equipments exceeds 13 kB or the plot is empty call HMP on-call

ITS#

General considerations#

If any of the plots listed on this page remains empty during a run, please call the ITS on-call. The only exception, i.e., where empty means good quality, is for the four plots on the lane status.

Quality summary#

its_summary The left panel summarises all ITS QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter.

Quality: BAD: contact ITS on-call expert
Quality: Medium: create a log entry
Quality: NULL: the plots are empty. Check in DCS if ITS is in STANBY. If not, inform the ITS on-call.

In BAD or MEDIUM cases, this canvas will duplicate the error message from the QC plot in the form "Flag: Unknown: ERROR MESSAGE". The bottom line shows an example of BAD status in Track Angular distribution and MEDIUM in nClusterPerTrack plots.

The right panel provides a time trend of the ITS QC summary. If ITS quality is BAD for the whole run duration, ITS should be tagged as BAD in the logbook.

Error count vs Error id#

[QC on FLP] its_pb1

The total number of decoding errors (the error ID is on the x-axis). The left figure provides an example of a good case, while the right corresponds to run with BAD quality;

The BAD quality flag will be triggered when a number of decoding errors exceed some limit; MEDIUM quality corresponds to the case when the number of errors is non-zero, but not significant for the BAD run. The following quality messages can be printed on the plot:

Quality::GOOD
Quality:: BAD: "definition of error ID"
Quality:: MEDIUM: "definition of error ID"

If the number of error with a certain ID show a BAD quality message, call the ITS on-call. In case of the MEDIUM quality, create a log entry linked to run and with tag = ITS. Please note that entries in the last bin are not an issue, because this bin is not related to any detector problems.

Fake-hit rate overview#

[QC on FLP] its_pb2_new Overview of the averaged fake-hit rate value for each ITS stave (given by triangles). The fake-hit rate value is displayed with the colored scale in hits/events/pixels. The left plot provides an example of a GOOD plot, and the right figure is an example of a BAD quality plot.

The following quality messages can be printed on the plot:

Quality::GOOD
Quality:: Medium: Max occupancy over 10^{-6}
Quality:: BAD: Max occupancy over 10^{-5}
Quality:: BAD: There are staves without hits

In case of BAD quality messages or empty triangles (some might stay empty for the first 25-30 mins of a run, do not call in this case), please call the ITS on-call.

Trigger count vs TriggerID and FeeID#

[QC on FLP] its_pb4 Plot summarizing trigger flags. The X-axis corresponds to the ID of Front-End Electronic; the Y-axis shows the list of all possible triggers that can be received by FEE. The coloured scale represents the counts in each bin. During physical run, we expect that all FEEs receive HB, ORBIT, SOC and TF trigger signals. The GOOD run is shown on the left panel, while the problematic one is depicted on the right. QC can plot the following error messages:

Quality::GOOD
Quality:: BAD: Trigger flag (TRIGGER_NAME) of bad quality

Call the ITS on-call in the case of BAD quality messages and also in the following cases:

there are x-bins without entries (i.e., vertical white lines on the plot)
entries are present in the empty lines in the example plot above.
the plot is completely empty

Lane Status Flag: ERROR/FAULT/WARNING#

[QC on FLP] its_pb5

This plot indicates the faction of lanes (colored axis) in ERROR/FAULT/WARNING status for each ITS stave. The empty plot corresponds to the GOOD run quality (left plot); in other cases (right plot), the following quality messages can appear on the plot:

Quality:: MEDIUM: ML/OL have staves in ERROR
Quality:: MEDIUM: Inner Barrel has stave with >2 chips in ERROR/FAULT/WARNING
Quality:: BAD: Layer 0/1/2/3/4/5/6 has >25% staves with lanes/chips in ERROR/FAULT/WARNING

If the BAD quality message is printed, call the ITS on-call.

Lane Status Global#

[QC on FLP] its_pb8

This plot shows the fraction of lanes into ERROR, FAULT, and WARNING statuses. The TOTAL bin gives the total fraction of lanes in any not OK status. The BAD quality will be triggered when the bin value exceeds the 10% threshold.

The following Quality messages can appear:

Quality::GOOD
Quality::BAD: >10% of the lanes are bad.

In case of BAD quality messages call the ITS on-call.

ITS Misconfiguration plot#

[QC on FLP] its_pb8

This plot shows the estimated readout rate for each FEE component of the ITS. Note that each FEE should have the same estimated frequency, as it is shown in the example figure.

The following Quality messages can appear:

Quality::GOOD
Quality::BAD: MISCONFIGURATION. CALL EXPERTS.

In case of BAD quality messages call the ITS on-call. Additionally, compare the estimated ITS readout rate with the ITS readout rate from the DCS shifter: these numbers should be the same, otherwise, call the ITS on-call.

Cluster Occupancy overview#

[QC on EPN] its_pb8

Overview of the cluster occupancy, i.e., number of clusters per event, for each stave (1 bin in the plot). The left figure gives example of the GOOD run, while on the right is the problematic distribution. Check the general trend, i.e. occupancy decreasing when going from the innermost to the outermost layers. This MO can have the following quality messages:

Quality:: MEDIUM: Layer_Stave has large cluster occupancy
Quality:: BAD: Layer_Stave has empty stave

Call the ITS on-call in case of anomalies in the plot or if the BAD quality message will appear.

Fraction of empty lanes#

[QC on EPN] its_pb8

This figure provides the fraction of empty lanes (without clusters) per IB/OL/ML/Total. When any of the bins exceeds the 10% limit, you will be notified with the Quality:: BAD: message. Please, call the ITS on-call in this case.

Angular Distribution#

[QC on EPN] its_pb8 its_pbtemp

Angular distribution of online reconstructed ITS tracks as a function of phi vs. eta (2D plot).
Possible Quality messages that can appear on the plot during a run:

Quality::GOOD: plot might still be bad! See example above and look at the arrows in the plot.
BAD: Asymmetric Phi distribution (OK if there are disabled ITS sectors)
BAD: Asymmetric Eta distribution (OK if there are disabled ITS sectors)
BAD: NO ITS TRACKS

Please try to reconfigure the detector (set run type again) in case the quality is BAD or if the plot shows several holes (blue regions in between the yellow parts as shown in the plot above), call ITS on-call is the issue persists. Please call the ITS on-call also if the plot remains empty during the run.

Number of clusters per track#

[QC on EPN] its_pb9

Distribution of the number of clusters per track. The plot shows a GOOD example of pp collisions run. The following messages can appear:

Quality::GOOD
Quality::Medium Mean is outside 5.2-6.2, ignore for COSMICS and TECHNICALS
Quality::BAD: 0 tracks with 4/5/6/7 clusters (OK if it's synthetic run)
Quality::BAD: NO ITS TRACKS

Call the ITS on-call in case of BAD quality messages, if a completely different plot is obtained, or if the plot stays empty after 5 min of data taking. In case of MEDIUM status, create log-book entry.

MCH#

Quality Summary#

[QC on EPN] mch-qualitysummary

The left panel shows a summary of the automated checked on the MCH data, in a human-readable format. The top line describes the aggregated quality status, followed by a message suggesting the appropriate action according to the quality level:

Bad: immediately inform the MCH on-call
Medium: write a logbook entry, tagging MCH
Null: the plots are completely empty. Check in DCS if MCH is in STANBY. If not, inform the MCH on-call.

The right panel shows a trending plot of the aggregated quality. The message in the left panel always corresponds to the most recent point in the trending plot.

If the quality in the trend plot is Bad for the whole duration of a run, MCH should be marked as Bad in the Bookkeeping flags for the run.

Quality Plots#

The following plots show the distribution of various estimators of the MCH data quality. Each horizontal bin shows the value of the monitored quantity, averaged over one Detection Element(DE). The vertical dashed lines show the boundaries between each of the 10 MCH chambers. An horizontal dashed line shows the threshold used by the checker to decide if a given detection element is considered good or bad.

The checker assigns an overall Good (green), Medium (orange) or Bad (red) quality flag to the plot, depending on the number and pattern of bad DEs. In general, the quality is still considered Good if only few DEs are bad. The quality is set to Medium if several DEs are Bad, but no significant impact on the detector acceptance is expected. If the number and pattern of bad DEs is such that the acceptance will be degraded, the quality is set to Bad.

The overall aggregated MCH quality is the logic AND of the qualities of the individual plots.

Fraction of Synchronized Boards#

[QC on EPN] $mch-syncfrac$ [QC on EPN] $mch-syncfracok$

The plot shows, for each Detection Element, the fraction of FEC boards that are properly synchronized. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards not in Error#

[QC on EPN] $mch-noerrorfrac$ [QC on EPN] $mch-noerrorfracok$

The plot shows, for each Detection Element, the fraction of FEC boards that do not have decoding errors. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards with Good Rate#

[QC on EPN] $mch-goodfrac$ [QC on EPN] $mch-goodfracok$

The plot shows, for each Detection Element, the fraction of FEC boards that have a correct hit rate. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Average Hit Rate#

[QC on EPN] mch-meanrate [QC on EPN] mch-meanrateok

The plot shows the average hit rate (in kHz) for each detection element. A given DE is coinsidered bad if the corresponding rate is below the horizontal dashed line.

Average Pseudo-efficiency#

[QC on EPN] mch-effnotok [QC on EPN] mch-effok

The plot shows the average pseudo-efficiency for each detection element. The detection efficiency is estimated from the correlation between the pre-clusters reconstructed in either cathode of each DE. A given DE is coinsidered bad if the corresponding efficiency is below the horizontal dashed line.

MFT#

Quality summary#

QualitySummaryGood QualitySummaryBad

The left panel summarises all MFT QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter. * Quality: BAD: contact MFT on-call expert * Quality: Medium: create a log entry * Quality: NULL: the plots are empty.

The bottom line shows an example of BAD status in Chips in error, Digit occupancy and Cluster occupancy plots.

The right panel provides a time trend of the MFT QC summary. If MFT quality is BAD for the whole run duration, MFT should be tagged as BAD in the logbook.

Chips in Error/Fault/Warning#

[QC on FLP - plot reset at each cycle] ReadoutChipsError ReadoutChipsFault ReadoutChipsWarning

Plot description

This plot is created on the FLPs. The histogram shows the number of MFT chips in Error/Fault/Warning and provides their list up to 20 entries.

Checks to be done

Check the message on the histogram.

Actions to be taken

Follow the instructions on the histogram message. Since MFT has auto-recovery of sensors implemented, call the on-call if the quality remains bad for more than 2 minutes.

Digit Occupancy Summary#

[QC on FLP - plot integrated over the run] DigitOccSummary

Plot description

This plot is created on the FLPs. The histogram shows the number of digits per ROF (Readout Frame) per MFT zone.

Checks to be done

There is an automatic checker for empty zones.

Actions to be taken

If several MFT zones are empty, the quality is bad (message in red), call the MFT on-call immediately.

Cluster Occupancy Summary#

[QC on EPN - plot integrated over the run] ClusterOccSummary

Plot description

This plot is created on the EPNs. The histogram shows the number of clusters per MFT zone.

Checks to be done

There is an automatic checker for empty zones.

Actions to be taken

If several MFT zones are empty, the quality is bad (message in red), call the MFT on-call immediately.

Distribution of the #clusters per ROF#

[QC on EPN - plot integrated over the run] clustersROF_pp

Plot description

This plot is created on the EPNs. The histogram shows the ROF size for clusters.

Checks to be done

The histogram should look similar to the good quality references (left) in this documentation.

Actions to be taken

If the plot looks significantly different from the good quality references call the MFT on-call.

MID#

Local boards occupancy map (DigitsQC)#

[QC on EPN] mid-locboard

The plot shows fired local Boards. In case of :

empty column.
empty or very high rate on several neighboring boards of the detector.
very low rate on several neighboring boards of the detector.

call expert

Hits multiplicity (DigitsQC)#

[QC on EPN] mid-hit-mult

These plots show hits multiplicity by plane for bending and non-bending

if mean value > 100.

call expert

PHS#

Physics PP runs#

Monitor if plots are not empty and timestamps are updated.

If plots are empty or not updated please inform the on-call.

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if expert is not informed yet. If PHS is not busy but you see this message then inform the on-call.

If you see other red messages please check known issues. If issue is not known please inform the on-call.

[QC on QC merger node] Global PHS quality#

Good plot:

alt text

The plot summarize PHS quality. If global quality is not good then please check below the details.

Number of cells increases assures that digits are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle" above.
Number of clusters increases assures that clusters are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle" above.
Cells check assures that cells occupancy is good. If not good then check Cells HG occupancy plots and follow the instructions.
Clusters check assures that mean cluster energy size is within allowed limits. If not good then please inform the on-call. If medium - put a log entry and inform the oncall during morning and afternoon shift. No need to call during night.
Errors check assures that occured number of errors is within limits. If not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues.

[QC on QC merger node] Global quality trend#

Good plot:

alt text

The trend shows evolution of global PHS quality. It can become medium for small portion of cycles due to failure of fit procedure. In this case run to be considered as good.

If the trend is not good for significant part of the run then run quality must be setted to bad. Put bad flag even if issue is known. Expert will have a look and adjust the qaulity later if needed.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

[QC on EPN] Cell HG occupancy in M1, 2, 3, 4#

Please check known issues

Good plots: alt text The plots represent number of cells seen in each channel. It should be more or less uniform.

If red messages say Not OK then put log entry and inform the oncall. Note that messages can be false positive. If you think the plots are not different too much from reference then just put log entry, no need to call during night. If they are differ too much (big empty/hot regions, etc.) then call the on-call. Please do not hesitate to call the on-call if you have any doubts: it's better to wake up us at night than loose data!

Actions to be taken by QC shifters#

Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz.
Follow the instructions above.

TOF#

Ignore alarms if TOF is not READY

Readout map (Slot Participating)#

[QC on FLP - plot integrated over the run]

Green panel: good quality	Red panel: bad quality

The plot shows a map of TOF readout slots per crate. The checker controls if enough crates are in the readout, if it detects lower than expected the quality is set to BAD. In case of red allarm please call TOF oncall.

Hit Multiplicity#

[QC on EPN - plot integrated over the run]

hitmultiplicity_good

The plot shows the number of hits detected by TOF, a checker provides instructions for the shifter based on measured counts. In case of yellow allarm please contact TOF on-call via email, in case of red alarm call TOF expert.

TPC#

TPC_Physics#

[QC on EPN]

To be checked - General:

The time stamp at the bottom should update every two 1 minutes during running.

rawDig_nClusters

When run is ongoing all Qualities in the list should be GOOD after 5 minutes. In case of a bad quality call OnCall.
If in the quality trending, the quality is constantly BAD for multiple cycles (>10 minutes) call On-Call.
In case the Raw task / Cluster task quality is bad you can check if there are holes in the N cluser plots and call On-Call.

cluster_per_track - In the plot "Cluster per Track" a prominent peak should be visibile arround 150. If structure differs significantly from reference call OnCall

eta_pos eta_neg eta_ratio - if Eta for positive or negative particle or eta ratio plot differs significantly from reference inform OnCall

cluster_per_track_phi - If Quality of Mean Cluster per Track per sector distribution shows bad quality call OnCall; if quality is Medium put log entry.

Mean_PID_versus_Sector - If "Mean dEdxTot of MIPs tracks per sector" show a bad quality call OnCall; if quality is Medium put log entry.

TRD#

Layout for physics runs in Pb-Pb#

Note:

all QC tasks for TRD are running on the EPNs
the layouts for TRD are currently undergoing major changes
- automatic checks will be added in the future

Data size per sector#

The TRD has 18 sectors which should all produce a similar data size per TF. In case on sector is deviating a lot (its mean is more/less than factor 3 from the rest) please write a bookkeeping entry tagging TRD.

Tracklet distribution in half-chambers#

This plot shows the number of tracklets per half-chamber. The x-axis is the sector number. If you see one column completely empty please notify the TRD oncall.

The crosses on the plot are from a static half-chamber status map which needs to be replaced by a dynamic one to correctly cross out half-chambers where no data is expected because of hardware issues.

Eta-phi distribution of ITS-TPC-TRD tracks#

This plot is currently missing its title in the QCG, needs to be fixed in QC code

Eta-phi distribution for ITS-TPC tracks matched to at least 3 TRD tracklets. The PHOS-hole from abs(eta) < 0.2 and phi ~ 5 leads to almost no tracks in that region. No need to call TRD expert if plot does not look as example above. We are currently adding automatic checks.

Pulse height plot based on ITS-TPC-TRD matched tracks#

Ignore in case eta-phi distribution plot is empty or very sparsly filled.

A clear peak should be visible between time bins 0-4, followed by a plateau and a falling edge around 20. If plot does not look approximately as in the example above (and the track eta-phi plot is filled) please create a bookkeeping entry tagging TRD.

Pulse height plot based on TRD-only data#

In case the pulse height plot based on matched tracks is filled you can ignore this plot.

A peak should be visible between time bins 0-5. Call the TRD oncall if that is not the case, as there might be an issue with the trigger settings. The plateau region at larger time bins might not look as smooth as in the above picture in case there is a lot of pile-up.

Number of tracklets per event and TF#

We would like to be notified via bookkeeping entry in case either more than one distinct peaks are appearing in the distributions or in case there are entries in the underflow bin in either of the two histograms.

ZDC#

Instructions for shifter in Pb-Pb

Please remember to always double check the current run number. The plots will update every minute.

Baseline [QC on FLP]#

Green: good quality	Orange: Warning quality	Red: bad quality

The plot shows the baseline mean values of each ZDC channel. The quality of the histogram is determined by verifying the deviation from the expected average value.

If the plot is empty or the text is red --> call the on-call.
if the plot text is orange --> insert a log entry

Align Plot [QC on FLP]#

Green: good quality	Red: bad quality

The plot shows if all channels are aligned. Most of the channels should be centered on sample 6. If any channel deviates by plus or minus one, it is considered good. - If the plot shows any errors. --> Call ZDC on-call - if the plot contains empty channels --> Call ZDC on-call

Raw Data Error [QC on FLP]#

Green: good quality	Red: bad quality

The Raw Data Error plot must be empty. The plot indicates the presence of the following errors: data corrupted, data loss, generic error. The plot resets every five minutes, so if it was a temporary error then it will disappear. - If the plot shows any errors. --> Call ZDC on-call

Summary ADC [QC on EPN]#

Green: good quality	Red: bad quality

The plot shows the baseline mean values of each ZDC ADC channel. The quality of the histogram is determined by verifying the deviation from the expected average value. - If the plot is empty or the text is red --> call the on-call. - if the plot text is orange --> insert a log entry

Summary TDC [QC on EPN]#

Green: good quality	Red: bad quality

The plot shows the baseline mean values of each ZDC TDC channel. The quality of the histogram is determined by verifying the deviation from the expected average value.

VTX#

Vertex distributions from matched central barrel tracks#

[QC on EPN] async_QC

The left plot shows the transversal x-y distribution of the reconstructed vertices, while the right plot shows the longitudinal vertex distribution.

The mean value of the z vertes distribution usually fluctuates by +/- 0.3 cm around the central value. The mean values in the x-y directions are usually very stable.

The RM/RC must be informed immediately if:

the mean x and y values are outside [-0.1 cm, +0.1 cm]
the mean z value is outside [-1.0 cm, +1.0 cm]
the standard deviation in z is larger than 6.0 cm

A bookkeeping entry tagging RC should be added if the mean z value is outside [-0.5 cm, +0.5 cm].