Skip to content

CPV#

Cosmics runs#

[QC on EPN] Digit Map in M2, M3, M4#

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS become busy. You can check with ECS shifter and inform the on-call if he is not informed yet. If PHS is not busy but you see this message then inform the on-call.

Message "Cold (hot) 3G cards (N/X)" is tolerated if N < 100.

Good plots: alt text

The plots represent number of digits seen in each channel. It should be more or less populated. In case when it is very different from reference please inform CPV on-call.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not OK then put a log entry and call the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

Minimal duration after SOR before taking any action required by these instructions: 10 min

Known issues#

  • In PEDESTAL runs QC plot "Pedestal sigma distribution M3 is bad". Acording to expert, noise conditions in CPV depend on general environment in ALICE, therefore, from time to time CPV pedestals can become wider which is reflected in QC-CPV calibration run plot with the message "Number of bad pedestal sigmas in module M3 (sometimes in M2) is larger than the upper limit". This is a known issue, can be ignored, but make a log entry when it happens. See: https://ali-bookkeeping.cern.ch/?page=log-detail&id=45015

  • In PHYSICS runs Red messages on QC Plot "Digit Map in M4". Problem with high voltage is preventing this module from running normally. Experts are trying to recover. No need to report. Run quality must be setted to "good" if there are no problems with other modules. See: https://ali-bookkeeping.cern.ch/?page=log-detail&id=53523. Quality aggregator is also expected to show bad quality for Digit oocupancy check and CalibDigit amplitude check.

  

EMC#

Don't call EMC expert during the night if run is not physics

[QC on FLP - plot integrated over the run] Error Type for SM (Super Module)#

Green: good quality Red: bad quality
ErrorTypeOK ErrorTypeError
  • If no entries, a green message "No Error: OK" inform you that everything is working properly.
  • In case of errors, a red message will appear: call EMCAL oncall if EMCAL is included in global runs. Take note of the error-type in the y axis.

[QC on FLP] Payload Size/events#

good quality bad quality empty
PayloadSizeEvOK PLEvError PLEvempty
  • If "Data OK" is shown, everything is fine.
  • If some of the DDL presents entries that are larger than the others, a red message will appears: please call EMCAL oncall.
  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

[QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL#

good quality bad quality
BunchOK BunchError
  • One peak should be visibile for all supermodules around 0 if EMCAL is in data taking.
  • If the peak moves to the range 20-50 "Min raw amplitude (ADC)" for any supermodule, or a second peak appears for any supermodule in that range , call EMCAL oncall
  • If the plot is empty, and EMCAL is included in the data taking: please call EMCAL oncall

[QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV and Cell Occupancy plots (CALIB) for E>0.5 GeV#

good quality bad quality (superMod not sending data) bad quaity (superMod noisy)
OccEbelowGOOD OccEbelow OccErr

Cell Occ. with E>0.2 GeV: the occupancy plots should be uniformly filled during Cosmics runs.
- if something noisy appears, please call the EMCAL oncall. - If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

Cell Occupancy plots with E>0.5 GeV (CAL)
- if some large empty areas (expect for the PHOS hole, present in the example for Cell Occ. with E<0.2 GeV) appear, please call EMCAL oncall.

[QC on EPN - plot integrated over the run] Cell Amplitude#

DigitAmplitude

  • no instructions for the moment.
  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.
  • If peaks appear but the text in histogram is okay, That's a problem due to bad channel. send a message to EMCAL expert via mattermost.

[QC on EPN - plot integrated over the run] Cell Time#

DigitTime

  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall. If the errors appear during nigh shift:
  • add a note in the logbook, don't call the on-call.
  • in case LHC is preparing for collisions the on-call should be notified at "Prepare Ramp" (even during the night in this case) in case of such errors occur in a technical run before, so that the cards can be recovered before taking physics data.

Number of triggers per Time Frame#

good quality bad quality
DigitTimeOK DigitTime
Trigger rate appox. const with minor fluctuations Trigger rate dropping by several events / timeframe or to almost 0

The number of physics triggers per Time Frame must stay constant while the run is ongoing. In case the number of trigger decreases call the EMCAL on-call (for technical and cosmics runs not during the night). The correct value will depend on the interaction rate and a checker will be implemented soon; for the moment please inform the oncall on case of changes during the run.

Known issues#

  • Do not call EMC oncall during the night for QC plots related errors for non PHYSICS runs, unless a PHYSICS run is expected next.

  • For the plot "Number of Physics triggers per timeframe" the expected value is currently approx 20. If value is below 15 for at least 2 consecutive timestamps, call on-call.

  • Payload Size/Event can be ignored in all runs till further notice

  • In plot "Raw data errors" (error rate) quality is bad if the rate is above 100 errors / minute for several minutes, and good if it is below.

  • An empty region is expected near the bottom-right corner for CAL (Calibration) triggers. Cell Occupancy EMCAL,DCAL with E>0.5 GeV/c CAL

  

FDD#

FDD: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 5 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

FDD: Amplitude vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_ampl_channel

Charge amplitude distribution vs channel ID [0-15]. It should be non-empty.

Units: Charge is expressed in ADC channels.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

FDD: Time vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_time_channel

The time distribution vs channel ID [0-15]. It should be non-empty.

Units: Time is expressed in TDC channels, channel = 13 ps.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

Known issues#

  • The Out of bunch collisions plot can be ignored in SYNTHETIC runs at the moment (i.e. no need to take any actions in case of not GOOD quality). The BC distribution is not simulated properly in MC and will always cause problems in this plot.

  

FT0#

FT0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

There will be many empty channels in FT0 plots for COSMICS. This can be safely ignored

FT0: Amplitude vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

ft0_ampl_channel

Charge amplitude distribution vs channel ID [0-207]. It should be non-empty.

Units: Charge is expressed in ADC channels.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

FT0: Time vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

ft0_time_channel

The time distribution vs channel ID [0-207]. It should be non-empty.

Units: Time is expressed in TDC channels, channel = 13 ps.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

Known issues#

  • channels 60-63 (FT0A side) are OFF in PHYSICS and COSMICS runs
  • channels 139 and 176-179 (FT0C side) are OFF in PHYSICS and COSMICS runs

  

FV0#

FV0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

FV0: Amplitude vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_ampl_channel

Charge amplitude distribution vs channel ID [0-47]. It should be non-empty.

Units: Charge is expressed in ADC channels.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

FV0: Time vs channel#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_time_channel

The time distribution vs channel ID [0-47]. It should be non-empty.

Units: Time is expressed in TDC channels, channel = 13 ps.

Actions:

  • if one or more empty channels (crosscheck known-issues file) - call FIT-on-call.

Known issues#

  

HMP#

Busy time#

[QC on FLP]busy_time

The plot shows the busy time for each detector DDL. In case more than three equipments exceeds 120 microsec or the plot is empty call HMP on-call

Event size#

[QC on FLP]event_size

The plot shows the event size for each detector DDL. In case more than three equipments exceeds 13 kB or the plot is empty call HMP on-call

Sum Q maps#

[QC on FLP]BigMapProfile

The plot shows the charge of all the detector channels in only one 2D map. It looks like the example plot shown above. In case it is completely empty call HMP on-call.

Charge vs HV sector#

[QC on FLP]HVsector

The plot shows the charge of all the detector HV sectors in only one 2D map. It looks like the example plot shown above. The white bands correspond to the faulty HV sectors that are off. In case more than two bands (w.r.t those shown here) became white call the an-call.

Occupancy#

[QC on FLP]Occupancy

The plot shows the occupancy for each detector DDL. In case more than three equipments exceeds 3% or the plot is empty call HMP on-call

Known issues#

For the moment links 3 and 12, exlcuded from data taking

  

ITS#

General considerations#

The ITS QC plots during the cosmic runs can suffer from the lack of statistics, therefore, report about QC error messages only after at least 15 minutes of data-taking; the exception is the (Lane Status Summary and Decoding Errors plots). If any other plot remains empty after 15 minutes, please call the ITS on-call.

Quality summary#

its_summary its_summary The left panel summarises all ITS QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter.

  • Quality: BAD: contact ITS on-call expert
  • Quality: Medium: create a log entry
  • Quality: NULL: the plots are empty. Check in DCS if ITS is in STANBY. If not, inform the ITS on-call.

In BAD or MEDIUM cases, this canvas will duplicate the error message from the QC plot in the form "Flag: Unknown: ERROR MESSAGE". The bottom line shows an example of BAD status in Track Angular distribution and MEDIUM in nClusterPerTrack plots.

The right panel provides a time trend of the ITS QC summary. If ITS quality is BAD for the whole run duration, ITS should be tagged as BAD in the logbook.

Error count vs Error id#

[QC on FLP]its_pb1

The total number of decoding errors (the error ID is on the x-axis). The left figure provides an example of a good case, while the right corresponds to run with BAD quality;

The BAD quality flag will be triggered when a number of decoding errors exceed some limit; MEDIUM quality corresponds to the case when the number of errors is non-zero, but not significant for the BAD run. The following quality messages can be printed on the plot:

  • Quality::GOOD
  • Quality:: BAD: "definition of error ID"
  • Quality:: MEDIUM: "definition of error ID"

If the number of error with a certain ID show a BAD quality message, call the ITS on-call. In case of the MEDIUM quality, create a log entry linked to run and with tag = ITS. Please note that entries in the last bin are not an issue, because this bin is not related to any detector problems.

Please ignore the last bin of this plot. The increase of counts there should be neglected.

Fake-hit rate overview#

[QC on FLP]its_pb2_new

Overview of the averaged fake-hit rate value for each ITS stave (given by triangles). The fake-hit rate value is displayed with the colored scale in hits/events/pixels. The left plot provides an example of a GOOD plot, and the right figure is an example of a BAD quality plot.

The following quality messages can be printed on the plot:

  • Quality::GOOD
  • Quality:: Medium: Max occupancy over 10^{-6}
  • Quality:: BAD: Max occupancy over 10^{-5}
  • Quality:: BAD: There are staves without hits

In case of BAD quality messages or empty triangles (some might stay empty for the first 25-30 mins of a run, do not call in this case), please call the ITS on-call.

Trigger count vs TriggerID and FeeID#

[QC on FLP]its_pb4

Plot summarizing trigger flags. The X-axis corresponds to the ID of Front-End Electronic; the Y-axis shows the list of all possible triggers that can be received by FEE. The coloured scale represents the counts in each bin. During physical run, we expect that all FEEs receive HB, ORBIT, SOC and TF trigger signals. The GOOD run is shown on the left panel, while the problematic one is depicted on the right. QC can plot the following error messages:

  • Quality::GOOD
  • Quality:: BAD: Trigger flag (TRIGGER_NAME) of bad quality

Please, note that the PHYSICS trigger is missing in cosmic runs. This results in a BAD quality message because of this trigger, which should be ignored.

call the ITS on-call in the case of BAD quality messages and also in the following cases: - there are x-bins without entries (i.e., vertical white lines on the plot) - entries are present in the empty lines in the example plot above. - the plot is completely empty

Lane Status Flag: ERROR/FAULT/WARNING#

[QC on FLP]its_pb5

This plot indicates the faction of lanes (colored axis) in ERROR/FAULT/WARNING status for each ITS stave. The empty plot corresponds to the GOOD run quality (left plot); in other cases (right plot), the following quality messages can appear on the plot:

  • Quality:: MEDIUM: ML/OL have staves in ERROR
  • Quality:: MEDIUM: Inner Barrel has stave with >2 chips in ERROR/FAULT/WARNING
  • Quality:: BAD: Layer 0/1/2/3/4/5/6 has >25% staves with lanes/chips in ERROR/FAULT/WARNING

If the BAD quality message is printed, call the ITS on-call.

Lane Status Global#

[QC on FLP]its_pb8

This plot shows the fraction of lanes into ERROR, FAULT, and WARNING statuses. The TOTAL bin gives the total fraction of lanes in any not OK status. The BAD quality will be triggered when the bin value exceeds the 10% threshold.

The following Quality messages can appear:

  • Quality::GOOD
  • Quality::BAD: >10% of the lanes are bad.

In case of BAD quality messages call the ITS on-call.

ITS Misconfiguration plot#

[QC on FLP]its_pb8

This plot shows the estimated readout rate for each FEE component of the ITS. Note that each FEE should have the same estimated frequency, as it is shown in the example figure.

The following Quality messages can appear:

  • Quality::GOOD
  • Quality::BAD: MISCONFIGURATION. CALL EXPERTS.

In case of BAD quality messages call the ITS on-call. Additionally, compare the estimated ITS readout rate with the ITS readout rate from the DCS shifter: these numbers should be the same, otherwise, call the ITS on-call.

Cluster Occupancy overview#

[QC on EPN]its_pb8

Overview of the cluster occupancy, i.e., number of clusters per event, for each stave (1 bin in the plot). The left figure gives example of the GOOD run, while on the right is the problematic distribution. Cluster occupancy should be uniform for comic runs.

  • Quality:: MEDIUM: Layer_Stave has large cluster occupancy
  • Quality:: BAD: Layer_Stave has empty stave

call the ITS on-call in case of anomalies in the plot or if the BAD quality message will appear.

Fraction of empty lanes#

[QC on EPN]its_pb8

This figure provides the fraction of empty lanes (without clusters) per IB/OL/ML/Total. When any of the bins exceeds the 10% limit, you will be notified with the Quality:: BAD: message. Please, call the ITS on-call in this case.

Number of clusters per track#

[QC on EPN]its_pb9

Distribution of the number of clusters per track. The plot shows a GOOD example of a cosmic run. The following messages can appear:

  • Quality::GOOD
  • Quality::Medium Mean is outside 5.2-6.2, ignore for COSMICS and TECHNICALS
  • Quality::BAD: 0 tracks with 4/5/6/7 clusters (OK if it's synthetic run)
  • Quality::BAD: NO ITS TRACKS

call the ITS on-call in case of BAD quality messages, if a completely different plot is obtained, or if the plot stays empty after 5 min of data taking. Ignore MEDIUM quality message for COSMIC data.

Known issues#

ITS KNOWN ISSUES#

  

MCH#

Quality Summary#

[QC on EPN]mch-qualitysummary

The left panel shows a summary of the automated checked on the MCH data, in a human-readable format. The top line describes the aggregated quality status, followed by a message suggesting the appropriate action according to the quality level:

  • Bad: immediately inform the MCH on-call

  • Medium: write a logbook entry, tagging MCH

  • Null: the plots are completely empty. Check in DCS if MCH is in STANBY. If not, inform the MCH on-call.

The right panel shows a trending plot of the aggregated quality. The message in the left panel always corresponds to the most recent point in the trending plot.

If the quality in the trend plot is Bad for the whole duration of a run, MCH should be marked as Bad in the Bookkeeping flags for the run.

Quality Plots#

The following plots show the distribution of various estimators of the MCH data quality. Each horizontal bin shows the value of the monitored quantity, averaged over one Detection Element(DE). The vertical dashed lines show the boundaries between each of the 10 MCH chambers. An horizontal dashed line shows the threshold used by the checker to decide if a given detection element is considered good or bad.

The checker assigns an overall Good (green), Medium (orange) or Bad (red) quality flag to the plot, depending on the number and pattern of bad DEs. In general, the quality is still considered Good if only few DEs are bad. The quality is set to Medium if several DEs are Bad, but no significant impact on the detector acceptance is expected. If the number and pattern of bad DEs is such that the acceptance will be degraded, the quality is set to Bad.

The overall aggregated MCH quality is the logic AND of the qualities of the individual plots.

Fraction of Synchronized Boards#

[QC on EPN]mch-syncfrac

The plot shows, for each Detection Element, the fraction of FEC boards that are properly synchronized. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards not in Error#

[QC on EPN]mch-noerrorfrac

The plot shows, for each Detection Element, the fraction of FEC boards that do not have decoding errors. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards with Good Rate#

[QC on EPN]mch-goodfrac

The plot shows, for each Detection Element, the fraction of FEC boards that have a correct hit rate. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Average Hit Rate#

[QC on EPN]mch-meanrate

The plot shows the average hit rate (in kHz) for each detection element. A given DE is coinsidered bad if the corresponding rate is below the horizontal dashed line.

Average Pseudo-efficiency#

[QC on EPN]mch-meanrate

The plot shows the average pseudo-efficiency for each detection element. The detection efficiency is estimated from the correlation between the pre-clusters reconstructed in either cathode of each DE. A given DE is coinsidered bad if the corresponding efficiency is below the horizontal dashed line.

Known issues#

  • In SYNTHETIC runs, MCH global quality may depend on the replay configuration

    • in p-p replay (500 kHz) : MCH run global quality is expected to be good. If quality is bad or medium, notify the on-call via a logbook entry
    • in PbPb replay : MCH run global quality is expected to be good. If quality is bad or medium, notify the on-call via a logbook entry
    • other replay settings : MCH run global quality may oscillates between good and bad. If quality is bad or medium, notify the on-call via a logbook entry

  • In TECHNICAL runs, MCH global quality depends on the DCS state of MCH

    • if MCH state is READY : MCH global quality should be good. If MCH global quality is bad a notification via a logbook entry is enough.
    • if MCH state is BEAM_TUNING (BEAM_TU) or STANDBY_CONFIGURED (STDB_CO): MCH global quality is expected to be bad. Only the "Decoding errors" is expected to have a good quality; if this is not the case a notification via a logbook entry is enough.

  • In COSMICS runs, the low number of tracks makes it difficult to compute the efficiency for all detection elements in each cycle. Hence oscillations between good and bad status should be expected.

    • if Bad Preclusters quality: Bad Mean Efficiency vs DE(B) and Mean Efficiency vs DE(NB) show multiple bins below threshold value. This is a known issue, no need to notify.
    • if the global quality is bad continuously for more than 1 hour, please make a notification via a logbook entry.

  • FLP Infologger

    • QC plots have been disabled and will generate some errors about not-found plots that can be ignored, for instance :
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/DecodingCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/PreclustersCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/DecodingCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/

  

MFT#

Quality summary#

QualitySummaryGood QualitySummaryBad

The left panel summarises all MFT QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter. * Quality: BAD: contact MFT on-call expert * Quality: Medium: create a log entry * Quality: NULL: the plots are empty.

The bottom line shows an example of BAD status in Chips in error, Digit occupancy and Cluster occupancy plots.

The right panel provides a time trend of the MFT QC summary. In cosmics, it usually takes ~5 minutes after the start of run for the occupancy plots to be files. If the quality stays BAD after ~5 minutes from the start of run, call the MFT on-call.

If MFT quality is BAD for the whole run duration, MFT should be tagged as BAD in the logbook.

Chips in Error/Fault/Warning#

[QC on FLP - plot reset at each cycle] ReadoutChipsError ReadoutChipsFault ReadoutChipsWarning

Plot description

This plot is created on the FLPs. The histogram shows the number of MFT chips in Error/Fault/Warning and provides their list up to 20 entries.

Checks to be done

Check the message on the histogram.

Actions to be taken

Follow the instructions on the histogram message.

Digit Occupancy Summary#

[QC on FLP - plot integrated over the run]DigitOccSummary

Plot description

This plot is created on the FLPs. The histogram shows the number of digits per ROF (Readout Frame) per MFT zone.

Checks to be done

Wait for ~5 minutes after the start of run for the histogram to be filled. Then check the message in the automatic checker for empty ladders.

Actions to be taken

If two adjacent MFT zones are empty after ~5 minutes from the start of run, the quality is bad (message in red), call the MFT on-call immediately.

Cluster Occupancy Summary#

[QC on EPN - plot integrated over the run]ClusterOccSummary

Plot description

This plot is created on the EPNs. The histogram shows the number of clusters per MFT zone.

Checks to be done

Wait for ~5 minutes after the start of run for the histogram to be filled. Then check the message in the automatic checker for empty ladders.

Actions to be taken

If two adjacent MFT ladders are empty after ~5 minutes from the start of run, the quality is bad (message in red), call the MFT on-call immediately.

Distribution of the #clusters per ROF#

[QC on EPN - plot integrated over the run]ClustersROF

Plot description

This plot is created on the EPNs. The histogram shows the ROF size for clusters.

Checks to be done

The histogram should look similar to the good quality reference in this documentation.

Actions to be taken

If the plot looks significantly different from the good quality reference, call the MFT on-call.

Known issues#

  

MID#

Local boards occupancy map (DigitsQC)#

[QC on EPN]mid-locboard

The plot shows fired local Boards.

In case of :

  • empty column call On Call mid-locboard-bad

  • empty or very high rate on several neighboring boards of the detector call On Call mid-locboard-bad2

Hits multiplicity (DigitsQC)#

[QC on EPN]mid-hit-mult

These plots show hits multiplicity by plane for bending and non-bending

  • if mean value > 100.

    call On Call

Known issues#

  • MID has some empty bins in SYNTETHIC runs on "Local boards Occupancy Map" plot. Ignore them till it will be fixed

  

PHS#

Cosmics runs#

Monitor if plots are not empty and timestamps are updated.

Ignore any red messages except "Number of entries has not changed in the past cycle".

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if he is not informed yet. If PHS is not busy but you see this message then inform the on-call.

[QC on EPN] Cell HG occupancy in M1, M2, M3, M4#

Good plots: alt text

The plots represent number of cells seen in each channel. It should be more or less populated. In case when it is very different from reference please inform PHS on-call.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

Minimal duration after SOR before taking any action required by these instructions: 10 min

Known issues#

  • COSMIC, TECHNICAL, PHYSICS_PP runs Cell HG occupancy, mod[1-4]: white horizontal stripes can be seen which differs from the reference plots. QC shifter can ignore these patterns, because they are caused by dynamic FEE mask which is changed by the PHOS experts without prior notice. Only completely empty plots should be reported by the QC shifter to PHOS/CPV on-call
  • COSMIC, TECHNICAL, PHYSICS_PP runs Cell HG occupancy, mod1 shows missing data in the area x=(32,47);z=(0,27) since end of March 2023. This is a known issue and is being investigated by PHS experts. No need to report it in the QC EOS reports and no calls to PHS/CPV shifters are needed.
  • COSMIC, TECHNICAL, PHYSICS_PP runs. Sometimes QC shifters report on the error message in EPN infoLogger: Trailer decoding error: Last RCU trailer word not found. These errors are persistent, they appear in all COSMIC, TECHNICAL, PHYSICS runs since the beginning of Run3, and caused by a bug in SRU firmware. All these errors should be ignored.

  

TOF#

Ignore alarms if TOF is not READY

Readout map (Slot Participating)#

[QC on FLP - plot integrated over the run]

Green panel: good quality Red panel: bad quality
slotpart_good slotpart_bad

The plot shows a map of TOF readout slots per crate. The checker controls if enough crates are in the readout, if it detects lower than expected the quality is set to BAD. In case of red allarm please call TOF oncall.

Hit Multiplicity#

[QC on EPN - plot integrated over the run]

IMPORTANT: please ignore warnings for this histogram untill further notice, see also TOF known issues paragraph.

Green panel: good quality Yellow panel: medium quality
hit_multiplicity_good hit_multiplicity_medium

The plot shows the number of hits detected by TOF, a checker provides instructions for the shifter based on measured counts. ~~In case of yellow allarm please contact TOF on-call via email, in case of red alarm call TOF expert.~~

Known issues#

Slot Participating will not update in REPLAY Runs, this is a known issue do not call the oncall.

  

TPC#

TPC_Physics#

[QC on EPN] rawDig_nClusters To be checked:

  • The time stamp at the bottom should update every two 2 minutes during running.

  • When run is ongoing all Qualities in the list should be GOOD after 3 update cycles (6 minutes).

  • **The quality will be BAD for multiple cycles, up to several hours, since tracks accumulate very slowly.

  • In case the Cluster task quality is bad you can check if there are holes in the N cluser plots and, if yes, call oncall.

Known issues#

Occasionally, the 'Calib' QO may go red; this is an issue with the interplay of the validity of two objects. Please ignore if this behaviour is not persistent (work in progress).

  

TRD#

Layout for cosmic runs#

Note:

  • all QC tasks for TRD are running on the EPNs

Data size per sector#

The TRD has 18 sectors which should all produce a similar data size per TF. There can be two bands forming for each sector as is the case in the example above, because the number of calibration triggers per TF (which are very large in data size) depends on the readout trigger rate and is not fixed. This is OK, please respect the info box on the plot. If it is not green, please write a bookkeeping entry tagging TRD.

Tracklet distribution in half-chambers#

This plot shows the number of tracklets per half-chamber. The x-axis is the sector number. If you see one column completely empty please notify the TRD oncall.

The crosses on the plot are from a static half-chamber status map which needs to be replaced by a dynamic one to correctly cross out half-chambers where no data is expected because of hardware issues.

Eta-phi distribution of ITS-TPC-TRD tracks#

This plot will be empty if either ITS or TPC is missing in the run. This is expected and not a problem.

Eta-phi distribution for ITS-TPC tracks matched to at least 3 TRD tracklets. The PHOS-hole from abs(eta) < 0.2 and phi ~ 5 leads to almost no tracks in that region. No need to call TRD expert if plot does not look as example above. We are currently adding automatic checks.

Pulse height plot based on TRD digit data only#

A peak should be visible between timebin 0 and 6. If that is not the case there might be an issue with the trigger settings. Please notify the TRD oncall in that case.

Number of tracklets per event and TF#

We would like to be notified via bookkeeping entry in case either more than one distinct peaks are appearing in the distributions or in case there are entries in the underflow bin in either of the two histograms.

Raw data statistics#

This plots summarizes statistics on the raw data, such as the number of collected timeframes nTF, the number of triggers nTrig the number of calibration triggers nCalTrig, the number of tracklets nTrkklts and the number of digits nDigits. Furthermore, the readout rate and the calibration trigger rate are determined from the collected data and shown in the plot. Please note that it is normal that the readout rate is lower than the interaction rate due to TRD dead times. The oncall shall only be called in case the plot is not filled at all.

Known issues#

QC plots for COSMICS are not consistent with the documentation.
- For the "Data sizes from HalfCRU header" plot shows deviations due to noisy sectors (values around 2500). This issue can be ignored till documentation is up to date.
- For the "Number of Tracklets per timeframe" plot has a double peak structure in COSMICS and a lot of entries in the underflow bin (around 90% of tracklets). The experts have been notified. The relevant log can be found here: https://ali-bookkeeping.cern.ch/?page=log-detail&id=80122. The issue may be fixed in future or the documentation updated.