Skip to content

CPV#

Physics HI runs#

Monitor if plots are not empty and timestamps are updated.

If plots are empty or not updated please inform the on-call.

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if expert is not informed yet. If PHS is not busy but you see this message then inform the on-call.

If you see other red messages please check known issues. If issue is not known please inform the on-call.

[QC on QC merger node] Global CPV quality#

Please check known issues

Good plot:

alt text

The plot summarize CPV quality. If global quality is not good then please check below the details.

  • Number of digits increases assures that digits are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle".

  • Number of clusters increases assures that clusters are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle".

  • Digit occupancy check assures that digit occupancy is good. If bad then check messages on Digit Map in M2,3,4 plots and follow the instructions.

  • Cluster size check assures that mean cluster size is within allowed limits. If bad please inform the on-call. If medium - put a log entry and inform the oncall during morning and afternoon shift. No need to call during night.

  • CalibDigit amplitude check assures that observed amplitude spectra are good. If bad please inform the oncall. Medium means lack of statistics: please check later.

  • Errors presence check that occured number of errors is within limits. If not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues.

[QC on QC merger node] Global quality trend#

Good plot:

alt text

The trend shows evolution of global CPV quality. It can become not good for small portion of cycles due to fluctuations. In this case run to be considered as good.

If the trend is not good for significant part of the run then run quality must be setted to bad. Put bad flag even if issue is known. Expert will have a look and adjust the qaulity later if needed.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

[QC on EPN] Digit Map in M2, 3, 4#

Please check known issues

Good plots: alt text The plots represent number of digits seen in each channel. It should be more or less uniform.

If red messages tell Hot 3G Cards (N1/N2) or Cold 3G Cards (N1/N2) then put log entry and ask the oncall to check the plots.

Bad example: HV Trip in module M2

Bad example: Number of entries has not changed in the past cycle

Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz

Actions to be taken by QC shifters#

  • Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz.
  • Inform on-call when quality is bad and it is not a known issue.

Known issues#

  • In PEDESTAL runs QC plot "Pedestal sigma distribution M3 is bad". Acording to expert, noise conditions in CPV depend on general environment in ALICE, therefore, from time to time CPV pedestals can become wider which is reflected in QC-CPV calibration run plot with the message "Number of bad pedestal sigmas in module M3 (sometimes in M2) is larger than the upper limit". This is a known issue, can be ignored, but make a log entry when it happens. See: https://ali-bookkeeping.cern.ch/?page=log-detail&id=45015

  • In PHYSICS runs Red messages on QC Plot "Digit Map in M4". Problem with high voltage is preventing this module from running normally. Experts are trying to recover. No need to report. Run quality must be setted to "good" if there are no problems with other modules. See: https://ali-bookkeeping.cern.ch/?page=log-detail&id=53523. Quality aggregator is also expected to show bad quality for Digit oocupancy check and CalibDigit amplitude check.

  

EMC#

[QC on FLP - plot integrated over the run] Raw Data Error#

Green: good quality Red: bad quality
ErrorTypeOK ErrorTypeError
  • If no entries, a green message "No Error: OK" inform you that everything is working properly.
  • In case of errors, a red message will appear: call EMCAL oncall if EMCAL is included in global runs. Take note of the error-type in the y axis.

[QC on FLP - plot integrated over the run] Payload Size/events#

good quality bad quality empty
PayloadSizeEvOK PLEvError PLEvempty
  • If "Data OK" is shown, everything is fine.
  • If some of the DDL presents entries that are larger than the others, a red message will appears: please call EMCAL oncall.
  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

[QC on FLP - plot integrated over the run] Bunch minimum amplitude EMCAL+DCAL#

good quality bad quality empty
BunchOK BunchError BunchEmpty
  • One peak should be visibile if EMCAL is in data taking.
  • If a second peak appears around "Min raw amplitude (ADC)" in the range 20-50, call EMCAL oncall
  • If the plot is empty, and EMCAL is included in the data taking: please call EMCAL oncall

[QC on EPN - plot integrated over the run] Cell Occupancy plots (PHYS) for E>0.2 GeV and E<0.2 GeV and Cell Occupancy plots (CALIB) for E>0.5 GeV#

good quality (high-E) good quality (high_E) bad quaity
OccEabove OccEbelow OccErr

IF PHYSICS RUN - The occupancy plots should be uniformly filled during Physics runs.
- if something noisy appears, please call the EMCAL oncall. - If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

Occupancy plots: Acceptance losses#

Missing acceptance (low-E) Missing acceptance (high-E)
MissingAccLowE MissingAccHighE

In case counts in a certain detector segment stop increasing the corresponding segment stops sending data. The corresponding area must be visible in both plots. - call EMCAL on-call

[QC on EPN - plot integrated over the run] Cell Amplitude#

DigitAmplitude

  • no instructions for the moment.
  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL oncall.

[QC on EPN - plot integrated over the run] Cell Time#

good quality bad quality
DigitTimeOK DigitTime
  • We expect a gaussian peak roughly centred at 0 and a structure of single lines with a spacing of 100 ns from noisy channels. Depending on the filling scheme smaller gaussian like peaks from from pileup could appear. In such cases the data is OK.
  • In case a second peak appears which is either of the same magnitude as the main peak or of magnitude 1/3 as the main peak and the peaks are separated by 100 ns call the EMCAL on-call
  • If the main peak deviates from 0 by more than 100 ns call the EMCAL on-call
  • If no gaussian like peak is present call the EMCAL on-call
  • If the plot is empty, and EMCAL is included in the data taking, call EMCAL on-call.

Number of triggers#

good quality bad quality
DigitTimeOK DigitTime
The number of physics triggers must increase while the run is ongoing. In case the number of trigger stays constant no data is received any more from the detector (most likely detector busy) - call the EMCAL on-call.

Known issues#

  • Do not call EMC oncall during the night for QC plots related errors for non PHYSICS runs, unless a PHYSICS run is expected next.

  • For the plot "Number of Physics triggers per timeframe" the expected value is currently approx 20. If value is below 15 for at least 2 consecutive timestamps, call on-call.

  • Payload Size/Event can be ignored in all runs till further notice

  • In plot "Raw data errors" (error rate) quality is bad if the rate is above 100 errors / minute for several minutes, and good if it is below.

  • An empty region is expected near the bottom-right corner for CAL (Calibration) triggers. Cell Occupancy EMCAL,DCAL with E>0.5 GeV/c CAL

  

FDD#

FDD: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 5 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

FDD: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the Vertex trigger.

Actions:

  • If WARNING - compare to FT0 and FV0 BC-trigger plots; if they also show WARNING - call FIT-on-call.
  • If ERROR - call FIT-on-call.

FDD: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FDD: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FDD: Fraction of charge in ADC range#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fdd_cfd_gate

Fraction of charge in ADC range in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FDD: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_cfd_gate

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

Known issues#

  • The Out of bunch collisions plot can be ignored in SYNTHETIC runs at the moment (i.e. no need to take any actions in case of not GOOD quality). The BC distribution is not simulated properly in MC and will always cause problems in this plot.

  

FT0#

FT0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

FT0: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

ft0_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the Vertex trigger.

Actions:

  • If ERROR - call FIT-on-call.
  • If WARNING - compare to FV0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.

FT0: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

ft0_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FT0: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FT0: Fraction of channels out of colliding BCs#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

ft0_channel_out_of_bunch

Fraction of channles fired out of colliding BCs. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FT0: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle] ft0_trg_validation

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

Known issues#

  • channels 60-63 (FT0A side) are OFF in PHYSICS and COSMICS runs
  • channels 139 and 176-179 (FT0C side) are OFF in PHYSICS and COSMICS runs

  

FV0#

FV0: General#

All QC plots are generated in 5 min. cycles. The histogram contents are reseted after each cycle. Please wait at least 10 min. from the start of the RUN to judge about the quality of collected data.

If a non-critical (i.e. expendable) QC task fails, please add a bookkeeping entry about it.

FV0: Out of bunch collisions#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_bc_triggers

BC vs trigger correlation for the events which were detected but are not aligned (out-of-bunch) with LHC filling scheme. The number of out-of-bunch events depends on the trigger settings. The check is performed for the TrgNchan trigger.

Actions:

  • If ERROR - call FIT-on-call.
  • If WARNING - compare to FT0 and FDD BC-trigger plots; if they also show WARNING - call FIT-on-call.

FV0: Fraction of events with CFD in ADC gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle] fv0_cfd_efficiency

Fraction of events with CFD in ADC gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FV0: Fraction of events with the CFD in time gate#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_cfd_gate

Fraction of events with CFD in time gate in each detector channel. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

FV0: Validation of hardware (HW) triggers in software (SW)#

[QC on EPN/QC nodes - plot integrated over the QC cycle]

fv0_cfd_gate

Fraction of only software or hardware (SW + HW) triggers. In ideal case both the FW and SW triggers or nighther of them should be present in a given event. Horizontal lines show the levels below which the warnings/errors are raised.

Actions:

  • If WARNING - no action.
  • If ERROR - call FIT-on-call.

Known issues#

  

HMP#

Busy time#

[QC on FLP]busy_time

The plot shows the busy time for each detector DDL. In case more than three equipments exceeds 120 microsec or the plot is empty call HMP on-call

Event size#

[QC on FLP]event_size

The plot shows the event size for each detector DDL. In case more than three equipments exceeds 13 kB or the plot is empty call HMP on-call

Sum Q maps#

[QC on FLP]BigMapProfile

The plot shows the charge of all the detector channels in only one 2D map. It looks like the example plot shown above. In case it is completely empty call HMP on-call.

Charge vs HV sector#

[QC on FLP]HVsector

The plot shows the charge of all the detector HV sectors in only one 2D map. It looks like the example plot shown above. The white bands correspond to the faulty HV sectors that are off. In case more than two bands (w.r.t those shown here) became white call the an-call.

Occupancy#

[QC on FLP]Occupancy

The plot shows the occupancy for each detector DDL. In case more than three equipments exceeds 3% or the plot is empty call HMP on-call

Known issues#

For the moment links 3 and 12, exlcuded from data taking

  

ITS#

General considerations#

If any of the plots listed on this page remains empty during a run, please call the ITS on-call. The only exception, i.e., where empty means good quality, is for the four plots on the lane status.

Quality summary#

its_summary its_summary The left panel summarises all ITS QC checks for the last QC cycle. The top line gives aggregated quality status with the text message suggesting actions for the QC shifter.

  • Quality: BAD: contact ITS on-call expert
  • Quality: Medium: create a log entry
  • Quality: NULL: the plots are empty. Check in DCS if ITS is in STANBY. If not, inform the ITS on-call.

In BAD or MEDIUM cases, this canvas will duplicate the error message from the QC plot in the form "Flag: Unknown: ERROR MESSAGE". The bottom line shows an example of BAD status in Track Angular distribution and MEDIUM in nClusterPerTrack plots.

The right panel provides a time trend of the ITS QC summary. If ITS quality is BAD for the whole run duration, ITS should be tagged as BAD in the logbook.

Error count vs Error id#

[QC on FLP]its_pb1

The total number of decoding errors (the error ID is on the x-axis). The left figure provides an example of a good case, while the right corresponds to run with BAD quality;

The BAD quality flag will be triggered when a number of decoding errors exceed some limit; MEDIUM quality corresponds to the case when the number of errors is non-zero, but not significant for the BAD run. The following quality messages can be printed on the plot:

  • Quality::GOOD
  • Quality:: BAD: "definition of error ID"
  • Quality:: MEDIUM: "definition of error ID"

If the number of error with a certain ID show a BAD quality message, call the ITS on-call. In case of the MEDIUM quality, create a log entry linked to run and with tag = ITS. Please note that entries in the last bin are not an issue, because this bin is not related to any detector problems.

Fake-hit rate overview#

[QC on FLP]its_pb2_new Overview of the averaged fake-hit rate value for each ITS stave (given by triangles). The fake-hit rate value is displayed with the colored scale in hits/events/pixels. The left plot provides an example of a GOOD plot, and the right figure is an example of a BAD quality plot.

The following quality messages can be printed on the plot:

  • Quality::GOOD
  • Quality:: Medium: Max occupancy over 10^{-6}
  • Quality:: BAD: Max occupancy over 10^{-5}
  • Quality:: BAD: There are staves without hits

In case of BAD quality messages or empty triangles (some might stay empty for the first 25-30 mins of a run, do not call in this case), please call the ITS on-call.

Trigger count vs TriggerID and FeeID#

[QC on FLP]its_pb4 Plot summarizing trigger flags. The X-axis corresponds to the ID of Front-End Electronic; the Y-axis shows the list of all possible triggers that can be received by FEE. The coloured scale represents the counts in each bin. During physical run, we expect that all FEEs receive HB, ORBIT, SOC and TF trigger signals. The GOOD run is shown on the left panel, while the problematic one is depicted on the right. QC can plot the following error messages:

  • Quality::GOOD
  • Quality:: BAD: Trigger flag (TRIGGER_NAME) of bad quality

Call the ITS on-call in the case of BAD quality messages and also in the following cases:

  • there are x-bins without entries (i.e., vertical white lines on the plot)
  • entries are present in the empty lines in the example plot above.
  • the plot is completely empty

Lane Status Flag: ERROR/FAULT/WARNING#

[QC on FLP]its_pb5

This plot indicates the faction of lanes (colored axis) in ERROR/FAULT/WARNING status for each ITS stave. The empty plot corresponds to the GOOD run quality (left plot); in other cases (right plot), the following quality messages can appear on the plot:

  • Quality:: MEDIUM: ML/OL have staves in ERROR
  • Quality:: MEDIUM: Inner Barrel has stave with >2 chips in ERROR/FAULT/WARNING
  • Quality:: BAD: Layer 0/1/2/3/4/5/6 has >25% staves with lanes/chips in ERROR/FAULT/WARNING

If the BAD quality message is printed, call the ITS on-call.

Lane Status Global#

[QC on FLP]its_pb8

This plot shows the fraction of lanes into ERROR, FAULT, and WARNING statuses. The TOTAL bin gives the total fraction of lanes in any not OK status. The BAD quality will be triggered when the bin value exceeds the 10% threshold.

The following Quality messages can appear:

  • Quality::GOOD
  • Quality::BAD: >10% of the lanes are bad.

In case of BAD quality messages call the ITS on-call.

ITS Misconfiguration plot#

[QC on FLP]its_pb8

This plot shows the estimated readout rate for each FEE component of the ITS. Note that each FEE should have the same estimated frequency, as it is shown in the example figure.

The following Quality messages can appear:

  • Quality::GOOD
  • Quality::BAD: MISCONFIGURATION. CALL EXPERTS.

In case of BAD quality messages call the ITS on-call. Additionally, compare the estimated ITS readout rate with the ITS readout rate from the DCS shifter: these numbers should be the same, otherwise, call the ITS on-call.

Cluster Occupancy overview#

[QC on EPN]its_pb8

Overview of the cluster occupancy, i.e., number of clusters per event, for each stave (1 bin in the plot). The left figure gives example of the GOOD run, while on the right is the problematic distribution. Check the general trend, i.e. occupancy decreasing when going from the innermost to the outermost layers. This MO can have the following quality messages:

  • Quality:: MEDIUM: Layer_Stave has large cluster occupancy
  • Quality:: BAD: Layer_Stave has empty stave

Call the ITS on-call in case of anomalies in the plot or if the BAD quality message will appear.

Fraction of empty lanes#

[QC on EPN]its_pb8

This figure provides the fraction of empty lanes (without clusters) per IB/OL/ML/Total. When any of the bins exceeds the 10% limit, you will be notified with the Quality:: BAD: message. Please, call the ITS on-call in this case.

Angular Distribution#

[QC on EPN]its_pb8 its_pbtemp

Angular distribution of online reconstructed ITS tracks as a function of phi vs. eta (2D plot).
Possible Quality messages that can appear on the plot during a run:

  • Quality::GOOD: plot might still be bad! See example above and look at the arrows in the plot.
  • BAD: Asymmetric Phi distribution (OK if there are disabled ITS sectors)
  • BAD: Asymmetric Eta distribution (OK if there are disabled ITS sectors)
  • BAD: NO ITS TRACKS

Please try to reconfigure the detector (set run type again) in case the quality is BAD or if the plot shows several holes (blue regions in between the yellow parts as shown in the plot above), call ITS on-call is the issue persists. Please call the ITS on-call also if the plot remains empty during the run.

Number of clusters per track#

[QC on EPN]its_pb9

Distribution of the number of clusters per track. The plot shows a GOOD example of pp collisions run. The following messages can appear:

  • Quality::GOOD
  • Quality::Medium Mean is outside 5.2-6.2, ignore for COSMICS and TECHNICALS
  • Quality::BAD: 0 tracks with 4/5/6/7 clusters (OK if it's synthetic run)
  • Quality::BAD: NO ITS TRACKS

Call the ITS on-call in case of BAD quality messages, if a completely different plot is obtained, or if the plot stays empty after 5 min of data taking. In case of MEDIUM status, create log-book entry.

Known issues#

ITS KNOWN ISSUES#

  

MCH#

Quality Summary#

[QC on EPN]mch-qualitysummary

The left panel shows a summary of the automated checked on the MCH data, in a human-readable format. The top line describes the aggregated quality status, followed by a message suggesting the appropriate action according to the quality level:

  • Bad: immediately inform the MCH on-call

  • Medium: write a logbook entry, tagging MCH

  • Null: the plots are completely empty. Check in DCS if MCH is in STANBY. If not, inform the MCH on-call.

The right panel shows a trending plot of the aggregated quality. The message in the left panel always corresponds to the most recent point in the trending plot.

If the quality in the trend plot is Bad for the whole duration of a run, MCH should be marked as Bad in the Bookkeeping flags for the run.

Quality Plots#

The following plots show the distribution of various estimators of the MCH data quality. Each horizontal bin shows the value of the monitored quantity, averaged over one Detection Element(DE). The vertical dashed lines show the boundaries between each of the 10 MCH chambers. An horizontal dashed line shows the threshold used by the checker to decide if a given detection element is considered good or bad.

The checker assigns an overall Good (green), Medium (orange) or Bad (red) quality flag to the plot, depending on the number and pattern of bad DEs. In general, the quality is still considered Good if only few DEs are bad. The quality is set to Medium if several DEs are Bad, but no significant impact on the detector acceptance is expected. If the number and pattern of bad DEs is such that the acceptance will be degraded, the quality is set to Bad.

The overall aggregated MCH quality is the logic AND of the qualities of the individual plots.

Fraction of Synchronized Boards#

[QC on EPN]mch-syncfrac [QC on EPN]mch-syncfracok

The plot shows, for each Detection Element, the fraction of FEC boards that are properly synchronized. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards not in Error#

[QC on EPN]mch-noerrorfrac [QC on EPN]mch-noerrorfracok

The plot shows, for each Detection Element, the fraction of FEC boards that do not have decoding errors. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Fraction of Boards with Good Rate#

[QC on EPN]mch-goodfrac [QC on EPN]mch-goodfracok

The plot shows, for each Detection Element, the fraction of FEC boards that have a correct hit rate. A given DE is coinsidered bad if the corresponding fraction is below the horizontal dashed line.

Average Hit Rate#

[QC on EPN]mch-meanrate [QC on EPN]mch-meanrateok

The plot shows the average hit rate (in kHz) for each detection element. A given DE is coinsidered bad if the corresponding rate is below the horizontal dashed line.

Average Pseudo-efficiency#

[QC on EPN]mch-effnotok [QC on EPN]mch-effok

The plot shows the average pseudo-efficiency for each detection element. The detection efficiency is estimated from the correlation between the pre-clusters reconstructed in either cathode of each DE. A given DE is coinsidered bad if the corresponding efficiency is below the horizontal dashed line.

Known issues#

  • In SYNTHETIC runs, MCH global quality may depend on the replay configuration

    • in p-p replay (500 kHz) : MCH run global quality is expected to be good. If quality is bad or medium, notify the on-call via a logbook entry
    • in PbPb replay : MCH run global quality is expected to be good. If quality is bad or medium, notify the on-call via a logbook entry
    • other replay settings : MCH run global quality may oscillates between good and bad. If quality is bad or medium, notify the on-call via a logbook entry

  • In TECHNICAL runs, MCH global quality depends on the DCS state of MCH

    • if MCH state is READY : MCH global quality should be good. If MCH global quality is bad a notification via a logbook entry is enough.
    • if MCH state is BEAM_TUNING (BEAM_TU) or STANDBY_CONFIGURED (STDB_CO): MCH global quality is expected to be bad. Only the "Decoding errors" is expected to have a good quality; if this is not the case a notification via a logbook entry is enough.

  • In COSMICS runs, the low number of tracks makes it difficult to compute the efficiency for all detection elements in each cycle. Hence oscillations between good and bad status should be expected.

    • if Bad Preclusters quality: Bad Mean Efficiency vs DE(B) and Mean Efficiency vs DE(NB) show multiple bins below threshold value. This is a known issue, no need to notify.
    • if the global quality is bad continuously for more than 1 hour, please make a notification via a logbook entry.

  • FLP Infologger

    • QC plots have been disabled and will generate some errors about not-found plots that can be ignored, for instance :
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/DecodingCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/PreclustersCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/
      • Requested resource does not exist: ali-qcdb.cern.ch:8083/qc/MCH/QO/DecodingCheck/1709911754085/PeriodName=LHC24aa/RunNumber=548050/

  

MFT#

Expendable MFT tasks#

The MFT currently has 4 post-processing tasks that are marked as non-critical (= they are allowed to crash while a run is ongoing):

  • MFTReadoutTrend
  • MFTOccupancyTrend
  • MFTTrendSlices
  • RefComp

If such tasks crash during a PHYSICS run, please immediately call the MFT on-call and create a log entry tagging MFT. If a crash occurs in COSMICS/SYNTHETIC/NOISE/..., a log entry is sufficient (no need to call).

Quality summary#

The left panel summarizes all MFT QC checks for the last QC cycle. The top row provides a summary of the quality status with a text message suggesting actions to be taken:

  • Quality: Bad - call the MFT on-call immediately
  • Quality: Medium - create a log entry tagging MFT
  • Quality: NULL - QC objects were not created

The right panel provides a time trend of the MFT quality summary.

Good quality example:

QualitySummaryGood

Bad quality example (triggered by the real-time cluster occupancy):

QualitySummaryBad

Chips in Error/Fault/Warning#

ReadoutChipsError ReadoutChipsFault ReadoutChipsWarning

  • Description: this plot is created on the FLPs and shows the number of MFT chips in Error/Fault/Warning. Up to the first 20 chips are explicitly listed.
  • Checks to be done: there is an automatic checker on the number of chips in E/F/W.
  • Actions to be taken: follow the instructions in the plot. Since the MFT has automatic chip recovery that is triggered if a certain amount of chips in E/F is reached, call the on-call if the quality remains Bad for more than 2 minutes.

Digit Occupancy Summary#

DigitOccSummary

  • Description: this plot is created on the FLPs and shows the number of digits per MFT zone per LHC orbit. It contains all data since SOR.
  • Checks to be done: there is an automatic checker for empty ladders (each MFT zone is composed of multiple ladders).
  • Actions to be taken: follow the instructions in the plot:
    • The quality turns Medium if some individual ladders are empty: create a log entry tagging MFT.
    • If at least two adjacent ladders are empty, the quality turns Bad: call the MFT on-call immediately.

Cluster Occupancy Summary#

ClusterOccSummary

  • Description: this plot is created on the EPNs and shows the number of clusters per MFT zone per LHC orbit. It contains all data since SOR.
  • Checks to be done: there is an automatic checker for empty ladders (each MFT zone is composed of multiple ladders).
  • Actions to be taken: follow the instructions in the plot:
    • The quality turns Medium if some individual ladders are empty: create a log entry tagging MFT.
    • If at least two adjacent ladders are empty, the quality turns Bad: call the MFT on-call immediately.

Real-time Cluster Occupancy Summary (last ~2 mins window)#

  • Description: this is the same plot as Cluster Occupancy Summary, but it only contains data from the last time window (duration approximately 2 mins).
  • Checks to be done: there is also an automatic checker for empty ladders with the same settings. The output of this checker corresponds to the "Real-time cluster occupancy" shown in the "Quality Summary" at the top.
  • Actions to be taken: follow the usual instructions given in the plot.

Track phi distribution, track eta distribution#

[Sample histograms need to be added]

  • Description: these plots are created on the EPNs and show the track phi and eta distributions (normalized by the number of LHC orbits).
  • Checks to be done: the histograms should look similar to the references shown in this documentation.
  • Actions to be taken: if the plots look significantly different from the references, call the MFT on-call.

Tracks X-Y distribution#

trackxy_pp

  • Description: this plot is created on the EPNs and shows the track position in the X-Y plane (normalized by the number of LHC orbits).
  • Checks to be done: the histogram should look similar to the reference (left) shown in this documentation.
  • Actions to be taken: if the plot looks significantly different from the reference, call the MFT on-call.

Distribution of the #clusters per ROF#

[Sample histograms need to be added]

  • Description: this plot is created on the EPNs and shows the number of clusters per MFT ROF (readout frame). The data are normalized by the number of LHC orbits.
  • Checks to be done: the histogram should look similar to the reference (left) shown in this documentation.
  • Actions to be taken: if the plot looks significantly different from the reference, call the MFT on-call.

Distribution of the #tracks per ROF#

[Sample histograms need to be added]

  • Description: this plot is created on the EPNs and shows the number of tracks per MFT ROF (readout frame). The data are normalized by the number of LHC orbits.
  • Checks to be done: the histogram should look similar to the reference (left) shown in this documentation.
  • Actions to be taken: if the plot looks significantly different from the good reference, call the MFT on-call.

Known issues#

  

MID#

Local boards occupancy map (DigitsQC)#

[QC on EPN]mid-locboard

The plot shows fired local Boards. In case of :

  • empty column.

  • empty or very high rate on several neighboring boards of the detector.

  • very low rate on several neighboring boards of the detector.

    call expert

Hits multiplicity (DigitsQC)#

[QC on EPN]mid-hit-mult

These plots show hits multiplicity by plane for bending and non-bending

  • if mean value > 100.

    call expert

Known issues#

  • MID has some empty bins in SYNTETHIC runs on "Local boards Occupancy Map" plot. Ignore them till it will be fixed

  

PHS#

Physics PP runs#

Monitor if plots are not empty and timestamps are updated.

If plots are empty or not updated please inform the on-call.

If you see "Number of entries has not changed in the past cycle" but run is still ongoing then most probable reason is that PHS bacome busy. You can check with ECS shifter and inform the on-call if expert is not informed yet. If PHS is not busy but you see this message then inform the on-call.

If you see other red messages please check known issues. If issue is not known please inform the on-call.

[QC on QC merger node] Global PHS quality#

Good plot:

alt text

The plot summarize PHS quality. If global quality is not good then please check below the details.

  • Number of cells increases assures that digits are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle" above.

  • Number of clusters increases assures that clusters are being produced. If bad then follow the procedure above If you see "Number of entries has not changed in the past cycle" above.

  • Cells check assures that cells occupancy is good. If not good then check Cells HG occupancy plots and follow the instructions.

  • Clusters check assures that mean cluster energy size is within allowed limits. If not good then please inform the on-call. If medium - put a log entry and inform the oncall during morning and afternoon shift. No need to call during night.

  • Errors check assures that occured number of errors is within limits. If not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues.

[QC on QC merger node] Global quality trend#

Good plot:

alt text

The trend shows evolution of global PHS quality. It can become medium for small portion of cycles due to failure of fit procedure. In this case run to be considered as good.

If the trend is not good for significant part of the run then run quality must be setted to bad. Put bad flag even if issue is known. Expert will have a look and adjust the qaulity later if needed.

[QC on EPN] Error occurance#

Good plot:

alt text

If plot is not good then please inform the oncall. If medium then put a log entry and inform the oncall during morning and afternoon shift. No need to call at night if there are no other issues. No need to put bad flag for run if there are no other issues.

[QC on EPN] Cell HG occupancy in M1, 2, 3, 4#

Please check known issues

Good plots: alt text The plots represent number of cells seen in each channel. It should be more or less uniform.

If red messages say Not OK then put log entry and inform the oncall. Note that messages can be false positive. If you think the plots are not different too much from reference then just put log entry, no need to call during night. If they are differ too much (big empty/hot regions, etc.) then call the on-call. Please do not hesitate to call the on-call if you have any doubts: it's better to wake up us at night than loose data!

Actions to be taken by QC shifters#

  • Minimal duration after SOR before taking any action required by these instructions: 5 min at 500 kHz.

  • Follow the instructions above.

Known issues#

  • COSMIC, TECHNICAL, PHYSICS_PP runs Cell HG occupancy, mod[1-4]: white horizontal stripes can be seen which differs from the reference plots. QC shifter can ignore these patterns, because they are caused by dynamic FEE mask which is changed by the PHOS experts without prior notice. Only completely empty plots should be reported by the QC shifter to PHOS/CPV on-call
  • COSMIC, TECHNICAL, PHYSICS_PP runs Cell HG occupancy, mod1 shows missing data in the area x=(32,47);z=(0,27) since end of March 2023. This is a known issue and is being investigated by PHS experts. No need to report it in the QC EOS reports and no calls to PHS/CPV shifters are needed.
  • COSMIC, TECHNICAL, PHYSICS_PP runs. Sometimes QC shifters report on the error message in EPN infoLogger: Trailer decoding error: Last RCU trailer word not found. These errors are persistent, they appear in all COSMIC, TECHNICAL, PHYSICS runs since the beginning of Run3, and caused by a bug in SRU firmware. All these errors should be ignored.

  

TOF#

Ignore alarms if TOF is not READY

Readout map (Slot Participating)#

[QC on FLP - plot integrated over the run]

Green panel: good quality Red panel: bad quality
slotpart_good slotpart_bad

The plot shows a map of TOF readout slots per crate. The checker controls if enough crates are in the readout, if it detects lower than expected the quality is set to BAD. In case of red allarm please call TOF oncall.

Hit Multiplicity#

[QC on EPN - plot integrated over the run]

hitmultiplicity_good

The plot shows the number of hits detected by TOF, a checker provides instructions for the shifter based on measured counts. In case of yellow allarm please contact TOF on-call via email, in case of red alarm call TOF expert.

Known issues#

Slot Participating will not update in REPLAY Runs, this is a known issue do not call the oncall.

  

TPC#

TPC_Physics#

[QC on EPN]

To be checked - General:

  • The time stamp at the bottom should update every two 1 minutes during running.

rawDig_nClusters

  • When run is ongoing all Qualities in the list should be GOOD after 5 minutes. In case of a bad quality call OnCall.

  • If in the quality trending, the quality is constantly BAD for multiple cycles (>10 minutes) call On-Call.

  • In case the Raw task / Cluster task quality is bad you can check if there are holes in the N cluser plots and call On-Call.

cluster_per_track - In the plot "Cluster per Track" a prominent peak should be visibile arround 150. If structure differs significantly from reference call OnCall

eta_pos eta_neg eta_ratio - if Eta for positive or negative particle or eta ratio plot differs significantly from reference inform OnCall

cluster_per_track_phi - If Quality of Mean Cluster per Track per sector distribution shows bad quality call OnCall; if quality is Medium put log entry.

Mean_PID_versus_Sector - If "Mean dEdxTot of MIPs tracks per sector" show a bad quality call OnCall; if quality is Medium put log entry.

Known issues#

Occasionally, the 'Calib' QO may go red; this is an issue with the interplay of the validity of two objects. Please ignore if this behaviour is not persistent (work in progress).

  

TRD#

Layout for physics runs in Pb-Pb#

Note:

  • all QC tasks for TRD are running on the EPNs
  • the layouts for TRD are currently undergoing major changes
    • automatic checks will be added in the future

Data size per sector#

The TRD has 18 sectors which should all produce a similar data size per TF. In case on sector is deviating a lot (its mean is more/less than factor 3 from the rest) please write a bookkeeping entry tagging TRD.

Tracklet distribution in half-chambers#

This plot shows the number of tracklets per half-chamber. The x-axis is the sector number. If you see one column completely empty please notify the TRD oncall.

The crosses on the plot are from a static half-chamber status map which needs to be replaced by a dynamic one to correctly cross out half-chambers where no data is expected because of hardware issues.

Eta-phi distribution of ITS-TPC-TRD tracks#

This plot is currently missing its title in the QCG, needs to be fixed in QC code

Eta-phi distribution for ITS-TPC tracks matched to at least 3 TRD tracklets. The PHOS-hole from abs(eta) < 0.2 and phi ~ 5 leads to almost no tracks in that region. No need to call TRD expert if plot does not look as example above. We are currently adding automatic checks.

Pulse height plot based on ITS-TPC-TRD matched tracks#

Ignore in case eta-phi distribution plot is empty or very sparsly filled.

A clear peak should be visible between time bins 0-4, followed by a plateau and a falling edge around 20. If plot does not look approximately as in the example above (and the track eta-phi plot is filled) please create a bookkeeping entry tagging TRD.

Pulse height plot based on TRD-only data#

In case the pulse height plot based on matched tracks is filled you can ignore this plot.

A peak should be visible between time bins 0-5. Call the TRD oncall if that is not the case, as there might be an issue with the trigger settings. The plateau region at larger time bins might not look as smooth as in the above picture in case there is a lot of pile-up.

Number of tracklets per event and TF#

We would like to be notified via bookkeeping entry in case either more than one distinct peaks are appearing in the distributions or in case there are entries in the underflow bin in either of the two histograms.

Raw data statistics#

This plots summarizes statistics on the raw data, such as the number of collected timeframes nTF, the number of triggers nTrig the number of calibration triggers nCalTrig, the number of tracklets nTrkklts and the number of digits nDigits. Furthermore, the readout rate and the calibration trigger rate are determined from the collected data and shown in the plot. Please note that it is normal that the readout rate is lower than the interaction rate due to TRD dead times. The oncall shall only be called in case the plot is not filled at all.

Known issues#

QC plots for COSMICS are not consistent with the documentation.
- For the "Data sizes from HalfCRU header" plot shows deviations due to noisy sectors (values around 2500). This issue can be ignored till documentation is up to date.
- For the "Number of Tracklets per timeframe" plot has a double peak structure in COSMICS and a lot of entries in the underflow bin (around 90% of tracklets). The experts have been notified. The relevant log can be found here: https://ali-bookkeeping.cern.ch/?page=log-detail&id=80122. The issue may be fixed in future or the documentation updated.

  

ZDC#

Instructions for shifter in Pb-Pb

Please remember to always double check the current run number. The plots will update every minute.

Baseline [QC on FLP]#

Green: good quality Orange: Warning quality Red: bad quality
baseline_good baseline_warning baseline_bad

The plot shows the baseline mean values of each ZDC channel. The quality of the histogram is determined by verifying the deviation from the expected average value.

  • If the plot is empty or the text is red --> call the on-call.
  • if the plot text is orange --> insert a log entry

Align Plot [QC on FLP]#

Green: good quality Red: bad quality
Align_good Align_bad

The plot shows if all channels are aligned. Most of the channels should be centered on sample 6. If any channel deviates by plus or minus one, it is considered good. - If the plot shows any errors. --> Call ZDC on-call - if the plot contains empty channels --> Call ZDC on-call

Raw Data Error [QC on FLP]#

Green: good quality Red: bad quality
error_good error_good

The Raw Data Error plot must be empty. The plot indicates the presence of the following errors: data corrupted, data loss, generic error. The plot resets every five minutes, so if it was a temporary error then it will disappear. - If the plot shows any errors. --> Call ZDC on-call

Summary ADC [QC on EPN]#

Green: good quality Red: bad quality
ADC_good ADC_bad

The plot shows the baseline mean values of each ZDC ADC channel. The quality of the histogram is determined by verifying the deviation from the expected average value. - If the plot is empty or the text is red --> call the on-call. - if the plot text is orange --> insert a log entry

Summary TDC [QC on EPN]#

Green: good quality Red: bad quality
TDC_good TDC_bad

The plot shows the baseline mean values of each ZDC TDC channel. The quality of the histogram is determined by verifying the deviation from the expected average value.

Known issues#

FLP InfoLoger: message: "Baseline Error in PED_Z%" message: "Rec Error in ADC_Z%" Detector ZDC. During SYNTHETIC runs this can be ignored.

  

VTX#

Vertex distributions from matched central barrel tracks#

[QC on EPN]async_QC

The left plot shows the transversal x-y distribution of the reconstructed vertices, while the right plot shows the longitudinal vertex distribution.

The mean value of the z vertes distribution usually fluctuates by +/- 0.3 cm around the central value. The mean values in the x-y directions are usually very stable.

The RM/RC must be informed immediately if:

  • the mean x and y values are outside [-0.1 cm, +0.1 cm]
  • the mean z value is outside [-1.0 cm, +1.0 cm]
  • the standard deviation in z is larger than 6.0 cm

A bookkeeping entry tagging RC should be added if the mean z value is outside [-0.5 cm, +0.5 cm].

[QC on EPN]async_QC

The RM/RC must be informed if the values are outside the limits