There are increasing economic and environmental incentives for ship owners and operators to develop tools to optimise operational decisions, particularly with the aim of reducing fuel consumption and/or maximising profit. Examples include real time operational optimisation, maintenance triggers and evaluating technological interventions. Performance monitoring is also relevant to charter party analysis, vessel benchmarking and to better inform policy decisions. The ship on-board systems and systems in which they operate are complex and it is common for data modelling and analysis techniques to be employed to extract trends. All datasets and modelling procedures have an inherent uncertainty and to aid the decision maker, the uncertainty can be quantified in order to fully understand the economic risk of a decision; an unacceptable risk requires further investment in data quality and data analysis techniques. This paper details and categorises the relevant sources of uncertainty in performance measurement, and presents a method to quantify the overall uncertainty in a ship performance indicator based on the framework of the “Guide to Uncertainty in Measurement using Monte Carlo Methods”. A sensitivity analysis conducted on the sources of uncertainty highlights the relative importance of each. The two major data acquisition strategies, continuous monitoring and noon reported, are compared.