.. _error_models:

Residual Error Model
######################
So far, we have looked at compartment models and different dosing regimes all under theoretical conditions. For example, our |CvT| curve for a one compartment model with absorption produces smooth curves with evenly spaced observation points (:numref:`fig_dep_one_cmp_noiseless`).

.. figure:: /_autogen/indiv_examples/compartment_models/odes/dep_one_cmp_cl/images/gen_dense/comb_spag_.*
    :name: fig_dep_one_cmp_noiseless
    :width: 50%
    :align: center

    |CvT| for a one compartment model with absorption, without measurement error (noise).

In practice, however, it is impossible to measure any quantity perfectly; all measurements are imperfect and contain some :term:`noise`. To weight each observation correctly, we may need to account for patterns of noise in the data. For example, noise often increases proportionally with signal so that higher concentrations contain more absolute error. Sometimes we know that observations have been collected in suboptimal conditions and may wish to allow the model to downweight them if they are inconsistent with other data.

We therefore turn to the different ways in which noise manifests itself on the observations and how we can incorporate this into a fitted model.


Error Models for Continuous Data
====================================
Specifically, we look at how to model errors that appear on a continuous quantity such as the measured concentration of drug, demonstrating the concept using a normal distribution (a popular choice for continuous measurements).


Additive Noise
----------------
.. note:: See the :ref:`sum_link_ao_gen_ao_fit_tut` for the :ref:`tut_script` used to generate results in this section. 

In the simplest case, the difference between the observed measurement and the model prediction will be a random variable of constant :term:`variance` (or, equivalently, constant standard deviation) such that the observations are scattered evenly around the ideal curve.

.. literalinclude:: /_autogen/indiv_examples/error_models/ao_tut/tut_sections/PREDICTIONS.pyml
    :language: pyml

With a standard deviation of 1.0, for example, and 100 randomly sampled time points between 1 and 40, we get a more realistic |CvT| curve (:numref:`fig_additive_noise`).

.. figure:: /_autogen/indiv_examples/error_models/ao_tut/images/gen_dense/comb_spag_.*
    :name: fig_additive_noise
    :width: 50%
    :align: center

    |CvT| for a one compartment model with absorption, plus additive noise.


Proportional Noise
-------------------
.. note:: See the :ref:`sum_link_po_gen_po_fit_tut` for the :ref:`tut_script` used to generate results in this section. 

While additive noise is independent of the strength of the signal (|eg| the magnitude of the measurement) at any point in time, proportional noise increases in proportion to the magnitude of the signal. For example, the standard deviation of the noise may be equal to 10% of the signal magnitude such that the error is greatest at the peak of the curve, reducing as the concentration falls (:numref:`fig_proportional_noise`)

.. literalinclude:: /_autogen/indiv_examples/error_models/po_tut/tut_sections/PREDICTIONS.pyml
    :language: pyml

.. figure:: /_autogen/indiv_examples/error_models/po_tut/images/gen_dense/comb_spag_.*
    :name: fig_proportional_noise
    :width: 50%
    :align: center

    |CvT| for a one compartment model with absorption, plus proportional noise.


Additive and Proportional Noise
-----------------------------------
.. note:: 
    See the :ref:`sum_link_pa_gen_pa_fit_tut` for the :ref:`tut_script` used to generate results in this section. 

In reality, most measurement processes are affected by both kinds of error such that the additive noise dominates at low signal magnitudes whereas proportional noise dominates at high signal magnitudes (:numref:`fig_mixed_noise`).

.. figure:: /_autogen/indiv_examples/error_models/pa_tut/images/gen_dense/000001.*
    :name: fig_mixed_noise
    :width: 50%
    :align: center

    |CvT| for a one compartment model with absorption, plus both proportional and additive noise

We must, however, take care when defining the variance of a mixed error model. The variance of the `sum of two normally distributed variables <https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables>`_ is the sum of their *variances*, and therefore the *standard deviation* of the sum does *not* equal the sum of their standard deviations:

.. math::

    a        &\sim N(\mu_a, \sigma_a^2) \\
    b        &\sim N(\mu_b, \sigma_b^2) \\
    Var(a+b) &= \sigma_a^2 + \sigma_b^2 \\
             &\neq (\sigma_a + \sigma_b)^2 \\
    Std(a+b) &\neq \sigma_a + \sigma_b

as shown in the |predictions| block of the |popy| input script:

.. literalinclude:: /_autogen/indiv_examples/error_models/pa_tut/tut_sections/PREDICTIONS.pyml
    :language: pyml

It is crucially important to specify the form of the total variance correctly when fitting if we are to estimate the additive and proportional variances correctly. To see the effect of specifying the variance incorrectly, compare the tutorial script :ref:`sum_link_pa_gen_pa_fit_badvar_fit` with its correct counterpart, :ref:`sum_link_pa_gen_pa_fit_tut`.
    
.. comment
    .. include:: error_models/fitting_results.txt

    Lognormal Noise
    -------------------
    A further error model uses the exponential to model the variance of the error:

    .. math::

        Y        &= f(x) \cdot \exp(\delta) \\
                 &= f(x) \cdot (1 + \exp(\delta) - 1) \\
                 &= f(x) + f(x) \cdot (\exp(\delta) - 1) \\
        Y - f(x) &= f(x) \cdot (\exp(\delta) - 1) \\

    where δ is a random variable, such that the error can be modelled in |popy| with a |predictions| section like:

    .. literalinclude:: /_autogen/indiv_examples/error_models/exp_tut/gen_sections/PREDICTIONS.pyml
        :language: pyml

    giving observations (:numref:`fig_exp_noise`) that look very similar to those generated with proportional noise (:numref:`fig_proportional_noise`).

    .. figure:: /_autogen/indiv_examples/error_models/exp_tut/images/gen_dense/000001.*
        :name: fig_exp_noise
        :width: 50%

        Amount-vs-Time for a one compartment model with absorption, plus exponential noise

    In practice, this is because the exponential function approximates the proportional noise model when the variance of δ is small as can be seen from the approximation,

    .. math::

        Y        &=       f(x) \cdot \exp(\delta) \\
                 &=       f(x) \cdot (1 + \delta + \delta^2/2! + \hdots + \delta^n/n! + \hdots) \\
                 &\approx f(x) \cdot (1 + \delta) \\

    derived from the Taylor expansion (a mathematical technique for expressing a function as the sum of a series of terms).

    There is, therefore, little reason to use the exponential model over the simpler, more popular, proportional model except in extreme cases, |ie| when there are many high-side outliers.

.. comment

    Other Error Models
    ====================
    The only distribution automatically available in |nonmem| is the |norm_dist|, which makes this reformulation of the likelihood syntax possible. To use other (none normal) distributions in |nonmem|, you have to write your own custom likelihood functions.

    The |popy| syntax is designed to be more explicit and allows for other (non-normal) distributions to be fitted (|eg| |bern_dist|, |poisson_dist| or |neg_bin_dist|), without specifying the likelihood as custom code. The |popy| system, however, still requires the user to define the means and variances correctly.

    Total Variance
    ================
    [Don't really know what this is]


    Discrete Errors
    ====================

    Negative binomial example: DDMORE 60, 66, 239, 261


    Counting
    ---------


    Categorical
    --------------
    Different kind of measurement that requires different error distribution altogether. Introduce the Bernoulli/Poisson/NegativeBinomial distribution here.


    Time-to-Event
    ----------------
    Different kind of measurement again.
