Choose evaluation splits

Problem

When you need this. A service predicts whether a user will return. Twelve observations from the same user are not twelve unrelated users. Randomly separating those rows can put the same user’s earlier and later activity in training, validation and test. That may answer a different question from the one your service needs to answer.

You will build four inspectable split plans for a synthetic log of 12 users observed daily over 12 days. The program records membership and overlap, without training a model or reporting a quality score. The most restrictive plan keeps a row only when its user’s assigned group and its observation period agree.

The first two lessons used Rice measurements to establish training and scoring. This lesson starts the synthetic user-return example used by the next five data and feature lessons. Repeated observations make user overlap and time boundaries visible; each lab bundles the inputs for its step.

Concepts

An entity is the user or other unit that can contribute several observations. An entity split asks how a system handles different users. A temporal split asks how it handles later observations. Combining them asks a stricter question about later observations from separately assigned user groups.

A gap deliberately excludes a time interval between partitions. Half-open periods include their start and exclude their end, making endpoint membership explicit. None of these plans is universally best: the target population and intended deployment decide which separation is appropriate.

Architecture

Each observation supplies an ID, entity and UTC time. A selected evaluation target determines group assignment and period boundaries. The planner records memberships, excluded rows and overlap diagnostics. It produces a split contract, not model predictions; later feature assembly must respect the information available at each retained observation.

Working code

Choose the population the evaluation should represent

Start with the intended use of the predictor. Will it serve existing users later, new users during a comparable period, or new users later? These are different targets, so one generic “random split” is insufficient as a specification.

An entity is the unit whose observations should remain together, here a user. In another application it could be a device, organization or document family. Choose it from the expected dependency, rather than assuming every unique row is independent. For an unseen-user estimate, all observations of a user must stay in one group. For a future estimate, the evaluation period must follow the training period. The scikit-learn guidance explains why grouped and time-dependent data need appropriate separation.

Download the complete lab and enter mlops-evaluation-splits/:

python3.12 -m venv .venv
source .venv/bin/activate
python inspect_splits.py
python -m unittest -v test_partitions.py

The original synthetic fixture contains only id, entity and observed_at. There are no real user records, labels, feature values or external dependencies. That narrow input lets you inspect separation before a training pipeline makes it harder to see.

Keep the split key separate from row order

The previous baseline used stable row keys. This program applies the same idea to either a row ID or an entity ID. Hashing the entity sends all of its rows to the same partition, regardless of the order in which those rows arrive.

"""Inspect row, entity and temporal boundaries; no model is trained."""
import hashlib
from datetime import datetime, timezone

NAMES = ('training', 'validation', 'test')
START = datetime(2026, 1, 1, tzinfo=timezone.utc)
TRAIN_END = datetime(2026, 1, 7, tzinfo=timezone.utc)
VALIDATION_START = datetime(2026, 1, 8, tzinfo=timezone.utc)
VALIDATION_END = datetime(2026, 1, 10, tzinfo=timezone.utc)
TEST_START = datetime(2026, 1, 11, tzinfo=timezone.utc)
END = datetime(2026, 1, 13, tzinfo=timezone.utc)
SALT = 'mlops-evaluation-splits-v1'
MODES = ('row_hash', 'entity_hash', 'future', 'unseen_future')


def instant(value):
    try:
        result = datetime.fromisoformat(value.replace('Z', '+00:00'))
    except (AttributeError, TypeError, ValueError) as error:
        raise ValueError('observation time must be an ISO UTC timestamp') from error
    if result.tzinfo is None or result.utcoffset().total_seconds() != 0:
        raise ValueError('observation time must have a UTC offset')
    return result


def validate(rows):
    seen = set()
    for row in rows:
        if not isinstance(row, dict) or set(row) != {'id', 'entity', 'observed_at'}:
            raise ValueError('each row needs exactly id, entity and observed_at')
        for key in ('id', 'entity'):
            if not isinstance(row[key], str) or not row[key] or len(row[key]) > 100:
                raise ValueError('IDs must be bounded nonempty strings')
        if row['id'] in seen:
            raise ValueError('duplicate row ID')
        seen.add(row['id'])
        instant(row['observed_at'])


def hash_partition(key):
    value = int.from_bytes(hashlib.sha256((SALT + ':' + key).encode()).digest()[:8])
    # Integer intervals avoid rounding the largest digest to an unassigned 1.0.
    for name, numerator, denominator in [('training', 7, 10), ('validation', 17, 20), ('test', 1, 1)]:
        if value * denominator < numerator * 2**64:
            return name
    raise AssertionError('unreachable digest endpoint')


def time_partition(at):
    if START <= at < TRAIN_END:
        return 'training'
    if VALIDATION_START <= at < VALIDATION_END:
        return 'validation'
    if TEST_START <= at < END:
        return 'test'
    return 'excluded'


def partition(rows, mode):
    if mode not in MODES:
        raise ValueError('unknown evaluation target')
    validate(rows)
    parts = {name: [] for name in (*NAMES, 'excluded')}
    for row in rows:
        entity_part = hash_partition(row['entity'])
        time_part = time_partition(instant(row['observed_at']))
        if mode == 'row_hash':
            name = hash_partition(row['id'])
        elif mode == 'entity_hash':
            name = entity_part
        elif mode == 'future':
            name = time_part
        else:
            name = time_part if entity_part == time_part else 'excluded'
        parts[name].append(row)
    if any(not parts[name] for name in NAMES):
        raise ValueError('each evaluation partition needs at least one row')
    return {name: sorted(items, key=lambda row: row['id']) for name, items in parts.items()}


def describe(parts):
    entities = {name: {row['entity'] for row in parts[name]} for name in NAMES}
    return {'rows': {name: len(items) for name, items in parts.items()},
            'entities': {name: len(items) for name, items in entities.items()},
            'train_validation_shared_entities': len(entities['training'] & entities['validation']),
            'train_test_shared_entities': len(entities['training'] & entities['test']),
            'ranges': {name: {'first': min(instant(row['observed_at']) for row in parts[name]).isoformat().replace('+00:00', 'Z'),
                              'last': max(instant(row['observed_at']) for row in parts[name]).isoformat().replace('+00:00', 'Z')} for name in NAMES}}

The hash intervals target 70% training, 15% validation and 15% test membership. With only 12 entities, the actual group counts are six, four and two. Those proportions are an observed consequence of the fixed salt, not a promise that each dataset will match the target percentages.

test_entity_split_has_no_shared_entity requires pairwise disjoint user sets. test_row_order_cannot_move_any_membership reverses the input for all four rules. Stable assignment does not prove that the chosen key captures every relationship: two accounts belonging to one person could still cross partitions.

Make time windows and gaps inspectable

The temporal plan uses UTC windows with inclusive starts and exclusive ends:

• Training: January 1 up to January 7.
• Validation: January 8 up to January 10.
• Test: January 11 up to January 13.

January 7 and January 10 are explicit gaps. A row at exactly midnight on January 7 is excluded, while a row at midnight on January 8 belongs to validation. The boundary test supplies those instants directly; a check using only midday observations would miss an endpoint error.

The one-day gaps are a scenario choice. Their duration must come from the real task’s information and label windows; adding a gap does not by itself prove that labels were available or features excluded later events. The next lesson adds those availability rules.

Timestamps without a UTC offset are refused. test_ranges_compare_instants_not_timestamp_spelling also checks that the summary compares parsed times, because different valid ISO spellings need not sort chronologically as raw strings.

Inspect overlap and excluded rows before fitting

Synthetic observations: 144; split membership only, no quality scores
row_hash: train=102; validation=23; test=19; excluded=0; train/test shared entities=9
entity_hash: train=72; validation=48; test=24; excluded=0; train/test shared entities=0
future: train=72; validation=24; test=24; excluded=24; train/test shared entities=12
unseen_future: train=36; validation=8; test=4; excluded=96; train/test shared entities=0
Temporal gaps and group isolation answer different deployment questions.

The row-hash plan shares nine users between training and test. The future plan shares all 12. The latter overlap is intentional when evaluating later activity from the same population; it would be wrong to present that result as performance on unseen users.

The joint plan keeps only 48 rows and excludes 96. It retains training-group users in the training window, validation-group users in the validation window, and test-group users in the test window. Other group/window combinations and both gap days remain in the exclusion record. Nothing is silently reassigned to recover a preferred sample size.

Four test observations from two users are enough to exercise this rule, not to support a useful model-performance conclusion. A stricter partition can be correctly implemented and still leave inadequate evaluation data. Change the collection plan or stated target rather than choosing the split that produces the most attractive score.

Save the membership contract with the experiment

python inspect_splits.py --check
python inspect_splits.py --save my-splits.json

The record binds the input checksum and hash salt to every retained and excluded ID. This makes a later change in membership inspectable. The driver is also included in the lab:

"""Materialize inspectable membership records for a synthetic observation log."""
import argparse
import hashlib
import json
from pathlib import Path
from partitions import MODES, partition, describe, SALT

ROOT = Path(__file__).resolve().parent


def record():
    raw = (ROOT / 'observations.json').read_bytes()
    rows = json.loads(raw)
    modes = {}
    for mode in MODES:
        parts = partition(rows, mode)
        modes[mode] = {'summary': describe(parts), 'ids': {name: [r['id'] for r in items] for name, items in parts.items()}}
    return {'mode': 'synthetic split inspection; no training or model evaluation',
            'input_sha256': hashlib.sha256(raw).hexdigest(), 'hash_salt': SALT,
            'observations': len(rows), 'modes': modes}


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--save', type=Path)
    parser.add_argument('--check', action='store_true')
    args = parser.parse_args()
    result = record()
    if args.check and result != json.loads((ROOT / 'result.json').read_text()):
        raise SystemExit('split record differs from the bundled fixture')
    if args.save:
        args.save.write_text(json.dumps(result, indent=2) + '\n')
    print(f"Synthetic observations: {result['observations']}; split membership only, no quality scores")
    for mode, item in result['modes'].items():
        s = item['summary']; r = s['rows']
        print(f"{mode}: train={r['training']}; validation={r['validation']}; test={r['test']}; "
              f"excluded={r['excluded']}; train/test shared entities={s['train_test_shared_entities']}")
    print('Temporal gaps and group isolation answer different deployment questions.')


if __name__ == '__main__':
    main()

Duplicate row IDs, malformed fields and unknown targets are refused. An empty training, validation or test partition also fails; the program does not produce a misleading empty report. It checks neither class balance nor feature availability because those fields are absent from this fixture.

Design exercise: suppose the service will support both returning and first-time users next month. State the two evaluation populations you need. Explain why a single aggregate score could conceal failure on one population, and what additional user-level sample counts you would record before comparing models.

Failure cases

A random-looking partition is not enough. The grouped plan must have no shared entities, and the temporal plan must assign boundary timestamps consistently. The tests compare memberships after reordering and retain excluded rows so missing data cannot disappear from the explanation.

Deployment

Save the chosen target, split parameters and exact membership with the dataset version. Reuse that contract when training and evaluating a candidate. Before applying it to real event data, define the observation timestamp and entity identity with the data producer; this synthetic log cannot settle those product semantics.

Scaling

Large repeated-entity datasets require efficient grouping and timestamp processing, but optimization must preserve membership. Inspect overlap and exclusion counts before parallelizing assignment. A more restrictive split may leave fewer usable rows, which changes the observations available; it does not automatically make the resulting score more accurate.

Key takeaways

• Entity and time separation answer different generalization questions.
• Combined constraints can exclude observations rather than assigning every row.
• Explicit memberships make a split reviewable before model fitting.

Best practices

• Choose the evaluation target before viewing model results.
• Keep gap and excluded-row records.
• Test endpoints and row-order independence.

References

• scikit-learn, cross-validation: grouped and chronological evaluation address different dependencies.
• Google Machine Learning Crash Course, dataset separation: validation supports development decisions; final testing stays separate.

Sources checked October 24, 2026. All displayed membership counts come from the bundled synthetic observation log and local execution.