Checking a model#
python -m modelly.lint reads your model’s source and reports two kinds of problems:
Errors (
MDL0xx): the model runs, but its result is wrong or cannot be reproduced from its seed.Warnings (
MDL1xx): the model is right, but its cost grows faster than it needs to, usually O(N²) per step where O(N) is possible.
Nothing is imported or executed: the files are parsed and walked, so the check takes milliseconds and is safe to run on any code.
python -m modelly.lint my_model/ # a folder, recursively
python -m modelly.lint model.py agents.py # or files
python -m modelly.lint --list-rules # one line per rule
python -m modelly.lint --explain MDL101 # one rule in detail: why, bad and good example
python -m modelly lint my_model/ # the same, through the umbrella command
model.py:14:12: MDL001 random draw outside the model's generators
hint: use self.random (stdlib API) or self.rng (numpy Generator) so the seed reproduces the run
model.py:17:19: MDL101 an agent scans the whole population: O(N) per agent, O(N^2) per step
hint: look at neighbours through self.cell.neighborhood or a space query, or move the phase to the model as columns
2 finding(s): 1 error(s), 1 warning(s)
Each finding is path:line:col: CODE message, the form editors and CI logs link to, followed by
a hint.
Options#
Option |
Effect |
|---|---|
|
keep only these comma-separated codes or prefixes ( |
|
drop these codes or prefixes |
|
the default: findings, hints and a summary line |
|
a JSON list of findings, for other tools |
|
|
|
one line per rule with its count, most frequent first |
|
only the |
|
apply the safe fixes, then report what is left |
|
print the safe fixes as a unified diff; nothing is written |
|
report, but exit 0 |
|
print why a rule matters, a bad and a good example, and exit |
|
print every rule and exit |
Exit status: 0 no findings (or --exit-zero), 1 at least one finding (with --diff:
at least one fix to apply), 2 a usage error
(a path that cannot be read, a file that does not parse, a bad configuration or an unknown rule
for --explain).
Errors#
MDL001: random draw outside the model’s generators#
The seed fixes a run only when every draw comes from the model. The global random and
np.random streams, an unseeded random.Random(), np.random.default_rng() or
default_rng(None), a bit generator built without a seed (PCG64()) and SystemRandom are
not owned by the model, so two runs with the same seed differ. Calls are read through the
module’s imports, so import numpy.random as npr and from random import choice are caught;
random.Random(42), default_rng(seed=7) and Generator(PCG64(7)) are not reported.
if random.random() < self.p: # MDL001
noise = np.random.normal(size=n) # MDL001
if self.random.random() < self.p: # the model's stdlib-style generator
noise = self.rng.normal(size=n) # the model's numpy Generator
MDL002: __init__ does not call super().__init__#
An agent whose __init__ skips super().__init__(model) is never registered: it is not in
model.agents, is never activated and has no unique_id. A model that skips it has no
generators or registry. Nothing raises at once; the failure comes later, far from the cause.
class Sheep(Agent):
def __init__(self, model):
self.energy = 5 # MDL002
class Sheep(Agent):
def __init__(self, model):
super().__init__(model) # registered, with a unique_id
self.energy = 5
MDL003: loop over a set#
A set of agents iterates in an order that follows memory addresses, which change from run to run, so every draw or update that depends on the order changes too.
for a in set(neighbours): # MDL003
for a in neighbours: # an AgentSet or a list keeps its order
for a in sorted(set(neighbours), key=lambda a: a.unique_id):
MDL004: mutable class attribute on an agent#
A list, dict or set in the class body is created once, with the class, and shared by every
agent: one agent’s append shows up in all of them. __slots__ and attributes annotated
ClassVar[...] are shared on purpose and not reported.
class Trader(Agent):
partners = [] # MDL004: one list, shared by every Trader
def __init__(self, model):
super().__init__(model)
self.partners = [] # one list per agent
MDL005: wall clock or OS entropy in model code#
time.time(), time.time_ns(), datetime.now(), datetime.utcnow(), date.today(),
uuid.uuid1(), uuid.uuid4(), os.urandom() and anything from secrets return a new value on
every run. Inside a model or agent class they make the outcome depend on the machine and the
moment instead of the seed. Timing code outside the classes, and time.perf_counter() for
profiling, are not reported.
class Sheep(Agent):
def step(self):
self.born = datetime.now() # MDL005
self.tag = uuid.uuid4() # MDL005
self.born = self.model.time # the model's clock
self.tag = self.unique_id # already unique and reproducible
MDL006: hash() or id() decides an outcome#
id() is a memory address, and hash() of a str or bytes is salted per process
(PYTHONHASHSEED). Ordering by them (key=id, key=lambda a: id(a)) anywhere, or calling
hash() in a model or agent method other than __hash__, gives a different result each run.
queue = sorted(self.agents, key=id) # MDL006
group = hash(self.name) % 4 # MDL006
queue = sorted(self.agents, key=lambda a: a.unique_id)
group = self.unique_id % 4
MDL007: a loop over the population removes agents#
Iterating an AgentSet walks a snapshot taken when the loop starts. An agent removed by an
earlier iteration (other.remove()) is therefore still visited and acts after its death.
agents.do(...) and agents.shuffle_do(...) check membership and skip it. Removing the loop’s
own agent (for a in ...: a.remove()) is fine and not reported.
for wolf in self.agents_by_type[Wolf]:
wolf.prey.remove() # MDL007: that prey may come up later
self.agents_by_type[Wolf].shuffle_do("hunt") # hunt() calls self.prey.remove()
MDL008: global statement in a model or agent method#
A module-level variable outlives the model. Two models in one process (a batch run, a test suite, a notebook re-run) share it and keep mutating it, so a run depends on what ran before.
BORN = 0
class Sheep(Agent):
def step(self):
global BORN # MDL008
BORN += 1
self.model.born += 1 # state on the model: each run starts fresh
MDL009: the model’s agent set is changed directly#
model.agents and model.agents_by_type[...] are the registry itself. add, discard,
remove or clear on them skips the per-type sets, the column tables and the data
collectors, which then disagree with each other. Creating an agent registers it; agent.remove()
deregisters it everywhere.
self.model.agents.discard(self) # MDL009
self.remove()
MDL010: a lambda or nested function is sent to worker processes#
Windows and macOS start process-pool workers with spawn: each worker is a fresh interpreter
that imports the function or class it runs by its module and qualified name. A lambda, or a
function or class defined inside another function (a closure), has no such name. Linux forks
its workers, so the same batch works there and fails on the other two.
The rule looks at every argument of run_batch, run_grid, run_scenarios,
RunConfiguration and Experiment, and at keyword arguments of any call named run,
model_class, model_cls or ending in factory (model_factory=). Lists, tuples, dict
values and functools.partial(f, ...) are looked into. A call with workers=1 runs every
task in this process and is not reported.
def main():
def run(task): # nested: a spawned worker cannot import it
return Sheep(task).run()
run_batch(run, tasks) # MDL010
run_batch(lambda t: t, tasks) # MDL010
def run(task): # module level: importable by every worker
return Sheep(task).run()
MDL012: network or language-model call outside modelly.llm#
A reply from a web service or a language model differs from one call to the next, and the
service may change or disappear. Inside a model or an agent, a call through requests, httpx,
urllib.request.urlopen, or the openai and anthropic SDKs (including a client kept in an
attribute: ....chat.completions.create, ....messages.create) is recorded nowhere, so the run
cannot be replayed, audited or held to a budget. Call through modelly.llm: it
caches every reply under a key of the call, replays offline and stops before a budget is passed.
class Voter(Agent):
def step(self):
reply = client.messages.create(model=MODEL, messages=[...]) # MDL012
self.vote = reply.content[0].text
class Voter(LLMAgent, Agent):
def step(self):
self.vote = self.decide(prompt, {"enum": ["yes", "no"]})
Warnings#
MDL101: an agent scans the whole population#
A loop, comprehension, select, agg, groupby, sort or map over self.model.agents
(or agents_by_type[...]) inside an agent method visits every agent once per agent: N² visits
per step. Look at neighbours through the space instead (self.cell.neighborhood, a
continuous-space query), or write the phase once at the model level over columns; see
vectorised step phases.
near = [a for a in self.model.agents if a.cell in self.cell.neighborhood] # MDL101
near = self.cell.neighborhood.agents
MDL102: nested loops over the population#
Two nested loops over the population compare every pair of agents, N² per step, even when only
neighbours can interact. Pair agents through the space (cells, pairs_within) or group them
first with agents.groupby.
for a in self.agents:
for b in self.agents: # MDL102
a.meet(b)
for a in self.agents:
for b in a.cell.neighborhood.agents:
a.meet(b)
MDL103: hand-written activation loop#
for a in self.agents: a.step() is self.agents.do("step"), a fixed order, and unlike do
it still visits agents removed earlier in the loop. Most models mean
self.agents.shuffle_do("step"), a new random order each step.
for agent in self.agents: # MDL103
agent.step()
self.agents.shuffle_do("step")
MDL104: deepcopy inside step#
deepcopy walks the whole object graph, which for an agent includes the model, the space and
every other agent. Done every step, it costs more than the step itself. Copy the fields you
need, or keep the previous value in a column.
self.previous = copy.deepcopy(self) # MDL104
self.previous_energy = self.energy
MDL105: step has more than 12 branches#
A long step hides its phase order. Split it into named phases (move, interact, update)
and call them in order, which is also the first move towards vectorising
them.
def step(self):
self.move()
self.eat()
self.reproduce()
MDL106: pandas DataFrame built inside step#
pd.DataFrame(...) and pd.concat(...) copy every row each time, so growing a table step by
step is quadratic in the number of steps. Record with a DataCollector, or append rows to a
list and build the frame once after the run.
def step(self):
self.log = pd.concat([self.log, pd.DataFrame([row])]) # MDL106
self.rows.append(row) # pd.DataFrame(self.rows) at the end
MDL107: per-agent loop that only writes Column attributes#
In a model method, a loop over the population whose body only assigns attributes declared as
Column (plain =/+=, or an if made of those, with no calls) is a whole-column update
written one agent at a time. The columns are those declared in the same module and those of the
agent classes it imports from another file of the linted tree (from .agents import Sheep, or
from pkg.agents import Sheep under a path given to the linter); that file is parsed once per
run, never imported. The column’s numpy array does it in C. Loops
that call anything (a draw, a method) are not reported: vectorising them changes the order of
draws or needs a decision.
class Sheep(Agent):
energy = Column(float, default=5.0)
class Pasture(Model):
def step(self):
for sheep in self.agents_by_type[Sheep]: # MDL107
sheep.energy -= 1
table = self.columns[Sheep]
table.energy -= 1 # one numpy operation
Fixing findings#
Four rules have a fix that is safe to apply without reading the code around it.
--diff shows what it would change; --fix writes it and then reports the findings that
are left for you.
python -m modelly.lint --diff my_model/ # review
python -m modelly.lint --fix my_model/ # apply
Rule |
Before |
After |
|---|---|---|
MDL001 |
|
|
MDL001 |
|
|
MDL002 |
an agent |
|
MDL002 |
a model |
|
MDL003 |
|
|
MDL003 |
|
|
MDL009 |
|
|
MDL009 |
|
|
Only the expression the finding points at is rewritten: formatting, comments and line endings
around it are kept. A finding is left alone when the right rewrite is a guess: a draw outside a
method (there is no self), a numpy function whose Generator method takes other arguments
(randint, rand), a base class whose __init__ signature is unknown, a model __init__ that
takes both rng and scenario, or agents.add(...) (an agent registers itself when it is
created). Every fix removes its own finding, so a second --fix changes nothing. A finding
silenced by # noqa, or dropped by select, ignore or per-file-ignores, is not fixed.
An imported name keeps its import (from random import choice stays when choice(xs) becomes
self.random.choice(xs)); remove it with your formatter or ruff check --fix.
Silencing a finding#
End the line the finding points at with # noqa: MDL101 to skip that rule there, # noqa: MDL101, MDL104 for several, or a bare # noqa to skip every rule on the line. Text after the
codes is free, so the comment can say why. For a loop or a multi-line call, the line is the
first one (for ...: or the line the call starts on).
for a in self.model.agents: # noqa: MDL101 runs once, at set-up
...
The checks are heuristics on the source: a loop over self.model.agents that runs once at
set-up is fine, and the comment is the place to say so.
Configuration#
[tool.modelly.lint] in the nearest pyproject.toml, searched from the working directory
upwards, sets the defaults:
[tool.modelly.lint]
select = ["MDL0", "MDL101"] # only these codes or prefixes (default: every rule)
ignore = ["MDL105"] # drop these codes or prefixes
per-file-ignores = { "tests/*" = ["MDL001"], "scripts/*.py" = ["MDL1"] }
Per-file patterns are globs on the path relative to the pyproject.toml folder, or on the bare
file name. --select and --ignore on the command line replace the file’s select and
ignore; per-file-ignores always applies. An unknown key is an error (exit 2), so a typo does
not silently disable the configuration.
In CI#
- run: pip install modelly-abm
- run: python -m modelly.lint --format github src/
With --format github, each finding shows as an annotation on the changed line of the pull
request.
In a notebook#
%load_ext modelly.ipython
%modelly lint model.py agents.py
%modelly lint --diff model.py
%modelly lint takes the options of the command line and prints the same report. A project
made with modelly new has its [tool.modelly.lint] settings and passes every
rule from the start.