In three lines

  • Just using an existing algorithm is no fun, and with one GPU I needed a very efficient structure
  • I picked three light candidates with strong recent results, RePPO, PQN and AGaLiTe, and ran them against a PPO baseline on the same island for 15 minutes each. Plain PPO came first at 14.0 days on average
  • In all four, the share of lives that fished peaked at 7.1%. So I decided to build our own brain, Mk.1, on PPO and a GRU, aimed at finding fish

There were two conditions

  • Light: 4,096 islands have to run together on one 8GB RTX 3070. If one training run takes days, there is no video
  • Varied behaviour: this is a simulation to watch, not a speedrun. A high score from moving the same way every time gives nothing to see
  • I also assumed the island needs memory: where the spring was, which bushes are picked clean

The candidates that were out

  • World models (DreamerV3 and others): they learn by simulating the world in imagination. That saves data, but a GPU simulator gives us plenty of data and the models are heavy
  • Transformer memory (TrXL): good memory, far heavier than the others. Too much to fit beside thousands of islands in 8GB

The three picks and the baseline

NameIn one lineWhy
PPO + GRUThe most common policy learning + recurrent memoryBaseline
RePPOPPO that assumes it can retry and pushes up the best outcomeExploration that gets bolder with retries
PQNQ-learning straight from parallel environments, no replay buffer or target networkSimple and fast
AGaLiTeA light transformer-style memory of fixed sizeTransformer memory on the cheap

Sources: RePPO code · PQN paper · AGaLiTe paper

  • All four trained for 15 minutes on the same island with the same observations and the same reward (1 point per day alive)
  • The learning rate fell toward the end, and the brain from the best moment was saved separately

PPO by the clock, AGaLiTe by experience

Line chart of mean days alive over the same 15 minutes. PPO ends at 14.0 days, AGaLiTe 11.8, PQN 9.4, RePPO 2.5

Made by HSL — 4,096 islands, 15 minutes each, 10/29 (runs/mk1_*/log.json)

AlgorithmMean survival after 15 minAt the same 18.35M stepsSteps per secondLives that fished (peak)
PPO + GRU14.0 days11.1 days44,0007.1%
AGaLiTe11.8 days11.8 days20,0006.1%
PQN9.4 days3.5 days50,0003.5%
RePPO2.5 days2.1 days45,0000.3%

Means from the training logs, one seed per algorithm, survival reward only, 10/29

  • By the clock, PPO wins. The simplest baseline did best
  • By experience, AGaLiTe is slightly ahead, but one step takes twice as long as with a GRU
  • PQN runs fastest and shot up late
  • RePPO stayed around 2 to 2.5 days and 95% died of thirst. I used the original code’s defaults, so without tuning for our island I cannot call it bad
  • With one seed each, the ranking itself is not reliable. I only took the broad direction from it

All four missed the fish

Peppo stands on the rocks beside a fishing spot. Fish show in the water inside the ring, and the bag panel lists 1 fish and a spear

Made by HSL — a scene this comparison never produced: a catch at a hidden spot (later brain, attempt 6,000)

Peak share of lives that fished for each algorithm: PPO 7.1%, AGaLiTe 6.1%, PQN 3.5%, RePPO 0.3%

Made by HSL — peak share of lives that fished (runs/mk1_*/log.json)

  • The island’s food is gone in a little over two weeks (part 2). Without fish, that is the limit
  • Even with the lower difficulty, the share of lives that fished peaked at 7.1%
  • So the real result of this match was not the ranking but that all four hit the same wall

So here is what Mk.1 aimed for

  • Almost nothing came from the three candidates. PPO + GRU did best, so it stayed as the base
  • AGaLiTe’s memory did not justify being twice as slow, so memory stayed a GRU for now
  • The new parts came from outside the comparison: curiosity that likes unvisited places (RND), a body that stays stable when stacked deeper (SimBa), and a Simplicial Embedding that splits up the policy’s features
  • Goal 1: learn faster on the same GPU
  • Goal 2: find the fishing breakthrough before the island’s food runs out
  • Goal 3: not move the same way every time, with different ways of surviving from life to life

If I did it again

  • Measure by both time and experience. The winner changed with the yardstick
  • Run several seeds from the start. A one-seed ranking had to be measured again later (part 4)
  • Track the behaviour that gets past the wall (here, fishing) next to mean survival. That number set the direction
  • Next part: building Mk.1 and fixing it until it became HyPPO