Not frequent
0/14
Reinforcement Learning
Bandits, MDPs, and the Q / policy fragment IAIO expects — currently the empty domain on this guide.
Bandits, MDPs, and the Q / policy fragment IAIO expects — currently the empty domain on this guide.
10 multiple choice at 2 marks and 2 fill-in-the-blank at 5, marked exactly as Round 1 marks them. Each answer is explained as soon as you check it. The clock is shown, not enforced.
IAIO lists sequential decisions as its own domain. CAIO's written paper rarely names "Q-learning" in a stem, but the camp and the international contest do, and a Round 1 question that looks like search is sometimes an MDP in disguise. This module is the missing piece: bandits first, then the tuple, then the two updates you can write by hand.
A multi-armed bandit is an MDP with one state. You pick an arm, see a reward, and the world does not move. The whole subject collapses to exploration versus exploitation: pull the arm you currently like, or pull one you might like more.
ε-greedy is the sitting-friendly rule. With probability 1 − ε take the current best mean; with probability ε pick something else. UCB1 replaces the coin-flip with a bonus that shrinks as arm is pulled. Write the bonus once; the rest is argmax.
States, actions, transition probabilities, rewards, discount. A policy is a choice (or a distribution) over actions in each state — not a number. A value is a number derived from a policy: is expected return from ; is expected return from after committing to .
The discount is a modelling choice. is a bandit. makes distant rewards matter. Contest questions that give a three-step trajectory and a want the finite geometric sum, not a word about infinite horizons.
The optimality equation, written so you can memorise it:
Tabular Q-learning is the sample version of that, off-policy: whatever action you actually took to get the data, the target still uses . SARSA is on-policy: the target uses the next action the behaviour policy did take. Mixing the two names is the usual two-mark trap.
Monte-Carlo methods wait for the episode to end and use the realised return. Temporal-difference methods bootstrap. Terminal states do not bootstrap — the target is just .
You will not be asked to run DQN. You will be asked:
Work those by hand. The code box below is a one-step Q update you can run; it is the same arithmetic the blank wants.
A tabular entry is updated with , , reward , and a terminal next state. Ten times the new is a whole number. What is that whole number?
At in UCB1, no arm has been pulled. Which arm does the rule select?
makes undefined / infinite. Every unused arm is infinitely optimistic, so UCB pulls each arm once before it starts comparing means. That first sweep the exploration.
Select an answer
Terminal target is , not . New value: . Ten times that is .