- Description:
- Network Topology = Graph = Adjacency Matrix over all nodes in the network.
- Flows = (src, dst) = Partial Routes = Dynamic State
- Policies =
Reachability to dst. Matrix of size #flows * #nodes. 1 for (flow_i, src_i) and (flow_i, dst_i).
(Soft) Isolation Policy = Flow i and j should not share any link. Matrix of size #flows * #flows. (flow_i, flow_j) = 1 and (flow_j, flow_i) = 1.
- Decompose this problem:
- Initial State (s_0): Each flow is at node src. Matrix of size #flows * #nodes. (flow_i, src_i) = 1
- We extend each flow by one hop at a time.
- Round Robin over Flows
- State: Topology, Partial Routes for each Flow, Current Active Flow, Next Hops (set of neighboring nodes which have not been visited and are reachable to the destination)
- Action: Which Next Hop to Choose?
- train_queue = interactions with the environment = current state (s), action (a), immediate reward (r), next state (s’)
- Reward: discounted over n-steps with gamma discounting factor = sum from i = 0 to n-steps gamma^i * r_i
- Graph Neural Net (Policy Model i.e. Actor-Critic Model) = inputs = state and output = next hop
- env.reset = state_0 = Topology is always constant.
The flows and policies have an upper bound =
You pick the same set of flows and same set of policies, OR
You randomly pick set of flows and set of policies.
- Agent = Wrapper Around the Brain Model
- action = agent.act(current_state) ← does inference on the policy i.e., actor model
- next_state, reward = env.step(action)
- repeated to get an episode (populate the train_queue)
- Batched Training : When train_queue is sufficiently full
- brain.optimize(): backprop on the actor-critic model → you get updated actor-critic model
- while True:
Optimizer+Brain initialization => start filling the train_queue
Environment => run episodes with the current version of the brain i.e., the actor-critic policy model
After recording episodes and filling the train_queue you train and update Brain