opentau.policies.value.reward
Reward calculation utilities for Value Policy training.
This module contains functions to calculate returns and discretize them into bins for value function training and advantage calculation.
Functions
|
Defines sparse Reward function for the pi0.6 policy to calculate advantage. |
Defines sparse Reward function for the pi0.6 policy to train value function network. |
- opentau.policies.value.reward.calculate_n_step_return(success: bool, n_steps_look_ahead: int, episode_end_idx: int, reward_normalizer: int, current_idx: int, c_neg: float = -100.0) float[source]
Defines sparse Reward function for the pi0.6 policy to calculate advantage.
- Parameters:
success – Defines if the episode was successful or failed.
n_steps_look_ahead – Number of steps to look ahead for calculating reward.
episode_end_idx – Index of the end of the episode.
reward_normalizer – Maximum length of the episode for normalization.
current_idx – Current index of the episode.
c_neg – Negative reward for failed episodes. Defaults to -100.0.
- Returns:
The normalized continuous reward for the n-step lookahead.
- Return type:
float
- opentau.policies.value.reward.calculate_return_bins_with_equal_width(success: bool, b: int, episode_end_idx: int, reward_normalizer: int, current_idx: int, c_neg: float = -100.0) tuple[int, float][source]
Defines sparse Reward function for the pi0.6 policy to train value function network.
- Parameters:
success – Defines if the episode was successful or failed.
b – Number of bins to discretize the reward into, including the special bin 0.
episode_end_idx – Index of the end of the episode, exclusive to the last step.
reward_normalizer – Maximum length of the episode for normalization.
current_idx – Current index of the episode.
c_neg – Negative reward for failed episodes. Defaults to -100.0.
- Returns:
- A tuple containing:
bin_idx: The index of the reward bin.
return_normalized: The normalized return value in range [-1, 0].
- Return type:
tuple[int, float]