Thanks for the metareview! Here are our thoughts for future readers.

For our thoughts on novelty, see Part 1 of our shared reply.

CQL and DT showed better or comparable performance with much lower compute

Raisin outscores CQL on 12 of the 15 tasks [1] and otherwise scores within 10%. (Similarly vs. DT [2,3].) Further, just like SAC-$N$, Raisin often beats CQL and DT by large margins, e.g. by ~500% on hopper-random [1].

Moreover, Raisin likely runs faster than CQL and DT since we use EDAC's code but with 5$\times$ fewer critics, and EDAC is already faster than CQL [1], which is faster than DT [3]. We are unsure which method uses less of the GPU, but [1] suggests the memory usage of CQL and Raisin roughly match.

does the conclusion generalizes to other RL algorithms involving bootstrapping (solving the Mean Squared Bellman Error) beyond SAC

We think our TD3 version of Raisin in Appendix E suggests this. Granted, we only tested the datasets we thought would be most difficult, and still only D4RL gym.

All that said, we also wish we had emphasized Raisin's empirical success mostly as further motivation for fixing pure RG.

[1] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble, An et al.

[2] RvS: What is Essential for Offline RL via Supervised Learning?, Emmons et al.

[3] Offline Reinforcement Learning with Implicit Q-Learning, Kostrikov et al.

Part 3: A Raisin analog for TD3

We've also now implemented a Raisin analog for TD3 to examine the generality of our approach and to show the advantage of Raisin over a method approximately equivalent to the prior work Bi-Res-DDPG (since TD3 is essentially an improved DDPG). As we expected, RA-TD3 with $N=1$ (so roughly the prior work Bi-Res-DDPG) hardly scores at all on hopper-expert, but adding our minimum-of-$N$-critics residual formulation gives scores similar to Raisin (the only difference is SAC vs TD3 as a base). We've added those results in Appendix E and show them here as well:

Part 4: Fixed point experiments

We've additionally added new experiments in Appendix F training RG ($\eta =100\mathrm{\%}$) and SG in new runs starting from weights at high scores to investigate the role of fixed points. When $N=1$, RG and SG attain high scores on hopper-expert and halfcheetah-random, respectively. We then restart training from those high-scoring weights using both RG and SG on each environment. On hopper-expert, SG trained from RG's high-scoring weights reverts to low scores, while new RG runs trained from those same weights retain high scores. Similarly, on halfcheetah-random, RG trained from SG's high-scoring weights falls to low scores while new SG runs starting from those same weights keep their high scores. Our results provide evidence in favor of fixed points playing an important role, though future work should study this in-depth. We also show those results here.

Resuming from SG on halfcheetah-random:

Resuming from RG on hopper-expert:

Thanks again for your review! In light of our additions and replies, might you consider raising your score?

Thanks a ton for the review! To recap the strengths you highlight:

• Raisin is simple
• The paper is written clearly
• Raisin roughly matches SOTA with dramatically reduced compute
• Aside from a fixed $N$ and tuning $\eta$, we left all other hyperparameters untouched

It seems that the novelty of this paper is limited, the methodology is a direct application of RA on SAC-N,

Please see Part 1 of our shared reply, "Novelty." To our knowledge, RA has strictly been proposed and tested as an online algorithm. The only use of deep RA we are aware of is the RA-DDPG work (Zhang et al. 2019), where even the highest improvement in AUC over all tasks is a factor of about 3$\times$. In contrast, we have discovered that RA works disproportionately well in the offline setting (for which it was not designed), giving a median 54$\times$ improvement in score over all tasks. Moreover, we find Raisin to be the best of multiple possible ways to combine RA and SAC-$N$, and that this combination performs better than the sum of its parts. We also find that tuning the residual weight differently for every dataset is critical.

Please also see Parts 3 and 4 of our shared reply, where we discuss new experiments we've added.

there are little theoretical discussion on how the performance get improved.

Please see Part 2 of our shared reply, "Why does Raisin work?" Also, please see our new experiments in Part 4 of our shared reply, which suggest the importance of fixed points.

In Section 4.1 comparison with TD3+BC-10, the authors simply claim that Raisin outscores TD3+BC-10. However, in table 1, there are some tasks that RAISIN scores lower than TD3+BC, for example on hopper-expert, walker2d-expert, walker2d-medium-expert, hopper-medium-expert, hopper-medium, halfcheetah-random, etc.

We wrote it that way in our initial draft because we did not consider 1% score differences significant. However, we have updated our draft to be more precise. For example, Raisin scores 90% or higher than the best score on twelve tasks, whereas TD3+BC-10 only achieves this for seven tasks. And TD3+BC-10's worst relative score on a task is only 18% of that task's best (walker2d-random), whereas Raisin's worst score on a task is still 73% of that task's best (hopper-medium).

The authors do not provide sufficient explanation regarding what are the properties in these tasks that makes the performance of RAISIN worse.

We agree that it would be nice to know the fundamental reason for the slightly lower scores on hopper-medium and walker2d-random. Two possible explanations are: (i) we left every hyperparameter other than $\eta$ completely untouched — Raisin may fully match SAC-$N$ with more optimized hyperparameter choices, such as a different learning rate; (ii) for simplicity, we intentionally ignored the slight stochasticity of MuJoCo, i.e., the double sampling problem, though there are ways to fix this, see Section 5.3.

Since this paper does not provide new techniques or methodologies that shed lights on future researches in offline RL, but only provides experimental studies on a combination of two existing methods. It is important for the authors to provide more thorough empirical explanations and interpretations. The experimental results are good for general papers, but for a paper without enough technical novelty, higher standard in experiments should be expected.

We argue that our work has sufficient novelty in method design. Raisin is not just a combination of two existing methods. We also have negative algorithmic results, where ensemble approaches attempting to give pure RG optimism don't improve pure RG's score on tasks like halfcheetah-random. We've expanded on this in Appendix G. Please see Part 1 of our shared reply for more discussions on novelty. Furthermore, we have added more experiments, as discussed in Parts 3 and 4 of our shared reply, and provided more thorough empirical explanations of why Raisin works — please see Parts 2 and 4 of our shared reply.

References

Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep Residual Reinforcement Learning. arXiv, May 2019. doi: 10.48550/arXiv.1905.01072.

Thanks again for the feedback! Given our additions and replies, might you consider raising your score?

Thanks a bunch for the feedback! To recap the strengths you emphasize:

• Raisin is performant
• Raisin achieves SOTA scores
• The paper is fairly clear and/or reproducible

The proposed RAISIN method can be viewed as a combination of RA and SAC-N, but in some sense lacking adequate motivation: RA method is helpful to the stable policy training, and the main issue of SAC-N is the large number of ensemble functions, but why simply combining the two would be good?

Please see Part 2 of our shared reply, "Why does Raisin work?" Also, please see our new experiments in Part 4 of our shared reply, which suggest the importance of fixed points.

The proposed method can be easily extended to most current popular offline rl algorithms, so it's better to add more experiments to verify its scalability.

We agree our approach can be applied beyond SAC — We have added experiments showing an analogous approach for TD3 works similarly. Please see Part 3 of our shared reply for more details. If you're referring instead to adding residual weight or the minimum-of-$N$-critics trick to IQL, it is possible that they would improve its scores. Nonetheless, we chose to use TD3+BC and SAC-$N$ for their simplicity. Both approaches take a well-tested online algorithm and add a single source of pessimism. In contrast, IQL already has two different sources of pessimism (expectile regression and AWR), each with its own pessimism hyperparameter.

About the pessimism settings in Appendix.A, when N=10, this paper chooses $\eta =0\mathrm{\%}$ on half of tasks, but this choice means an actual equivalence with SAC-10, so I doubt whether the proposed method will work in this case.

We agree that $\eta =0\mathrm{\%}$ means an equivalence with SAC-10, as noted in Section 3. To clarify, our proposed method is to tune $\eta$ per task, including tasks where $\eta =0\mathrm{\%}$ (i.e., SAC-10) performs best. In other words, SAC-10 is part of our proposed method Raisin. We agree the need to tune $\eta$ per task is a potential weakness. In the Related Work section, we mention a few approaches that could eliminate the need to tune $\eta$ per dataset. As we mention in the paper, ultimately replacing SG entirely (for more robust convergence) may be ideal, but Raisin already gives SOTA scores even where the optimal setting is equivalent to SAC-10 (i.e., SG).

Technical Novelty And Significance: 2: The contributions are only marginally significant or novel. Empirical Novelty And Significance: 2: The contributions are only marginally significant or novel.

Please see Part 1 of our shared reply, "Novelty." To our knowledge, RA has strictly been proposed and tested as an online algorithm. The only use of deep RA we are aware of is the RA-DDPG work (Zhang et al. 2019), where even the highest improvement in AUC over all tasks is a factor of about 3$\times$. In contrast, we have discovered that RA works disproportionately well in the offline setting (for which it was not designed), giving a median 54$\times$ improvement in score over all tasks. Moreover, we find Raisin to be the best of multiple possible ways to combine RA and SAC-$N$, and that this combination performs better than the sum of its parts. We also find that tuning the residual weight differently for every dataset is critical.

Please also see Parts 3 and 4 of our shared reply, where we discuss new experiments we've added.

References

Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep Residual Reinforcement Learning. arXiv, May 2019. doi: 10.48550/arXiv.1905.01072.

Thanks again for reviewing! In light of our replies and additions, might you consider increasing your score?

Thanks a ton for the review! To summarize the strengths you point out:

• We give extensive experiments to show that Raisin is computationally efficient and scores well on a diverse range of datasets
• The paper is clear and easy to follow, includes a detailed discussion of SG vs. RG, and experiments from different angles
• The novelty is "good", and the reproducibility is "okay" even before we release the code

Most of this paper’s efforts are on the experimental comparison of SG and RA SAC algorithms. And the difference between them is adding the gradient of the next states into the loss function. Though these two approaches have significant differences, the RA approach has already been investigated by previous researchers, as mentioned in the RA-DDPG work.

Please see Part 1 of our shared reply, "Novelty." To our knowledge, RA has strictly been proposed and tested as an online algorithm. Additionally, the only use of deep RA we are aware of is indeed the RA-DDPG work (Zhang et al. 2019), where even the highest improvement in AUC over all tasks is a factor of about 3$\times$. In contrast, we have discovered that RA works disproportionately well in the offline setting (for which it was not designed), giving a median 54$\times$ improvement in score over all tasks. Moreover, we find Raisin to be the best of multiple possible ways to combine RA and SAC-$N$, and that this combination performs better than the sum of its parts. We also find that tuning the residual weight differently for every dataset is critical for performance.

In the baseline comparison, the author should also involve past work that extended RA to DDPG to prove their models' effectiveness and novelty. Since in the introduction, the author talked in detail and claimed RA is in nature beneficial to offline learning reinforcement learning, they should provide more information to prove it, not just compare between SAC methods. [...] The author conducted many experiments to support their claims, but in order to prove that RG could outperform SG, more investigations, such as different agents or tasks, should be involved.

We've added new experiments with a Raisin analog for TD3 (which also show that even an improved version of the prior work Bi-Res-DDPG does not score very well). Please see Part 3 of our shared reply.

This paper aims to prove that RA-SAC is more versatile than the SG counterparts, and compared to traditional SAC, it needs fewer critics, thus significantly reducing the computing cost. However, as introduced in section 1, the RA method is slower to converge. So the author should also provide convergence curve comparison, not just RA-SAC’s.

While slow convergence indeed seems to arise if using only a single critic and initialization that is too high (as shown in Figure 2 of our paper), with our proposed algorithm ($N=10$ and a standard near-zero initialization), we do not see this slow convergence empirically. The original SAC-N paper (which uses SG) requires 3M gradient steps to converge, which Raisin matches.

However, some of the propositions are not clearly explained. For example, on page 2, the theoretical concerns are important but not mentioned though it seems this paper is focusing on experiments.

Thank you for this comment. We've now clarified what those theoretical concerns are (and some counterpoints). Please let us know if there are other propositions you'd like us to explain more clearly.

Reproducibility: the technical description is okay and not in detail. The producibility could be increased if the author releases their codebase.

We present the pseudocode in Appendix B. We will also release clean code upon acceptance.

References

Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep Residual Reinforcement Learning. arXiv, May 2019. doi: 10.48550/arXiv.1905.01072.