Language Models Can Rewrite Their Own Weights

(arxiv.org)

50 points | by lwhsiao 49 minutes ago

8 comments

  • gavinray 6 minutes ago
    Didn't Google's "sleep" paper already show this?

    https://research.google/blog/the-sleep-framework-a-new-way-f...

      > During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
  • vlovich123 0 minutes ago
    Isn’t this the same paper from before where it found a single specific bug in its own weights and fixed that infinite loop it would sometimes hit, a bug that was findable without the LLM? Experiments on other LLM weights failed to find any improvements if I recall correctly. Haven’t read this paper but the intro sounded familiar
  • xhrpost 17 minutes ago
    >We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training

    Any increase in fidelity must come from somewhere...

    >t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.

    I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.

    • p-e-w 6 minutes ago
      The “test material” can simply be interactions with the environment. It’s just learning.
    • Jerrrrrrrrry 10 minutes ago
      negentropy
  • pdntspa 12 minutes ago
    Reminds me of Westworld season 1
  • test6554 14 minutes ago
    Well, it was nice knowing you all.
  • zodwick 29 minutes ago
    Not the RSI we wanted, bu the RSI we deserve
  • esafak 21 minutes ago
    > We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.

    "Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??

    You can't throw rockets up in the air without worrying where they are going to land. It is your department.

    • jrflo 5 minutes ago
      If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

      That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.

    • snowram 15 minutes ago
      It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.

      Until it doesn't, then comes responsability shifting.

    • lenerdenator 19 minutes ago
      > What do you think is going to happen when robots can modify themselves like this without any oversight??

      This is a field heavily focused on shifting liability away from itself. They don't think about that at all.

    • thaumasiotes 9 minutes ago
      > You can't throw rockets up in the air without worrying where they are going to land.

      It works for firing off celebratory bullets.

  • tsunamifury 7 minutes ago
    I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.

    The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.