• MalReynolds@slrpnk.net
    link
    fedilink
    English
    arrow-up
    6
    arrow-down
    2
    ·
    8 months ago

    Cheap fuckers cheaping out, shocker (context is (V)RAM). AI speedrunning enshittification, who’d of thunk.

    • pixxelkick@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      arrow-down
      1
      ·
      8 months ago

      Uh… no its just the free models being free, theyre lower cost intentionally to provide free options for people who dont wanna pay subscription fees.

      (context is (V)RAM)

      Eh sort of, its more operating costs, the larger the context size the more expensive the model is to run, literally in terms of power consumption.

      Keep in mind we are on the scale of fractions of cents here, but multiply that by millions of users and it adds up fast.

      But the end result is that the agent will fuck stuff up, and will even quickly /forget/ it fucked that up if you dont catch it asap

      A lot of them have a context window that can be wiped out within like, 2 minutes of steady busywork…

        • pixxelkick@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          1
          ·
          8 months ago

          What are you talking about.

          No? I never said that.

          I just explained /why/ it happened, I literally nowhere in my post said, or implied, someone should pay for more expensive models. What are you smoking?

          You just have to be aware they have very short memory when using a cheap model and assume anything you wrote 1 minute ago has already left its memory, which is why they produce pretty dumb output if you try and depend on that… so… dont depend on that.

            • pixxelkick@lemmy.world
              link
              fedilink
              English
              arrow-up
              1
              ·
              8 months ago

              You: just the cheap ones

              I never said that. I just said that the cheap ones are especially shitty.

              People on this site really lack reading comprehension it seems.

                • pixxelkick@lemmy.world
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  8 months ago

                  The only people who have these issues, are people who are using the tools wrong or poorly.

                  Using these models in a modern tooling context is perfectly reasonable, going beyond just guard rails and instead outright only giving them explicit access to approved operations in a proper sandbox.

                  Unfortunately that takes effort and know-how, skill, and understanding how these tools work.

                  And unfortunately a lot of people are lazy and stupid, and take the “easy” way out and then (deservedly) get burned for it.

                  But I would say, yes, there are safe ways yo grant an llm “access” to data in a way where it does not even have the ability to muck it up.

                  My typical approach is keeping it sandbox’d inside a docker environment, where even if it goes off the rails and deletes something important, the worst it can do is cause its docker instance to crash.

                  And then setting up via MCP tooling that commands and actions it can prefer are explicit opt in whitelist. It can only run commands I give it access to.

                  Example: I grant my LLMs access to git commit and status, but not rebase or checkout.

                  Thus it can only commit stuff forward, but it cant even change branches, rebase, nor push either.

                  This isnt hard imo, but too many people just yolo it and raw dawg an LLM on their machine like a fuckin idiot.

                  These people are playing with fire imo.