I didn’t say it wasn’t, the sheer amount of amazing models that have come out of beaverAI and theDrummer and more is testament to that. It’s not niche though lol.
How about building the skills and know-how to run models locally?
APIs are only cheap now because they're heavily subsidized. The moment the free money dries up, expect API costs to skyrocket similarly to how hardware prices have. Thing is, even if you can access hardware at reasonable prices, you'll still need the know-how of how to build a good machine that can run larger models for a decent price and how to setup the software stack to run those models.
You see it on this sub all the time, people throwing a ton of money on consumer hardware and then hitting wall after wall with compatibility or bottlenecks despite spending a pretty penny. I'm sure in ten years we'll have low cost turnkey inference solutions, but in the meantime, we'll have to learn how to build balanced systems depending the hardware we can find.
Unlike what scam Altman / Elon cuck says , medical and genetic data can be used to optimise answers locally in a secure and safe environment using medgemma.
Could I ask what your workflow looks like when working with genetic data? I’ve never thought of that! Might make that DNA test I did a while back more useful that telling me I might be lactose intolerant.
Genetic not DNA sequences, aka genetic heritage for me. It means when I’m trying to debug my own body for fitness and health the information is optimised for me not the general public.
Actually understanding the technology that is - for better or worse - one of the major shifts for the short and long term future of our species. And you only learn to understand it with a hands on approach.
What do you need to pay to get the performance of a subscription locally? In other words, how much do you have to spend upfront to run a SOTA open model like GLM-5 at good speeds (and decent precision level)?
here's a "besides privacy and porn": censorship. i don't want my coding model to sass me because it thinks i'm writing malware. fully managed cloud models are always going to have Some Bullshit.
scenarios where you control the entire software stack and are just paying for someone else to run it less so, but there's a lot of overlap between the skills you need to do that and the skills you need to run local anyway
Privacy is already so nice, you can copy paste your api keys, passwords and whatever into the chat without having to worry about it (respecting least privilege ofc).
Latency and bandwidth sensitive tasks can benefit a lot. You can use something like Frigate that sends things to a multimodal model for image classification which can set off a local automation. If the process were totally cloud based then you are talking about several hops back and forth to a data center that can degrade user experience.
Because eventually, when everyone is hooked and completely dependent on the cloud models, they will ramp up the price 10-100x to recoup their investments and capex.
“Besides privacy?” excellent summation of our entire digital experience right now.
Porn is a perfectly rewarding way to learn how to use AI models and gain understanding of prompting.
I mean Google images was not developed to look up images of cats
I didn’t say it wasn’t, the sheer amount of amazing models that have come out of beaverAI and theDrummer and more is testament to that. It’s not niche though lol.
Doomsday scenario, bad internet, internet outage, impractical internet cost, safeguarding trade secret etc.
How about building the skills and know-how to run models locally?
APIs are only cheap now because they're heavily subsidized. The moment the free money dries up, expect API costs to skyrocket similarly to how hardware prices have. Thing is, even if you can access hardware at reasonable prices, you'll still need the know-how of how to build a good machine that can run larger models for a decent price and how to setup the software stack to run those models.
You see it on this sub all the time, people throwing a ton of money on consumer hardware and then hitting wall after wall with compatibility or bottlenecks despite spending a pretty penny. I'm sure in ten years we'll have low cost turnkey inference solutions, but in the meantime, we'll have to learn how to build balanced systems depending the hardware we can find.
Unlike what scam Altman / Elon cuck says , medical and genetic data can be used to optimise answers locally in a secure and safe environment using medgemma.
Could I ask what your workflow looks like when working with genetic data? I’ve never thought of that! Might make that DNA test I did a while back more useful that telling me I might be lactose intolerant.
Genetic not DNA sequences, aka genetic heritage for me. It means when I’m trying to debug my own body for fitness and health the information is optimised for me not the general public.
I can run them with solar power :)
Ooh fancy…
To just mess around :)
for summarization, basic VLM capabilities, ocr, etc local is easily much cheaper to run. esp at scale.
Actually understanding the technology that is - for better or worse - one of the major shifts for the short and long term future of our species. And you only learn to understand it with a hands on approach.
If the only reason was privacy, it would already be worth every penny invested.
I added a token counter feature to my code completion plugin.
The other day, I used 1'000'000 tokens in little under an hour and I didn't even use a single suggestion.
It didn't cost me a dime.
Cost? Flexibility? Ownership? Fitting the theme of the subreddit?
How about avoiding surprise API bills?
imagine the porno ai could generate. im very interested <3
an adventure where you meet alien life forms and hook them up to dildo machines. yay !! good use of my time.
definitely dont want that leaking on the internet. privacy.
Average localllama enjoyer
Gotta know what you want out of ai... I mean life
cost, the cost of local use is far far less than subscriptions.
What do you need to pay to get the performance of a subscription locally?
In other words, how much do you have to spend upfront to run a SOTA open model like GLM-5 at good speeds (and decent precision level)?
here's a "besides privacy and porn": censorship. i don't want my coding model to sass me because it thinks i'm writing malware. fully managed cloud models are always going to have Some Bullshit.
scenarios where you control the entire software stack and are just paying for someone else to run it less so, but there's a lot of overlap between the skills you need to do that and the skills you need to run local anyway
porn?
My own edification and planning for unreliable internet
Privacy is already so nice, you can copy paste your api keys, passwords and whatever into the chat without having to worry about it (respecting least privilege ofc).
Cost (if you don’t buy hardware to do it)
Reliability of access
Not giving my todo-list to a third party
"Privacy" also applies to a ton of business and client confidentiality that cloud models may not work for. That's not a small thing.
Latency and bandwidth sensitive tasks can benefit a lot. You can use something like Frigate that sends things to a multimodal model for image classification which can set off a local automation. If the process were totally cloud based then you are talking about several hops back and forth to a data center that can degrade user experience.
Because eventually, when everyone is hooked and completely dependent on the cloud models, they will ramp up the price 10-100x to recoup their investments and capex.