I’ve tried coding and every one I’ve tried fails unless really, really basic small functions like what you learn as a newbie compared to say 4o mini that can spit out more sensible stuff that works.
I’ve tried explanations and they just regurgitate sentences that can be irrelevant, wrong, or get stuck in a loop.
So. what can I actually use a small LLM for? Which ones? I ask because I have an old laptop and the GPU can’t really handle anything above 4B in a timely manner. 8B is about 1 t/s!
for coding tasks you need web search and RAG. It’s not the size of the model that matters, since even the largest models find solutions online.
Any suggestions for solutions?
Learning/practice, and any use that feeds in sensitive data you want to keep on-prem.
Unless you’re set to retire within the next 5 years, the best reason is to keep your resume up to date with some hands-on experience. With the way they’re trying to shove AI into every possible application, there will be few (if any) industries untouched. If you don’t start now, you’re going to be playing catch up in a few years.
As cool and neato as I find AI to be, I haven’t really found a good use case for it in the selfhosting/homelabbing arena. Most of my equipment is ancient and lacking the GPU necessary to drive that bus.
Have you tried RAG? I believe that they are actually pretty good for searching and compiling content from RAG.
So in theory you could have it connect to all of you local documents and use it for quick questions. Or maybe connected to your signal/whatsapp/sms chat history to ask questions about past conversations
No, what is it? How do I try it?
RAG is basically like telling an LLM “look here for more info before you answer” so it can check out local documents to give an answer that is more relevant to you.
You just search “open web ui rag” and find plenty kf explanations and tutorials
It’ll work for quick bash scripts and one-off things like that. But there’s not usually enough context window unless you’re using a 24G GPU or such.
Yeah shell scripts are one of those things that you never remember how to do something and have to always look it up!
Snippets are a great use.
I use StableCode on my phone as a programming tutor for learning Python. It is outstanding in both speed and in accuracy for this task. I have it generate definitions which I copy and paste into Anki the flashcard app. Whenever I’m on a bus or airplane I just start studying. Wish that it could also quiz me interactively.
Please be very careful. The python code it’ll spit out will most likely be outdated, not work as well as it should (the code isn’t “thought out” as if a human did it.
If you want to learn, dive it, set yourself tasks, get stuck, and f around.
I know what you mean. All the code generated with ai was loaded with problems. Specifically it kept forcing my api keys into the code without using environmental variables. But for basic coding concepts it has so far been perfect. even a 3b model seemingly generates great definitions
Currently I’ve been using a local AI (a couple different kinds) to first - take the audio from a Twitch stream; so that I have context about the conversation, convert it to text, and then use a second AI; an LLM fed the first AIs translation + twitch chat and store ‘facts’ about specific users so that they can be referenced quickly for a streamer who has ADHD in order to be more personable.
That way, the guy can ask User X how their mothers surgery went. Or he can remember that User K has a birthday coming up. Or remember that User G’s son just got a PS5 for Christmas, and wants a specific game.
It allows him to be more personable because he has issues remembering details about his users. It’s still kind of a big alpha test at the moment, because we don’t know the best way to display the ‘data’, but it functions as an aid.
Surely none of that uses a small LLM <= 3B?
Yes. The small LLM isn’t retrieving data, it’s just understanding context of text enough to know what “Facts” need to be written to a file. I’m using the publicly released Deepseek models from a couple of months ago.
Some questions and because you don’t actually understand, also, the answers.
- what does the LLM understand the context of, (other user’s data owned by Twitch)
- How is the LLM fed that data? (You store it and feed it to the LLM)
- Do you use Twitch’s data and its users data through an AI without their consent? (Most likely, yes)
- Do you have consent from the users to store ‘facts’ about them (You’re pissy, so obviously not)
- Are you then storing that processed data? (Yes, you are, written to a file)
- Is the purpose this data processing commercial (Yes, it is, designed to increase viewer count for the user of this system - and before you retort “OMG it helps twitch too”… Uhm no, Twitch has the viewers if not watching him, watching someone else)
I mean yeah, it’s a use case, but own up to the fact that you’re wrong. Or be pissy. I don’t care.
So this wasn’t a post actually asking what a small LLM was good for, it was just an opportunity you could use to dump on LLM usage I take it. So this whole thing was made in bad faith?
With the comments about “vibe coding” and such, all it looks like you’re doing here is arguing the “merits” of how it’s being used, and you’re not interested in its actual usage at all.
Nobody is being pissy here except you. Small LLMs can be used for tasks such as this, and it doesn’t have to be twitch - It could be an assistant that you build for reminders in your personal life - using it on twitch is a minor detail that you seem to have latched onto because you just want to dump on LLM usage.
Go to /c/fuck_ai for that.
I gave you an example that it’s good for, and all you want to do is argue the merits of how I’m using it (even though it falls perfectly within Twitches TOS and use cases)
You’re conflating me asking how to use these tools with you who’s misusing them. I see you still don’t accept what you’re doing is wrong. But go you.
