// HACKER NEWS — CYBERSECURITY
Show HN: TERMy – A fast terminal assistant that does not use LLMs
I love research and development, you may have heard of me because of PJON (Padded Jittering Operative Network). It is a network protocol I started developing in 2010, which was recently implemented in silicon by the ETH Zurich university thanks to the research of Pius Sieber.
I had a chance to focus for 2 months on my personal projects since early July, during the strange times of AI price hikes and the end of subsidized tokenmaxing. I was curious to see if I could develop from scratch a terminal assistant capable of handling simple natural language requests. I have a bad memory and got used to ask to copilot "activate the virtual environment" or similar trivial operations spending a non negligible sum every month. I started thinking, maybe I can do something to make my workflow more efficient? Do I really need trillions of parameters to accomplish those tasks?
I started an open-ended research on the feasibility of implementing a generative model at home and training it from scratch on the computer I used to play Kerbal Space Program in the early 2010s "upgraded" with 16GB of RAM, NVIDIA GTX 1050 Ti (4GB VRAM) and a i7-4790K (4.0GHz 8 cores) CPU. In my experiments I tend to look for minimalism, so I imposed myself a constrained environment to be forced to work towards an elegant and efficient solution.
I first developed a framework to train and evaluate transformers, which I implemented from scratch in Python. I have started with something very similar to NanoGPT with 100-200M parameters, then I added flash attention, and all the expected optimizations, I even tried novel architectures like Mamba. The results were generally unsatisfactory, creepy if not outright scary, like the following:
(the word excrements was used instead of an expletive composed of 4 letters that I am not willing to publish in here)
All my tests produced models that were prone to enter in loops repeating the same sentence, and even if trained on QA they were rarely able to consistently answer questions, specially if technical. I trained models on a collection of royalty free books from the Project Gutemberg, a lot of open-source software, and various datasets available on Huggingface.
I quickly understood that this approach was not feasible; a proper run would have required at least a month of training non-stop. I was amazed by how my models looked alive and magical, but I was also ashamed because they were incredibly wasteful and effectively useless.
I pivoted to ollama and open-weight models and developed howto, yet another terminal harness that uses a pre-prompt to force the model to answer only with terminal commands. Results were generally unsatisfactory because of the time required to get a response. Models like ornith:9b, mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM.
Then I remembered about the blockchain craze, when everyone wanted to fit a blockchain somewhere and sell it as the next big thing. I didn't want to waste my time and money like all those people did in the previous hype cycle, so I started building a terminal assistant from scratch with a new set of constraints:
The first things I needed was a set of conventions to rely on, so I drafted the NDF 0.0 (NPC-Forge Dataset Format which specifies the dataset format of NPC-Forge. The following object contains category, input sentences, textual response, thinking traces, permission gating and tool calls to be executed in a format compatible with VS code.