en English · auto-generated
Samsung Found a Way to Store AI in 0.1 Bits Per Weight #ai #llm #samsung English subtitles
Report a problem
Advertisement
English transcript
Samsung research just found a way to
store each weight of a language model in
a tenth of a bit, which sounds
impossible because a bit is already the
smallest unit of data a computer has,
but they did it. And with that trick,
they squeezed a 70 billion parameter
model that normally needs 138 GB of
memory in less than 2 GB, which is small
enough to fit on a cheap USB stick. So
let me explain how that actually works.
So this project is called Little Bit. So
the way it works is that a normal
quantization takes each weight and it
stores it in fewer bits, like four or
two. And once you go below one bit per
weight, there's nothing left to round.
And most previous methods completely
fall apart there. But Little Bit takes a
bit of a different route. It relies on
the fact that weight matrices inside an
LLM are full of redundancy, which
mathematicians describe as being low
rank. So that means that instead of
storing the big matrix, it approximates
it as the product of two much thinner
matrices. Then it binarizes those thin
matrices, so every value becomes either
plus one or minus one. And to recover
the information lost in that step, it
adds three small sets of 16 bit scales,
one for the rows, one for the columns,
and one learns how important each latent
dimension is. And on top of that,
there's a second residual path that
learns to correct the errors made by the
first one. And because the matrices are
so thin, the average storage per
original weight can drop to 0.55, 0.3,
or even 0.1 bits. And for context, at
0.3 bits on Llama 2, 7 billion parameter
model, the previous best sub one bit
method, STB LLM, completely collapsed to
a perplexity of around 1,800 while
Little Bit stayed at 12. And even at 0.1
bits, Little Bit did better than STB LLM
did at 0.7 bits. And in their second
paper, Little Bit 2, they improved their
method by fixing how training starts.
Little Bit 2 now uses what's called a
singular value decomposition, which is
the classic technique that splits a
matrix into its most important patterns.
And that's where the two thin matrices
come from. The problem is that those
patterns are spiky, meaning most of
their values sit close to zero. And when
you binarize them, anything near zero
flip its sign from positive to negative
at the slightest nudge. So little bit
too fixes this with a neat bit of math.
If you rotate both factor matrices by
the same orthogonal rotation, their
product stays exactly the same. So they
search for the rotation that pushes the
values away from zero and towards plus
or minus one using a method called joint
iterative quantization. And the
improvement is pretty significant. On
Llama 3, the 8 billion parameter model
at one bit per weight, perplexity
dropped from 16.3 to 11.5, which is a
big jump in quality at exactly the same
size. And the coolest part is that the
code for both papers is open source and
it's on GitHub. So if you have a GPU to
spare, you can actually train your own
compressed versions of models like
Llama, Qwen 3, or Gemma 3. Just keep in
mind that it's a research project under
a non-commercial license and there's no
Llama CPP or Ollama support yet. But if
someone builds a proper runtime for it,
this could be how we end up running
surprisingly large models on phones and
tiny edge devices. And if you want to
stay up to date with the latest in AI
research and engineering, be sure to
subscribe to the Better Stack channel.
store each weight of a language model in
a tenth of a bit, which sounds
impossible because a bit is already the
smallest unit of data a computer has,
but they did it. And with that trick,
they squeezed a 70 billion parameter
model that normally needs 138 GB of
memory in less than 2 GB, which is small
enough to fit on a cheap USB stick. So
let me explain how that actually works.
So this project is called Little Bit. So
the way it works is that a normal
quantization takes each weight and it
stores it in fewer bits, like four or
two. And once you go below one bit per
weight, there's nothing left to round.
And most previous methods completely
fall apart there. But Little Bit takes a
bit of a different route. It relies on
the fact that weight matrices inside an
LLM are full of redundancy, which
mathematicians describe as being low
rank. So that means that instead of
storing the big matrix, it approximates
it as the product of two much thinner
matrices. Then it binarizes those thin
matrices, so every value becomes either
plus one or minus one. And to recover
the information lost in that step, it
adds three small sets of 16 bit scales,
one for the rows, one for the columns,
and one learns how important each latent
dimension is. And on top of that,
there's a second residual path that
learns to correct the errors made by the
first one. And because the matrices are
so thin, the average storage per
original weight can drop to 0.55, 0.3,
or even 0.1 bits. And for context, at
0.3 bits on Llama 2, 7 billion parameter
model, the previous best sub one bit
method, STB LLM, completely collapsed to
a perplexity of around 1,800 while
Little Bit stayed at 12. And even at 0.1
bits, Little Bit did better than STB LLM
did at 0.7 bits. And in their second
paper, Little Bit 2, they improved their
method by fixing how training starts.
Little Bit 2 now uses what's called a
singular value decomposition, which is
the classic technique that splits a
matrix into its most important patterns.
And that's where the two thin matrices
come from. The problem is that those
patterns are spiky, meaning most of
their values sit close to zero. And when
you binarize them, anything near zero
flip its sign from positive to negative
at the slightest nudge. So little bit
too fixes this with a neat bit of math.
If you rotate both factor matrices by
the same orthogonal rotation, their
product stays exactly the same. So they
search for the rotation that pushes the
values away from zero and towards plus
or minus one using a method called joint
iterative quantization. And the
improvement is pretty significant. On
Llama 3, the 8 billion parameter model
at one bit per weight, perplexity
dropped from 16.3 to 11.5, which is a
big jump in quality at exactly the same
size. And the coolest part is that the
code for both papers is open source and
it's on GitHub. So if you have a GPU to
spare, you can actually train your own
compressed versions of models like
Llama, Qwen 3, or Gemma 3. Just keep in
mind that it's a research project under
a non-commercial license and there's no
Llama CPP or Ollama support yet. But if
someone builds a proper runtime for it,
this could be how we end up running
surprisingly large models on phones and
tiny edge devices. And if you want to
stay up to date with the latest in AI
research and engineering, be sure to
subscribe to the Better Stack channel.