I hope that through the introduction of this article, I can give some inspiration and help to people who don't know much about lzw algorithm and its application in gif images, but are eager to know. Just to attract jade, I hope the brothers in the garden will put forward valuable opinions.
What is the full name of 1? LZW？
Lempel-Ziff-Welch (LZW).
2. What is the introduction and compression principle of 2.LZW?
LZW compression algorithm is a novel compression method, which was founded by Lempel-Ziff-Welch and named after them. It adopts advanced string table compression, puts each first string in a string table and uses a number to represent the string. The compressed file only stores numbers, not strings, which greatly improves the compression efficiency of image files. Miraculously, this string table can be established correctly during compression or decompression, and then discarded after compression or decompression.
In LZW algorithm, firstly, a string table is established, and each string that appears for the first time is put into the string table and represented by a number, which is related to the position of the string in the string table and stored in a compressed file. If the string appears again, it can be replaced by the number representing it, which is stored in the file. Discard the string table after compression. For example, if the "print" string is represented by 266 when it is compressed, it will be represented by 266 every time it appears again, and the "print" string will be stored in the string table. When the number 266 is encountered in the image decoding process, the character string "print" represented by 266 can be found from the character string table, and the character string table can be regenerated according to the compressed data during decompression.
3. Before introducing the algorithm in detail, list some concepts and vocabulary related to the algorithm.
1)' character': a character, a basic data element, occupies 1 individual bytes in an ordinary text file, but in an image, it is an index value representing the color of a given pixel.
2) "charstream": the character stream in the data file.
3) "Prefix": Prefix. Like the meaning of this word, it represents the most direct character before a character. The length of prefix character can be 0, and prefix and character can form a string.
4)' Suffix': Suffix, which is a character. A string can be composed of (a, b). A is a prefix and b is a suffix. When the length of a is 0, it stands for Root, root.
5)' code: code, which is used to indicate the position coding of a string.
6) "Item", a code and the string it represents.
4. The simple example of compression algorithm is not to completely realize lzw algorithm, but to look at the idea of LZW algorithm from the most intuitive point of view.
LZW compressed raw data ABCCAABCDDAACCDB.
The original data only contains four characters (a, b, c, d), which can be represented by a 2-2 digit, 0-A, 1-B, 2-C, 3-d. From the most intuitive point of view, there are repeated characters in the original string: ABCCAABCDDAACCDB, 4 stands for AB, 4.
Scope of application of 5.5. lempel ziv welch encoding
In order to distinguish the value (code) representing a string from the original single data value (string), it is necessary to make their numerical fields not coincide, and use 0-3 to represent A-D, so AB must be replaced by a numerical value greater than 3. For another example, the original numerical range can be represented by 8 bits, so it is considered that the original numerical range is 0 ~ 255, and the label range generated by the compression program cannot be 0 ~. We can only start with 256, but this will go beyond the range of 8-bit representation, so we must expand the number of data bits by at least one bit, but doesn't this increase the space occupied by 1 character? However, one character can represent several characters. For example, 255 used to be 8 digits, but now it is worthwhile to use 256 to represent the two numbers 254 and 255. From this principle, it can be seen that the application scope of LZW algorithm is that the original data string should have a large number of repeated substrings, and the more repetitions, the better the compression effect. On the contrary, the worse it is, the more it may actually increase rather than decrease.
Special marks in 6.6. lempel ziv welch encoding
As new strings are discovered, tags will continue to grow. If the original data is too large, the generated label table will become larger and larger. Operating this system at this time will cause efficiency problems. How to avoid this problem? The way Gif uses lzw algorithm is that when the tag set is large enough, it obviously won't increase, so it just needs to start from scratch. At this position, a label is inserted, which means that I start to construct the dictionary again. All the previous labels are invalid and I start to use a new one.
At this time, another problem appeared. How big is it? What is the appropriate size for this tag set? Theoretically, the larger the label set, the higher the compression ratio, but the higher the overhead. Generally according to the processing speed and memory space to choose. GIF specification stipulates that 12 bit, if the expression range exceeds 12 bit, it will be reinvented. In order to improve the compression ratio, GIF uses variable word length. For example, if the original data is 8 bits, add 1 bit at the beginning, and the initial word length will become 9 bits, and then start labeling. When the tag is added to 5 12, that is, when it is greater than 9, it means that the following tags can only be represented by 10 bits, so from here on, the following word length is 10. And so on, to 2 12, that is, 4096, insert a clear number here, starting from the back and starting from 9 digits.
The value of the CLEAR flag specified by GIF is the maximum value of the original data word length plus 1. If the original data word length is 8, then the clear flag is 256, and if the original data word length is 4, then it is 16. In addition, GIF also specifies an END flag, whose value is the CLEAR flag plus 1. Because the number of bits specified in GIF is 1 (monochrome image), 4 (16 color) and 8 (256 color), if only 1 bit is extended to represent four states in the case of 1 bit, then adding a clear flag and an end flag will be used up, so/kloc. In the other two cases, the initial word length is 5 bits and 9 bits respectively. The pressure routine is referenced here.
//
//Lempel-Ziff compression based on "compression". GIF modifier
//David Rowley (mgardi@watdcsu.waterloo.edu)
//General definition
Static read-only int bit =12;
Static read-only int HSIZE = 5003// 80% occupancy rate
// GIF image compression-modified "compression"
//
//Based on: compress.c file compression ala IEEE computer, June 1984.
//
//Author: Spencer W. Thomas (decvax! Harper. Utah -cs! Utah -gr Thomas)
//Jim McKee (decvax! Mcvax! Jim)
//Steve Davies (decvax! vax 135！ petsd！ Peola. srd)
// Ken Turkowski (decvax！ decwrl！ turtlevax！ Ken)
//James A. Woods (decvax! ihnp4！ Aymis. Jaw)
// Joe Orost (decvax！ vax 135！ petsd！ Joe)
Int n _ bits// number of digits/number of codes
Int maxbits = BITS// Maximum digits/code that can be set by users.
Int maxcode// the maximum code for a given n_bits.
int maxmax code = 1 & lt； & lt bit; //Never generate this code.
int[]htab = new int[HSIZE]； //This is a package for hashing. You can find 1 key in it soon.
int[]codetab = new int[HSIZE]；
Int hsize = HSIZE// Used to dynamically resize the table.
int free _ ent = 0； //The first unused entry
//Block compression parameters-After all the codes are used up,
//and the compression ratio changes and starts again.
bool clear _ flg = false
//algorithm: in
//prefix code/next character combination. We made a variant of Knuth.
//Algorithm D (Volume 3, sec. 6.4) Together with the counterpart of G. Knodt-Prime Minister
//auxiliary detector. Here, the first exploration of modularity is to make way.
//To a faster XOR operation. Block compression is also performed using.
//Adaptive reset to clear the code table during compression.
The//ratio decreases, but after the table is filled. Variable length output
//The code will be resized and a special cleanup code will be generated.
//Used for decompressor. Post-addition: Build a table according to the following contents
//File size can significantly improve the speed of small files. Please instruct.
//Give the question about this implementation to ames! Chin.
int g _ init _ bits
int ClearCode
int EOFCode
//output
//
//Output the given code.
//Input:
// code: an integer with n_bits. If ==-1, then EOF. This hypothesis
//Then n _ bits = & ltwordsize-1.
//Output:
//Output the code to a file.
//Suppose:
//The character length is 8 bits.
//algorithm:
//Maintain a buffer with a length of binary characters (so that 8 codes will
//just fits it). Insert each using the VAXINV instruction.
//Code in turn. When the buffer is full, empty it and start over.
int cur _ accum = 0；
int cur _ bits = 0；
Int [] mask =
{
0x0000，
0x000 1，
0x0003，
0x0007，
0x000F，
0x00 1F，
0x003F，
0x007F，
0x00FF，
0x0 1FF，
0x03FF，
0x07FF，
0x0FFF，
0x 1FFF，
0x3FFF，
0x7FFF，
0x ffff }；
//The number of characters in this "packet" so far.
int a _ count
//Define the storage of the packet accumulator.
Byte[] accum = new byte [256];
// -
public LZWEncoder(int width，int height，byte[] pixels，int color_depth)
{
ImgW = width;
ImgH = height;
Pixel = pixel;
initCodeSize = Math。 Max(2，color _ depth)；
}
//Add a character at the end of the current packet, if it is 254.
//character to refresh the packet to disk.
Voideadd (byte c, stream output)
{
accum[a _ count++]= c；
if(a _ count & gt； = 254)
Flushing (outflow);
}
//Clear the hash table
//Clear the block compressed table
Void ClearTable (stream output)
{
ResetCodeTable(hsize)；
free _ ent = clear code+2；
clear _ flg = true
Output (clear code, output);
}
//Reset the code table
//All initialized to-1
void ResetCodeTable(int hsize)
{
for(int I = 0； I & lthsize++i)
htab =- 1；
}
void Compress(int init_bits，Stream outs)
{
int fcode
int I/* = 0 */；
int c；
int ent
int disp
int hsize _ reg
int hshift
//Set global variable: g_init_bits-initial number of bits.
//The word length of the original data. In a gif file, the word length of the original data can be 1 (monochrome image), 4( 16 color) and 8(256 color).
//Add 1 at the beginning.
//But when the original data length is 1, it starts from 3.
//So the original length is1->; 3,4-> 5,8-> 9
//? The word length of the original data is 1. Why is the starting length 3? ?
//If+1=2, only four states can be represented, and the clearcode and endcode will be used up. So it must be extended to 3.
g _ init _ bits = init _ bits
//Set the necessary values
//Do you need to add a clear mark?
//GIF In order to improve compression ratio, variable word length (VCL) is adopted. For example, if the original data is 8 bits, start with 1 bit (8+ 1=9).
//When the label reaches 2 9 = 512, which exceeds the maximum value that can be represented by the current length of 9, then the following labels must be represented by 10.
//By analogy, when the label reaches 2 12, it cannot be expanded any more, because the maximum value is 12. You need to insert a clear code in the position of 2 12 = 4096, which means that from now on, you have to start from 9 digits.
clear _ flg = false
n _ bits = g _ init _ bits
//Get the maximum value that can be represented by n digits (in a gif picture, it usually starts with 3, 5 and 9, so the maxcode is usually 7,365,438+0,565,438+065,438+0).
max code = max code(n _ bits)；
//means that I have started to construct a dictionary from here, and the previous tags are invalid.
//Start using a new tag. What is the appropriate size for this tag set? It is said that in theory, the greater the compression ratio, the higher the compression ratio (personally, it is not necessarily good to feel too big).
//However, the processing cost has also increased exponentially.
//gif specifies that the value of clearcode is the value+1that can be represented by the maximum word length of the original data; For example, if the original data length is 8, then clearcode =1< <; (9- 1)=256
clear code = 1 & lt； & lt(init _ bits- 1)；
//The end flag is clearcode+ 1.
eof code = clear code+ 1；
//Cancel this end.
free _ ent = clear code+2；
//Quantity cleared
a _ count = 0； //Clear the packet
//Get the next pixel from the image
ent = next pixel()；
h shift = 0；
for(fcode = hsize； fcode & lt65536; fcode *= 2)
++ h shift；
//Set the hash code range
h shift = 8-h shift； //Set the hash code range limit
hsize _ reg = hsize
//Clear the fixed-size hash table that stores labels, which is equivalent to a dictionary.
ResetCodeTable(hsize _ reg)； //Clear the hash table
Output (clear code, output);
outer_loop : while ((c = NextPixel())！ = EOF)
{
fcode =(c & lt； & ltmaxbits)+ent；
I =(c & lt； < Otolaryngology; // xor hash
//Hey hey, sample, here we go again, I know you.
if (htab == fcode)
{
ent = codetab
Continue;
}
//This boy is new here.
else if(htab & gt； = 0) // Non-empty slot
{
disp = hsize _ reg-I； //Secondary hash (after G. Knodt)
If (i == 0)
disp = 1；
do
{
if((I-= disp)& lt； 0)
I+= hsize _ reg；
if (htab == fcode)
{
ent = codetab
goto outer _ loop
}
} while(htab & gt； = 0);
}
Output (ent, outs);
//It can be seen that ent is a prefix and the character symbol currently being processed is a suffix.
ent = c；
//Judge whether the terminator is beyond the range that can be represented by the current digits.
if(free _ ent & lt； maxmaxcode)
{
//If there is no super
codetab = free _ ent++； //code-& gt； HashTable
//Establish the corresponding index in the//hash table.
htab = fcode
}
other
//The description is beyond the range that can be expressed at present. Empty the dictionary and start over.
Clearable (out);
}
//Output the final code.
Output (ent, outs);
Outs (EOFCode);
}
// -
Common space coding (streaming operating system)
{
os。 WriteByte (conversion. ToByte(init codesize))； //Write "Initial Code Size" bytes
//How many pixels does this image contain?
Remaining = imgW * imgH// Reset navigation variable.
//The currently processed pixel index
curPixel = 0；
Compress(initCodeSize + 1，OS)； //compress and write pixel data
os。 write byte(0)； //Write block terminator
}
//Refresh the packet to disk and reset the accumulator.
Invalid flushing (outflow)
{
if(a _ count & gt； 0)
{
Out. WriteByte (conversion. ToByte(a _ count))；
Out. Write(accum，0，a _ count)；
a _ count = 0；
}
}
///& lt； Summary & gt
///Get the maximum value that can be represented by n digits.
///& lt； /summary & gt；
///& lt； Param name = "n _ bits"> digits, generally n _ bits = 9.
///& lt； The maximum value of returns & gt, for example, n_bits=8, then the return value is 2 8-1= 255.
int MaxCode(int n_bits)
{
return( 1 & lt； & ltn _ bits)- 1；
}
// -
//Returns the next pixel of the image.
// -
///& lt； Summary & gt
///Get the next pixel from the image
///& lt； /summary & gt；
///& lt； returns & gt& lt/returns & gt；
private int NextPixel()
{
//How many pixels are left unprocessed?
//If not, return the end flag.
If (residual == 0)
Return to EOF
//Otherwise, process the next one and set the number of unprocessed pixels to-1.
-surplus;
//Pixel currently being processed
int temp = cur pixel+ 1；
//If the currently processed pixel is within the pixel range
if(temp & lt； pixAry。 GetUpperBound( 0))
{
//Next pixel
byte pix = pixAry[cur pixel++]；
Return pixel & amp0xff
}
Returns 0xff
}
///& lt； Summary & gt
///Output words to the output stream
///& lt； /summary & gt；
///& lt； Param name = "code"> words to output.
///& lt； Param name = "outs"> output stream
Void output (int code, stream output)
{
//Get the maximum flag value that the current flag bit can represent.
Current cumulative & = masks [cur _ bits];
if(cur _ bits & gt； 0)
cur _ accum | =(code & lt； & ltcur _ bits)；
other
//If the flag bit is 0, the current label is the input stream.
cur _ accum = code
//Maximum Chinese character length of current energy symbol (9-10-1-12-9-10. . . . . . . )
cur _ bits+= n _ bit；
//If the current maximum length is greater than 8
while(cur _ bits & gt； = 8)
{
//Output a byte to the stream.
Add ((bytes) (cur _ accum & amp0xff), outs
//Move the current label 8 bits to the right.
cur _ accum & gt& gt= 8;
cur _ bits-= 8；
}
//If the next entry is too large for the code size,
//Then increase, if possible.
if(free _ ent & gt； maxcode || clear_flg)
{
if (clear_flg)
{
max code = max code(n _ bits = g _ init _ bits)；
clear _ flg = false
}
other
{
++ n _ bits；
if (n_bits == maxbits)
maxcode = maxmaxcode
other
max code = max code(n _ bits)；
}
}
if (code == EOFCode)
{
//At EOF, write the rest of the buffer.
while(cur _ bits & gt； 0)
{
Add ((bytes) (cur _ accum & amp0xff), outs
cur _ accum & gt& gt= 8;
cur _ bits-= 8；
}
Flushing (outflow);
}
}
}
}