[HOW TO] Lollipop DAT files | Compression and Decompression

Introduction
You probably know already that starting from Android 5.x (Lollipop) compiled roms (aosp,cm,stock) are not compressed anymore the way they used to be on previous android versions. On previous versions all content inside /system folder that has to be extracted within our device was either uncompressed (simple /system folder inside our flashable zip) or compressed in a system.img file, which it is a ext4 compressed file; both of these, anyway, were readable and we could see all system files (app,framework, etc).
The problem comes in 5.x versions, this method is not used anymore. Why? Because roms started to be always larger, so it is necessary to compress them even more.


What does Android 5.x zips (full roms, but also otas) contain?
Android 5.x flashable zips are made this way:

boot.img (we all know what it is)
file_contexts
META-INF (folder containing scripts)
system.new.dat (compressed /system partition)
system.patch.dat
system.transfer.list (see explanation below)

What does updater-script contains then?
The updater-script uses a brand new function: block_image_update(), this method basically decompresses necessary files inside the device. Let's study it.
From google git source code, if we go inside the new file /bootable/recovery/updater/blockimg.c, we find at the end of it the registration of the function block_image_update() as the method BlockImageUpdateFn() which starts at line 254. Here finally we find all information we need to know about the decompression of the .dat file(s). First file we analyze is system.transfer.list which Google tells us:

Quote:

The transfer list is a text file containing commands to transfer data from one place to another on the target partition.

But what each line means?:

Quote:

First line is the version number of the transfer list; 1 for android 5.0.x, 2 for android 5.1.x
Second line is the total number of blocks being written
Third line is how many stash entries are needed simultaneously (only on version >= 2)
Fourth line is the maximum number of blocks that will be stashed simultaneously (only on version >= 2)
Fifth line and subsequent lines are all individual transfer commands.

Ok, but how to decompress the system.new.dat file?
howellzhu and luxi78, two guys from chinese development forums managed to write some binaries to decompress/compress this file. Big thanks to them for initial sources which I've compiled from!


You can use/modify these files and/or include them in your work as long as proper credits and a link to this thread are given.

If you have questions or problems write here


Are you building from source and want to revert to the old flashing method without DAT files?
Do this (thx @Elluel):
1 - Delete this line
2 - Make sure you have this commit


Credits
howellzhu & luxi78 - for initial source codes