How to extract APK files from IMG (Parsing Android firmware). Custom Android: making your own firmware from stock without digging into the sources How to open dat firmware file

10.11.2020

A detailed step by step guide to extract system.img and system.new.dat files to get the system dumps on windows. Latest 2017 working method!

What is a system.new.dat file?

It is a part of Android OS. It mainly consists of default system apps, wallpapers, ringtones etc. This file is used for extracting system dump of Android OS.

What is a system.img file?

It’s the Android file system, which means all the support files that Android needs, the applications, but also the framework, the Dalvik VM, initialization scripts and so on. It is also used for ROM development purposes.

Since it is not possible to extract these files on windows PC with a normal extractor tool. We are going to use Auto Tool v3.0. This is a special software made for extracting this type of files. Follow the steps given below and you will simply learn how to extract these files on Windows PC.

Table of Contents

Prerequisites

Download and install the Java SE Development Kit:
Download and install Python 3.6.x and Python 2.7.x:

Note: While installing Python 3.6.x don’t forget to check the box at the bottom which says Add Python 3.6 to PATH.

Note: When installing Python 2.7.x, Go to Advanced option > Check the box says Compile .py files to bytecode after installation.

Necessary Downloads

We are going to need the below tools, download them:

Download the Auto Tool 3.0:
Download and install 7zip extractor tool:

Steps to Extract System.new.dat file on windows

Step 1

Step 2. Run the Auto Tool v3.0.exe

Step 3. Now type 1 and hit enter for adding the syatem.new.dat file to be extracted. A new window will be popped up(Name of the window will be – Input_Dat), copy the file in this window.

Step 4. Once you copy, it will show you Press any key to continue… Now press any key. A process of converting your system.new.dat file into EXT4 format will get started.

Step 5. Once the process is completed you will see that EXT4 superblock info stating the successful completion of the process. Now again press any key to continue the extraction of EXT4 file.

Step 6. When the above process is completed it will show Done. Go To Folder System_unpack press any key to continue. The process is completed you will get the extracted file in folder System_unpack which will be located in the directory of Auto Tool 3.0.

Congratulations! You have successfully extracted the system.dat file on your windows PC.

Steps to Extract System.img on Windows

Step 1. Download and extract the Auto tool 3.0 file. (Use 7zip tool for extraction)

Step 2. Run the Auto Tool v3.0.exe. After running it you will see a screen like this.

Step 3. Now type 4 and hit enter. A window will be popped up, copy the system.img file in that folder and press any key to continue. (Process of converting the file into system.img file to EXT4 will be started).

Step 4. Once the file is converted into EXT4 it will say press any key to continue. Press any key any and the extraction process will get started.

Step 5. When the process gets completed it will show display a message as " Done. Go To the Folder System_unpack press any key to continue”. You will get your extracted file in the folder System_unpack which will be in the same directory as Auto Tool 3.0.

Congratulations! You have successfully extracted the system.img file on your windows PC.

In this tutorial, we'll see how we can convert and extract System.img and System.new.dat files on a Windows PC. The system.img is a part of the Android OS. The image file contains all the default applications, framework, Dalvik VM (Virtual Machine), initialization scripts and other necessary support files which an Android OS needs or comprises of.

The system.new.dat file is somewhat similar to the system.img file. It can be used to derive or extract system dump of any Android OS. This dump generally consists of the stock or default apps, ringtones, wallpapers, themes and a few other necessary system files.

Both the system.img and the system.new.dat cannot be extracted using conventional file extractions tools such as WinZip or 7-Zip. To perform this task one requires special tools that are specifically built for this purpose.

Sadly, performing this task on a Windows PC is difficult. There are a lot of limitations when on Windows platform. There are a few tools which guarantee to do this. However, none were found promising to me except one.

Auto Tool v3.0 functions as it should and is best out of its bunch. The link to download Auto Tool v3.0 is listed below. I have also listed the steps to extract system.img and system.new.dat files. If you need to then you can also refer to it.

Prerequisites

Java SE Development Kit – Download and Install
Download and Install both Python 3.6.x and Python 2.7.x
Auto Tool v3.0 – Download for x64 | Download for x86

Important Points to Note

While installing Python 3.6.x, make sure to check the box that says Add Python 3.6 to PATH.
While installing Python 2.7.x, click on Advanced Options and check the box that says Compile .py files to bytecode after installation.

How to Extract System.new.dat file on Windows

Make sure you downloaded the Auto Tool v3.0 linked above in the prerequisites section.
Extract the file in a new folder.
The Android-Tool folder will be hidden (You can unhide it if you want, doesn’t make a difference).
Next, open the Auto Tool v3.0.
Then, type 1 and hit enter (A new folder will pop up).
The system.new.dat is generally accompanied by two more files (system.transfer.list and file_contexts). Copy all the three files in the folder that popped up.
Then, as it says, press any key to continue (The file will be converted to EXT4 format).
After the process is complete, the terminal will display EXT4 superblock info stating the successful completion of the process.
Next, press any key to continue (The unpacking of the EXT4 file will begin).
Wait, a message will be displayed saying, Done. Go to Folder System_Unpack (Press any key to continue).

How to Extract System.img File on Windows

That's it, guys. You can find the extracted files in the System_Unpack folder located in the same directory as Auto Tool v3.0.

Make sure to clean folder DAT and folder IMG before reusing them or you might end up mixing some other components into the current extraction firmware.

Seems like there is also a ‘readme’ file which lists all the features and prerequisites. Type 0 to open it. The steps for conversion of DAT files to IMG files or vice-versa is more or less same therefore I’m not mentioning them. However, if you guys have any queries or questions related to this post do let me know in the comments below.

Let's start with the fact that you need Linux. In Windows, you can only disassemble the firmware, but you won’t be able to put it back together purely. technical reasons. Now about the firmware. They are usually distributed in the form of ZIP archives flashed through custom recovery. It is one of them that we will need for experiments. I recommend starting the Romodel journey with some custom firmware that is as close as possible to AOSP, because it is often easier to understand than the stock one.

Unpack the archive with the firmware into any folder.
Download the script from the link and unzip it into any folder.
We launch the ext file (if it complains about the lack of Java, just skip by pressing y; Java is only needed for packaging).
Now select unpacking by pressing button 1 and then Enter.
A new folder named extract_* will appear next to the ext file and the tools folder. Copy the files system.new.dat and system.transfer.list into it.
After copying the files, press Enter and wait. After a while, you will need to press Enter again, enter the administrator password and press Enter again.
Ready. The contents of system are in the extract_*/output folder.

Manual method

Unpack the archive with the firmware into any folder (for example, rom):

$ mkdir ~/rom $ unzip path_to_archive -d ~/rom/

Download the tools we need into this folder:

$ cd ~/rom $ wget https://github.com/xpirt/sdat2img/raw/master/sdat2img.py

Let's run the script:

$ chmod +x sdat2img.py $ ./sdat2img.py system.transfer.list system.new.dat system.img

It converts the system.new.dat file into a raw image called system.img . Mount the image to the mnt subfolder:

$ mkdir mnt $ sudo mount -t ext4 -o loop system.img ~/rom/mnt

Android Directory Structure

After unpacking system, the following directory structure will appear:

app - pre-installed applications with standard privileges;
priv-app- pre-installed applications with elevated privileges, including some system components;
bin- binary files in ELF format, an analogue of the /bin and /usr/bin directories in Linux. Contains various system components used by higher-level system components;
etc- settings files. A complete analogue of /etc in Linux, used, however, only by those same system components. Android Applications store individual settings in the /data/data/ directories;
fonts- fonts. By default, it contains only Roboto branded fonts;
framework- libraries of Java classes used by the system and applications. There is also a file framework-res.apk containing a complete description of the interface operating system, including all graphic files;
lib And lib64- Linux libraries used by low-level system components. An analogue of the /lib and /usr/lib directories in Linux, including standard libraries such as libc, libz, libssl. On devices with ARMv7 architecture and lower, the lib64 directory will be missing;
media- media files: ringtones, notification sounds, interface sounds and OS loading animation;
tts- files needed by the speech synthesizer;
usr- a directory that usually contains files needed for applications to run from the bin directory. Essentially an analogue of /usr/share ;
vendor- files supplied by the device manufacturer. Usually contains binary firmware for various hardware components, for example a Wi-Fi module;
xbin- optional directory; custom firmwares use it to store things like the bash interpreter, SSH, PowerTOP, BusyBox, and other useful tools;
build.prop- a file containing information about the assembly, as well as various low-level settings;
addon.d- contains scripts that are launched after installing the firmware. GApps also write their own script here, thanks to which they are revived after reinstalling the firmware.

Now that we're familiar with the basic structure of Android, let's start making changes.

Removing and adding applications

All pre-installed programs can be found in two folders:

/system/app/;
/system/priv-app/.

They differ from each other in access privileges. If programs from app have the same permissions as third party programs(for example, installed from Play Store), then applications from the priv-app can use privileged APIs (privileged rights). You can learn more about this from.

To preinstall an application into the firmware, simply drop its APK file into /system/app/ . You can, of course, create a separate folder, but in our case this makes no sense, because the folder is used to store libraries and odex files, which we simply do not have. To remove it, simply delete the folder with it.

You can go further and replace stock applications with analogues. For example, to replace the calendar, delete the folder Calendar and copy our favorite com.rpagyc.simplecalendar.apk to /system/app. And you don’t have to copy it. Then the firmware will be without a calendar.

The main thing to remember is that stock programs can be related to each other. Therefore, deleting one program may well lead to the complete inoperability of another (for example, CalendarProvider and Calendar: by deleting the first, you will make not only the stock calendar inoperable, but also any other calendar). Fortunately, in pure AOSP firmware there are not many interconnections.

Changing the loading animation

The animation is stored as PNG images, packed into the archive /system/media/bootanimation.zip without compression. Inside the archive there are:

desc.txt- a file describing the animation;
part0- a folder with animation files that are played first;
part1- a folder with animation files that are played second;
part?- the last folder, the images from which are played back at the end.

The desc.txt file might contain something like

1920 1080 60 p 1 0 part0 p 0 0 part1

The purpose of these lines is intuitive: 1920 × 1080 is the image resolution, 60 is the number of frames per second. Part0 and part1 indicate the folders from which the animation will be played and the playback sequence. In general, there can be either one part or several (three or more).

The images located in the part folders are numbered with five numbers in playback order: 00000.png, 00001.png, 00002.png... These images can be replaced with your own, thereby creating an original animation. Or you can simply delete the bootanimation.zip file. Then the device will show animation from standard Android. Or use a ready-made collection of animations on w3bsit3-dns.com.

Changing the sound design

In fact, all the sounds that the system plays are stored in the /system/media/audio folder. Inside it you will find the following folders:

alarms- alarm clock melodies;
notifications- notification sounds;
ringtones- ringtones;
ui- system sounds, such as low battery, camera focusing, selection of interface elements.

In alarms, notifications, ringtones you can add as many melodies as you like. You can take them, for example, here:

standard ringtones from different Nokia phones and smartphones;

And a little life hack: deleting files from the ui folder will not lead to failures and errors, but to disappearance system sounds. Therefore, you can easily turn off the sound of taking a picture from the camera, taking a screenshot, simply by erasing the files containing these sounds (their names are intuitive).

Adding fonts

Fonts are stored in fonts . You can find archives with font files on w3bsit3-dns.com and XDA. To install, just copy and replace the ttf files from the archive to the fonts folder.

Changing system settings (build.prop)

IN system image there is an interesting build.prop file containing many useful information about the device hardware and default settings for various stock applications. To be fair, I note that this is not always the case. For example, in Gigaset ME and ME Pro build.prop is divided into two parts. One part contains settings for Gigaset ME, and for ME Pro some lines are duplicated, but the keys (smartphone name, etc.) are different. This was done to ensure more or less correct operation of the same firmware on different devices.

Build.prop contains (or may contain) great amount settings. Some of them do not change anything, some improve one thing at the expense of another, but there are those that are really useful:

ro.product.model And ro.product.manufacturer- smartphone model and manufacturer’s name. By replacing these lines, you can make the Play Store think that you have a different smartphone, which will open access to more software. For all those little known Chinese smartphones these lines can become life-saving;
hw.qemu.mainkeys- takes only two values: 0 - show on-screen navigation keys, 1 - do not show buttons. The absence of a line corresponds to 0;
debug.sf.nobootanimation- value 1 disables loading animation, which slightly increases its speed. Setting it to 0 or deleting the line returns the animation to its place;
ro.telephony.default_network- tells the system which mode it should be switched to mobile network while loading;
ro.sf.lcd_density- Display DPI, the most accurate value for the display can be calculated through a convenient website. But no one forbids you to set a higher or lower value to your liking: higher values make the interface elements larger, lower values make the interface elements smaller;
ro.config.vc_call_vol_steps- number of volume steps during a call (default 8);
ro.config.media_vol_steps- number of media volume steps (default 15).

We implement Google Apps in the firmware

Almost always custom firmware comes without Google services and application store. The developers suggest that we install them separately using the GApps package. However, it can be integrated directly into the firmware.

First you need to download the GApps package. I recommend taking the Open GApps archives. You choose Android version, processor architecture and configuration option (Pico, Nano, Stock...), which determines how many various applications Google contains an archive. I recommend downloading the Pico version. It contains only the Play Store and a set of libraries necessary for its operation.

Integrating GApps into the firmware is done like this:

Unpack the GApps ZIP archive using any archiver.
Go to the Core folder.
We see many archives with the extension .tar.lz. Extract using lzip.
After unpacking, copy the files from the folders to the corresponding folders in the system. It’s easy to guess what to throw where from the directory structure in the archive. For example, configupdater (from the screenshot) needs to be placed in the priv-app folder.
Go to the GApps folder (located next to Core) and follow steps 3 and 4 for the files in it.
That's it, we have integrated GApps into our firmware!

Free place

It is necessary to understand that the space for installing firmware is limited. You cannot install firmware whose size exceeds the size of the system partition of the device. You can view its value using:

$ adb shell df /system

Second option: put a terminal on the device and type the command

$df/system

You can find out the partition size in bytes by installing BusyBox on your smartphone and running the command in the terminal

$ busybox df -B 1 /system

Or the same with ADB:

$ adb shell busybox df -B 1 /system

The space occupied by the firmware will be approximately equal to the size of the system when unpacked. In general, when creating firmware, it is necessary to take into account that the user can also flash various modifications on top of it (SuperSU, Xposed) or transfer applications to the system section. For example, the minimum package Google applications(Pico) requires a minimum of 150 MB of additional installation space.

If necessary, the size of the firmware file can be reduced by removing not only unnecessary programs from /system/app (/system/priv-app) and call sounds from system/media/audio and bootanimation.zip, but also:

/system/tts/lang_pico- languages of the primitive voice engine Pico TTS, Google voice engine will not be affected;
/system/usr/srec/config/- offline languages. You can download it online later if needed.

Assembly

After making changes, you need to put everything back together. First, let's pack the system section into system.new.dat . Download the tools we need:

$ wget https://github.com/xpirt/img2sdat/raw/master/img2sdat.py $ wget https://github.com/xpirt/img2sdat/raw/master/blockimgdiff.py $ wget https://github. com/xpirt/img2sdat/raw/master/common.py $ wget https://github.com/xpirt/img2sdat/raw/master/rangelib.py $ wget https://github.com/xpirt/img2sdat/raw/ master/sparse_img.py $ sudo apt-get install android-tools-fsutils

Let's convert our folder back to a RAW image. Let's call it system_new.img:

$ sudo make_ext4fs -T 0 -S file_contexts -l 1073741824 -a system system_new.img output/

1073741824 is changed to the size of the system partition in bytes. It would be advisable to even make it a little smaller. We make a sparse image from the RAW image:

$ img2simg system_new.img system_snew.img

Let's convert our image into system.transfer.list and system.new.dat, which should be thrown into the archive with the firmware, but first we'll delete the old files:

$ rm -rf system.transfer.list $ rm -rf system.new.dat $ rm -rf system.patch.dat $ chmod +x img2sdat.py $ ./img2sdat.py system_snew.img

Let's separate the firmware files from the excess husk (files that we downloaded for work. For this it is convenient to use the archive with the firmware). Deleted? Now you need to pack the firmware into a ZIP archive (with any archiver).

All that remains is to sign the archive. This can be done both on Android itself using ZipSigner, and on a PC (you will need Java installed):

$ wget https://github.com/appium/sign/raw/master/dist/sign.jar $ java -jar file.zip

Underwater rocks

While building system.new.dat you may encounter several problems caused by constant changes in the mechanisms for generating Android firmware. The method described above should work well in the case of firmware based on Android 5.1; newer ones may have difficulties, so you will need to use other versions of the build tools. Unfortunately, we cannot describe all the nuances of the assembly, so you may have to Google it.

Installation

To install custom firmware, you need a custom TWRP recovery, which allows you to install unsigned or signed firmware with a test key (this is exactly what we created). In the magazine, we have repeatedly described the process of installing it, and in the forum threads dedicated to your device, there is usually enough information to do this.

TWRP

conclusions

This article describes only the tip of a huge iceberg called “firmware modification”. “Serious” firmware not only complements the kernel and the firmware itself with stock applications with many functions (which are often taken from other kernels and firmware), organizing or even changing the principles of their interaction, but can also radically change the operating principles of the OS. True, such a hack is no longer Android, but a separate OS, even if Play services can be installed there (by the way, such actions, to put it mildly, are not encouraged by Google). Well, don’t forget: all shells from manufacturers - TouchWiz, ZenUI, HTC Sense and so on - are just ordinary custom ones, maximally tied to the hardware of the device and to each other.

Mega utility Android IMG Repack Tools for all beginners and advanced romodelers, developers, for editing Android images.

Android IMG Repack Tools what is it and why?

Android IMG Repack Tools utility from user under nickname A.S._id(xda, w3bsit3-dns.com), designed to edit Android images:

unpack Android image (img, ext4)
editing files (manually)
collect the image

And also work with kernels (boot.img) and recovery (recovery). The utility is designed to work both under Linux and Windows (you must install Cygwin).

Android IMG Repack Tools will be useful to absolutely all Android Romodels for creating new custom firmware, as well as software developers.

Where to find and download Android IMG Repack Tools?

Download and latest version Android utilities IMG Repack Tools can be found from the official project page on XDA.

Installing Android IMG Repack Tools on your computer

Since Android OS is based on Linux kernel, then it would still be more correct to work with the Android IMG Repack Tools utility from a Linux-based operating system.

OS pre-configuration for Android IMG Repack Tools

If you have Ubuntu versions 14.04 and older (and Mint based on it), then you will need to install GCC 5 from the repository (open a terminal and type the following commands):

sudo add-apt-repository ppa:ubuntu-toolchain-r/test

sudo apt-get update

sudo apt-get install gcc-5 g++-5

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5

For Ubuntu users 16.04 (and Mint based on it), then you need to install first:

1. GNU Automake 1.14 (download and unpack)

2. Go to the folder with Gnu Automake, open a terminal and execute the commands one by one: ./configure make sudo make install

3. Install packages: sudo apt-get install git-core gnupg flex bison gperf libsdl-dev libesd0-dev build-essential zip curl libncurses5-dev zlib1g-dev valgrind libreadline6-dev gcc-multilib g++-multilib libc6-dev x11proto-core -dev libx11-dev libz-dev gawk texinfo automake libtool cvs libsdl-dev

Setting up Android IMG Repack Tools

1. After downloading Android IMG Repack Tools, unpack, go, open terminal

2. Type the command to launch the utility./main_menu

If you have Mint installed, then click on the main_menu file

3. Go to the utility section " 0 - Configure & clean tools menu»

4. Select " 1 - Build android_img_repack_tools“, in order to download and configure the tools that work with Android images.

Working with the Android IMG Repack Tools utility

Expect an update to the article with instructions on how to use the utility soon!