diff options
| author | Jordan Wiens <jordan@psifertex.com> | 2024-02-19 15:04:00 -0500 |
|---|---|---|
| committer | Jordan Wiens <jordan@psifertex.com> | 2024-02-23 11:14:06 -0500 |
| commit | 448f40be71dffa86a6581c3696627ccc1bdf74f2 (patch) | |
| tree | cd79923c85986c0135b41eb7432e54218f1a56e5 /docs/dev | |
| parent | 748c2295e31c03afa1d4b1b776a59f9fa9a48d34 (diff) | |
4.0 documentation
- Refactored Type Documentation
- Added Projects
- Added Type Archives
- Added New Sidebar Documentation
- Added String Concepts
- Added Light/Dark Mode
- Added New Tab Documentation
- Added BNIL Guide: HLIL docs
- Added new cookbook examples
- Added migration guide
- Added script for building docsets
- Added documentation for themes
- Updated all images to Ninja Edit
- API Docs : Documents BasicBlockEdge and BasicBlock
- API Docs : Documents CoreVariable, Variable, and VariableNameAndType
- API Docs : Corrects note on `BinaryView.update_analysis` and `BinaryView.update_analysis_and_wait` to represent that analysis is run by default for you now.
- Many, many other changes
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/annotation.md | 297 | ||||
| -rw-r--r-- | docs/dev/bnil-hlil.md | 158 | ||||
| -rw-r--r-- | docs/dev/bnil-llil.md | 10 | ||||
| -rw-r--r-- | docs/dev/bnil-mlil.md | 16 | ||||
| -rw-r--r-- | docs/dev/bnil-overview.md | 8 | ||||
| -rw-r--r-- | docs/dev/concepts.md | 64 | ||||
| -rw-r--r-- | docs/dev/cookbook.md | 163 | ||||
| -rw-r--r-- | docs/dev/flags.md | 6 | ||||
| -rw-r--r-- | docs/dev/index.md | 4 | ||||
| -rw-r--r-- | docs/dev/plugins.md | 2 | ||||
| -rw-r--r-- | docs/dev/themes.md | 138 | ||||
| -rw-r--r-- | docs/dev/typelibraries.md | 288 | ||||
| -rw-r--r-- | docs/dev/uidf.md | 39 | ||||
| -rw-r--r-- | docs/dev/workflows.md | 1 |
14 files changed, 780 insertions, 414 deletions
diff --git a/docs/dev/annotation.md b/docs/dev/annotation.md index d952baa6..60cb9681 100644 --- a/docs/dev/annotation.md +++ b/docs/dev/annotation.md @@ -1,11 +1,11 @@ # Applying Annotations -This document is organized into five sections describing how to work with various types of annotations in Binary Ninja using the API. Note that its [companion documentation](../guide/type.md) describes how to interact with many of the same elements via the UI. +This document is organized into five sections describing how to work with various types of annotations in Binary Ninja using the API. Note that its [companion documentation](../guide/types/index.md) describes how to interact with many of the same elements via the UI. 1. [Symbols](#symbols) covers how to work with Symbols in a binary 1. [Types](#types) documents creating and interacting with types through the API 1. [Tags](#tags) describes how to create tags and bookmarks -1. [Type Libraries](#type-libraries) explains how to work with Type Libraries, including multiple sources of information from which Binary Ninja can automatically source for type information from and how you can add to them +1. [Type Libraries](typelibraries.md) explains how to work with Type Libraries, including multiple sources of information from which Binary Ninja can automatically source for type information from and how you can add to them 1. [Signature Libraries](#signature-library) explains how to work with the signature library which match statically compiled functions which are then matched with type libraries ## Symbols @@ -20,7 +20,7 @@ From an API perspective there are several helper functions available for working 'newName' ``` -Other objects or variables may need a [symbol](https://api.binary.ninja/binaryninja.types.Symbol.html) created and applied (this can also be done even if not using a helper function on objects such as functions): +Other objects or variables may need a [symbol](https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.Symbol) created and applied (this can also be done even if not using a helper function on objects such as functions): ```py >>> mysym = Symbol(SymbolType.FunctionSymbol, here, "myVariableName") @@ -31,7 +31,7 @@ Other objects or variables may need a [symbol](https://api.binary.ninja/binaryni Note that `here` and `bv` are used in many of the previous examples. These shortcuts and [several others](../guide/index.md#script-python-console) are only available when running in the Binary Ninja python console and are used here for convenience. -Valid symbol types [include](https://api.binary.ninja/binaryninja.enums.SymbolType.html): +Valid symbol types [include](https://api.binary.ninja/binaryninja.enums-module.html#binaryninja.enums.SymbolType): | SymbolType | Description | | ---------- | ----------- | @@ -439,7 +439,7 @@ Note that most of the APIs that take a Type object also take a C-style type stri ### Headers -Importing a header goes through the same code path as parsing source directly. You will just have to read the file and specify the appropriate command-line arguments as an array. See [user type guide](../guide/type.md#import-header-file) for directions for choosing arguments. +Importing a header goes through the same code path as parsing source directly. You will just have to read the file and specify the appropriate command-line arguments as an array. See [user type guide](../guide/types/typeimportexport.md#import-header-file) for directions for choosing arguments. ```python >>> with open('C:\projects\stdafx.h', 'r') as f: @@ -473,292 +473,7 @@ Exporting a header uses the `TypePrinter.print_all_types` api, and outputs a str ## Type Libraries -Type Libraries are collections of type information (structs, enums, function types, etc.) stored in a file with the extension `.bntl`. - -Relative to the binaryninja executable, the default type library location is `../Resources/typelib` on macOS and `./typelib` on Linux and Windows. Individual .bntl files are organized in subdirectories named for the supported architecture. Users may include their own type libraries - -The information in a type library is contained in two key-value stores: - -1. named types: key is the type name, value is the type -1. named objects: key is external symbol name, value is the type - -### How Binary Ninja Loads Type Libraries - -When a binary is opened, its platform is determined, all .bntl's are processed, and those matching the platform of the loaded binary are registered. A debug log will show: - -``` -Registered library 'libc.so.6' with platform 'linux-x86_64' -``` - -Then, those with either a filename or an alternative name matching the exact text of the binary's import command are imported, much like the native linker/loader. For example, in ELF, the .dynstr entry is used. - -``` -elf: searching for 'libc.so.6' in type libraries -Type library 'libc.so.6' imported -``` - -Type libraries for linux are ideally named after their realname, preserving the library minor version from which they were generated, and the soname in the alternatives list. In practice, naming them after their soname suffices. Using the linkname with no alternatives will prevent your library from loading. - -This requested name should be a soname, like "libfoo.so.1" but could be a linkname like "libfoo.so". (The ldconfig tool is responsible for creating symlinks from soname to realnames, like `/usr/lib/libfoo.so.1` -> `/usr/lib/libfoo.so.1.0`. See [tldp.org](https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html) for more information.). - -Binary Ninja's logic for determining a match is straightforward: - -``` -typelibname.removesuffix('.bntl') == requestedname or requestedname in alternativenames -``` - -Therefore, without any alternative names, `libc.so.bntl` will not be loaded by Binary Ninja if an ELF requests libc.so.6. - -We recommend and use the following convention: - -Type libraries should be named for the filename from which they were generated with the phrase ".bntl" added. When the source library contains additional minor and release number, like `libfoo.so.1.2.3` Binary Ninja would not load the resulting type library `libfoo.so.1.2.3.bntl` for an ELF requesting soname `libfoo.so.1`. Therefore the alternative names list should include the most specific version numbers, incrementally stripped down to the soname, and finally a linkname for good measure. - -Example: - -`libfoo.so.1.2.3` is used to generated `libfoo.so.1.2.3.bntl` - -The alternative names list should have: - -``` -libfoo.so.1.2.3 <-- includes version, minor, release (most specific) -libfoo.so.1.2 <-- includes version, minor (less specific) -libfoo.so.1 <-- includes version (soname) -libfoo.so <-- linkname -``` - -### Acquiring a Handle - -The platform class exposes handles to these imported type libraries with its `type_libraries` list and its `get_type_libraries_by_name()` function: - -```python ->>> bv.platform.type_libraries -[<typelib 'libm.so.6':x86_64':x86_64>, <typelib 'SYSCALLS':x86_64] ->>> bv.platform.get_type_libraries_by_name('libm.so.6') -[<typelib 'libm.so.6':x86_64>] -``` - -That requires the type library having been loaded. A more direct way is to load from a file path with the `load_from_file()` from `Typelibrary` class from `typelibrary` module: - -```python ->>> typelibrary.TypeLibrary.load_from_file('/path/to/libm_x86_64.so.6.bntl') -<typelib 'libm_x86_64.so.6':x86_64> -``` - -### Contents of Libraries - -The following demonstrates attributes of interest on a loaded type library in variable `tl`: - -```python - print(' name: %s' % tl.name) - print(' arch: %s' % tl.arch) - print(' guid: %s' % tl.guid) - print('dependency_name: %s' % tl.dependency_name) - print('alternate_names: %s' % tl.alternate_names) - print(' platform_names: %s' % tl.platform_names) - - print(' named_objects: %d' % len(tl.named_objects)) - for (key, val) in tl.named_objects.items(): - print('\t"%s" %s' % (str(key), str(val))) - - print(' named_types: %d' % len(tl.named_types)) - for (key,val) in tl.named_types.items(): - print('\t"%s" %s' % (str(key), str(val))) -``` - -Named objects (via dictionary `.named_objects` are functions signatures and a module's exported variables. Named types (via dictionary `.named_types`) are the textual named you might use to declare a variable. For example, - -For example, `.named_objects['fegetexceptionflag']` looks up its function prototype: `int32_t (fexcept_t* flagp)` and `.named_types['fexcept_t']` looks up `uint16_t`, its typedef. - -### Creating - -You may also wish to use the [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py) example script included both online and offline in your installation path. - -Types entered by manual entry can be exported from the binary view using `export_type_to_library()`. - -Manual creation is achieved by creating a new type library, associating the correct platform and architecture, adding types, finalizing, and writing to a file. Example: - -```python -arch = binaryninja.Architecture['x86_64'] - -struct = Structure() -struct.append(Type.pointer(arch, Type.char()), 'name') -struct.append(Type.int(4), 'age') -struct.append(Type.int(4), 'height') -struct.append(Type.int(4), 'weight') - -typelib = binaryninja.typelibrary.TypeLibrary.new(arch, 'test.so.1.4') -typelib.add_named_type('human', binaryninja.types.Type.structure_type(struct)) -typelib.add_alternate_name('test.so.1') #don't forget this step! -typelib.add_alternate_name('test.so') -typelib.finalize() -typelib.write_to_file('test.so.1.bntl') -``` - -### Other Type Library Questions - -_What's a named type vs. just a type?_ - -Some variable definitions have type information, but don't produce a type name useful for future definitions, examples: - -- `enum {A=1,B=2} foo;` : foo has type with no type name (it does have a variable name) -- `struct {int A; int B;} bar;` : bar has type with no type name - -In C, enum and struct definitions can create a new type name as a byproduct of a definition by using a "tag name": - -- `enum MyEnum {A=1,B=2} foo;` : foo has the type named `MyEnum` -- `struct MyStruct {int A; int B;} bar;` : bar has the type named `MyStruct` - -In the second set of examples, the types are named, and that name could be used to declare other variables, like `enum MyEnum bar2;` and `struct MyStruct bar2`. - -Functions' types are not named. The function name is considered the name of a function object, and the function's type is anonymous. - -In summary: - -```c -typedef int foo; // type:int, name:foo - -// structs, without and with a "tag name" -struct {int A; int B;} foo; // type:struct{int A, intB;}, name:<anonymous> -struct MyStruct {int A; int B;} foo; // type:struct{int A, intB;}, name:MyStruct - -// enumerations, without and with a "tag name" -enum {A=1,B=2} foo; // type:enum{A=1,B=2}, name:<anonymous> -enum MyEnum {A=1,B=2} foo; // type:enum{A=1,B=2}, name:MyEnum - -// functions -int main(int ac, char **av); // type int ()(int, char **), name:<anonymous> -typedef int (MyFunc)(int ac, char **av); // type int ()(int, char **), name:MyFunc - -``` - -_How does Binary Ninja decide when to use a typelibrary (.bntl) file?_ - -Type Libraries are loaded when the corresponding library is imported by a BinaryView. (i.e. if an exe imports `ntdll.dll`, binja will look in the bv's platform for type libraries named ntdll.bntl and load the first one it finds) - -_What's the difference between a named type and a named object?_ - -A named type is a type with a name that can identify it. For example, `color` is the name of type `enum {RED=0, ORANGE=1, YELLOW=2, ...}`. - -A named object is the name of an external/imported symbol for which the type library has type information. For example, `MessageBoxA` is the name of a function whose type is `int ()(HWND, LPCSTR, LPCSTR, UINT)`. - -_How do I find what type of type a type object is? How many are there?_ - -I've seen "types of types", "sorts of types", "kinds of types", "classes of types" used to differentiate the varieties of possible types, and there are probably more. Binary Ninja uses "class", example: - -``` ->>> type_obj.type_class -<TypeClass.FunctionTypeClass: 8> -``` - -In [enums.py](https://api.binary.ninja/_modules/binaryninja/enums.html#TypeClass) we can see Binary Ninja currently thinks of types falling into 13 classes: `Void`, `Bool`, `Integer`, `Float`, `Structure`, `Enumeration`, `Pointer`, `Array`, `Function`, `VarArgs`, `Value`, `NamedTypeReference`, `WideCharType`. - -Compare this to LLDB, which also uses the term "class", and currently has 19 of them: `Array`, `BlockPointer`, `Builtin`, `Class`, `ComplexFloat`, `ComplexInteger`, `Enumeration`, `Function`, `MemberPointer`, `ObjCObject`, `ObjCInterface`, `ObjCObjectPointer`, `Pointer`, `Reference`, `Struct`, `Typedef`, `Union`, `Vector`, `Other`. - -Compare this to GDB, which uses the term "type code" and has 25 of them. - -_Where are function parameter names stored?_ - -While technically not part of the type, having names of function parameters is very useful and can thus be optionally stored in a type. - -Function types (types with `.type_class == FunctionTypeClass`) have a `.parameters` attribute, a list of [`FunctionParameter`](https://api.binary.ninja/binaryninja.types.FunctionParameter.html) objects. When those objects have `.name==''` you get the bare bones function types like `int ()(int, char **)`. When those objects have their `.name` populated you get the more meaningful `int ()(int argc, char **argv)`. - -_How do I manually load a type library?_ - -``` ->>> bv.add_type_library(TypeLibrary.load_from_file('test.bntl')) -``` - -_How can I manually load a type object?_ - -``` ->>> bv.import_library_object('_MySuperComputation') -<type: int32_t (int32_t, int32_t, char*)> -``` - -_Why doesn't the types view show the types imported from type libraries?_ - -Because the type libraries added to a binary view only makes their type information _available_ for use. The types view will show a type from a type library only after it is used (on demand). - -Try this experiment: - -- note `bv.type_libraries`, `bv.types` -- add a type library with `bv.add_type_library(TypeLibrary.load_from_file('test.bntl'))` -- note that `bv.type_libraries` is extended, but `bv.types` is unchanged! -- note `bv.get_type_by_name('Rectangle')` returns nothing -- set the type of some data to `struct Rectangle` (using `y` in linear view or via any other method described above) -- `bv.types` is extended, and the types view shows `struct Rectangle` in the auto types - -_What's a named type reference?_ - -Named Type References are a way to refer to a type by name without having its declaration immediately available. - -For example, examine this struct from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py): - -``` -struct Rectangle2 { - int width; - int height; - struct Point center; -} -``` - -We don't know at this moment what a `struct Point is`. Maybe we've already added it. Maybe we'll add it later. Maybe it's in another type library. But we want to add a Rectangle now. So we leave the center field as a reference to the named type `struct Point`. - -Load the resulting `test.bntl` in your binary and try to set some data to type `struct Rectangle2` and you'll be met with this message: - -``` -TypeLibrary: failed to import type 'Point'; referenced but not present in library 'libtest.so.1` -``` - -This makes sense! Now go to types view and `define struct Point { int x; int y; }` and try again, success! - -``` -100001000 struct rectangle_unresolved data_100001000 = -100001000 { -100001000 int32_t width = 0x5f0100 -100001004 int32_t height = 0x5f030005 -100001008 struct Point center = -100001008 { -100001008 int32_t x = 0x655f686d -10000100c int32_t y = 0x75636578 -100001010 } -100001008 } -``` - -You should repeat the experiment using `struct Rectangle` and see that you're allowed to create variables with type containing _pointers_ to unresolved type references. - -_How are types represented?_ - -By a hierarchy of objects from [api/python/types.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/types.py) referencing one another. The "glue" object is [`binaryninja.types.Type`](https://api.binary.ninja/binaryninja.types.Type.html#binaryninja.types.Type) and depending on the complexity of the type it represents (stored in its `.type_class` attribute), it could have an attribute with more information. For instance, if the `binaryninja.types.Type` has `.type_class == FunctionTypeClass` then its `.parameters` attribute is a list of [`binaryninja.types.FunctionParameter`](https://api.binary.ninja/binaryninja.types.FunctionParameter.html). See [typelib_dump.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_dump.py) for how this can work. - -As an example, here is the hierarchical representation of `type struct Rectangle` from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py) - -``` -typelib.named_types["Rectangle"] = ----------------------------------- -Type class=Structure - Structure - StructureMember "width" - Type class=Integer width=4 - StructureMember "height" - Type class=Integer width=4 - StructureMember "center" - Type class=Pointer - Type class=NamedTypeReference - NamedTypeReference <named type: struct Point> -``` - -Here is the representation of `type int ()(int, int)` named `MyFunctionType` from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py): - -_When do named objects get used?_ - -When a binary is loaded and its external symbols is processed, the symbol names are searched against the named objects from type libraries. If there is a match, it obeys the type from the type library. Upon success, you'll see a message like: - -``` -type library test.bntl found hit for _DoSuperComputation -``` - -At this moment, there is no built in functionality to apply named objects to an existing Binary Ninja database. +Type Libraries have outgrown this section and now live [in their own file](typelibraries.md) ## Signature Library diff --git a/docs/dev/bnil-hlil.md b/docs/dev/bnil-hlil.md new file mode 100644 index 00000000..157b6a59 --- /dev/null +++ b/docs/dev/bnil-hlil.md @@ -0,0 +1,158 @@ +# Binary Ninja Intermediate Language Series, Part 3: High Level IL +# Binary Ninja Intermediate Language: High Level IL + +The High Level Intermediate Language (HLIL) is Binary Ninja's decompiler output. Much like [LLIL](./bnil-llil.md) and [MLIL](./bnil-mlil.md), this representation is tree based and has many of the same instructions. This representation is distinct in a few key ways. + + + +1. High level language concepts are recovered. +1. Expressions are folded + +## Purposes of HLIL + +* Simplified representation +* Small discrete operations +* Enables source-level forms of queries and analysis + +In the rest of this article we will explore the instruction set. + +## The Instruction Set + +The instruction set is made up of [`HighLevelILInstruction`](https://api.binary.ninja/binaryninja.highlevelil-module.html#binaryninja.highlevelil.HighLevelILInstruction) objects. Let's start exploring by using the python console to poke around at some instructions. Open up a binary in Binary Ninja and retrieve an HLIL instruction:+ + +``` +>>> current_il_instruction +<HighLevelILVarInit: uint64_t rax_2 = zx.q(rax_1 - 0x6c)> +>>> type(current_il_instruction) +<class 'binaryninja.highlevelil.HighLevelILVarInit'> +``` + +`current_il_instruction` is mapped to whatever il instruction is currently selected viewed and is not generally available to those writing plugins or scripts. You can see a full list of our [magic variables here](../guide/index.md#magic-console-variables). + +There are a number of properties that can be queried on the [`HighLevelILInstruction`](https://api.binary.ninja/binaryninja.highlevelil-module.html#binaryninja.highlevelil.HighLevelILInstruction) object, and the validity of these properties changes depending on what the current operation is. The properties in HLIL are extremely similar to those in MLIL, with one notable exception being that `HLIL_CALL` instruction objects do not have a `.output` property, but instead `HLIL_CALL`s that have return values will be the `inst.right` of a `HighLevelILVarInit`, `HighLevelILVarAssign`, or similar. + + +### Control Flow + +* `HLIL_JUMP` - Branch to the `dest` expression's address +* `HLIL_CALL` - Branch to the `dest` expression function, saving the return address, with the list of parameters `params` +* `HLIL_RET` - Return to the calling function. +* `HLIL_NORET` - This instruction will never be executed, the instruction before it is a call that doesn't return +* `HLIL_IF` - Branch to the `true`/`false` HLIL instruction identifier depending on the result of the `condition` expression +* `HLIL_GOTO` - Branch to the `dest` expression id +* `HLIL_TAILCALL` - This instruction calls the expression `dest` using `params` as input and `output` for return values +not exist +* `HLIL_SYSCALL` - Make a system/service call with parameters `params` and output `output` +* `HLIL_WHILE` - +* `HLIL_DO_WHILE` - +* `HLIL_FOR` - +* `HLIL_SWITCH` - +* `HLIL_CASE` - +* `HLIL_BREAK` - +* `HLIL_CONTINUE` - + +### Variable Reads and Writes + +* `HLIL_VAR_DECLARE` - A declaration of `var` +* `HLIL_VAR_INIT` - Initializes `dest` to the result of an expression `src` +* `HLIL_ASSIGN` - Sets a variable `dest` to the result of an expression `src` +* `HLIL_ASSIGN_UNPACK` - +* `HLIL_VAR` - A variable expression `src` +* `HLIL_VAR_PHI` - A `PHI` represents the combination of several prior versions of a variable when differnet basic blocks coalesce into a single destination and it's unknown which path was taken. +* `HLIL_MEM_PHI` - A memory `PHI` represents memory modifications that could have occured down different source basic blocks similar to a `VAR_PHI`. +* `HLIL_ADDRESS_OF` - The address of variable `src` +* `HLIL_CONST` - A constant integral value `constant` +* `HLIL_CONST_DATA` - A constant data reference `constant data reference` +* `HLIL_CONST_PTR` - A constant integral value which is used as a pointer `constant` +* `HLIL_EXTERN_PTR` - A symbolic pointer `constant` + `offset` to a symbol that exists outside the binary +* `HLIL_FLOAT_CONST` - A floating point constant `constant` +* `HLIL_IMPORT` - A `constant` integral value representing an imported address +* `HLIL_LOW_PART` - `size` bytes from the low end of `src` expression +* `HLIL_STRUCT_FIELD` - +* `HLIL_ARRAY_INDEX` - +* `HLIL_SPLIT` - A split pair of variables `high`:`low` which can be used a single expression +* `HLIL_DEREF` - Dereferences `src` +* `HLIL_DEREF_FIELD` - + +### Arithmetic Operations + +* `HLIL_ADD` - Adds `left` expression to `right` expression +* `HLIL_ADC` - Adds with carry the `left` expression to the `right` expression with carry from the `carry` expression +* `HLIL_SUB` - Subtracts the `right` expression from the `left` expression +* `HLIL_SBB` - Subtraction with borrow the `right` expression from the `left` expression with carry from the `carry` expression +* `HLIL_AND` - Bitwise AND `left` expression with the `right` expression +* `HLIL_OR` - Bitwise OR `left` expression with the `right` expression +* `HLIL_XOR` - Bitwise XOR `left` expression with the `right` expression +* `HLIL_LSL` - Logical shift left the `left` expression by the number of bits stored in the `right` expression +* `HLIL_LSR` - Logical shift right the `left` expression by the number of bits stored in the `right` expression +* `HLIL_ASR` - Arithmetic shift right the `left` expression by the number of bits stored in the `right` expression +* `HLIL_ROL` - Rotate left the `left` expression by the number of bits stored in the `right` expression +* `HLIL_RLC` - Rotate left with carry the `left` expression and the `carry` expression by the number of bits stored in the `right` expression +* `HLIL_ROR` - Rotate right the `left` expression by the number of bits stored in the `right` expression +* `HLIL_RRC` - Rotate right with carry the `left` expression and the `carry` expression by the number of bits stored in the `right` expression +* `HLIL_MUL` - Single-precision multiply the `left` expression with the `right` expression +* `HLIL_MULU_DP` - Double-precision unsigned multiply the `left` expression with the `right` expression, result expression is twice the size of the input expressions +* `HLIL_MULS_DP` - Double-precision signed multiply the `left` expression with the `right` expression, result expression is twice the size of the input expressions +* `HLIL_DIVU` - Unsigned single-precision divide `left` expression by the `right` expression +* `HLIL_DIVU_DP` - Unsigned double-precision divide `left` expression by the `right` expression +* `HLIL_DIVS` - Signed single-precision divide `left` expression by the `right` expression +* `HLIL_DIVS_DP` - Signed double-precision divide `left` expression by the `right` expression +* `HLIL_MODU` - Unsigned single-precision modulus of `left` expression by the `right` expression +* `HLIL_MODU_DP` - Unsigned double-precision modulus of `left` expression by the `right` expression +* `HLIL_MODS` - Signed single-precision modulus of `left` expression by the `right` expression +* `HLIL_MODS_DP` - Signed double-precision modulus of `left` expression by the `right` expression +* `HLIL_NEG` - Sign inversion of `src` expression +* `HLIL_NOT` - Bitwise inversion of `src` expression +* `HLIL_FADD` - IEEE754 floating point addition of `left` expression with `right` expression +* `HLIL_FSUB` - IEEE754 floating point subtraction of `left` expression with `right` expression +* `HLIL_FMUL` - IEEE754 floating point multiplication of `left` expression with `right` expression +* `HLIL_FDIV` - IEEE754 floating point division of `left` expression with `right` expression +* `HLIL_FSQRT` - IEEE754 floating point square root of `left` expression with `right` expression +* `HLIL_FNEG` - IEEE754 floating point sign negation of `src` expression +* `HLIL_FABS` - IEEE754 floating point absolute value of `src` expression +* `HLIL_FLOAT_TO_INT` - IEEE754 floating point to integer conversion of `src` expression +* `HLIL_INT_TO_FLOAT` - Integer to IEEE754 floating point conversion of `src` expression +* `HLIL_FLOAT_CONV` - Convert bytes in `src` expression to IEEE754 floating point +* `HLIL_ROUND_TO_INT` - Rounds the IEEE754 floating point number `src` expression +* `HLIL_FLOOR` - Computes the floating point floor of the IEEE754 number in `src` +* `HLIL_CEIL` - Computes the floating point floor of the IEEE754 number in `src` +* `HLIL_FTRUNC` - Computes the floating point truncation of the IEEE754 number in `src` +* `HLIL_SX` - Sign extends the `src` expression +* `HLIL_ZX` - Zero extends the `src` expression +* `HLIL_ADD_OVERFLOW` - Calculates overflow of the addition of `left` expression with `right` expression +* `HLIL_BOOL_TO_INT` - Converts a bool `src` to an integer + +### Comparison Instructions + +* `HLIL_CMP_E` - Compare expression evaluates to true if `left` expression is equal to `right` +* `HLIL_CMP_NE` - Compare expression evaluates to true if `left` expression is not equal to `right` +* `HLIL_CMP_SLT` - Compare expression evaluates to true if `left` expression is signed less than `right` +* `HLIL_CMP_ULT` - Compare expression evaluates to true if `left` expression is unsigned less than `right` +* `HLIL_CMP_SLE` - Compare expression evaluates to true if `left` expression is signed less than or equal to `right` +* `HLIL_CMP_ULE` - Compare expression evaluates to true if `left` expression is unsigned less than or equal to `right` +* `HLIL_CMP_SGE` - Compare expression evaluates to true if `left` expression is signed greater than or equal to `right` +* `HLIL_CMP_UGE` - Compare expression evaluates to true if `left` expression is unsigned greater than or equal to `right` +* `HLIL_CMP_SGT` - Compare expression evaluates to true if `left` expression is signed greater than `right` +* `HLIL_CMP_UGT` - Compare expression evaluates to true if `left` expression is unsigned greater than `right` +* `HLIL_TEST_BIT` - Test if bit `right` in expression `left` is set +* `HLIL_FCMP_E` - Floating point compare expressions - evaluates to true if `left` expression is equal to `right` +* `HLIL_FCMP_NE` - Floating point compare expressions - evaluates to true if `left` expression is not equal to `right` +* `HLIL_FCMP_LT` - Floating point compare expressions - evaluates to true if `left` expression is less than `right` +* `HLIL_FCMP_LE` - Floating point compare expressions - evaluates to true if `left` expression is less than or equal to `right` +* `HLIL_FCMP_GE` - Floating point compare expressions - evaluates to true if `left` expression is greater than or equal to `right` +* `HLIL_FCMP_GT` - Floating point compare expressions - evaluates to true if `left` expression is greater than `right` +* `HLIL_FCMP_O` - Floating point compare expressions - evaluates to true if both `left` and `right` expressions are ordered (not NaN) +* `HLIL_FCMP_UO` - Floating point compare expressions - evaluates to true if either `left` or `right` expression is unordered (NaN) + +### Miscellaneous Instructions + +* `HLIL_NOP` - No operation +* `HLIL_BP` - Breakpoint instruction +* `HLIL_TRAP` - Interrupt/trap instruction with `vector` expression +* `HLIL_INTRINSIC` - Intrinsic instruction defined by the architecture +* `HLIL_UNDEF` - The expression performs undefined behavior +* `HLIL_UNIMPL` - The expression is not implemented +* `HLIL_UNIMPL_MEM` - The expression is not implemented but does access `src` memory +* `HLIL_BLOCK` - +* `HLIL_LABEL` - +* `HLIL_UNREACHABLE` -
\ No newline at end of file diff --git a/docs/dev/bnil-llil.md b/docs/dev/bnil-llil.md index df520e4e..97ce11bc 100644 --- a/docs/dev/bnil-llil.md +++ b/docs/dev/bnil-llil.md @@ -1,6 +1,6 @@ # Binary Ninja Intermediate Language: Low Level IL -Make sure to checkout the [BNIL overview](bnil-overview.md) first if you haven't already. Or feel free to skip to [part 2](bnil-mlil.md) which covers MLIL. This developer guide is intended to cover some of the mechanics of the LLIL to distinguish it from the other ILs in the BNIL family. +Make sure to checkout the [BNIL overview](bnil-overview.md) first if you haven't already. Or feel free to skip to [part 2](bnil-mlil.md) which covers MLIL, or [part 3](bnil-hlil.md) which covers HLIL. This developer guide is intended to cover some of the mechanics of the LLIL to distinguish it from the other ILs in the BNIL family. If you've already read the introduction, let's get right into the details of LLIL! @@ -12,11 +12,11 @@ The Lifted IL is very similar to the LLIL and is primarily of interest for Archi Since doing is the easiest way to learn let's start with a simple example binary and step through analyzing it using the python console. - + - Download [chal1](../files/chal1) and open it with Binary Ninja - - Next, bring up the `Low Level IL` view by clicking in the options pane at the bottom of the screen - (or alternatively, use the `i` key) + - Next, bring up the `Low Level IL` view by clicking in the view drop down at the top of the pane + (or alternatively, use the `i` key to cycle view levels) - Navigate to main (`g`, then "main", or double-click it in the function list) - Finally, bring up the python console using: `~` @@ -38,7 +38,7 @@ This will print out all the LLIL instructions in the current function. How does First we use the global magic variable `current_function` which gives us the python object [`function.Function`](https://api.binary.ninja/binaryninja.function-module.html#binaryninja.function.Function) for whatever function is currently selected in the UI. The variable is only usable from the python console, and shouldn't be used for headless plugins. In a script you can either use the function that was passed in if you [registered your plugin](https://api.binary.ninja/binaryninja.plugin-module.html#binaryninja.plugin.PluginCommand.register_for_function) to handle functions, or you can compute the function based on [a specific address](https://api.binary.ninja/binaryninja.binaryview-module.html?highlight=get_functions_at#binaryninja.binaryview.BinaryView.get_functions_at), or maybe even just iterate over all the functions in a BinaryView (`for func in bv.functions:`). -Next we get the [`lowlevelil.LowLevelILFunction`](http://api.binary.ninja/binaryninja.lowlevelil.LowLevelILFunction.html) from the `Function` class: `current_function.low_level_il`. Iterating over the `LowLevelILFunction` class provides access to the [`lowlevelil.LowLevelILBasicBlock`](http://api.binary.ninja/binaryninja.lowlevelil.LowLevelILBasicBlock.html) classes for this function. Inside the loop we can now iterate over the `LowLevelILBasicBlock` class which provides access to the individual [`lowlevelil.LowLevelILInstruction`](http://api.binary.ninja/binaryninja.lowlevelil.LowLevelILInstruction.html) classes. +Next we get the [`lowlevelil.LowLevelILFunction`](https://api.binary.ninja/binaryninja.lowlevelil-module.html#binaryninja.lowlevelil.LowLevelILFunction) from the `Function` class: `current_function.low_level_il`. Iterating over the `LowLevelILFunction` class provides access to the [`lowlevelil.LowLevelILBasicBlock`](https://api.binary.ninja/binaryninja.lowlevelil-module.html#binaryninja.lowlevelil.LowLevelILBasicBlock) classes for this function. Inside the loop we can now iterate over the `LowLevelILBasicBlock` class which provides access to the individual [`lowlevelil.LowLevelILInstruction`](https://api.binary.ninja/binaryninja.lowlevelil-module.html#binaryninja.lowlevelil.LowLevelILInstruction) classes. Finally, we can print out the attributes of the instruction. We first print out `address` which is the address of the corresponding assembly language instruction. Next, we print the `instr_index`, this you can think of as the address of the IL instruction. Since translating assembly language is a many-to-many relationship, we may see multiple IL instructions needed to represent a single assembly language instruction, and thus each IL instruction needs to have its own index separate from its address. Finally, we print out the instruction text. diff --git a/docs/dev/bnil-mlil.md b/docs/dev/bnil-mlil.md index ee7c1d91..bdade82c 100644 --- a/docs/dev/bnil-mlil.md +++ b/docs/dev/bnil-mlil.md @@ -24,7 +24,7 @@ In the rest of this article we will explore the variable object, the type object ## The Variable Object -First, it's important to understand what we mean when we talk about a MLIL variable. Continuing from our example above we can get a [`Variable`](https://api.binary.ninja/binaryninja.function.Variable.html) object. +First, it's important to understand what we mean when we talk about a MLIL variable. Continuing from our example above we can get a [`Variable`](https://api.binary.ninja/binaryninja.variable-module.html#binaryninja.variable.Variable) object. ``` >>> inst.output @@ -36,7 +36,7 @@ First, it's important to understand what we mean when we talk about a MLIL varia Variables in MLIL have a very specific meaning, that is not completely obvious at first. They represent a single storage location within the scope of a single function. To those not well versed in program analysis, a storage location is where a value is located at a given point in time. In the process of compilation a compiler conducts a step called _Register Allocation_; this is the process of figuring out how to map the potentially infinite number of variables specified in the original source code to a finite set of registers. When there are more variables and intermediate values than registers available, the compiler _spills_ them on to the stack. Thus a single high-level-language variable can be mapped across a number of storage locations. A variable can simultaneously be in multiple registers and on the stack at the same time. However, unlike high-level-language variables, MLIL variables represent one and only one storage location. Binary Ninja's High Level IL (HLIL) will be responsible for storing this mapping. -So let's look at the properties available on a [`Variable`](https://api.binary.ninja/binaryninja.function.Variable.html) object. +So let's look at the properties available on a [`Variable`](https://api.binary.ninja/binaryninja.variable-module.html#binaryninja.variable.Variable) object. ### `source_type` @@ -59,7 +59,7 @@ enum VariableSourceType ### `storage` -The `storage` property changes meaning depending on the [`VariableSourceType`](https://api.binary.ninja/binaryninja.enums.VariableSourceType.html). When a variable is of type `RegisterVariableSourceType`, its `storage` property represents the index into the register list for the given architecture. If the `source_type` is `StackVariableSourceType`, its `storage` property represents the stack offset of the variable. +The `storage` property changes meaning depending on the [`VariableSourceType`](https://api.binary.ninja/binaryninja.enums-module.html#binaryninja.enums.VariableSourceType). When a variable is of type `RegisterVariableSourceType`, its `storage` property represents the index into the register list for the given architecture. If the `source_type` is `StackVariableSourceType`, its `storage` property represents the stack offset of the variable. ``` >>> var @@ -95,7 +95,7 @@ Type objects are described in detail in the next section. ## The Type Object -Type objects are very similar to standard C types. A Type object's type can be determined through the object’s `type_class` property. Valid types are in the [`TypeClass`](https://api.binary.ninja/binaryninja.enums.TypeClass.html) enumeration: +Type objects are very similar to standard C types. A Type object's type can be determined through the object’s `type_class` property. Valid types are in the [`TypeClass`](https://api.binary.ninja/binaryninja.enums-module.html#binaryninja.enums.TypeClass) enumeration: ``` enum TypeClass @@ -168,7 +168,7 @@ A value type is simply a constant value. It is used mainly in demangling for typ ### FunctionTypeClass -The function type describes the return type, parameter list, and calling convention of a [function](https://api.binary.ninja/binaryninja.function.Function.html), among many other properties. +The function type describes the return type, parameter list, and calling convention of a [function](https://api.binary.ninja/binaryninja.function-module.html#binaryninja.function.Function), among many other properties. * `can_return` - boolean value indicating if the function can return * `calling_convention` - the calling convention this function uses @@ -193,7 +193,7 @@ Array types function similarly to pointer types however the array type knows how ### EnumerationTypeClass -`Enumeration` types function much the same way they do in C, providing a mapping between a name and corresponding constant. The object itself contains a `members` property and a list of [`EnumerationMember`](https://api.binary.ninja/binaryninja.types.EnumerationMember.html) objects each containing a name and value. +`Enumeration` types function much the same way they do in C, providing a mapping between a name and corresponding constant. The object itself contains a `members` property and a list of [`EnumerationMember`](https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.EnumerationMember) objects each containing a name and value. ### StructureTypeClass @@ -219,7 +219,7 @@ Most of the above should be self-explanatory except for the `UnknownNamedTypeCla ## The Instruction Set -The instruction set is made up of [`MediumLevelILInstruction`](https://api.binary.ninja/binaryninja.mediumlevelil.MediumLevelILInstruction.html) objects. Let's start exploring by using the python console to poke around at some instructions. Open up a binary in Binary Ninja and retrieve an MLIL instruction: +The instruction set is made up of [`MediumLevelILInstruction`](https://api.binary.ninja/binaryninja.mediumlevelil-module.html#binaryninja.mediumlevelil.MediumLevelILInstruction) objects. Let's start exploring by using the python console to poke around at some instructions. Open up a binary in Binary Ninja and retrieve an MLIL instruction: ``` >>> inst = current_mlil[8] @@ -230,7 +230,7 @@ The instruction set is made up of [`MediumLevelILInstruction`](https://api.binar `current_mlil` is mapped to whatever function is currently being viewed and is not generally available to those writing plugins, as your plugin could be headless. The bracket operators tell the API to get the MLIL instruction at index 8 for the current function. -There are a number of properties that can be queried on the [`MediumLevelILInstruction`](https://api.binary.ninja/binaryninja.mediumlevelil.MediumLevelILInstruction.html) object, and the validity of these properties changes depending on what the current operation is. If we look at the `operation` of `inst` we can see it is a `MLIL_CALL` instruction. +There are a number of properties that can be queried on the [`MediumLevelILInstruction`](https://api.binary.ninja/binaryninja.mediumlevelil-module.html#binaryninja.mediumlevelil.MediumLevelILInstruction) object, and the validity of these properties changes depending on what the current operation is. If we look at the `operation` of `inst` we can see it is a `MLIL_CALL` instruction. ``` >>> inst.operation diff --git a/docs/dev/bnil-overview.md b/docs/dev/bnil-overview.md index 480b9873..b0ffefe2 100644 --- a/docs/dev/bnil-overview.md +++ b/docs/dev/bnil-overview.md @@ -12,7 +12,7 @@ Except that overview isn't telling the whole story! The real stack of BNIL _actu  -This short introduction is a very brief guide to BNIL with much more details in [Part 1: LLIL](bnil-llil.md) and [Part 2: MLIL](bnil-mlil.md). +This short introduction is a very brief guide to BNIL with much more details in [Part 1: LLIL](bnil-llil.md), [Part 2: MLIL](bnil-mlil.md), and [Part 3: HLIL](bnil-hlil.md). ## Summary @@ -20,10 +20,10 @@ While the image above shows a bit of the logical relationships between each IL i - Lifted IL is intended to be a straight-forward translation from native instruction semantics to LLIL instructions. Note that Lifted IL and LLIL share the same instructions and are substantively similar with a few small differences. - Low Level IL (LLIL) is converted from Lifted IL, LLIL removes NOP instructions and folds flags into conditional instructions but otherwise contains the same operations as Lifted IL. -- Low Level IL SSA Form is an [SSA](conceps.md#static-single-assignment-basics) form of LLIL. +- Low Level IL SSA Form is an [SSA](concepts.md#static-single-assignment-basics) form of LLIL. - Mapped Medium Level IL is a translation layer between LLIL that is rarely needed for analysis but is useful for the translation process. - Medium Level IL (MLIL) translates registers and memory accesses into variables, types are associated with variables, platform information is used to generate call sites with types (both inferred and explicit) and parameters, data flow is calculated and constants are propagated. -- Medium Level IL SSA Form is an [SSA](conceps.md#static-single-assignment-basics) form of MLIL. +- Medium Level IL SSA Form is an [SSA](concepts.md#static-single-assignment-basics) form of MLIL. - High Level IL (HLIL) [builds on](https://api.binary.ninja/binaryninja.function-module.html#binaryninja.function.Function.request_debug_report) (use the `high_level_il` report type) MLIL by adding higher level control flow instructions, a number of dead-code and variable passes as well as other simplification/folding. It also includes an [AST](https://api.binary.ninja/binaryninja.highlevelil-module.html#binaryninja.highlevelil.HighLevelILInstruction.ast). @@ -117,7 +117,7 @@ Here's what that instruction might look like when selected with the IL Hierarchy Be warned though! HLIL in particular is very tree-based. LLIL and MLIL are much safer to use the above paradigm of simply iterating through top-level instructions. -Make sure to also check out the specifics of each IL level for more details: [LLIL](https://docs.binary.ninja/dev/bnil-llil.html), [MLIL](https://docs.binary.ninja/dev/bnil-mlil.html) (HLIL not yet complete) +Make sure to also check out the specifics of each IL level for more details: [LLIL](bnil-llil.md), [MLIL](bnil-mlil.md), [HLIL](bnil-hlil.md) ## Visitors diff --git a/docs/dev/concepts.md b/docs/dev/concepts.md index 5dcf0a3a..daf120b1 100644 --- a/docs/dev/concepts.md +++ b/docs/dev/concepts.md @@ -1,12 +1,5 @@ # Important Concepts -## Mapping between ILs - -ILs in general are critical to how Binary Ninja analyzes binaries and we have much more [in-depth](bnil-overview.md) documentation for BNIL (or Binary Ninja Intermediate Language -- the name given to the family of ILs that Binary Ninja uses). However, one important concept to summarize here is that the translation between each layer of IL is many-to-many. Going from disassembly to LLIL to MLIL can result in more or less instructions at each step. Additionally, at higher levels, data can be copied, moved around, etc. You can see this in action in the UI when you select a line of HLIL and many LLIL or disassembly instructions are highlighted. - -APIs that query these mappings are plural. So for example, while `current_hlil.llil` will give a single mapping, `current_hlil.llils` will return a list that may contain multiple mappings. - - ## REPL versus Scripts @@ -24,15 +17,29 @@ Another difference in the REPL / scripting console is that the scripting console  -## Operating on IL versus Native +## Auto vs User + +In the Binary Ninja API, there are often two parallel sets of functions with `_auto_` and `_user_` in their names. For example: [add_user_segment](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.add_user_segment) and [add_auto_segment](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.add_auto_segment). So what's the difference? Auto functions are those that are expected to be run _automatically_, every time the file is loaded. So for example, if you're writing a [custom file loader](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/nsf.py) that will parse a particular binary format, you would use `_auto_` functions because each time the file is opened your loader will be used. The results of auto functions are **not** saved in a `.bndb` database because it's expected that whatever produced them originally will again when the file is re-opened or analyzed. This means that even if you're writing a plugin to make changes on a file during analysis, you likely want to use the `_user_` set of APIs so that the changes your plugin causes will separately be saved to the database as if they were done by a user. + +## Concepts for ILs + +### Mapping between ILs + +ILs in general are critical to how Binary Ninja analyzes binaries and we have much more [in-depth](bnil-overview.md) documentation for BNIL (or Binary Ninja Intermediate Language -- the name given to the family of ILs that Binary Ninja uses). However, one important concept to summarize here is that the translation between each layer of IL is many-to-many. Going from disassembly to LLIL to MLIL can result in more or less instructions at each step. Additionally, at higher levels, data can be copied, moved around, etc. You can see this in action in the UI when you select a line of HLIL and many LLIL or disassembly instructions are highlighted. + +APIs that query these mappings are plural. So for example, while `current_hlil.llil` will give a single mapping, `current_hlil.llils` will return a list that may contain multiple mappings. + + + +### Operating on IL versus Native Generally speaking, scripts should operate on ILs. The available information far surpasses the native addresses and querying properties and using APIs almost always beats directly manipulating bytes. However, when it comes time to change the binary, there are some operations that can only be done at a simple virtual address. So for example, the [comment](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.set_comment_at) or [tag](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.add_tag) APIs (among others) work off of native addressing irrespective of IL level. -## Instruction Index vs Expression Index +### Instruction Index vs Expression Index It is easy to confuse ExpressionIndex and InstructionIndex properties in the API. While they [are both integers](https://github.com/Vector35/binaryninja-api/blob/dev/python/highlevelil.py#L49-L50) they mean different things and it's important to keep them straight. The Instruction Index is a unique index for that given IL level for that given function. However, because BNIL is [tree-based](bnil-overview.md), when there are nested expresses the expression index may be needed. These indexes are also unique per-function and per-IL level, but they are _distinct_ from instruction indexes even though they may occasionally be similar since they each start at 0 for a given function! -## Static Single Assignment Basics +### Static Single Assignment Basics Our [BNIL Overview](bnil-overview.md) mentions Static Single Assignment (SSA) without explaining what it is. You can of course always check out [Wikipedia](https://en.wikipedia.org/wiki/Static_single-assignment_form), but here's a quick summary of what it is, why we expose multiple SSA forms of our ILs and how you can use it to improve your plugin writing with BNIL. @@ -86,4 +93,39 @@ One reason that having an "end" might be useful in a function (as opposed to the So how do you tell Binary Ninja how big a function is? The answer is you don't directly, but you can instead direct its analysis. For example, if a function is improperly not marked as a noreturn function, edit the function properties via the right-click menu and set the property and all calls to it will end analysis at that point in any callees. -Likewise, you can do things like change memory permissions, patch in invalid instructions, [change an indirect branch's targets](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/jump_table.py), or use [UIDF](https://binary.ninja/2020/09/10/user-informed-dataflow.html) to influence analysis such that it ends where desired.
\ No newline at end of file +Likewise, you can do things like change memory permissions, patch in invalid instructions, [change an indirect branch's targets](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/jump_table.py), or use [UIDF](https://binary.ninja/2020/09/10/user-informed-dataflow.html) to influence analysis such that it ends where desired. + +## Working with Strings + +### Unicode Support + +If you're opening a file with non-ascii string encodings, make sure to use one of the "open with options" [methods](../guide/index.md#loading-files) and choose the appropriate code pages. These settings are under the `analysis` / `unicode` section of the settings dialog. If you're not sure which code page you should be enabling, consider using a library like [chardet](https://chardet.readthedocs.io/en/latest/usage.html). For example, here's a file both with and without the "CJK Unified Ideographs" block being enabled: + + + + + +???+ Warning "Tip" + Note that there is a small bug and some string length calculations will be off with certain characters as shown above. + +### Wide Strings in the UI + +Many Windows applications use wide (UTF-16 or UTF-32) strings. By default these should be identified during initial string analysis as long as the default settings are still enabled. + + + +However, you can always change a variable to a `wchar_t` with `[SHIFT] a` and to a `wchar32_t` with `[CTRL-SHIFT] a` in the UI. + +### "Strings" versus Strings + +The strings view in Binary Ninja are based on all identified strings found in a manner similar to the command-line "strings" tool (with `-a` if you're using a GPL Binutils version). However, this is not the same as a string having a string type in Binary Ninja! + +Several settings under analysis limits are relevant to identifying these strings: + + - Maximum String Length: The maximum string length that will be copied (defaults to 16384) + - Minimum String Length: The minimum string length for strings created during auto-analysis (defaults to 4) + - Maximum String Search: Maximum number of strings before string identification halts (defaults to 1048576) + +### Custom String Formats + +There is currently no support for custom types with different encoding mechanisms. Follow [#1334](https://github.com/Vector35/binaryninja-api/issues/1334) for more updates on when that feature is added. That said, it is possible to use a custom [DataRenderer](https://api.binary.ninja/binaryninja.datarender-module.html#binaryninja.datarender.DataRenderer) to improve the rendering using a custom type, but those strings will still not show up in the strings list or when included as variables in decompilation, only in linear view.
\ No newline at end of file diff --git a/docs/dev/cookbook.md b/docs/dev/cookbook.md index 61cd6386..abd1a089 100644 --- a/docs/dev/cookbook.md +++ b/docs/dev/cookbook.md @@ -7,31 +7,110 @@ One of the best ways to learn a complicated API is to simply find the right exam - [Gist Collection](https://gist.github.com/psifertex/6fbc7532f536775194edd26290892ef7): Jordan's collection of python examples usually created for (or contributed by) customers - [Offline examples](https://github.com/Vector35/binaryninja-api/tree/dev/python/examples): These examples are especially useful because they're included in your offline install as well, just look in the examples/python subfolder wherever Binary Ninja installed - That said, most of those examples tend to be more complex and so the following recipes are meant to be simple but useful building-blocks with which to learn useful techniques: + That said, most of those examples tend to be more complex and so the following recipes are meant to be simple but useful building-blocks with which to learn useful techniques. Many of them also make use of the built-in Python console's [magic variables](../guide/index.md#magic-console-variables): ## Recipes -### Accessing cross references +#### Getting all functions in a binary -This recipe is useful for iterating over all of the HLIL cross-references of a given interesting function: +```python +for func in bv.functions: + print(func.name) + print(func.start) + print(func.parameter_vars) + print(func.function_type) +``` + +#### Getting a specific function ```python -for ref in current_function.caller_sites: - print(ref.hlil) +func = bv.get_functions_by_name(here)[0] # Multiple functions can share the same name! +func = bv.get_function_at(here) # Shortcut for the next one +func = bv.get_functions_at(here)[0] # Binary Ninja support functions that overlap! +func = bv.get_function_containing(here) # Functions that contain the given address +# Just a note that using address to work with functions is fine +# But when working with ILs, addresses are approximate and can change for any given instruction ``` -But what if you don't have that function yet? +#### All forms of a function: + +```python +for func in bv.functions: + low_level_il = func.llil + low_level_il_ssa = func.llil.ssa_form -### Getting a function by name + medium_level_il = func.mlil + medium_level_il_ssa = func.mlil.ssa_form + + # Decompilation: + high_level_il = func.hlil + high_level_il_ssa = func.hlil.ssa_form + + base_function = <any>_level_il.source_function # Some helpers are only on the base function object! +``` + +#### All decompiled instructions in a binary ```python -bv.get_functions_by_name('_start') +for func in bv.functions: + for inst in func.hlil.instructions: + print(f"{inst.address} : {inst}") ``` -### Finding the function with the most bytes +or ```python -max(bv.functions, key=lambda x: x.total_bytes) +for func in bv.functions: + for bb in func.hlil: + for inst in bb: + print(f"{inst.address} : {inst}") +``` + +or + +```python +for inst in bv.hlil_instructions: + print(f"{inst.address} : {inst}") +``` + +#### Getting the decompiled instruction at an address + +```python +func = bv.get_functions_containing(here)[0] # You should probably be more robust than this +llil_inst = func.get_llil_at(here) # LLIL have the "closest" mapping to actual addresses, but you should still consider this volatile/fuzzy +hlil_inst = llil_inst.hlil # This is also very approximate + +# What's "more correct" walking down instead: +hlil_inst.mlil # Approximate "direct" mapping down +hlil_inst.mlils # All mlil instructions that contributed to this hlil instruction - most correct! +hlil_inst.llil # Approximate "direct" mapping down +hlil_inst.llils # All llil instructions that contributed to this hlil instruction - most correct! +# Be careful when working with address and mappings! We try to make them work as well as possible +# (and in most cases using the direct mapping is _fine_) +# But you should always be aware that they are approximate and can change! +``` + + +#### All callers of a function + +```python +current_function.callers +``` + +#### All locations where a function is called + +```python +for site in current_function.caller_sites: + addr = site.address + inst = site.hlil +``` + +#### All calls and call instructions in a function: + +```python +for site in current_function.call_sites: + addr = site.address + inst = site.hlil ``` ### Finding the most "connected" function @@ -42,6 +121,12 @@ As defined by having the highest sum of incoming and outgoing calls. Adjust acco max(bv.functions, key=lambda x: len(x.callers + x.callees)) ``` +### Finding the largest function (by most bytes) + +```python +max(bv.functions, key=lambda x: x.total_bytes) +``` + ### Querying possible values of a function parameter Is that memcpy length a bit too big? @@ -67,6 +152,64 @@ Make sure to check out the much more in-depth [applying annotations](annotation. current_function.type = Type.function(Type.void(), []) ``` +### Accessing cross references + +This recipe is useful for iterating over all of the HLIL cross-references of a given interesting function: + +```python +for ref in current_function.caller_sites: + print(ref.hlil) +``` + +#### Common variable APIs + +```python +for func in bv.functions: + all_vars = func.vars # This isn't the most meaningful thing to do, because.... + hlil_vars = func.hlil.vars # ...you probably only want the variables used in the IL you're looking at + hlil_aliased_vars = func.hlil.aliased_vars # ...but don't forget about aliased variables! + parameter_vars = func.parameter_vars # ...or parameter variables! + + var = hlil_vars[0] + if var.source_type == StackVariableSourceType: + print(var.storage) # var.storage is the variables stack offset, but ONLY IF the source type is `StackVariableSourceType` + + # There are many ways to *estimate* the size of a variable on the stack + print(var.offset_to_next_variable) # Distance to the next variable that Binary Ninja has identified on the stack + print(abs(var.storage)) # Absolute maximum size the variable can be until it overwrites the saved return pointer! + print(abs(var.type.width)) # If Binary Ninja gave the variable a type, or you manually applied a type, then you can get the size from that type + + # SSA + hlil_ssa_vars = func.hlil.ssa_vars # You can also get ssa variables + def_inst = func.hlil.ssa_form.get_ssa_variable_definition(ssa_vars[0]) # But if you want definitions, you need to use the ssa form + use_insts = func.hlil.ssa_form.get_ssa_variable_uses(ssa_vars[0]) # There's only ever one ssa definition, but potentially many uses +``` + +#### Working with Tags + +```python +# Data tags +bv.add_tag(here, "Crashes", "Description") + +# Function tags +current_function.add_tag("Important", "Look at this later!") + +# Function address tags +current_function.add_tag("Bug", "I think there's an overflow here?", here) +``` + +#### Logging + +```python +log.log_debug("Debug logs are hidden by default") +log.log_info("Info logs are displayed in the console") +log.log_warn("Warning logs will print in yellow text") +log.log_error("Errors are red!") +log.log_alert("This pops up a dialogue box!") + +log.log_error("You can add your own filter group easily to any of these APIs", "My Log Group") +``` + ### Find a variable's definition and all uses using SSA ```python diff --git a/docs/dev/flags.md b/docs/dev/flags.md index 2aabd244..e87c6a2d 100644 --- a/docs/dev/flags.md +++ b/docs/dev/flags.md @@ -182,9 +182,9 @@ This also means that you can mark instructions as producers of a group of many f You, the architecture author: * inform Binja of your architecture's flags by defining: - * flag names - * flag "roles" which are just their textbook behavior, if they qualify - * flag "write types" which groups of flags often set together + * flag names + * flag "roles" which are just their textbook behavior, if they qualify + * flag "write types" which groups of flags often set together * assign instructions as flag **producers** by passing `flag=` keyword parameter during lifting * assign instructions as flag **consumers** by passing IL flag expressions as operands during lifting diff --git a/docs/dev/index.md b/docs/dev/index.md index d9abcde6..7407d7bb 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -1,6 +1,6 @@ # Using the Binary Ninja API -Welcome to the Binary Ninja API documentation. Much like the [User Manual](../guide/index.md), some larger sections have been split off into their own sections on the left, while the table of contents for this documentation is on the right. +Welcome to the Binary Ninja API documentation. Much like the [User Guide](../guide/index.md), some larger sections have been split off into their own sections on the left, while the table of contents for this documentation is on the right. ## Language Specific Bindings @@ -34,5 +34,5 @@ The C++ API is what the Binary Ninja UI itself is built using so it's a robust a The Rust API is still experimental and lacks complete coverage for all core APIs. Documentation is available at: - - [Rust API](https://github.com/Vector35/binaryninja-api/tree/dev/rust) + - [Rust API](https://dev-rust.binary.ninja/) diff --git a/docs/dev/plugins.md b/docs/dev/plugins.md index 79a39982..f489f94b 100644 --- a/docs/dev/plugins.md +++ b/docs/dev/plugins.md @@ -206,7 +206,7 @@ class, and you should only ever handle Refs or bare pointers. When in doubt, fee Managing build infrastructure to build cross-platform native plugins can be a headache even for stables, let alone trying to track all dev releases. To help with that, we've published an [example plugin](https://github.com/CouleeApps/binja-ci-tests) that includes GitHub -actions to build on MacOS, Linux, and Windows. Combining this with something like a [plugin loader](https://github.com/rikodot/binja_native_plugin_loader) can simplify +actions to build on MacOS, Linux, and Windows. Combining this with something like a [plugin loader](https://github.com/rikodot/binja_native_sigscan_loader) can simplify both publishing and using native plugins. ### UI Plugins diff --git a/docs/dev/themes.md b/docs/dev/themes.md index 1c2931ed..0f2b7caf 100644 --- a/docs/dev/themes.md +++ b/docs/dev/themes.md @@ -1,4 +1,4 @@ -User themes are loaded from JSON files (with the `.bntheme` extension) found in the `themes` or `community-themes` subdirectories of your [user folder](../guide/index.md#user-folder). The default, full path to these folders is the following on each supported platform: +User themes are loaded from JSON files (with the `.bntheme` extension) found in the `themes` or `community-themes` (consider cloning the [community themes repo](https://github.com/Vector35/community-themes)) subdirectories of your [user folder](../guide/index.md#user-folder). The default, full path to these folders is the following on each supported platform: - macOS: `~/Library/Application Support/Binary Ninja/{themes,community-themes}` - Windows: `%APPDATA%\Binary Ninja\{themes,community-themes}` @@ -110,37 +110,33 @@ Colors marked "*required*" must be specified. Unmarked colors will hold default #### Tokens - + -1. `addressColor` (*required*) - Used to color memory addresses, e.g. `0x100003c5b` -2. `modifiedColor` (*required*) -3. `insertedColor` (*required*) -4. `notPresentColor` (*required*) -5. `selectionColor` (*required*) -6. `outlineColor` (*required*) -7. `registerColor` (*required*) - Used to color register names in code views, e.g. `rax` -8. `numberColor` (*required*) - Used to color number literals in code view, e.g. `0xf0` -9. `codeSymbolColor` (*required*) - Used to color local function names in code views, e.g. `sub_100003c50` -10. `dataSymbolColor` (*required*) - Used to color data symbols in code views, e.g. `data_100003e2c` -11. `stackVariableColor` (*required*) - Used to color stack variables in code views, e.g `var_8` -12. `importColor` (*required*) - Used to color imported function names in code views, e.g. `printf` -13. `instructionHighlightColor` (*required*) -14. `relatedInstructionHighlightColor` -15. `tokenHighlightColor` (*required*) -16. `annotationColor` (*required*) - Used to color annotations, such as hints -17. `commentColor` - Used to color comments -18. `opcodeColor` (*required*) - Used to color instruction opcodes in code views -19. `stringColor` (*required*) - Used to color string literals in code views, e.g. `"Hello, world!"` -20. `typeNameColor` (*required*) - Used to color user-defined type names in code views, e.g. `my_struct` -21. `fieldNameColor` (*required*) - Used to color structure member names in code views -22. `keywordColor` (*required*) - Used to color keywords in code views, e.g. `for` in HLIL -23. `uncertainColor` (*required*) - Used to color uncertain data in code views, such as variable types with low confidence -24. `exportColor` -25. `nameSpaceColor` -26. `nameSpaceSeparatorColor` -27. `operationColor` -28. `gotoLabelColor` -29. `tokenSelectionColor` +1. `addressColor` (*required*) - Used to color memory address tokens (e.g. `0x100003c5b`) +2. `registerColor` (*required*) - Used to color register names (e.g. `rax`) +3. `numberColor` (*required*) - Used to color number literals (e.g. `0xf0`) +4. `codeSymbolColor` (*required*) - Used to color local function names (e.g. `sub_100003c50`) +5. `dataSymbolColor` (*required*) - Used to color data symbols (e.g. `data_100003e2c`) +6. `stackVariableColor` (*required*) - Used to color stack variables (e.g `var_8`) +7. `importColor` (*required*) - Used to color imported function names (e.g. `printf`) +8. `annotationColor` (*required*) - Used to color annotations (e.g. hints), not shown in picture above +9. `commentColor` - Used to color code comments +10. `opcodeColor` (*required*) - Used to color instruction opcodes (e.g. `ebfe`) +11. `stringColor` (*required*) - Used to color string literals (e.g. `"Hello, world!"`) +12. `typeNameColor` (*required*) - Used to color user-defined type names (e.g. `my_struct`) +13. `fieldNameColor` (*required*) - Used to color structure member names +14. `keywordColor` (*required*) - Used to color keywords (e.g. `for` in HLIL) +15. `uncertainColor` (*required*) - Used to color uncertain data (e.g. variable types with low confidence) +16. `exportColor` - Used to color exported symbols, not shown in picture above +17. `nameSpaceColor` - Used to color a namespace, not shown in picture above +18. `nameSpaceSeparatorColor` - Used to color the separator between a namespace and the rest of the symbol, not shown in picture above +19. `operationColor` - *Currently unused* +20. `gotoLabelColor` - Used to color a `goto`'s label, not shown in picture above +21. `tokenSelectionColor` - Used to draw the background of tokens selected with click-and-drag +22. `outlineColor` (*required*) - Used to draw a box around tokens selected with click-and-drag +23. `tokenHighlightColor` (*required*) - Used to color the background of a token that has been selected with a single click or an arrow key +24. `instructionHighlightColor` (*required*) - Used to color the background of an instruction that has been selected with a single click or an arrow key +25. `relatedInstructionHighlightColor` - Used to color the background of instructions related to an instruction that has been selected with a single click or an arrow key The following colors are used for the Rainbow Braces setting (`ui.rainbowBraces`): @@ -153,20 +149,24 @@ The following colors are used for the Rainbow Braces setting (`ui.rainbowBraces` #### Hex View - + Each byte in hex view is given a background color based on its value. Values between `0x00` and `0xFF` will use a color interpolated between the `Dark` and `Light` colors specified below. -1. `backgroundHighlightDarkColor` - Used as the background color for bytes of value `0x00` -2. `backgroundHighlightLightColor` - Used as the background color for bytes of value `0xFF` -3. `boldBackgroundHighlightDarkColor` (*required*) -4. `boldBackgroundHighlightLightColor` (*required*) -5. `alphanumericHighlightColor` (*required*) - Used to color alphanumeric characters in hex views, takes precedence over printableHighlightColor -6. `printableHighlightColor` (*required*) - Used to color printable characters in hex views +1. `alphanumericHighlightColor` (*required*) - Used to color bytes that are alphanumeric characters if Color Highlight/ASCII and Printable is enabled in view options (default) +2. `printableHighlightColor` (*required*) - Used to color bytes that are non-alphanumeric printable characters if Color Highlight/ASCII and Printable is enabled in view options (default) +3. `modifiedColor` (*required*) - Used to color bytes that have been modified if Color Highlight/Modification is enabled in view options +4. `insertedColor` (*required*) - Used to color bytes that have been inserted if Color Highlight/Modification is enabled in view options +5. `notPresentColor` (*required*) - Used to color bytes that do not have a value +5. `backgroundHighlightDarkColor` (*required*) - Used as the background color for bytes of value `0x00` if Contrast/Normal is enabled in view options (default) +6. `backgroundHighlightLightColor` (*required*) - Used as the background color for bytes of value `0xFF` if Contrast/Normal is enabled in view options (default) +7. `boldBackgroundHighlightDarkColor` (*required*) - Used as the background color for bytes of value `0x00` if Contrast/High is enabled in view options +8. `boldBackgroundHighlightLightColor` (*required*) - Used as the background color for bytes of value `0xFF` if Contrast/High is enabled in view options +9. `selectionColor` (*required*) - Used as the background color on any bytes that have been selected via click-and-drag #### Linear View - + 1. `linearDisassemblyFunctionHeaderColor` (*required*) - Used as the background for function headers in linear view @@ -179,7 +179,7 @@ Each byte in hex view is given a background color based on its value. Values bet #### Graph View - + Both the graph background and individual graph nodes are actually painted as a gradient. To get a flat background instead, set the `Dark` and `Light` colors to the same color value. @@ -201,7 +201,7 @@ Both the graph background and individual graph nodes are actually painted as a g #### Highlighting - + 1. `blackStandardHighlightColor` (*required*) 2. `blueStandardHighlightColor` (*required*) @@ -215,24 +215,24 @@ Both the graph background and individual graph nodes are actually painted as a g #### Tab Bar -1. `tabBarTabActiveColor` -2. `tabBarTabHoverColor` -3. `tabBarTabInactiveColor` -4. `tabBarTabBorderColor` -5. `tabBarTabGlowColor` +1. `tabBarTabActiveColor` - Used to color the background of the currently active tab +2. `tabBarTabHoverColor` - Used to color the background of any tab the mouse is currently hovering over +3. `tabBarTabInactiveColor` - Used to color the background of any inactive tab +4. `tabBarTabBorderColor` - Used to draw the border around the currently active tab +5. `tabBarTabGlowColor` - Used to draw a glow or shadow around the border of the currently active tab #### Feature Map -1. `featureMapBaseColor` -2. `featureMapNavLineColor` -3. `featureMapNavHighlightColor` -4. `featureMapDataVariableColor` -5. `featureMapAsciiStringColor` -6. `featureMapUnicodeStringColor` -7. `featureMapFunctionColor` -8. `featureMapImportColor` -9. `featureMapExternColor` -10. `featureMapLibraryColor` +1. `featureMapBaseColor` - Used to color the background +2. `featureMapNavLineColor` - Used to color the line(s) that represent where you are in the binary +3. `featureMapNavHighlightColor` - Used as a highlight outside of the navigation line(s) +4. `featureMapDataVariableColor` - Used to highlight any area containing data variables +5. `featureMapAsciiStringColor` - Used to highlight any area containing ASCII strings +6. `featureMapUnicodeStringColor` - Used to highlight any area containing Unicode strings +7. `featureMapFunctionColor` - Used to highlight any area containing functions +8. `featureMapImportColor` - Used to highlight any area containing imported functions that are not from a library (see below) +9. `featureMapExternColor` - Used to highlight any area containing symbols that are externs +10. `featureMapLibraryColor` - Used to highlight any area containing symbolic functions or functions that are imported from a library #### Side Bar @@ -245,11 +245,13 @@ Both the graph background and individual graph nodes are actually painted as a g #### Mini-Graph -1. `miniGraphOverlayColor` + + +1. `miniGraphOverlayColor` - Used to color the area of the mini graph that is currently being displayed in graph view #### Script Console - + 1. `scriptConsoleOutputColor` - Used to color normal output in the console 2. `scriptConsoleWarningColor` - Used to color warnings in the console @@ -258,12 +260,20 @@ Both the graph background and individual graph nodes are actually painted as a g #### Panes -1. `activePaneBackgroundColor` -2. `inactivePaneBackgroundColor` + + +The image above shows an active pane on the left and an inactive pane on the right. + +1. `activePaneBackgroundColor` - Used to color the background of a pane that is active +2. `inactivePaneBackgroundColor` - Used to color the background of a pane that is inactive #### Status Bar -1. `statusBarServerConnectedColor` -2. `statusBarServerDisconnectedColor` -3. `statusBarServerWarningColor` -4. `statusBarProjectColor` + + +The image above shows the Enterprise button in the "connected" state (connected to a server named "dev") with an open project (named "Alex's Stuff"). + +1. `statusBarServerConnectedColor` - Used for the Enterprise button while connected to a server +2. `statusBarServerDisconnectedColor` - Used for the Enterprise button while disconnected from a server +3. `statusBarServerWarningColor` - Used for the Enterprise button when there is an issue with the server connection +4. `statusBarProjectColor` - Used for the Project button diff --git a/docs/dev/typelibraries.md b/docs/dev/typelibraries.md new file mode 100644 index 00000000..dccb9102 --- /dev/null +++ b/docs/dev/typelibraries.md @@ -0,0 +1,288 @@ +## Type Libraries + +Type Libraries are collections of type information (structs, enums, function types, etc.) stored in a file with the extension `.bntl`. + +Relative to the binaryninja executable, the default type library location is `../Resources/typelib` on macOS and `./typelib` on Linux and Windows. Individual .bntl files are organized in subdirectories named for the supported architecture. Users may include their own type libraries in the `typelib` folder in their [user folder](../guide/index.md#user-folder). + +The information in a type library is contained in two key-value stores: + +1. named types: key is the type name, value is the type +1. named objects: key is external symbol name, value is the type + +## How Binary Ninja Loads Type Libraries + +When a binary is opened, its platform is determined, all .bntl's are processed, and those matching the platform of the loaded binary are registered. A debug log will show: + +``` +Registered library 'libc.so.6' with platform 'linux-x86_64' +``` + +Then, those with either a filename or an alternative name matching the exact text of the binary's import command are imported, much like the native linker/loader. For example, in ELF, the .dynstr entry is used. + +``` +elf: searching for 'libc.so.6' in type libraries +Type library 'libc.so.6' imported +``` + +Type libraries for linux are ideally named after their realname, preserving the library minor version from which they were generated, and the soname in the alternatives list. In practice, naming them after their soname suffices. Using the linkname with no alternatives will prevent your library from loading. + +This requested name should be a soname, like "libfoo.so.1" but could be a linkname like "libfoo.so". (The ldconfig tool is responsible for creating symlinks from soname to realnames, like `/usr/lib/libfoo.so.1` -> `/usr/lib/libfoo.so.1.0`. See [tldp.org](https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html) for more information.). + +Binary Ninja's logic for determining a match is straightforward: + +``` +typelibname.removesuffix('.bntl') == requestedname or requestedname in alternativenames +``` + +Therefore, without any alternative names, `libc.so.bntl` will not be loaded by Binary Ninja if an ELF requests libc.so.6. + +We recommend and use the following convention: + +Type libraries should be named for the filename from which they were generated with the phrase ".bntl" added. When the source library contains additional minor and release number, like `libfoo.so.1.2.3` Binary Ninja would not load the resulting type library `libfoo.so.1.2.3.bntl` for an ELF requesting soname `libfoo.so.1`. Therefore the alternative names list should include the most specific version numbers, incrementally stripped down to the soname, and finally a linkname for good measure. + +Example: + +`libfoo.so.1.2.3` is used to generated `libfoo.so.1.2.3.bntl` + +The alternative names list should have: + +``` +libfoo.so.1.2.3 <-- includes version, minor, release (most specific) +libfoo.so.1.2 <-- includes version, minor (less specific) +libfoo.so.1 <-- includes version (soname) +libfoo.so <-- linkname +``` + +## Acquiring a Handle + +The platform class exposes handles to these imported type libraries with its `type_libraries` list and its `get_type_libraries_by_name()` function: + +```python +>>> bv.platform.type_libraries +[<typelib 'libm.so.6':x86_64':x86_64>, <typelib 'SYSCALLS':x86_64] +>>> bv.platform.get_type_libraries_by_name('libm.so.6') +[<typelib 'libm.so.6':x86_64>] +``` + +That requires the type library having been loaded. A more direct way is to load from a file path with the `load_from_file()` from `Typelibrary` class from `typelibrary` module: + +```python +>>> typelibrary.TypeLibrary.load_from_file('/path/to/libm_x86_64.so.6.bntl') +<typelib 'libm_x86_64.so.6':x86_64> +``` + +## Contents of Libraries + +The following demonstrates attributes of interest on a loaded type library in variable `tl`: + +```python + print(' name: %s' % tl.name) + print(' arch: %s' % tl.arch) + print(' guid: %s' % tl.guid) + print('dependency_name: %s' % tl.dependency_name) + print('alternate_names: %s' % tl.alternate_names) + print(' platform_names: %s' % tl.platform_names) + + print(' named_objects: %d' % len(tl.named_objects)) + for (key, val) in tl.named_objects.items(): + print('\t"%s" %s' % (str(key), str(val))) + + print(' named_types: %d' % len(tl.named_types)) + for (key,val) in tl.named_types.items(): + print('\t"%s" %s' % (str(key), str(val))) +``` + +Named objects (via dictionary `.named_objects` are functions signatures and a module's exported variables. Named types (via dictionary `.named_types`) are the textual named you might use to declare a variable. For example, + +For example, `.named_objects['fegetexceptionflag']` looks up its function prototype: `int32_t (fexcept_t* flagp)` and `.named_types['fexcept_t']` looks up `uint16_t`, its typedef. + +## Creating + +You may also wish to use the [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py) example script included both online and offline in your installation path. + +Types entered by manual entry can be exported from the binary view using `export_type_to_library()`. + +Manual creation is achieved by creating a new type library, associating the correct platform and architecture, adding types, finalizing, and writing to a file. Example: + +```python +arch = binaryninja.Architecture['x86_64'] + +struct = Structure() +struct.append(Type.pointer(arch, Type.char()), 'name') +struct.append(Type.int(4), 'age') +struct.append(Type.int(4), 'height') +struct.append(Type.int(4), 'weight') + +typelib = binaryninja.typelibrary.TypeLibrary.new(arch, 'test.so.1.4') +typelib.add_named_type('human', binaryninja.types.Type.structure_type(struct)) +typelib.add_alternate_name('test.so.1') #don't forget this step! +typelib.add_alternate_name('test.so') +typelib.finalize() +typelib.write_to_file('test.so.1.bntl') +``` + +## Other Type Library Questions + +_What's a named type vs. just a type?_ + +Some variable definitions have type information, but don't produce a type name useful for future definitions, examples: + +- `enum {A=1,B=2} foo;` : foo has type with no type name (it does have a variable name) +- `struct {int A; int B;} bar;` : bar has type with no type name + +In C, enum and struct definitions can create a new type name as a byproduct of a definition by using a "tag name": + +- `enum MyEnum {A=1,B=2} foo;` : foo has the type named `MyEnum` +- `struct MyStruct {int A; int B;} bar;` : bar has the type named `MyStruct` + +In the second set of examples, the types are named, and that name could be used to declare other variables, like `enum MyEnum bar2;` and `struct MyStruct bar2`. + +Functions' types are not named. The function name is considered the name of a function object, and the function's type is anonymous. + +In summary: + +```c +typedef int foo; // type:int, name:foo + +// structs, without and with a "tag name" +struct {int A; int B;} foo; // type:struct{int A, intB;}, name:<anonymous> +struct MyStruct {int A; int B;} foo; // type:struct{int A, intB;}, name:MyStruct + +// enumerations, without and with a "tag name" +enum {A=1,B=2} foo; // type:enum{A=1,B=2}, name:<anonymous> +enum MyEnum {A=1,B=2} foo; // type:enum{A=1,B=2}, name:MyEnum + +// functions +int main(int ac, char **av); // type int ()(int, char **), name:<anonymous> +typedef int (MyFunc)(int ac, char **av); // type int ()(int, char **), name:MyFunc + +``` + +_How does Binary Ninja decide when to use a typelibrary (.bntl) file?_ + +Type Libraries are loaded when the corresponding library is imported by a BinaryView. (i.e. if an exe imports `ntdll.dll`, binja will look in the bv's platform for type libraries named ntdll.bntl and load the first one it finds) + +_What's the difference between a named type and a named object?_ + +A named type is a type with a name that can identify it. For example, `color` is the name of type `enum {RED=0, ORANGE=1, YELLOW=2, ...}`. + +A named object is the name of an external/imported symbol for which the type library has type information. For example, `MessageBoxA` is the name of a function whose type is `int ()(HWND, LPCSTR, LPCSTR, UINT)`. + +_How do I find what type of type a type object is? How many are there?_ + +I've seen "types of types", "sorts of types", "kinds of types", "classes of types" used to differentiate the varieties of possible types, and there are probably more. Binary Ninja uses "class", example: + +``` +>>> type_obj.type_class +<TypeClass.FunctionTypeClass: 8> +``` + +In [enums.py](https://api.binary.ninja/_modules/binaryninja/enums.html#TypeClass) we can see Binary Ninja currently thinks of types falling into 13 classes: `Void`, `Bool`, `Integer`, `Float`, `Structure`, `Enumeration`, `Pointer`, `Array`, `Function`, `VarArgs`, `Value`, `NamedTypeReference`, `WideCharType`. + +Compare this to LLDB, which also uses the term "class", and currently has 19 of them: `Array`, `BlockPointer`, `Builtin`, `Class`, `ComplexFloat`, `ComplexInteger`, `Enumeration`, `Function`, `MemberPointer`, `ObjCObject`, `ObjCInterface`, `ObjCObjectPointer`, `Pointer`, `Reference`, `Struct`, `Typedef`, `Union`, `Vector`, `Other`. + +Compare this to GDB, which uses the term "type code" and has 25 of them. + +_Where are function parameter names stored?_ + +While technically not part of the type, having names of function parameters is very useful and can thus be optionally stored in a type. + +Function types (types with `.type_class == FunctionTypeClass`) have a `.parameters` attribute, a list of [`FunctionParameter`](https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.FunctionParameter) objects. When those objects have `.name==''` you get the bare bones function types like `int ()(int, char **)`. When those objects have their `.name` populated you get the more meaningful `int ()(int argc, char **argv)`. + +_How do I manually load a type library?_ + +``` +>>> bv.add_type_library(TypeLibrary.load_from_file('test.bntl')) +``` + +_How can I manually load a type object?_ + +``` +>>> bv.import_library_object('_MySuperComputation') +<type: int32_t (int32_t, int32_t, char*)> +``` + +_Why doesn't the types view show the types imported from type libraries?_ + +Because the type libraries added to a binary view only makes their type information _available_ for use. The types view will show a type from a type library only after it is used (on demand). + +Try this experiment: + +- note `bv.type_libraries`, `bv.types` +- add a type library with `bv.add_type_library(TypeLibrary.load_from_file('test.bntl'))` +- note that `bv.type_libraries` is extended, but `bv.types` is unchanged! +- note `bv.get_type_by_name('Rectangle')` returns nothing +- set the type of some data to `struct Rectangle` (using `y` in linear view or via any other method described above) +- `bv.types` is extended, and the types view shows `struct Rectangle` in the auto types + +_What's a named type reference?_ + +Named Type References are a way to refer to a type by name without having its declaration immediately available. + +For example, examine this struct from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py): + +``` +struct Rectangle2 { + int width; + int height; + struct Point center; +} +``` + +We don't know at this moment what a `struct Point is`. Maybe we've already added it. Maybe we'll add it later. Maybe it's in another type library. But we want to add a Rectangle now. So we leave the center field as a reference to the named type `struct Point`. + +Load the resulting `test.bntl` in your binary and try to set some data to type `struct Rectangle2` and you'll be met with this message: + +``` +TypeLibrary: failed to import type 'Point'; referenced but not present in library 'libtest.so.1` +``` + +This makes sense! Now go to types view and `define struct Point { int x; int y; }` and try again, success! + +``` +100001000 struct rectangle_unresolved data_100001000 = +100001000 { +100001000 int32_t width = 0x5f0100 +100001004 int32_t height = 0x5f030005 +100001008 struct Point center = +100001008 { +100001008 int32_t x = 0x655f686d +10000100c int32_t y = 0x75636578 +100001010 } +100001008 } +``` + +You should repeat the experiment using `struct Rectangle` and see that you're allowed to create variables with type containing _pointers_ to unresolved type references. + +_How are types represented?_ + +By a hierarchy of objects from [api/python/types.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/types.py) referencing one another. The "glue" object is [`binaryninja.types.Type`](https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.Type) and depending on the complexity of the type it represents (stored in its `.type_class` attribute), it could have an attribute with more information. For instance, if the `binaryninja.types.Type` has `.type_class == FunctionTypeClass` then its `.parameters` attribute is a list of [`binaryninja.types.FunctionParameter`](https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.FunctionParameter). See [typelib_dump.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_dump.py) for how this can work. + +As an example, here is the hierarchical representation of `type struct Rectangle` from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py) + +``` +typelib.named_types["Rectangle"] = +---------------------------------- +Type class=Structure + Structure + StructureMember "width" + Type class=Integer width=4 + StructureMember "height" + Type class=Integer width=4 + StructureMember "center" + Type class=Pointer + Type class=NamedTypeReference + NamedTypeReference <named type: struct Point> +``` + +Here is the representation of `type int ()(int, int)` named `MyFunctionType` from [typelib_create.py](https://github.com/Vector35/binaryninja-api/blob/dev/python/examples/typelib_create.py): + +_When do named objects get used?_ + +When a binary is loaded and its external symbols is processed, the symbol names are searched against the named objects from type libraries. If there is a match, it obeys the type from the type library. Upon success, you'll see a message like: + +``` +type library test.bntl found hit for _DoSuperComputation +``` + +At this moment, there is no built in functionality to apply named objects to an existing Binary Ninja database. diff --git a/docs/dev/uidf.md b/docs/dev/uidf.md index a6422246..8a421dfa 100644 --- a/docs/dev/uidf.md +++ b/docs/dev/uidf.md @@ -1,8 +1,8 @@ ## User Informed Data Flow -Binary Ninja now implements User-Informed DataFlow (UIDF) to improve the static reverse engineering experience of our users. This feature allows users to set the value of a variable at the Medium-level IL layer and have the internal dataflow engine propagate it through the control-flow graph of the function. Besides constant values, Binary Ninja supports various `PossibleValueSet` states as containers to help inform complex variable values. +Binary Ninja now implements User-Informed DataFlow (UIDF) to improve the static reverse engineering experience of our users. This feature allows users to set the value of a variable and have the internal dataflow engine propagate it through the control-flow graph of the function. Besides constant values, Binary Ninja supports various `PossibleValueSet` states as containers to help inform complex variable values. -## Usage +## Sample Usage For the purpose of demonstration, we are going to use a simple [crackme](https://github.com/Vector35/uidf-example). @@ -14,18 +14,18 @@ Let's first look at `rax_2`. We see that the conditional at Medium Level IL (MLI First, note the definition site of `rax_2` is instruction 7 and then right click there to see "Set User Variable Value...". Users can also bind the action to a hotkey or access it via the command palette. - + The dialog box to set the variable value contains two fields: * `PossibleValueSet` type: `PossibleValueSet` is used to represent the values that a variable can take during analysis. This can also be representative of the values that the variable would take during concrete execution of the program. For instance if the variable value is `<const 0x0>`, it implies that the mapped register would always contain the value `0x0` during the lifetime of the variable. `UndeterminedValue` on the other hand suggests that the analysis could not figure out an appropriate value and representation in the available `PossibleValueSet`s. For the program analysis folks, this is analogous to the top of the dataflow lattice. An `EntryValue` represents the bottom. `PossibleValueSet`s provide complex container types such as `InSetOfValues`, `SignedRangeValue`, `UnsignedRangeValue`, etc. * Value: Represents the concrete value which is to be informed to the analysis. Please refer to the parse_possiblevalueset [documentation](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.parse_possiblevalueset) for a detailed description of the parser rules for the various `PossibleValueSet`s. - + -The parser checks if the string can be parsed into a [`PossibleValueSet`](https://api.binary.ninja/binaryninja.function.PossibleValueSet.html). The behavior of this parser is similar to the `Goto Address` dialog box or the [`parse_expression` API](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.parse_expression). This enables the use of keywords such as `$here`, `$start/$end`, etc. +The parser checks if the string can be parsed into a [`PossibleValueSet`](https://api.binary.ninja/binaryninja.variable-module.html#binaryninja.variable.PossibleValueSethtml). The behavior of this parser is similar to the `Goto Address` dialog box or the [`parse_expression` API](https://api.binary.ninja/binaryninja.binaryview-module.html#binaryninja.binaryview.BinaryView.parse_expression). This enables the use of keywords such as `$here`, `$start/$end`, etc. -And of course, like everything else in Binary Ninja you can accomplish this using the API. `PossibleValueSet`s can be created through instantiation of the [`PossibleValueSet`](https://api.binary.ninja/binaryninja.function.PossibleValueSet.html) class or through parsing a value string: +And of course, like everything else in Binary Ninja, you can accomplish this using the API. `PossibleValueSet`s can be created through instantiation of the [`PossibleValueSet`](https://api.binary.ninja/binaryninja.variable-module.html#binaryninja.variable.PossibleValueSet) class or through parsing a value string: ```python >>> # Creating a PossibleValueSet object through instance creation @@ -62,7 +62,7 @@ Let's set the value of the variable `rax_2` to `ConstantValue <0x0>`. Here's how Looking at the binary this input should lead to "Incorrect" output. Setting the variable value automatically triggers a re-analysis of the function. This leads to a dataflow analysis pass over the function seeded with the user-informed value for that specific variable. On hovering over the variable token, we see the value that it takes is the one we informed. - + The API makes it easy to query for the value of the conditional to check if it has been statically evaluated. We can also get a list of all the currently informed variable values through the API: @@ -74,13 +74,24 @@ The API makes it easy to query for the value of the conditional to check if it h >>> ``` +## Variable View Sidebar + +The variable sidebar also supports both setting and seeing the values of UIDF variables. Note that because the UIDF values are location-sensitive you'll want to make sure you've clicked into an appropriate location in the MLIL or HLIL to see the appropriate values annotated in the variable view. + + + +## Jump Table Example + +???+ Warning "Tip" + Since these docs were originally written, this jump table no longer behaves the same way and is automatically solved during Binary Ninja's normal analysis. The documentation is being left in for illustrative purposes as it's still a valuable consideration even if this exact example no longer applies. + Let us see the High Level IL (HLIL) representation of `main` before UIDF: - + Below is the HLIL of `main` after UIDF. If constant propagation leads to conditionals which can be evaluated at analysis time, its value is set to `if (true) or if (false)` at the MLIL layer. More so, the branch of the conditional where the control flow is guaranteed to not reach is eliminated from HLIL. The sub-graph of the CFG which results in the "Correct" output is eliminated. Here, HLIL benefits from dataflow calculations made at MLIL and the updated CFG is indicative of that. - + We believe that UIDF is great for experimenting around with during the initial phase of the RE life cycle. If the value for a variable at a location is already informed, right-clicking on the variable at any usage site where the variable has the same SSA version will show an option to "Clear User Variable Value". This clears the informed value and triggers reanalysis. And as always, you can just use undo if setting the user variable value was the last action (using CMD/CTRL+Z). @@ -100,7 +111,7 @@ Another example of using UIDF would be to construct jump-tables with known bound Now, with the introduction of UIDF, there's another easy way to help inform the jump-table analysis. Consider the MLIL representation for an indirect jump as shown below: - + Let us inform value for the variable `eax` to be in the range `1` to `0x13`. @@ -113,12 +124,12 @@ Let us inform value for the variable `eax` to be in the range `1` to `0x13`. A jump table will be created with a base of `0x804891c` with 14 possible targets for the jump at 0x804855b. Below is a cropped screenshot of the resulting HLIL which consists of a switch construct which depends on the variable value. This [binary](http://pages.cs.wisc.edu/~xmeng/Testsuite.tar.gz) is available on [BN Cloud](https://cloud.binary.ninja/bn/fb2131dd-e726-4009-8428-80f9f0b5e8ae) and is originally written by [Xiaozhu Meng](http://pages.cs.wisc.edu/~xmeng). - + ## Design notes -Static-only reverse engineering limits the ability to test hypothesis formulated during the initial analysis phase. To that end, analysts usually turn to emulation/debugging to experiment with a binary. This can often be time-consuming if the focus is on checking dependencies between program variables or for verifying the dataflow, or worse, may be impossible for some platforms that are difficult to dynamically analyze. With UIDF, we aim to provide our users with the ability to interact with the dataflow engine more richly and partly achieve the same results without leaving the flow graph or linear analysis window. UIDF could also aid in better understanding targets of indirect jumps or fixing jump tables. Research [[1](#ref1),[2](#ref2)] shows that it is imperative that binary analysis tools adopt a more user-guided approach to solve RE/VR tasks and UIDF tries to be a step up that slope. We aim to continue work in this direction! +Static-only reverse engineering limits the ability to test hypotheses formulated during the initial analysis phase. To that end, analysts usually turn to emulation/debugging to experiment with a binary. This can often be time-consuming if the focus is on checking dependencies between program variables or for verifying the dataflow, or worse, may be impossible for some platforms that are difficult to dynamically analyze. With UIDF, we aim to provide our users with the ability to interact with the dataflow engine more richly and partly achieve the same results without leaving the flow graph or linear analysis window. UIDF could also aid in better understanding targets of indirect jumps or fixing jump tables. Research [[1](https://acmccs.github.io/papers/p347-shoshitaishviliA.pdf),[2](https://dl.acm.org/doi/fullHtml/10.1145/3290607.3313040)] shows that it is imperative that binary analysis tools adopt a more user-guided approach to solve RE/VR tasks and UIDF tries to be a step up that slope. We aim to continue work in this direction! -UIDF primarily operates on the MLIL layer. Binary Ninja performs constant propagation, resolves call parameters and lifts stack load/stores into a variables. This combined with the SSA form enables an effective dataflow analysis pipeline. MLIL also provides control over dead-code elimination and allows us to prevent removal of redundant instructions (eg. variable definitions for informed variables) and basic blocks (removed due to the resulting dataflow). Binary Ninja's tiered IL representation allows the higher layers to benefit from simplifying transformations at the lower layers. Through fixing jump tables or branch elimination, UIDF influences HLIL output as a user would expect without having drastic changes to the MLIL representation. +UIDF primarily operates on the MLIL layer. Binary Ninja performs constant propagation, resolves call parameters and lifts stack load/stores into a variables. However, the UI can apply a UIDF constraint on an HLIL variable or MLIL variable and apply it appropriately. This combined with the SSA form enables an effective dataflow analysis pipeline. MLIL also provides control over dead-code elimination and allows us to prevent removal of redundant instructions (eg. variable definitions for informed variables) and basic blocks (removed due to the resulting dataflow). Binary Ninja's tiered IL representation allows the higher layers to benefit from simplifying transformations at the lower layers. Through fixing jump tables or branch elimination, UIDF influences HLIL output as a user would expect without having drastic changes to the MLIL representation. -Looking towards the future, improvements to the dataflow solver engine would inherently increase the efficacy of UIDF. This could include support for more operations on the complex [`PossibleValueSet`](https://api.binary.ninja/binaryninja.function.PossibleValueSet.html) containers or the ability to play well with program structure such as loops. +Looking towards the future, improvements to the dataflow solver engine would inherently increase the efficacy of UIDF. This could include support for more operations on the complex [`PossibleValueSet`](https://api.binary.ninja/binaryninja.variable-module.html#binaryninja.variable.PossibleValueSet) containers or the ability to play well with program structure such as loops. diff --git a/docs/dev/workflows.md b/docs/dev/workflows.md index 678f6510..7ff91503 100644 --- a/docs/dev/workflows.md +++ b/docs/dev/workflows.md @@ -13,7 +13,6 @@ This capability is experimental with no guarantees of API stability or future co - [What is Workflows](#what-is-workflows) - [Getting Started](#getting-started) - - [Availability](#availability) - [Enable](#enable) - [Explore](#explore) - [Examples](#examples) |
