Merge branch 'dev' into test_stack_adjust

23 files changed, 143 insertions, 47 deletions

diff --git a/docs/about/open-source.md b/docs/about/open-source.md
index be9725d0..81ef1e69 100644
--- a/docs/about/open-source.md
+++ b/docs/about/open-source.md
@@ -32,6 +32,7 @@ The previous tools are used in the generation of our documentation, but are not
 
 * Other
     - [yasm] ([yasm license] - 2-clause BSD) used for assembling x86 and x64
+    - [xed] ([xed license] - Apache License 2.0) used for disassembling x86, x64, and x16
     - [capstone] ([capstone license] - 3-clause BSD) used in the PPC architecture module as an example of how to wrap an external disassembler
 
 * Upvector update library
@@ -97,5 +98,7 @@ Please note that we offer no support for running Binary Ninja with modified Qt l
 [unsign]: https://github.com/steakknife/unsign
 [yasm license]: https://github.com/yasm/yasm/blob/master/BSD.txt
 [yasm]: http://yasm.tortall.net/
+[xed]: http://www.github.com/intelxed/xed/
+[xed license]: http://www.github.com/intelxed/xed/blob/master/LICENSE
 [zlib license]: http://www.zlib.net/zlib_license.html
 [zlib]: http://www.zlib.net/
diff --git a/docs/dev/bnil-llil.md b/docs/dev/bnil-llil.md
index 7748d140..71cedd96 100644
--- a/docs/dev/bnil-llil.md
+++ b/docs/dev/bnil-llil.md
@@ -2,7 +2,7 @@
 
 The Binary Ninja Intermediate Language (BNIL) is a semantic representation of the assembly language instructions for a native architecture in Binary Ninja.  BNIL is actually a family of intermediate languages that work together to provide functionality at different abstraction layers. This developer guide is intended to cover some of the mechanics of the LLIL to distinguish it from the other ILs in the BNIL family. 
 
-![BNIL-LLIL Selected](/images/BNIL.png)
+![BNIL-LLIL Selected](/img/BNIL.png)
 
 The Lifted IL is very similar to the LLIL and is primarily of interest for Architecture plugin authors. If you're writing an analysis plugin, you'll always want to be working at LLIL or higher. During each stage of the lifting process a number of transformations take place, and each layer of IL can have different instructions. Because of this, you can not rely on an instruction from one layer existing in another.
 
@@ -10,7 +10,7 @@ The Lifted IL is very similar to the LLIL and is primarily of interest for Archi
 
 Since doing is the easiest way to learn lets start with a simple example binary and step through analyzing it using the python console. 
 
-![Low Level IL Option >](/images/llil_option.png)
+![Low Level IL Option >](/img/llil_option.png)
 
  - Download [chal1](../files/chal1) and open it with Binary Ninja
  - Next, bring up the `Low Level IL` view by clicking in the options pane at the bottom of the screen
@@ -143,7 +143,7 @@ Now with some knowledge of the `LowLevelIL` class lets try to do something with
 ```
 >>> for block in current_function.low_level_il:
 ...  for instr in block:
-...   if instr.operation == LowLevelILOperation.LLIL_SET_REG and instr.dest == 'rdx':
+...   if instr.operation == LowLevelILOperation.LLIL_SET_REG and instr.dest.name == 'rdx':
 ...    print instr.address, instr.instr_index, instr
 ... 
 4196490 14 rdx = [rax].q
diff --git a/docs/dev/documentation.md b/docs/dev/documentation.md
index 830bd4ec..16a1c9ee 100644
--- a/docs/dev/documentation.md
+++ b/docs/dev/documentation.md
@@ -27,7 +27,7 @@ To contribute to the Binary Ninja documentation, first sign the [contribution li
 Changing documentation for the API itself is fairly straight forward. Use [doxygen style comment blocks](https://www.stack.nl/~dimitri/doxygen/manual/docblocks.html) in C++ and C, and [restructured text blocks](http://thomas-cokelaer.info/tutorials/sphinx/docstring_python.html) for python for the source. The user documentation is located in the `api/docs/` folder and the API documentation is generated from the config in the `api/api-docs` folder.
 
 !!! Tip "Tip"
-    When updating user documentation, the `mkdoc serve` feature is particularly helpful.
+    When updating user documentation, the `mkdocs serve` feature is particularly helpful.
 
 [contribution license agreement]: https://binary.ninja/cla.pdf
 [Vector 35]: https://vector35.com/
diff --git a/docs/docs.css b/docs/docs.css
index ebd56907..6009a104 100644
--- a/docs/docs.css
+++ b/docs/docs.css
@@ -1,37 +1,93 @@
+/*
 code {
-	color: #000;
+color: #000;
 }
 
-.admonition {
-	background: rgb(128, 198, 223);
-	color: #fff;
+.article pre code {
+background: rgba(0, 0, 0, 0);
 }
+ */
 
-.tip {
-	background: rgb(110, 110, 110);
-	color: #fff;
+.admonition.warning .admonition-title, .admonition.warning {
+background-color: #d73726 !important;
+color: #fff !important;
+}
+
+.admonition.tip .admonition-title, .admonition.tip {
+background: #6e6e6e !important;
+color: #fff !important;
+}
+
+a {
+color: #d73726;
 }
 
+a:visited {
+color: #990000;
+}
+
+a:hover {
+color: #ff8080;
+}
+
+
 img[alt$=">"] {
-  float:right;
-  display: inline-block;
-  padding-left: 10px;
+	float:right;
+	display: inline-block;
+	padding-left: 10px;
 }
 
 img[alt$="<"] {
-  float:left;
-  display: inline-block;
-  padding-right: 10px;
+	float:left;
+	display: inline-block;
+	padding-right: 10px;
 }
 
 img[alt$="><"] {
-    display: block;
-    max-width: 100%;
-    height: auto;
-    margin: auto;
-    float: none!important;
+	display: block;
+	max-width: 100%;
+	height: auto;
+	margin: auto;
+	float: none!important;
 }
 
-.article pre code {
-	background: rgba(0, 0, 0, 0);
+@media only screen and (min-width: 76.25em) {
+	.md-main__inner {
+		max-width: none;
+	}
+	.md-sidebar--primary {
+		left: 0;
+	}
+	.md-sidebar--secondary {
+		right: 0;
+		margin-left: 0;
+		-webkit-transform: none;
+		transform: none;
+	}
+	.wy-nav-content {
+		max-width: 100%;
+	}
 }
+
+pre {
+    overflow-x: auto;
+}
+
+.wy-side-nav-search {
+	background-color: #d73726;
+	color: #fff;
+}
+
+.toctree-l1 a:visited, .toctree-l1 a:hover, .toctree-l1 a {
+color: #ccc;
+}
+
+.wy-nav-top {
+	background-color: #d73726 !important;
+	color: #fff !important;
+}
+
+.wy-nav-top a:visited, .wy-nav-top a:hover, .wy-nav-top a, .wy-side-nav-search a:visited, .wy-side-nav-search a:hover, .wy-side-nav-search a {
+color: #fff;
+}
+
diff --git a/docs/getting-started.md b/docs/getting-started.md
index 7e7e4acb..5148b9d5 100644
--- a/docs/getting-started.md
+++ b/docs/getting-started.md
@@ -33,7 +33,7 @@ Contents of the user folder includes:
 - `repositories/`: Folder containing files and plugins managed by the [Plugin Manager API](https://api.binary.ninja/binaryninja.pluginmanager-module.html)
 - `settings.json`: Advanced settings (see [settings](#settings))
 
-![license popup >](/images/license-popup.png "License Popup")
+![license popup >](/img/license-popup.png "License Popup")
 
 ## License
 
@@ -57,12 +57,12 @@ You can load files in many ways:
 5. Opening a file from a URL via the `⌘-l` or `⌃-l` hotkey
 6. Opening a file using the binaryninja: url handler. For security reasons, the url handler requires you to confirm a warning before opening a file via the url handler. The url handler can open remote URLs like: `binaryninja:https://captf.com/2015/plaidctf/pwnable/datastore_7e64104f876f0aa3f8330a409d9b9924.elf`, or even local files like `binarynina://bin/ls` in cases where you wish to script up Binary Ninja from a local webapp.
 
-![recent files](/images/recent.png "Recent Files")
+![recent files](/img/recent.png "Recent Files")
 
 
 ## Analysis
 
-![auto analysis ><](/images/analysis.png "Auto Analysis")
+![auto analysis ><](/img/analysis.png "Auto Analysis")
 
 As soon as you open a file, Binary Ninja begins its auto-analysis.
 
@@ -76,7 +76,7 @@ Errors or warnings during the load of the binary are also shown in the status ba
 
 Navigating code in Binary Ninja is usually a case of just double-clicking where you want to go. Addresses, references, functions, jmp edges, etc, can all be double-clicked to navigate. Additionally, The `g` hotkey can navigate to a specific address in the current view.
 
-![graph view](/images/view-choices.png "Different Views")
+![graph view](/img/view-choices.png "Different Views")
 
 Switching views happens multiple ways. In some instances, it's automatic (clicking a data reference from graph view will navigate to linear view as data is not shown in the graph view), and there are multiple ways to manually change views as well. While navigating, you can use the view hotkeys (see below) to switch to a specific view at the same location as the current selection. Alternatively, the view menu in the bottom-right can be used to change views without navigating to any given location.
 
@@ -111,11 +111,11 @@ Switching views happens multiple ways. In some instances, it's automatic (clicki
 
 ### Graph View
 
-![graph view](/images/graphview.png "Graph View")
+![graph view](/img/graphview.png "Graph View")
 
 The default view in Binary Ninja when opening a binary is a graph view that groups the basic blocks of disassembly into visually distinct blocks with edges showing control flow between them.
 
-![graph view context >](/images/graphcontext.png "Graph View Contet Menu")
+![graph view context >](/img/graphcontext.png "Graph View Contet Menu")
 
 Features of the graph view include:
 
@@ -129,7 +129,7 @@ Features of the graph view include:
 
 ### View Options
 
-![options ><](/images/options.png "options")
+![options ><](/img/options.png "options")
 
 Each of the views (Hex, Graph, Linear) have a variety of options configurable in the bottom-right of the UI.
 
@@ -162,7 +162,7 @@ Current options include:
 
 ### Hex View
 
-![hex >](/images/hex.png "hex view")
+![hex >](/img/hex.png "hex view")
 
 The hexadecimal view is useful for view raw binary files that may or may not even be executable binaries. The hex view is particularly good for transforming data in various ways via the `Copy as`, `Transform`, and `Paste from` menus. Note that `Transform` menu options will transform the data in-place, and that these options will only work when the Hex View is in the `Raw` mode as opposd to any of the binary views (such as "ELF", "Mach-O", or "PE").
 
@@ -170,15 +170,15 @@ Note that any changes made in the Hex view will take effect immediately in any o
 
 ### Xrefs View
 
-![xrefs >](/images/xrefs.png "xrefs")
+![xrefs >](/img/xrefs.png "xrefs")
 
 The xrefs view in the lower-left shows all cross-references to a given location or reference. Note that the cross-references pane will change depending on whether an entire line is selected (all cross-references to that address are shown), or whether a specific token within the line is selected.
 
-One fun trick that the xrefs view has up its sleeve: when in [Hex View](#hexview), a large range of memory addresses can be selected and the xrefs pane will show all xrefs to any location within that range of data.
+One fun trick that the xrefs view has up its sleeve: when in [Hex View](#hex-view), a large range of memory addresses can be selected and the xrefs pane will show all xrefs to any location within that range of data.
 
 ### Linear View
 
-![linear](/images/linear.png "linear view")
+![linear](/img/linear.png "linear view")
 
 Linear view is a hybrid view between a graph-based disassembly window and the raw hex view. It lists the entire binary's memory in a linear fashion and is especially useful when trying to find sections of a binary that were not properly identified as code or even just examining data.
 
@@ -187,7 +187,7 @@ Linear view is most commonly used for identifying and adding type information fo
 
 ### Function List
 
-![function list >](/images/functionlist.png "Function List")
+![function list >](/img/functionlist.png "Function List")
 
 The function list in Binary Ninja shows the list of functions currently identified. As large binaries are analyzed, the list may grow during analysis. The function list starts with known functions such as the entry point, exports, or using other features of the binary file format and explores from there to identify other functions.
 
@@ -198,7 +198,7 @@ The function list also highlights imports, and functions identified with symbols
 
 ### Script (Python) Console
 
-![console >](/images/console.png "Console")
+![console >](/img/console.png "Console")
 
 The integrated script console is useful for small scripts that aren't worth writing as full plugins.
 
@@ -247,7 +247,7 @@ Binary Ninja supports loading PDB files through the built in PDB plugin. When se
 
 ## Preferences/Updates
 
-![preferences >](/images/preferences.png "Preferences")
+![preferences >](/img/preferences.png "Preferences")
 
 Binary Ninja automatically updates itself by default. This functionality can be disabled in the preferences by turning off the `Update to latest version automatically` option. Updates are silently downloaded in the background and when complete an option to restart is displayed in the status bar. Whenever Binary Ninja restarts next, it will replace itself with the new version as it launches.
 
@@ -259,15 +259,21 @@ Settings are stored in the _user_ directory in the file `settings.json`. Each to
 
 |Plugin     | Setting                  | Type         | Default                                        | Description                                                                                   |
 |----------:|-------------------------:|-------------:|-----------------------------------------------:|:----------------------------------------------------------------------------------------------|
+| analysis  | autorunLinearSweep       | boolean      | True                                           | Automatically run linear sweep when opening a binary for analysis                             |
+| analysis  | enabledUnicodeBlocks     | list(string) | []                                             | Defines which Unicode blocks to consider when searching for strings                           |
+| analysis  | enableUTF8               | boolean      | True                                           | Whether or not to consider UTF-8 code points when searching for strings                       |
+| analysis  | enableUTF16              | boolean      | True                                           | Whether or not to consider UTF-16 code points when searching for strings                      |
+| analysis  | enableUTF32              | boolean      | True                                           | Whether or not to consider UTF-32 code points when searching for strings                      |
 | analysis  | max-function-size        | integer      | 65536                                          | Any functions over this size will not be automatically analyzed and require manual override   |
 | core      | linux\_ca\_bundle        | string       | ""                                             | Certificate authority (.pem or .crt) file to be used for secure downloads                     |
 | core      | linux\_ca\_dir           | string       | ""                                             | Certificate authority directory (for distributions without a CA bundle)                       |
 | ui        | activeContent            | boolean      | True                                           | Allow Binary Ninja to connect to the web to check for updates                                 |
 | ui        | colorblind               | boolean      | True                                           | Choose colors that are visible to those with red/green colorblind                             |
 | ui        | debug                    | boolean      | False                                          | Enable developer debugging features (Additional views: Lifted IL, and SSA forms)              |
-| ui        | recent-file-limit        | integer      | 10                                             | Specify limit for number of recent files                                                       |
+| ui        | recent-file-limit        | integer      | 10                                             | Specify limit for number of recent files                                                      |
+| ui        | scriptingProvider        | string       | "Python"                                       | Specify the registered ScriptingProvider that controls the 'Console' in the UI                |
 | pdb       | local-store-absolute     | string       | ""                                             | Absolute path specifying where the pdb symbol store exists on this machine, overrides relative path |
-| pdb       | local-store-relative     | string       | "symbols"                                      | Path *relative* to the binaryninja _user_ directory, sepcifying the pdb symbol store            |
+| pdb       | local-store-relative     | string       | "symbols"                                      | Path *relative* to the binaryninja _user_ directory, sepcifying the pdb symbol store          |
 | pdb       | auto-download-pdb        | boolean      | True                                           | Automatically download pdb files from specified symbol servers                                |
 | pdb       | symbol-server-list       | list(string) | ["http://msdl.microsoft.com/download/symbols"] | List of servers to query for pdb symbols.                                                     |
 | python    | interpreter              | string       | "python27.{dylib,dll,so.1}"                    | Python interpreter to load if one is not already present when plugins are loaded              |
@@ -275,13 +281,13 @@ Settings are stored in the _user_ directory in the file `settings.json`. Each to
 Below is an example `settings.json` setting various options:
 ```
 {
-	"ui" :
-	{
-		"activeContent" : false,
-		"colorblind" : false,
-		"debug" : true
-		"recent-file-limit" : 10
-	}
+    "ui" :
+    {
+        "activeContent" : false,
+        "colorblind" : false,
+        "debug" : true
+        "recent-file-limit" : 10
+    }
     "pdb" :
     {
         "local-store-absolute" : "C:\Symbols",
@@ -290,6 +296,11 @@ Below is an example `settings.json` setting various options:
     }
 }
 ```
+
+## Unicode Support
+
+Currently, Unicode support for Big Endian strings is very limited. Also, UTF-16 only supports Basic Latin code points.
+
 ## Getting Support
 
 Vector 35 offers a number of ways to get Binary Ninja [support](https://binary.ninja/support/).
diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md
index 5e310dfb..0aafd4a7 100644
--- a/docs/guide/troubleshooting.md
+++ b/docs/guide/troubleshooting.md
@@ -94,6 +94,32 @@ ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 /usr/lib/x86_64-linux-gnu/libssl
 
 One Gentoo user [reported][issue672] a failed SSL certificate when trying to update. The solution was to copy over `/etc/ssl/certs/ca-certificates.crt` from another Linux distribution.
 
+### NixOS
+
+Here's a customer-provided nix derivation file for the Binary Ninja demo. Adapt as necessary for other versions, or hop onto our slack (specifically the #unsupported-distros channel) to find out more:
+
+```
+{ stdenv, autoPatchelfHook, makeWrapper, fetchurl, unzip, libGL, glib, fontconfig, xlibs, dbus, xkeyboard_config }:
+stdenv.mkDerivation rec {
+  name = "binary-ninja-demo";
+  buildInputs = [ autoPatchelfHook makeWrapper unzip libGL stdenv.cc.cc.lib glib fontconfig xlibs.libXi xlibs.libXrender dbus ];
+  src = fetchurl {
+    url = "https://cdn.binary.ninja/installers/BinaryNinja-demo.zip";
+    sha256 = "1yq2kgrhrwdi7f66jm1w5sc6r49hdhqnff9b0ysr5k65w9kxhl1k";
+  };
+  
+  buildPhase = ":";
+  installPhase = ''
+    mkdir -p $out/bin
+    mkdir -p $out/opt
+    cp -r * $out/opt
+    chmod +x $out/opt/binaryninja
+    makeWrapper $out/opt/binaryninja \
+          $out/bin/binaryninja \
+          --prefix "QT_XKB_CONFIG_ROOT" ":" "${xkeyboard_config}/share/X11/xkb"
+  '';
+}```
+
 ## API
 
  - If the GUI launches but the license file is not valid when launched from the command-line, check that you're using the right version of Python. Only a 64-bit Python 2.7 is supported at this time. Additionally, the [personal][purchase] edition does not support headless operation.
diff --git a/docs/images/BNIL.png b/docs/img/BNIL.png
index 37fc9997..37fc9997 100644
--- a/docs/images/BNIL.png
+++ b/docs/img/BNIL.png
diff --git a/docs/images/analysis.png b/docs/img/analysis.png
index 088d4f32..088d4f32 100644
--- a/docs/images/analysis.png
+++ b/docs/img/analysis.png
diff --git a/docs/images/console.png b/docs/img/console.png
index e4401dea..e4401dea 100644
--- a/docs/images/console.png
+++ b/docs/img/console.png
diff --git a/docs/favicon.ico b/docs/img/favicon.ico
index e99e1d31..e99e1d31 100644
--- a/docs/favicon.ico
+++ b/docs/img/favicon.ico
diff --git a/docs/images/functionlist.png b/docs/img/functionlist.png
index 2fb2d475..2fb2d475 100644
--- a/docs/images/functionlist.png
+++ b/docs/img/functionlist.png
diff --git a/docs/images/graphcontext.png b/docs/img/graphcontext.png
index d453f5e1..d453f5e1 100644
--- a/docs/images/graphcontext.png
+++ b/docs/img/graphcontext.png
diff --git a/docs/images/graphview.png b/docs/img/graphview.png
index 0d190a9f..0d190a9f 100644
--- a/docs/images/graphview.png
+++ b/docs/img/graphview.png
diff --git a/docs/images/hex.png b/docs/img/hex.png
index f30440a6..f30440a6 100644
--- a/docs/images/hex.png
+++ b/docs/img/hex.png
diff --git a/docs/images/license-popup.png b/docs/img/license-popup.png
index d61b2b13..d61b2b13 100644
--- a/docs/images/license-popup.png
+++ b/docs/img/license-popup.png
diff --git a/docs/images/linear.png b/docs/img/linear.png
index 8eaeb50f..8eaeb50f 100644
--- a/docs/images/linear.png
+++ b/docs/img/linear.png
diff --git a/docs/images/llil_option.png b/docs/img/llil_option.png
index ee42c3b9..ee42c3b9 100644
--- a/docs/images/llil_option.png
+++ b/docs/img/llil_option.png
diff --git a/docs/images/logo.png b/docs/img/logo.png
index c1e1ecb1..c1e1ecb1 100644
--- a/docs/images/logo.png
+++ b/docs/img/logo.png
diff --git a/docs/images/options.png b/docs/img/options.png
index 10af6920..10af6920 100644
--- a/docs/images/options.png
+++ b/docs/img/options.png
diff --git a/docs/images/preferences.png b/docs/img/preferences.png
index 29b3033e..29b3033e 100644
--- a/docs/images/preferences.png
+++ b/docs/img/preferences.png
diff --git a/docs/images/recent.png b/docs/img/recent.png
index 13f3ebe7..13f3ebe7 100644
--- a/docs/images/recent.png
+++ b/docs/img/recent.png
diff --git a/docs/images/view-choices.png b/docs/img/view-choices.png
index 9ea16dcf..9ea16dcf 100644
--- a/docs/images/view-choices.png
+++ b/docs/img/view-choices.png
diff --git a/docs/images/xrefs.png b/docs/img/xrefs.png
index ab7f056f..ab7f056f 100644
--- a/docs/images/xrefs.png
+++ b/docs/img/xrefs.png