Docs for IL modification

2 files changed, 414 insertions, 0 deletions

diff --git a/docs/dev/bnil-modifying.md b/docs/dev/bnil-modifying.md
new file mode 100644
index 00000000..6de6e703
--- /dev/null
+++ b/docs/dev/bnil-modifying.md
@@ -0,0 +1,410 @@
+# Modifying ILs
+
+Modifying ILs during lifting is a complex process with many factors to consider.
+
+APIs for modifying ILs are under active development, with support depending on which level of IL and which language you use.
+
+Key:
+- ✅ Full support
+- ⚠️ Partial support
+- ❌ No support
+
+| IL Level        | C++ | Python | Rust |
+|-----------------|:---:|:------:|:----:|
+| Lifted IL       |  ✅  |   ✅    |  ✅   |
+| Low Level IL    |  ✅  |   ✅    |  ✅   |
+| Medium Level IL | ⚠️* |   ✅    |  ❌   |
+| High Level IL   | ⚠️* |   ❌    |  ❌   |
+
+\* Modifying Medium and High Level IL in C++, while some APIs exist, is incomplete and certain more complicated operations are not possible yet. 
+
+## Choosing A Level of IL To Modify
+
+One of the most important considerations to make when modifying ILs is, "What level of IL do I need to modify?"
+Depending on what effects you want to have, different levels will work better.
+
+- Modifying **Lifted IL** is similar to a generic Architecture Extension, letting you change instructions directly out of the lifter.
+Note that Lifted IL has no SSA forms, dataflow, stack offsets, or even resolved flags.
+- Modifying **Low Level IL** lets you affect operations on registers, which eventually affect values of variables (in MLIL).
+Depending on where in the pipeline you insert your action, stack offsets may or may not be calculated, and most dataflow information is not available.
+Notably, any `PossibleValueSet` calculations require MLIL or higher, so only `RegisterValue` dataflow results are available (and only after generating SSA form).
+- Modifying **Medium Level IL** lets you modify uses of variables with the ability to solve for values in many cases, but is too late to let you change IL to create new variables properly. Changes can affect HLIL structuring.
+- Modifying **High Level IL** lets you affect the final presentation of the decompiled code directly, but requires working with the expression AST and cannot affect things like value set analysis. HLIL does not have the same guarantees that variables and statements exist in the original code, and expressions can be changed completely from their original forms or be created from thin air.
+
+There are certain ILs which you cannot modify, for a variety of reasons:
+
+- **SSA Forms** are generated by the core from the various Non-SSA Form ILs at each stage of analysis. You should not modify them, as they are considered analysis derived from the Non-SSA Form. They will likely be regenerated from the source Non-SSA Form without consulting any changes you might make to them. 
+- **Disassembly** should be modified using an Architecture Extension, not a Workflow Activity.
+- **Lifted IL SSA Form** does not exist, as Lifted IL gets lifted directly to LLIL, which does have an SSA form.
+- **Mapped MLIL** is lifted directly from LLIL SSA Form and not user-editable. Generally speaking, users and developers of Workflow Actions should not need to consult Mapped MLIL.
+- **Pseudo-C** and other Language Representation forms are not a distinct level of IL, but are simply alternate renderings of HLIL.
+
+## Considerations
+
+Based on your IL of choice, there are a bunch of places you can insert your modification action, which affects the information available to use or modify.
+
+- **Lifted IL** modification needs to happen before the `core.function.analyzeAndExpandFlags` stage, which translates Lifted IL into Low Level IL. After that stage, Lifted IL is never again consulted.
+  - **You probably do not want to modify Lifted IL with a Workflow**. Instead, consider modifying the Architecture directly ([most of which are Open Source on our GitHub](https://github.com/Vector35/binaryninja-api/tree/dev/arch)) or making your Activity modify Low Level IL.
+  - Modifying flag definitions and usages is likely impossible, as the Semantic Flags system will consult the Architecture to determine what IL to generate when a flag is used, not your Workflow Action. There is currently no way to change this behavior.
+- **Low Level IL** modification is generally done right before the `core.function.generateMediumLevelIL` stage, after the final Low Level IL function is emitted. After Medium Level IL is generated, changes to the Low Level IL will have limited effects on the other ILs (because most things refer to MLIL). 
+  - If you want to make changes to LLIL that can affect the stack pointer resolution, you will want to insert your action before `core.function.analyzeStackAdjustment` instead.
+- **Medium Level IL** modification is generally done right after `core.function.generateMediumLevelIL` since (as of writing) the MLIL translation is all done in that one monolithic step and future steps are simply processing and annotations. This may change some time in the future.
+- **High Level IL** modification is generally done right after `core.function.generateHighLevelIL` since HLIL generation is also monolithic. This also may change eventually.
+
+## Writing The Plugin
+
+Once you figure out which level of IL you want to modify, and where in the Workflow pipeline you want to do modifications, it is time to write some code!
+Writing Workflow Activities 
+
+### Terminology
+
+- An **IL Function** is a collection of **IL Instructions**, grouped into **IL Basic Blocks**
+- An **IL Basic Block** is a contiguous sequence of **IL Instructions** between a start and end **Instruction Index**, which only has incoming branches at the start and outgoing branches at the end.
+- An **IL Expression** is a single-operation expression, which may have child expressions or be the child of another expression
+- An **Expression Index** points to an **IL Expression** and may be used as an operand of another **IL Expression**.
+- An **IL Instruction** is a top-level **IL Expression**.
+- An **Instruction Index** points to an **IL Instruction** and may be contained within the start and end regions of an **IL Basic Block**.
+
+### Registering Your Plugin
+
+In order for your modifications to run, you need to register a Workflow Activity.
+There are a couple of steps involved in this process:
+
+First, you need to make a clone of the default workflow, making it mutable and allowing you to insert your Activity. 
+You can either name the clone the same thing as the metaAnalysis workflow to make your Activity available by default,
+or clone to a different name, which will require users to select that workflow in Open with Options.
+```py
+# This Workflow will replace metaAnalysis and be used by default 
+wf = Workflow("core.function.metaAnalysis").clone("core.function.metaAnalysis")
+
+# Users will need to pick this Workflow in Open with Options
+wf = Workflow("core.function.metaAnalysis").clone("MyCustomWorkflow")
+```
+
+Then, define a new Activity on the Workflow, which allows you to run your code to make modifications.
+
+```py
+def rewrite_action(context: AnalysisContext):
+    # Actual modification code goes here (documented below)...
+
+# Define the custom activity configuration
+wf.register_activity(Activity(
+    configuration=json.dumps({
+        "name": "extension.my_extension.do_the_thing",
+        "title": "My IL Modifications",
+        "description": "Does some IL modifications.",
+        "eligibility": {
+            "auto": {
+                "default": False  # Controls if the workflow is enabled by default
+            }
+        }
+    }),
+    action=rewrite_action
+))
+
+# Add the new action to the workflow, with position as described above
+wf.insert_after("core.function.generateMediumLevelIL", [
+    "extension.my_extension.do_the_thing"
+])
+wf.register()
+```
+
+A more in-depth explanation of the Workflows system is available [here](./workflows.md), with complete documentation of the Eligibility system and Workflows in general. This guide, however, will just give enough of an example for you to get to writing IL modification code.
+
+### Replacing Trivial Instructions
+
+Replacing exactly one instruction with exactly one other instruction is relatively simple:
+
+1. Create the new expression in the function, and note its **Expression Index**
+2. Find the instruction you wish to remove, and get its **Expression Index** as well
+3. Use `*LevelILFunction::ReplaceExpr` (Python: `*LevelILFunction.replace_expr`), passing the **Expression Index** of the instruction to remove and the **Expression Index** of the replacement
+4. Now, the backing expression for the target instruction has been replaced by your new expression. There may be dangling expressions no longer referenced by any instructions; this is fine. They will simply be skipped during later stages of lifting.
+5. Call `*LevelILFunction::Finalize` (Python: `*LevelILFunction.finalize`) to BFS traverse the function's **IL Instructions** starting with instruction 0, reconstructing the **IL Basic Blocks** of the function
+6. Call `*LevelILFunction::GenerateSSAForm` (Python: `*LevelILFunction.generate_ssa_form`) to reconstruct the SSA Form of the function, updating dataflow calculations
+
+If you wish to replace more than one instruction, or replace an instruction with more than one new instructions, you will need to use the more complicated method described below.
+
+### Adding Instructions and Replacing Multiple Instructions
+
+If you want to insert new instructions into a function, or want to replace an instruction with than one instruction, you will need to construct a new **IL Function** based on the original **IL Function** but with your changes.
+This is a rather cumbersome process and it can be easy to make mistakes.
+
+First, you need to construct a new **IL Function** based on the existing function.
+Let's call the new function `new_func` and the existing function `old_func`.
+
+```py
+# ... workflow boilerplate
+def wf_activity(context: AnalysisContext):
+    old_func = context.mlil  
+    # Create a new IL Function based on the old one.
+    # Make sure you use the LLIL from the analysis context, and *do not* go through Function.llil (it has not been updated yet)
+    new_func = MediumLevelILFunction(old_mlil.arch, low_level_il=context.llil)
+    # Tell the new function that we are copying from an existing function (this transfers various metadata like block labels)
+    new_func.prepare_to_copy_function(old_mlil)
+
+    # continues ...
+```
+
+Then, going block-by-block, copy the instructions from `old_func` to `new_func`.
+When you reach a part where you want to insert new instructions (or replace instructions), do that instead of simply copying the old instructions.
+
+```py
+    # previous code...
+    
+    # Copy each block in the old function to the new function
+    for old_block in old_mlil.basic_blocks: 
+        # Copy block labels and tell new IL instructions they came from this block
+        new_mlil.prepare_to_copy_block(old_block)
+        
+        # Copy each instruction from the old block to the new function
+        # Blocks "contain" instructions by knowing their start and end instruction indices
+        # Any instructions in that range as in the block
+        for old_instr_index in range(old_block.start, old_block.end):
+            # Retrieve instruction from old function
+            old_instr: MediumLevelILInstruction = old_mlil[old_instr_index]
+
+            # Make sure the new instruction we insert has the correct location information
+            new_mlil.set_current_address(old_instr.address, old_block.arch)
+
+            # continues...
+```
+
+For each instruction, determine if you want to insert new instructions before it, or replace it with something else. You _probably_ want to do this step ahead of time and apply all the changes at once, but if your changes are simple enough you can get away with doing it inside the loop.
+
+```py
+            # previous code...
+
+            if want_changes:
+                # Create new instructions in the function for whatever changes you desire
+
+                # All ILs: Be sure to pass the location of the previous instruction when creating a new expression so mappings work
+                new_expr = new_mlil.nop(ILSourceLocation.from_instruction(old_instr))
+                # MLIL and higher: pass the location of the previous instruction to append() so cross-IL mappings are preserved
+                new_mlil.append(new_expr, ILSourceLocation.from_instruction(old_instr))
+            else:
+                # Copy the instruction as-is
+                
+                # Copy the instruction from the old function to the new function using `copy_to` (deep copy)
+                # MLIL and higher: also pass the location of the previous instruction to append() so cross-IL mappings are preserved
+                new_mlil.append(old_instr.copy_to(new_mlil), ILSourceLocation.from_instruction(old_instr))
+
+    # continues ...
+```
+
+And finally, regenerate dataflow:
+
+```py
+    # previous code ...
+
+    # Create IL basic blocks
+    new_mlil.finalize()
+    # Do dataflow, etc
+    new_mlil.generate_ssa_form()
+    # Assign to the context to apply modifications
+    context.mlil = new_mlil
+```
+
+## Useful Tools
+
+Modifying ILs can be quite tricky, and there are not that many guard rails to prevent you from breaking analysis or even crashing Binary Ninja.
+Here are a couple of tips and tricks to help you with this process:
+
+### Show IL Opcodes Render Option
+
+In Linear and Graph views, you can enable Show IL Opcodes under the View Options menu on the toolbar.
+This debug rendering mode includes the operation of every IL Expression in the output text, giving a very verbose but visual display of expressions.
+In addition, it applies to Find In Text results, so you can enable Show IL Opcodes and then search for e.g. `LLIL_ADC` to find all `LLIL_ADC` expressions.
+
+```
+Show IL Opcodes off:
+  10 @ 1400553e7  bool rcx = temp0 s< 0x12817497a9ff848a
+  11 @ 1400553f1  uint32_t rdx_1 = zx.d(rcx)
+  12 @ 1400553f4  uint32_t rdx_2 = rdx_1 << 3
+  13 @ 1400553f7  uint64_t rcx_1 = zx.q(rdx_2)
+  14 @ 1400553f9  uint64_t rcx_2 = zx.q((rcx_1 + (rcx_1 << 1)).d)
+```
+
+```
+Show IL Opcodes on:
+  10 @ 1400553e7  (MLIL_SET_VAR.b bool rcx = (MLIL_CMP_SLT.q (MLIL_VAR.q temp0) s< (MLIL_CONST.q 0x12817497a9ff848a)))
+  11 @ 1400553f1  (MLIL_SET_VAR.d uint32_t rdx_1 = (MLIL_ZX.d zx.d((MLIL_VAR.b rcx))))
+  12 @ 1400553f4  (MLIL_SET_VAR.d uint32_t rdx_2 = (MLIL_LSL.d (MLIL_VAR.d rdx_1) << (MLIL_CONST.d 3)))
+  13 @ 1400553f7  (MLIL_SET_VAR.q uint64_t rcx_1 = (MLIL_ZX.q zx.q((MLIL_VAR.d rdx_2))))
+  14 @ 1400553f9  (MLIL_SET_VAR.q uint64_t rcx_2 = (MLIL_ZX.q zx.q((MLIL_LOW_PART.d ((MLIL_ADD.q (MLIL_VAR.q rcx_1) + (MLIL_LSL.q ((MLIL_VAR.q rcx_1) << (MLIL_CONST.b 1))))).d))))
+```
+
+### `current_il_expr` in the Python Console
+
+When you need to inspect a particular expression in a given IL function, it can often be quite cumbersome trying to locate it via the API.
+If you have the UI open, you can skip traversing the entire IL tree and simply click the expression of interest.
+From there, you can use `current_il_expr` in the Python console to interact with that expression directly.
+
+```
+>>> current_il_expr
+<MediumLevelILCmpSlt: temp0 s< 0x12817497a9ff848a>
+>>> current_il_expr
+<MediumLevelILSetVar: rcx_1 = zx.q(rdx_2)>
+
+>>> current_il_expr
+<MediumLevelILVar: rdx_1>
+>>> current_il_expr.possible_values
+<unsigned ranges: [<range: 0x0 to 0x1>]>
+```
+
+Compare this to `current_il_instruction`, which only gives you the top-level instruction:
+
+```
+>>> current_il_instruction
+<MediumLevelILSetVar: rdx_2 = rdx_1 << 3>
+>>> current_il_expr
+<MediumLevelILVar: rdx_1>
+```
+
+### IL Sidebar
+
+Included in the [BN Inspectors plugin](https://github.com/CouleeApps/bn_inspectors) is a sidebar that lists all IL Expressions and IL Instructions.
+It shows you all the expressions' and instructions' indices in a big table, making it easier to determine where things are located. 
+It can be quite useful for determining the raw operand values of various expressions and finding dangling expressions that get lost after particularly messy calls to `replace_expr`.
+It also shows what types the expressions have (MLIL and higher). 
+
+![img](../img/il_inspector.png)
+
+### Custom Debug Reports
+
+Trying to debug analysis modifications can be tricky, since usually you can only see the end result after all logic has completed. Luckily, Binary Ninja has a built-in feature that lets you build up a report where you can visualize your changes in stages. We've used this in the past, e.g. [during the development of HLIL](https://binary.ninja/2024/07/15/debug-visualizations.html#high-level-il-debug-report), we used a Debug Report to visualize the control flow structuring. You can write your own Debug Reports too, and visualize changes made in your Workflow Activity.
+
+Debug Reports are usually requested via a Python script in the scripting console:
+
+```py
+current_function.request_debug_report("my_report")
+```
+
+In your Activity, you can check for one of these requests and then, if detected, generate a Debug Report and present it to the user:
+
+```py
+def rewrite_action(context: AnalysisContext):
+    report = None
+    if context.function.check_for_debug_report("my_report"):
+        report = ReportCollection()
+    
+    # ... do changes and build report
+
+    if report is not None:
+        show_report_collection("My Debug Report", report)
+```
+
+It can be exceedingly helpful to put your entire modification script inside a `try`/`except` block and present the report even on failure:
+
+```py
+    # ...
+
+    try:
+        # ... do changes and build report
+    finally:
+        if report is not None:
+            show_report_collection("My Debug Report", report)
+```
+
+Then, you can build up the `ReportCollection` object by adding reports to it with the various Report APIs.
+
+**Note:** If you are making function graphs using the various `create_graph` APIs, make sure to use `create_graph_immediate` as it will construct the graph from the function as it exists when called, instead of lazily once the view is rendered (otherwise you will only see the end state of the function). 
+
+```py
+# For ReportCollection, create graph with the settings I want
+settings = DisassemblySettings()
+settings.set_option(DisassemblyOption.ShowAddress, True)
+
+# Create graph *immediate* from the function
+init_graph = old_func.create_graph_immediate(settings=settings)
+
+# And add it to my debug report
+report.append(FlowGraphReport("Initial", init_graph, context.view))
+```
+
+You can also use the various graph APIs to style the report and highlight nodes or instructions:
+
+```py
+init_graph = old_func.create_graph_immediate(settings=settings)
+
+# Go through the nodes in the graph and highlight some of them
+for node in init_graph.nodes:
+    if some_condition:
+        node.highlight = HighlightStandardColor.RedHighlightColor
+    if other_condition:
+        node.highlight = HighlightStandardColor.GreenHighlightColor
+    
+    # Update the graph with the highlighted node
+    init_graph.replace(node_index, node)
+
+# And then you can add as normal
+report.append(FlowGraphReport("Graph with Highlights", init_graph, context.view))
+```
+
+### Dry Run Mode
+
+Due to the nature of Workflows, a custom Debug Report is only able to be triggered if your Activity gets run.
+But what if you don't want to apply your changes yet?
+This might be because you suspect your changes are bugger, or so you can introspect the unchanged function while still seeing diagnostics from your changes.
+Either way, sometimes it is helpful to have a variant of your Activity that does all of the logic _except_ for apply the updated analysis.
+There's no built-in solution for this, but you can model it pretty effectively by having two activities and a Dry Run switch.
+
+First, modify your Activity function with another parameter for if this is a dry run:
+
+```py
+def rewrite_action(context: AnalysisContext, dry_run: bool):
+    new_mlil = context.mlil  
+    # ... logic here
+      
+    # Only update analysis if this is NOT a dry run
+    if not dry_run:
+        context.mlil = new_mlil
+```
+
+Then, when registering your Workflow Activity, make two copies of the Activity, having one be for the dry run:
+
+```py
+# Workflow setup boilerplate
+wf = Workflow("core.function.metaAnalysis").clone("core.function.metaAnalysis")
+
+# Define your activity twice, once as a "dry run" enabled by default
+wf.register_activity(Activity(
+    configuration=json.dumps({
+        "name": "extension.my_extension.do_the_thing.dry_run",
+        "title": "My IL Modifications Dry Run",
+        "description": "My IL Modifications (dry run)",
+        "eligibility": {
+            "auto": {
+                "default": True  # Dry run enabled by default
+            }
+        }
+    }),
+    action=lambda context: rewrite_action(context, True)  # Dry run
+))
+
+# ... and one that will actually apply changes, disabled by default
+wf.register_activity(Activity(
+    configuration=json.dumps({
+        "name": "extension.my_extension.do_the_thing",
+        "title": "My IL Modifications",
+        "description": "My IL Modifications",
+        "eligibility": {
+            "auto": {
+                "default": False  # Real run disabled by default
+                                  # (or enabled by default if you are confident)
+            }
+        }
+    }),
+    action=lambda context: rewrite_action(context, False)  # Not dry run
+))
+```
+
+Finally, insert both actions into the workflow at the same place.
+Insert the dry run first so that, even if the real action is enabled, the dry run still runs on the unchanged state.
+
+```py
+wf.insert_after("core.function.generateMediumLevelIL", [
+    "extension.my_extension.do_the_thing.dry_run",
+    "extension.my_extension.do_the_thing"
+])
+wf.register()
+```
diff --git a/docs/dev/bnil-overview.md b/docs/dev/bnil-overview.md
index b0ffefe2..fd56d5d9 100644
--- a/docs/dev/bnil-overview.md
+++ b/docs/dev/bnil-overview.md
@@ -153,3 +153,7 @@ The visitor receives 4 operands:
 0x4012d8 arg1 + 0x1404
 0x4012de arg1 + 0x1400
 ```
+
+## Modifying ILs
+
+See [Modifying ILs](bnil-modifying.md).