# Modifying ILs Modifying ILs during lifting is a complex process with many factors to consider. ## Support APIs for modifying ILs are under active development, with support depending on which level of IL and which language you use. Key: - ✅ Full support - ⚠️ Partial support - ❌ No support | IL Level | C++ | Python | Rust | |-----------------|:---:|:------:|:----:| | Lifted IL | ✅ | ✅ | ✅ | | Low Level IL | ✅ | ✅ | ✅ | | Medium Level IL | ✅ | ✅ | ❌ | | High Level IL | ⚠️* | ❌ | ❌ | \* Modifying High Level IL in C++, while some APIs exist, is incomplete and [certain more complicated operations are not possible yet](#notes-on-hlil). ## Choosing A Level of IL To Modify One of the most important considerations to make when modifying ILs is, "What level of IL do I need to modify?" Depending on what effects you want to have, different levels will work better. - Modifying **Lifted IL** is similar to a generic Architecture Extension, letting you change instructions directly out of the lifter. Note that Lifted IL has no SSA forms, dataflow, stack offsets, or even resolved flags. - **You probably do not want to modify Lifted IL with a Workflow**. Instead, consider modifying the Architecture directly ([most of which are Open Source on our GitHub](https://github.com/Vector35/binaryninja-api/tree/dev/arch)) or making your Activity modify Low Level IL. - Modifying **Low Level IL** lets you affect operations on registers, which eventually affect values of variables (in MLIL). Depending on where in the pipeline you insert your action, stack offsets may or may not be calculated, and most dataflow information is not available. Notably, any `PossibleValueSet` calculations require MLIL or higher, so only `RegisterValue` dataflow results are available (and only after generating SSA form). - Modifying **Medium Level IL** lets you modify uses of variables with the ability to solve for values in many cases, but is too late to let you change IL to create new variables properly. Changes can affect HLIL structuring. - Modifying **High Level IL** lets you affect the final presentation of the decompiled code directly, but requires working with the expression AST and cannot affect things like value set analysis. HLIL does not have the same guarantees that variables and statements exist in the original code, and expressions can be changed completely from their original forms or be created from thin air. There are certain ILs which you cannot modify, for a variety of reasons: - **SSA Forms** are generated by the core from the various Non-SSA Form ILs at each stage of analysis. You should not modify them, as they are considered analysis derived from the Non-SSA Form. They will likely be regenerated from the source Non-SSA Form without consulting any changes you might make to them. - **Disassembly** should be modified by changing the Architecture plugin if it is open source, or by using an Architecture Extension if it is not. A Workflow Activity cannot affect Disassembly. - **Mapped MLIL** is lifted directly from LLIL SSA Form and not user-editable. Generally speaking, users and developers of Workflow Actions should not need to consult Mapped MLIL. - **Pseudo-C** and other Language Representation forms are not a distinct level of IL, but are simply alternate renderings of HLIL. ## Considerations Based on your IL of choice, there are a bunch of places you can insert your modification action, which affects the information available to use or modify. - **Lifted IL** modification needs to happen before the `core.function.analyzeAndExpandFlags` stage, which translates Lifted IL into Low Level IL. After that stage, Lifted IL is never again consulted. - Modifying flag definitions and usages is likely impossible, as the Semantic Flags system will consult the Architecture to determine what IL to generate when a flag is used, not your Workflow Action. There is currently no way to change this behavior. - **Low Level IL** modification is generally done right before the `core.function.generateMediumLevelIL` stage, after the final Low Level IL function is emitted. After Medium Level IL is generated, changes to the Low Level IL will have limited effects on the other ILs (because most things refer to MLIL). - If you want to make changes to LLIL that can affect the stack pointer resolution, you will want to insert your action before `core.function.analyzeStackAdjustment` instead. - **Medium Level IL** modification is generally done right after `core.function.generateMediumLevelIL` since (as of writing) the MLIL translation is all done in that one monolithic step and future steps are simply processing and annotations. This may change some time in the future. - **High Level IL** modification is generally done right after `core.function.generateHighLevelIL` since HLIL generation is also monolithic. This also may change eventually. ## Terminology - An **IL Function** is a collection of **IL Instructions**, grouped into **IL Basic Blocks** - An **IL Basic Block** is a contiguous sequence of **IL Instructions** between a start and end **Instruction Index**, which only has incoming branches at the start and outgoing branches at the end. - An **IL Expression** is a single-operation expression, which may have child expressions or be the child of another expression. - The `*LevelILInstruction` class represents a single **IL Expression**. The class name may be confusing since **IL Instructions** are a different concept. - An **Expression Index** points to an **IL Expression** and may be used as an operand of another **IL Expression**. - An **IL Instruction** is an **IL Expression** that has been added as a top-level instruction to the function with a call to `*LevelILFunction.append()` (C++: `*LevelILFunction::AddInstruction`) - An **Instruction Index** is assigned to an **IL Expression** that is added as an **IL Instruction**. Child expressions of an **IL Instruction** do not inherently have an **Instruction Index**, though the API tries to provide this for you. ## Writing a Transformation Once you figure out which level of IL you want to modify, and where in the Workflow pipeline you want to do modifications, it is time to write some code! ### Registering Your Plugin In order for your modifications to run, you need to register a Workflow Activity. There are a couple of steps involved in this process: First, you need to make a clone of the default workflow, making it mutable and allowing you to insert your Activity. You can either name the clone the same thing as the metaAnalysis workflow to make your Activity available by default, or clone to a different name, which will require users to select that workflow in Open with Options. Also note that, as of writing, Objective-C support registers its own workflow as `core.function.objectiveC`. If you want your Workflow Activity to apply to Objective-C files, you will need to do this whole process twice, once for `core.function.metaAnalysis` and once for `core.function.objectiveC`. The same applies if any of your other plugins register a custom workflow that is not `core.function.metaAnalysis`. === "Python" ```py # This Workflow will replace metaAnalysis and be used by default wf = Workflow("core.function.metaAnalysis").clone("core.function.metaAnalysis") # Users will need to pick this Workflow in Open with Options wf = Workflow("core.function.metaAnalysis").clone("MyCustomWorkflow") # As of 5.1: To modify functions in binaries using the Objective-C workflow, # you need to do the entire rest of this section twice: # once as above, and once for the Objective-C workflow as shown here. wf = Workflow("core.function.objectiveC").clone("core.function.objectiveC") ``` === "C++" ```c++ // This Workflow will replace metaAnalysis and be used by default auto wf = Workflow::Instance("core.function.metaAnalysis")->Clone("core.function.metaAnalysis"); // Users will need to pick this Workflow in Open with Options auto wf = Workflow::Instance("core.function.metaAnalysis")->Clone("MyCustomWorkflow"); // As of 5.1: To modify functions in binaries using the Objective-C workflow, // you need to do the entire rest of this section twice: // once as above, and once for the Objective-C workflow as shown here. auto wf = Workflow::Instance("core.function.objectiveC")->Clone("core.function.objectiveC"); ``` Then, define a new Activity on the Workflow, which allows you to run your code to make modifications. === "Python" ```py def rewrite_action(context: AnalysisContext): # Actual modification code goes here (documented below)... # Define the custom activity configuration wf.register_activity(Activity( configuration=json.dumps({ "name": "extension.my_extension.do_the_thing", "title": "My IL Modifications", "description": "Does some IL modifications.", "eligibility": { "auto": { "default": False # Controls if the workflow is enabled by default } } }), action=rewrite_action )) # Add the new action to the workflow, with position as described above wf.insert_after("core.function.generateMediumLevelIL", [ "extension.my_extension.do_the_thing" ]) wf.register() ``` === "C++" ```c++ void RewriteAction(Ref context) { // Actual modification code goes here (documented below)... } BINARYNINJAPLUGIN bool CorePluginInit() { // Rest of your plugin init here ... // Define the custom activity configuration wf->RegisterActivity(new Activity(R"~( { "name": "extension.my_extension.do_the_thing", "title": "My IL Modifications", "description": "Does some IL modifications.", "eligibility": { "auto": { "default": false } } } )~", RewriteAction)); wf->InsertAfter("core.function.generateMediumLevelIL", std::vector{ "extension.my_extension.do_the_thing" }); Workflow::RegisterWorkflow(wf); } ``` A more in-depth explanation of the Workflows system is available [here](./workflows.md), with complete documentation of the Eligibility system and Workflows in general. This guide, however, will just give enough of an example for you to get to writing IL modification code. ### Replacing Trivial Instructions Replacing exactly one instruction with exactly one other instruction is relatively simple: 1. Create the new expression in the function, and note its **Expression Index** 2. Find the instruction you wish to remove, and get its **Expression Index** as well 3. Use `*LevelILFunction.replace_expr` (C++: `*LevelILFunction::ReplaceExpr`), passing the **Expression Index** of the instruction to remove and the **Expression Index** of the replacement 4. Now, the backing expression for the target instruction has been replaced by your new expression. There may be dangling expressions no longer referenced by any instructions; this is fine. They will simply be skipped during later stages of lifting. 5. Call `*LevelILFunction.finalize` (C++: `*LevelILFunction::Finalize`) to BFS traverse the function's **IL Instructions** starting with instruction 0, reconstructing the **IL Basic Blocks** of the function 6. Call `*LevelILFunction.generate_ssa_form` (C++: `*LevelILFunction::GenerateSSAForm`) to reconstruct the SSA Form of the function, updating dataflow calculations If you wish to replace more than one instruction, or replace an instruction with more than one new instruction, you will need to use the more complicated method described below. ### Adding Instructions and Replacing Multiple Instructions If you want to insert new instructions into a function, or want to replace an instruction with than one instruction, you will need to construct a new **IL Function** based on the original **IL Function** but with your changes. This is a rather cumbersome process and it can be easy to make mistakes. First, you need to construct a new **IL Function** based on the existing function. Let's call the new function `new_func` and the existing function `old_func`. === "Python" ```py # ... workflow boilerplate def rewrite_action(context: AnalysisContext): old_func = context.mlil # Create a new IL Function based on the old one. # Make sure you use the LLIL from the analysis context, # and *do not* go through Function.llil (it will not have been updated yet) new_func = MediumLevelILFunction(old_func.arch, low_level_il=context.llil) # Tell the new function that we are copying from an existing function # (this transfers various metadata like some block labels) new_func.prepare_to_copy_function(old_func) # continues ... ``` === "C++" ```c++ void RewriteAction(Ref context) { auto oldFunc = context->GetMediumLevelILFunction(); // Create a new IL Function based on the old one. // Make sure you use the LLIL from the analysis context, // and *do not* go through Function::GetLowLevelIL (it will not have been updated yet) // Also (C++ specific), make sure you use Ref<> to hold // a strong reference so the new function doesn't get deleted Ref newFunc = new MediumLevelILFunction( oldFunc->GetArchitecture(), oldFunc->GetFunction(), context->GetLowLevelILFunction() ); // Tell the new function that we are copying from an existing function // (this transfers various metadata like some block labels) newFunc->PrepareToCopyFunction(oldFunc); // Continues... } ``` ???+ Note "Note" You may notice that the sample code above mentions always using `AnalysisContext.llil` (C++: `AnalysisContext::GetLowLevelIL()`) to access the relevant Low Level IL function, and you may be wondering why the sample doesn't instead use `AnalysisContext.function.llil` (C++: `AnalysisContext::GetFunction()->GetLowLevelILFunction()`). This is because you must **always access IL Functions directly through `AnalysisContext`** when writing Workflow Activities, as that is where the current, up-to-date analysis information is stored. During analysis, the `LowLevelILFunction` object stored in the `AnalysisContext` object contains the in-progress analysis information from the current run of the Workflow, whereas the `LowLevelILFunction` object stored on the Function object is from the previous run of the Workflow and contains stale analysis data. Then, going block-by-block, copy the instructions from `old_func` to `new_func`. When you reach a part where you want to insert new instructions (or replace instructions), do that instead of simply copying the old instructions. === "Python" ```py # previous code... # Copy each block in the old function to the new function for old_block in old_func.basic_blocks: # Copy some block labels and tell new IL instructions they came from this block new_func.prepare_to_copy_block(old_block) # Copy each instruction from the old block to the new function # Blocks "contain" instructions by knowing their start and end instruction indices # Any instructions in that range as in the block for old_instr_index in range(old_block.start, old_block.end): # Retrieve instruction from old function old_instr: MediumLevelILInstruction = old_func[old_instr_index] # Make sure the new instruction we insert has the correct location information new_func.set_current_address(old_instr.address, old_block.arch) # continues... ``` === "C++" ```c++ // ... previous code // Copy each block in the old function to the new function for (auto& oldBlock: oldFunc->GetBasicBlocks()) { // Copy some block labels and tell new IL instructions they came from this block newFunc->PrepareToCopyBlock(oldBlock); // Copy each instruction from the old block to the new function // Blocks "contain" instructions by knowing their start and end instruction indices // Any instructions in that range as in the block for (size_t oldInstrIndex = oldBlock->GetStart(); oldInstrIndex < oldBlock->GetEnd(); oldInstrIndex++) { // Retrieve instruction from old function auto oldInstr = oldFunc->GetInstruction(oldInstrIndex); // Make sure the new instruction we insert has the correct location information newFunc->SetCurrentAddress(oldBlock->GetArchitecture(), oldInstr.address); // continues... ``` For each instruction, determine if you want to insert new instructions before it, or replace it with something else. You _probably_ want to do this step ahead of time and apply all the changes at once, but if your changes are simple enough you can get away with doing it inside the loop. === "Python" ```py # previous code... if want_changes: # Create new instructions in the function for whatever changes you desire # All ILs: Be sure to pass the location of the previous instruction # when creating a new expression, so mappings work new_expr = new_func.nop(ILSourceLocation.from_instruction(old_instr)) # MLIL and higher: pass the location of the previous instruction # to append(), so cross-IL mappings are preserved new_func.append(new_expr, ILSourceLocation.from_instruction(old_instr)) else: # Copy the instruction as-is # Copy the instruction from the old function to the new function # using `copy_to` (deep copy) # MLIL and higher: also pass the location of the previous instruction # to append() so cross-IL mappings are preserved new_func.append(old_instr.copy_to(new_func), ILSourceLocation.from_instruction(old_instr)) # continues ... ``` === "C++" ```C++ // previous code... if (wantChanges) { // Create new instructions in the function for whatever changes you desire // All ILs: Be sure to pass the location of the previous instruction // when creating a new expression, so mappings work auto newExpr = newFunc->Nop(oldInstr); // MLIL and higher: pass the location of the previous instruction // to AddInstruction(), so cross-IL mappings are preserved newFunc->AddInstruction(newExpr, oldInstr); } else { // Copy the instruction as-is // Copy the instruction from the old function to the new function // using `CopyTo()` (deep copy) // MLIL and higher: also pass the location of the previous instruction // to AddInstruction() so cross-IL mappings are preserved newFunc->AddInstruction(oldInstr.CopyTo(newFunc), oldInstr); } } // continues ... ``` And finally, regenerate dataflow: === "Python" ```py # previous code ... # Create IL basic blocks new_func.finalize() # Do dataflow, etc new_func.generate_ssa_form() # Assign to the context to apply modifications context.mlil = new_func ``` === "C++" ```C++ // previous code ... // Create IL basic blocks newFunc->Finalize(); // Do dataflow, etc newFunc->GenerateSSAForm(); // Assign to the context to apply modifications context->SetMediumLevelILFunction(newFunc); } ``` ### Transforming Multiple Times If your action requires sufficiently complex logic that you want to transform the function twice, there are a few extra things to consider. Due to limitations currently in the Python bindings, IL mappings will only update properly if you assign the intermediate MLIL function to the Analysis Context before transforming again: === "Python" ```py # previous code ... # Need to commit this for mappings to work (as of 5.1 at least) context.mlil = new_func # Now we want to modify the function again... # The newly created function is now our "old function" old_func = new_func # All of the rest of this is the same as before new_func = MediumLevelILFunction(old_func.arch, low_level_il=context.llil) new_func.prepare_to_copy_function(old_func) ``` === "C++" ```C++ // previous code ... // Need to commit this for mappings to work (as of 5.1 at least) context->SetMediumLevelILFunction(new_func); // Now we want to modify the function again... // The newly created function is now our "old function" oldFunc = newFunc; // All of the rest of this is the same as before Ref newFunc = new MediumLevelILFunction( oldFunc->GetArchitecture(), oldFunc->GetFunction(), context->GetLowLevelILFunction() ); newFunc->PrepareToCopyFunction(oldFunc); ``` You may realize this is rather at odds with the section below on having a [dry run mode](#dry-run-mode). Sadly, there is currently no good solution for that, since the IL mappings are generated during assignment to the Analysis Context. This may change in future versions, but is required for now. ### Using IL Labels and GOTO/IF Instructions When you are trying to insert `*LIL_GOTO`, `*LIL_IF`, and similar instructions, you will need to specify the IL destination as a `*LevelILLabel`. To properly use a label, you must call `mark_label` on the **IL Function** _right before emitting the instruction that the label targets_, which is always at the start of an **IL Basic Block**. Your `*LIL_GOTO` instruction can come either before or after its target, but you must make sure to use the same `*LevelILLabel` object in the call to `*LevelILFunction.goto()` (C++: `*LevelILFunction::Goto()`) and `*LevelILFunction.mark_label()` (C++: `*LevelILFunction::MarkLabel`). Labels are not preserved when constructing a new function during transformation, so in practice you should not assume calls to `get_label_for_source_instruction` will always return a usable label. This means, if you are emitting any of these control-flow instructions, you will need to transform the entire function in the process, as described in [Adding Instructions and Replacing Multiple Instructions](#adding-instructions-and-replacing-multiple-instructions), so you are able to call `mark_label` at the appropriate time. An example of label creation and usage is as follows: === "Python" ```py def rewrite_action(context: AnalysisContext): # Setup as described in the sections above old_func = context.mlil new_func = MediumLevelILFunction(old_func.arch, low_level_il=context.llil) new_func.prepare_to_copy_function(old_func) # Keep a running list of labels for MLIL_GOTO targets # Stored in a map of { :