summaryrefslogtreecommitdiff
path: root/view
AgeCommit message (Collapse)Author
2025-02-10Fix some issues regarding allocations within the sharedcache plugin FFIkat
2025-02-10[SharedCache] Make `m_exportInfos` map's values a `shared_ptr`WeiN76LQh
This avoids expensive copying when returning a value from the map in `SharedCache::GetExportListForHeader`. Additionally it ensures that the value stays alive and at the same location in memory if `m_exportInfos` is modified and requires its storage to be re-allocated. I was unable to use a `unique_ptr` instead of a `shared_ptr` because of copy semantics with `m_exportInfos` in `ViewStateCacheStore`. I don't see things being any worse using `shared_ptr` instead of `unique_ptr` anyway and it means less code changes.
2025-02-10[SharedCache] Only setup undo actions and bulk modify in ↵WeiN76LQh
`SharedCache::FindSymbolAtAddrAndApplyToAddr` if a symbol is found
2025-02-10[SharedCache] Improve the types for `m_exportInfos`WeiN76LQh
This commit changes 2 things; 1. `m_exportInfos` is now a map where its values are also a map rather than a vector of pairs. The reason for this is that `SharedCache::FindSymbolAtAddrAndApplyToAddr` is a hot path which does by far the most accesses to `m_exportInfos`. In that function it must find the correct symbol for a given address so a map lookup will be much quicker than iterating a vector. The other use cases of `m_exportInfos` would prefer a vector but they are executed very infrequently. 2. The symbols are stored in `m_exportInfos` as references to the `Symbol` type. This makes more sense because otherwise there is a lot of time spent converting to and from a `Symbol` type and a pair of `BNSymbolType` + a `std::string`.
2025-02-10[SharedCache] Add parameter to `SharedCache::InitializeHeader` to check if ↵WeiN76LQh
`m_exportInfos` was modified This probably makes more sense than the current solution of using execution of the callback parameter to determine if `m_exportInfos` was modified.
2025-02-10[SharedCache] Use `m_exportInfos` as an export list cacheWeiN76LQh
`SharedCache::ParseExportTrie` is getting called a lot during DSC library loading and analysis. In large part due to the hot path `SharedCache::FindSymbolAtAddrAndApplyToAddr`. Its unnecessary for it to be being called more than once per DSC header as the export list symbol information is stored in `SharedCache::m_exportInfos`. This commit adds the function `SharedCache::GetExportListForHeader`, which will either return the header's list of symbol information cached in `SharedCache::m_exportInfos` or call `SharedCache::ParseExportTrie` and cache the results in `SharedCache::m_exportInfos`. This should also improve the execution time of `SharedCache::LoadAllSymbolsAndWait`. Further improvement here would be to add locking to `SharedCache::GetExportListForHeader` so that races don't result in redundant parsing of the export trie for the same header if multiple threads call `SharedCache::GetExportListForHeader` at the same time for the same header. This only really matters during initial loading because from what I can tell that parses all the export trie's anyway.
2025-02-04Update to Qt 6.8.2Rusty Wagner
2025-02-04Avoid crashing when initial type parse fails in DSCViewkat
2025-01-30[SharedCache] Vision Pro, tvOS, iOS Simulator supportkat
2025-01-27[SharedCache] Optimize `ReadExportNode`Mark Rowe
`ReadExportNode` is called a lot during the initial load of the shared cache and thus impacts how long it takes for the UI to become responsive. This is a collection of optimizations that cut the time spent within `ReadExportNode` by 50%: 1. Pass iterators to `ReadExportNode` rather than a `DataBuffer` + offset. The lack of inlining in `DataBuffer`'s `operator[]` kills performance. Ideally this would have used `std::span`, but that would require bumping the minimum C++ version to C++20. 2. Add `MMappedFileAccessor::ReadSpan` so that `ReadExportNode` can operate directly on the mapped data without first copying it. 3. Removes a call to `GetAnalysisFunctionsForAddress` whose result was unused. 4. Use `std::find` to find the nul at the end of strings rather than assembling the string a character at a time. This avoids repeatedly growing the string. 5. Avoid the usual `Symbol` constructor in favor of `BNCreateSymbol`. The `Symbol` constructor has over head in two forms: 1. It has a `NameSpace` as an argument. Creating / destroying this allocates and deallocates memory We could create a single instance and reuse it for all calls, but... 2. The constructor copies all fields of the `NameSpace` to the heap before calling `BNCreateSymbol` and then deallocates them afterwards. This seems unnecessary, and adds a non-trivial amount of overhead. This can go back to directly constructing the `Symbol` once the constructors are improved.
2025-01-27Port sharedcache view to MemoryRegions, removing the need for the fake Raw ↵kat
view workaround
2025-01-25Rust refactorMason Reed
Moves a bunch of stuff out of src/types.rs that did not belong: - Confidence - Variable - Function specific stuff - Refactored InstructionInfo, see the msp430 and riscv examples. - Renamed Function::from_raw to Function::ref_from_raw and fixed places where the ref was incremented twice - Fixed FunctionRecognizer leaking functions (see above) - Fixed some apis incorrectly returning Result where Option is clearer - Started to move destructured types to the From trait for converting to an from ffi types, see Location for an example - Started to remove bad module level imports (importing std modules like mem all over the place) - Moved some wrapper handle types to named handle field (this improves readability), see CoreArchitecture for an example - Removed some unintuitive getters, this is bad practice for Rust code, just access the field directly, see DataVariable for an example - General code cleanup, purposely did not run rustfmt, that will be a single seperate commit More rust cleanup - Fixed invalid views being able to invoke UB when dealing with databases - Cleaned up some helper code in dwarf_import - Fixed inverted is_null checks causing crashes! Oops! More rust cleanup Still a WIP, I think branch info is still invalid, need to figure out the issue there. - Fixed some invalid Ref lifetimes when constructing indirectly, see Array<DataVariable> for example - Added some more comments - Removed some "magic" functions like MLIL Function::ssa_variables There are still a bunch of invalid lifetimes that aren't crashing us due to the usage of those API's not living long enough. But they ARE an issue. More rust cleanup Trying to comment more TODO's as I go along. - Renamed llil::Function to llil::LowLevelILFunction for clarity and consistency - Take base structures by ref in StructureBuilder::set_base_structures to prevent premature drops - Added more raw to wrapper conversions - Removed UB prone apis - Getting rid of more core module references, use std! - Removed improper Guard usage in array impls for wrapper types with no context - Untangling the UB of the Label api, still only half done :/ More rust cleanup - Misc formatting - Made Logger ref counted - Fixed leaking name of logger every time something was logged - Fixed the last (hopefully) of the unresolved labels - Simplified some code Fix leak in DebugInfo::AddType componentArray was never freed Add more HLIL related functions to rust More rust cleanup improve the CustomBinaryView init process Canonicalize path in `create_delete_empty` test Link core in rust When linking you must depend on the -sys crate. This is because linker arguments (what says where to find binaryninjacore) are NOT transitive. The top level application crate MUST provide it. For more information see: - https://github.com/rust-lang/cargo/issues/9554#issuecomment-857882964 - https://github.com/oxidecomputer/omicron/pull/225 Remove vendored pdb crate Use cargo to manage the git repo ref instead Fix misc rustdoc warnings Move actual plugins out of `rust/examples` and into `plugins` This is where all shipped public plugins that are not arch/view/platform/lang will be at from now on Originally they were in the rust workspace, meaning they all shared a Cargo.lock which is ill-advised. More rust cleanup - More clarification on plugin/executable requirements - Made examples actually rust examples - Add Display impl for InstructionTextToken - Renamed feature "noexports" to "no_exports" Move under_construction.png to assets directory This really did bother me Remove unneeded `extern crate bindgen` Replace nop'd log::info with println We don't register a compatible log sink so they will just get sent into the void Move inline tests into tests directory This is being done in the hopes of curbing the multi-thousand line files that will occur once we flesh out the tests Format rust code Update rust ci Still need to add support for running tests in ci More rust cleanup - Architecture id's are now typed accordingly - Fix some clippy lints - Make instruction index public in LLIL - Removed useless helper functions - LLIL expressions and instruction indexes are now typed accordingly Generate binaryninjacore-sys documentation This should show binaryninjacore-sys alongside binaryninja crate More rust cleanup - Remove lazy_static dependency - Remove hacky impl Debug for Type and just use the display impl - Add more debug impls - Reorder some top level namespace items - Add first type test Remove unneeded script helper in rust api More rust cleanup - Added main thread handler api - Register a headless main thread handler by default in initialization - Refactor QualifiedName to be properly owned - Loosened some type constraints on some apis involving QualifiedName - Fixed some apis that were crashing due to incorrect param types - Removed extern crate cruft for log crate - Simplified headless initialization using more wrapper apis - Fixed segments leaking because of no ref wrapper, see BinaryViewExt::segment_at - Added rstest to manage headless init in unit tests - Added some more unit tests - Refactored demangler api to be more ergonomic - Fixed minidump plugin not building More rust cleanup - Fixup usage of QualifiedName in plugins - Make QualifiedName more usable Implement rust TypeParser fix Platform TypeParser related functions separate User and System implementations of TypeParserResult Implement rust TypeContainer More rust cleanup - Hopefully fixed the rust.yml CI - Added TypePrinter API (this is still WIP and will crash) - Added TypeParser API - Added TypeContainer API - More work around QualifiedName apis Oh your suppose to do this Add workflow_dispatch trigger to rust.yml More rust fixes - Swapped some usage of raw 255 to MAX_CONFIDENCE, no one likes magic numbers - New InstructionTextToken API, complete with owned data, this still needs a lot of testing. - InstructionTextTokenKind describes a destructured InstructionTextToken, this should make token usage much clearer, some docs pending - Added some misc Default and Debug impls - Updated TypePrinter and architectures to use new InstructionTextToken API Misc formatting changes More rust cleanup - Fixed MANY memory leaks (most due to QualifiedName) - Made StructureBuilder more builder and less structure - Fixed CMakeLists.txt that were globbing entire api, resulting in 100 second slowdown on cmake generation - Added more Debug impl's - Added some more tests - Fixed TypeParserResult UB - Moved the From impls to blank impl for clarity, we have multiple different variants of core to rust for some structures, it should be explicit which one you are choosing. - PossibleValueSet should now be able to allocate so we can go from rust to core with those variants that require allocation - Misc doc code formatting Misc clippy lints and clippy CI Co-authored-by: Michael Krasnitski <michael.krasnitski@gmail.com> Fix typo in rust CI Misc rust formatting Fix misc typos and add typos to rust CI Add cargo workspace This will help tooling and external contributors get a map of the rust crates within binaryninja-api More rust cleanup - Format all rust plugins - Fix some tests that were out of date - Simplify WARP tests to only binaries, building object files from source is a pain - Link to core in all rust plugins - Fix some memory leaks - Update warp insta snapshots - Fix some misc clippy lints Run rust tests in CI This commit also coincides with the creation of the "testing" environment which exposes a BN_SERIAL secret for pulling a headless Binary Ninja Install missing wayland dependency in github CI Apparently its needed for linux file picker for the WARP integration Set the BINARYNINJADIR so rust can find binaryninjacore in CI The chances of this working are low Misc remove unused dependency Rust misc formatting fixes Improve initialization in rust headless scripts Provide sensible errors and validation to rust headless scripts, solves https://github.com/Vector35/binaryninja-api/issues/5796 Add BN_LICENSE environment variable to rust CI We pass the serial to download binary ninja, but we never provided the license for core initialization Fix typo More rust cleanup - Improved binary view initialization (see init_with_opts) - Allow floating license to free itself before initialization - Add initialization unit test - Add better Debug impls for some common types - Use Path api for opening binary views, this is not breaking as it uses the AsRef impl - Bump rayon dependency and constrain dependencies to x.x Update readme and include that directly in the rustdocs More rust documentation changes Add format comment to InitializationOptions::with_license Misc formatting and clippy lint allow More rust cleanup - Remove under_construction.png from the build.rs it has been removed - Use InstructionIndex in more places - Add missing PartialOrd and Ord impls for id types More rust cleanup - Make workflow cloning explicit - Add workflow tests - Add missing property string list getting for settings - Remove IntoActivityName (see https://github.com/Vector35/binaryninja-api/pull/6257) More rust cleanup This commit is half done Misc rust formatting More rust cleanup - Renamed common name conflictions (I will put my justification in the PR) - Fixed invalid instruction retrieval for LLIL - Added common aliases for llil function, instruction and expression types (see my comment in the PR) - Refactored the instruction retrieval for LLIL, MLIL and HLIL - Added instruction index types to MLIL and HLIL - Moved llil module to lowlevelil module (mlil and hlil will be moved as well) - Added preliminary LLIL unit testing Fix typos Misc clippy fixes More rust cleanup - Normalized modules - Split some code out into own files - Fixed some UB in type archive and projects - Improved API around type archives and projects substantially - Added ProgressExecutor abstraction for functions which have a progress callback - Improved background task documentation and added unit tests - Added worker thread api and unit tests - Moved some owned types to ref types, this is still not complete, but this is the path forward. - Add external location/library accessors to the binary view - Added some misc documentation - Replaced mod.rs with the module name source file Still need to normalize some paths and also update some documentation surrounding that change. Update some tests and examples Fix background task tests colliding We were creating multiple background tasks with the same progress text on multiple threads More rust cleanup - Fixed progress executor freeing itself after one iteration - Updated the last of the doc imports - Moved mainthread to main_thread - Made project creation and opening failable We could probably AsRef<ProgressExecutor> to get around the allocation and free inside the function bodies, not high priority as those functions are long running anyways. Move binary view initialization into function body for LLIL test Normalize test file names More rust cleanup - Updated README to clarify offline documentation - Refactored settings api - Added settings example to show dumping settings value and specific properties - Use the workspace to depend on binaryninja and binaryninjacore-sys - Remove binaryninjacore-sys as a workspace member (its not really required) Update workflow test to new settings api More rust cleanup - Rename Label to LowLevelILLabel - Update the functions label map automatically This fixed a major api blunder where the label map is returned as a reference and originally resulted in UB prone lifetime semantics. It was temporarily "fixed" with explicit label updates in the architecture implementation code. But that was less than ideal and was easy to mess up. Now the label map will be updated automatically as the location of labels is now tracked. Misc clippy lints More rust cleanup - Get rid of RawFunctionViewType - Add better Debug impl for Function More rust cleanup - Fixed the documentation icon using local files (thank you @mkrasnitski) - Fixed labels being updated and overwriting the label location used to update the label map More rust cleanup - Added unit tests for MLIL and HLIL - "Fixed" MLIL, LLIL, and HLIL having issues regarding Instruction vs Expression indexes - Renamed CallingConvention to CoreCallingConvention and removed architecture generic - Renamed CallingConventionBase to CallingConvention - Simplified calling convention code and fixed some bugs with implicit registers - Added impl Debug to MLIL and HLIL instructions Still need to at some point add an Expression to MLIL and HLIL. We also might want to look into having the Instruction kind just return the expression kind. Misc clippy lint More rust cleanup - Allow calling conventions to be registered for multiple architectures - Swapped a unreachable statement to an unimplemented statement More rust cleanup - Fixed the issue with PDB types, this has caused me an insane amount of grief - Fixed LLIL visit_tree missing LLIL_LOAD expressions - Added LLIL visitor test - Made all WARP file pickers use the rfd crate Use the dev branch of Binary Ninja in rust CI Misc rust fmt More rust cleanup - Refactored BinaryReader and BinaryWriter - Added some warnings to high_level_il and medium_level_il modules that they are unstable - Add BinaryReader and BinaryWriter tests - Changed BinaryView len to return u64 (that is what the core returns) - Misc formatting changes - Remove extern uses in lib.rs Add impl Debug for BinaryReader and BinaryWriter Turn off broken tests Add more info to the rust README.md More rust cleanup - Make EdgeStyle type not wrap raw - Regression tests for WARP will run on all bins in the out dir now impl rust Collaboration and Remote API Fix typo Update collaboration API Makes collaboration more in line with the refactor. Still a lot of work to do. Namely still need: - Proper errors - _with_opts functions - More ergonomic api - Better connection procedure - Updated documentation - A LOT of unit tests - An example - Typed id's for everything (i dont want BnString as the id!!!) - NEED to refactor the progress callbacks into the new progress api, but we should pull in some of the stuff the collab progress has - Elimination of apis that are dumb helpers Separate out the rust testing and use pull_request_target pull_request_target allows PR's to access the headless license, for this to be safe we need to prevent people from running the job. To prevent the job from being ran we add an environment requirement on testing that a reviewer must review the code and then manually approve it to run. More rust cleanup - Use GroupId instead of u64 - Use ProgressCallback in place of ProgressExecutor - Misc cleanup of FileMetadata - Add `save_to_path` and `save_to_accessor` to save modified binaries - Added binary_view unit tests - Added collaboration unit tests - Fixed a few issues with the collaboration apis - Renamed Command registration functions so that there is no import ambiguity - Split out RemoteUndoEntry - Collaboration apis now have a explicit `_with_progress` set of apis - Misc clippy lint fixes Fix some typos More rust cleanup - Add extra info to README.md - Refactor components api - Add components unit test Add testing and documentation to contributing section in README.md Fix misc doc comments
2025-01-20Allow overriding common loader settings when automatic load file parsing fails.Brian Potchik
2025-01-15Remove unused calls to defineTypedef which were causing hangsPeter LaFosse
2025-01-15[SharedCache] Avoid copying strings on each call to VM::MappingAtAddressMark Rowe
`PageMapping` was storing the path to the file it points within. This was causing unnecessary work within `VM::MappingAtAddress` as the `PageMapping`, and thus the path, is copied into the return value. This copying was more expensive than the map lookup. The file path is now stored within a `LazyMappedFileAccessor` class that wraps the `SelfAllocatingWeakPtr`. This means the path is still available via the `PageMapping`, but it does not need to be copied as often. This includes two additional improvements / optimizations while I was touching the code in question: 1. `MMappedFileAccessor::Open` no longer performs two hash lookups when a file accessor already exists. 2. `VM::MapPages` takes the path by const reference to avoid an unnecessary copy.
2025-01-10[SharedCache] Avoid crashing the product whenever bugs occur in ser/deser, ↵kat
fix compilation issue on linux
2025-01-10[SharedCache] Fix handling of relative selectors in macOS shared cachesMark Rowe
Find the relative selector base address in the Objective-C optimization data pointed to by the shared cache header, rather than via `__objc_scoffs`. This is only present on iOS, and not for every iOS version that encodes selectors via direct offsets. This also includes some related improvements: 1. Direct selectors get their own pointer type so they're rendered correctly in the view. 2. Method lists encoded as lists of lists are now handled. 3. The `dyld_cache_header` type added to the view is truncated to the length in the loaded cache. This ensures it is applied to the view. 4. A couple of methods that process method IMPs and selectors are updated to check whether the address is valid before attempting to process them. They would otherwise fail by throwing an exception if they proceed, but checking for validity is quicker and makes exception breakpoints usable.
2025-01-10[SharedCache] Track whether non-image regions are data vs codeMark Rowe
`BackingCache` now tracks the `dyld_cache_mapping_info` for its mappings so it has access to the memory protections for the region. This means it can avoid marking some regions as containing code when they don't, reducing the amount of analysis work that has to be done. Using `dyld_cache_mapping_info` also makes references to mappings easier to understand due to its named fields vs the nested `std::pair`s that were previously in use.
2025-01-10[SharedCache] Cache type libraries in the view-specific stateMark Rowe
They're surprisingly expensive to look up.
2025-01-10[SharedCache] Split view-specific state into a separate structMark Rowe
The existing view-specific state was stored in several global unordered maps. Many of these were accessed without locking, including `viewSpecificMutexes`, which is racy in the face of multiple threads. View-specific state is stored in a new heap-allocated `ViewSpecificState` struct that is reference counted via `std::shared_ptr`. A static map holds a `std::weak_ptr` to each view-specific state, keyed by session id. `SharedCache` retrieves its view-specific state during its constructor. Since `ViewSpecificState` is reference counted it will naturally be deallocated when the last `SharedCache` instance that references it goes away. Its corresponding entry will remain in the static map, though since it only holds a `std::weak_ptr` rather than any state it will not use much memory. The next time view-specific state is retrieved any expired entries will be removed from the map.
2025-01-08[SharedCache] Fix uninitialized `loaded` field for mappings returned by ↵WeiN76LQh
`BNDSCViewGetAllImages`
2024-12-26[SharedCache] Add the ability to manually trigger Objective-C processingWeiN76LQh
A problem with processing Objective-C sections at the time a library is loaded is that some of the references within those sections may refer to unload sections. This results in things like selectors not being correctly typed and named. This commit provides a way for users to manually trigger Objective-C parsing against sections for a specific library or all libraries, via the API. Combined with the previous commit a user can use the API to batch load a number of libraries and skip Objective-C processing for each one and then run it across the entire of the DSC once they are all loaded. Further improvement would be to provide a way to trigger Objective-C processing through the UI.
2024-12-26[SharedCache] Add the ability to skip Objective-C processing when loading a ↵WeiN76LQh
library This is useful when batch loading libraries to avoid extra processing (once the next commit has landed).
2024-12-26[SharedCache] Fix post-processing of Objective-C sectionsWeiN76LQh
Prior to this commit the function `DSCObjCProcessor::PostProcessObjCSections` never does anything because it doesn't use the correct names to get the Objective-C sections of the recently loaded library. In fact it never does anything because the DSC never has sections with the names its searching for. This commit passes the `baseName` (the name of the library that was loaded), which is what other Objective-C section processing code does. Combining the base name with the section names it will now find them and process them as intended. This was resulting in alot of Objective-C related stuff being missed. There is however still an issue of the fact that the way this DSC plugin works means it only analyzes Objective-C sections once. This catches alot of things but there are a number of cases where other libraries need to be loaded first due to information being referenced in another library. For instance errors like `Failed to determine base classname for category` can be caused by the class reference in the category being to a class outside of the loaded library. Once the library containing the class has been loaded, the section containing the category should be re-proccessed.
2024-12-25[SharedCache] Remove unnecessary lock in `SharedCache` destructorWeiN76LQh
From what I can tell the lock being taken in `SharedCache::~SharedCache` was purely for the decrement of `sharedCacheReferences`, however its an atomic so a lock isn't necessary. The lock being taken is extremely contentious and therefore often slow to be acquired. This resulted in a surprising amount of execution time spent in the `SharedCache` destructor. Nothing hugely significant but a quick and easy win to remove this single line of code.
2024-12-24Fix Shared Cache Plugin out-of-tree buildVisual Ehrmanntraut
2024-12-23Update metadata version to prevent v2 loader incorrectly trying to load v3 ↵kat
symbol list We should think through a better way of handling upgrades, as v3 can load v2 dbs just fine, but there is no clean way to upgrade the info currently. We need SharedCache ser/des functions in DSCView.cpp
2024-12-23[SharedCache] Serialize `SharedCache::m_symbolInfos`WeiN76LQh
`SharedCache::m_symbolInfos` isn't being serialized but there is an attempt to deserialize it. This commit adds in the code to serialize it. I slightly modified the format because I didn't really understand how it was expected to be serialized based on the deserialization code. The deserialization code looked wrong to me but its likely a misunderstanding on my part. I kept it similar to how `m_exportInfos` is serialized.
2024-12-14Pass length to free flag conditions list callback and remove libc from rust apiMason Reed
Allows language bindings like rust to free conditions lists sanely
2024-12-13Bump supported rust version to 1.83.0Mason Reed
2024-12-10[SharedCache] Fix slide info parsing and complete v2 supportWeiN76LQh
There seem to be a number of issues with slide info parsing. I decided the simplest fix was to largely mimick what `dyld` is doing. Part of this process was to remove creating a vector of page starts and processing them in another loop below the one where they were read out. It wasn't clear to me the original design decision to separate it into 2 loops like that. I think this was part of the problem that was causing issues. By adding rewrites in the same loop where page starts are being read out, it was much easier to mimick the code in `dyld` which I assume has to be correct. So as long as my copying was correct then I believe this should work as intended. Not everything has been thoroughly tested but I'm pretty confident v3 and v5 are now working as intended. v2 should be but less testing of it has been done.
2024-12-10[SharedCache] Fix padding issue with v5 slide infoWeiN76LQh
The issue this commit fixes was causing `SharedCache::ParseAndApplySlideInfoForFile` to completely fail to work with v5 slide info, which had a lot of knock on effects, i.e. lots of Objective-C analysis was failing due to invalid pointers which hadn't been fixed up. `dyld_cache_slide_info5` has 4 bytes of padding before `value_add`. Whilst `value_add` is not actually being read from, `SharedCache::ParseAndApplySlideInfoForFile` will read at a location in the file based on the size of the structure `dyld_cache_slide_info5`. This being off by 4 bytes basically broke v5 slide info fixups. With this fix many more Objective-C functions have names and a lot more `msgSend` calls are fixed up.
2024-12-10Fix a compilation error on Windowskat
2024-12-10[SharedCache] Use basic copy-on-write for viewStateCacheMark Rowe
Copying the state from the cache into a new `SharedCache` object is done with a global lock held and is so expensive that it results in much of the shared cache analysis running on a single thread, with others blocked waiting to acquire the lock. The cache now holds a `std::shared_ptr` to the state. New `SharedCache` objects take a reference to the cached state and only create their own copy of it the first time they perform an operation that would mutate it. The cached copy is never mutated, only replaced, so there is no danger of modifying the state out from under a `SharedCache` object. Since the copy happens at first mutation, it is performed without any global locks held. This avoids blocking other threads. This cuts the initial load time of a macOS shared cache from 3 minutes to 70 seconds, and cuts the time taken to load and analyze AppKit from multiple hours to around 14 minutes.
2024-12-10[SharedCache] Preserve flags when splitting a memory regionMark Rowe
The flags field was being left uninitialized which could result in the region being mishandled during later analysis.
2024-12-10[SharedCache] Have VM track ranges rather than pagesMark Rowe
There are typically only a few dozen mappings, while there can be millions of pages. This reduces the amount of time spent populating the mapping from region to file accessor along with the memory usage of the same.
2024-12-10[SharedCache] Switch to SAX-based writer API for JSON serializationAuthor: Mark Rowe
2024-12-10[SharedCache] Rework metadata serialization to reduce memory overheadMark Rowe
api/MetadataSerializable.hpp is removed in favor of including core/MetadataSerializable.hpp. Both headers defined types with the same name leading to One Definition Rule violations and surprising behavior. The serialization and deserialization context are now created on-demand during serialization rather than being a member of `MetadataSerializable`. This reduces the size of every serializable object by ~220 bytes. The context is passed explicitly as an argument to `Serialize` / `Deserialize`. As a result, `Serialize` / `Deserialize` can now be free functions rather than member functions. Since `MetadataSerializable` is not used for dynamic dispatch, the virtual methods are removed and the class is updated to be a class template using CRTP. This allows delegating to the derived class's `Load` and `Store` methods without the additional size overhead of the vtable pointer in every serializable object. These changes reduce the memory footprint of Binary Ninja after loading the macOS shared cache and loading a single dylib from it from 8.3GB to 4.6GB.
2024-12-05Implement SHN_COMMON symbol handlingMitchell Johnson
2024-11-25Rename 'defaultAnalysis' to 'baseAnalysis' in the Workflow system.Brian Potchik
2024-11-22Fix CXXABI postfixed externs being duplicated in ELFMason Reed
2024-11-13[SharedCache] Fix a bndb deserialization error caused by last commitkat
2024-11-13Append export and symbol info to correct arraysMark Rowe
This is both a correctness fix and avoids leaking a massive amount of memory.
2024-11-13Don't leak DataBuffersMark Rowe
`MMappedFileAccessor::ReadBuffer` was returning a heap-allocated `DataBuffer`, but no callers were ever deleting it. There does not appear to be any reason to heap allocate the `DataBuffer` as the type is effectively a smart pointer wrapper around `BNDataBuffer`. Switch to returning it by value instead. Additionally, `MMappedFileAccessor::ReadBuffer` was allocating a buffer, copying data into it, and then handing that allocation to the `DataBuffer` constructor. The constructor copies data into a new allocation it owns so this allocation is unnecessary and was being leaked.
2024-11-13make abbreviation match longer name for DSC plugin infoJordan Wiens
2024-11-13Bulk add segments in the binary view to improve performanceXusheng
2024-11-12[SharedCache] Better failure state for triage viewkat
2024-11-11Add actual logger api to rustMason Reed
2024-11-09fix DSC first-popup dialogJordan Wiens
2024-11-06reorder DSC triage tabs and small text changesJordan Wiens