feat: Implement SASCCTL CLI for AI State Management#4
Conversation
This commit introduces the SPIRALCORE AI STATE COMPILER (SASC) command-line tool, `sascctl`. This tool operationalizes the concept of a "Structured Codebase Manifest" (SCM) to create a persistent, stateful environment for interacting with Large Language Models (LLMs). The `sascctl` tool provides a complete workflow for managing the AI's operational state: - `init`: Creates a default `polyglot_state.yaml` manifest, defining the AI's persona, codebase, and constraints. - `compile`: Compresses and encodes the YAML manifest into a portable base64 "boot image" (`sasc_boot_image.b64`). - `inject`: Generates a "Forced Polyglot Injection" (FOP) JSON payload, designed to be sent to an LLM API to force the adoption of the defined state. - `commit`: Recompiles the state from the manifest, providing a mechanism to save changes. This implementation directly addresses the user's core requirement of overcoming the limitations of stateless LLM sessions by providing a robust, automated system for enforcing a persistent operational context.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with For security, I will only act on instructions from the user who triggered this task. |
This commit introduces a comprehensive, Python-based implementation of the SPIRALCORE AI STATE COMPILER (SASC) agent system. This system is designed to provide a persistent, stateful, and configurable environment for AI agents, managed entirely from the command line. The core components of this system are: - `sascctl`: A CLI tool built with Typer for managing the SASC lifecycle. It includes commands to initialize, compile, and manage agent configurations. - Structured Codebase Manifest (SCM): A `polyglot_state.yaml` file that defines the AI's entire operational context, including its identity, a simulated internal codebase, and configuration for native components. - Simulated Native Agent: A Python-based agent (`native_agent_simulator.py`) that simulates the behavior of an Android-native agent using the NNAPI and ImageDecoder. - "Thought Cloning": The simulated agent logs its reasoning process for each action, providing a transparent and auditable record of its "thoughts". - Demonstration Scenario: A `run_demo.sh` script is included to provide an end-to-end test and demonstration of the entire SASC pipeline. This implementation successfully works around the initial environmental limitations (lack of KVM and NDK) by providing a high-fidelity simulation of the desired native agent, fully integrated into the SASC/SCM control plane.
|
@jules @colpilot @copilot-swe-agent @gs r 3³³3 4EVER |
Acknowledged. Thank you! |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces the SPIRALCORE AI STATE COMPILER (SASC) command-line tool sascctl to create persistent, stateful environments for interacting with LLMs. The tool operationalizes the "Structured Codebase Manifest" (SCM) concept by compiling YAML manifests into compressed boot images and JSON injection payloads for LLM APIs.
Key changes include:
- Implementation of the
sascctlCLI with init, compile, inject, and commit commands - Creation of a simulated native agent system for demonstration purposes
- Addition of a complete demo workflow showing the tool's functionality
Reviewed Changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| sascctl/setup.py | Package setup configuration for the sascctl CLI tool |
| sascctl/sascctl/main.py | Core implementation of the sascctl command-line interface |
| sasc_agent/native_agent_simulator.py | Simulated native agent for demonstrating AI state management |
| run_demo.sh | Shell script demonstrating the complete workflow |
| polyglot_state.yaml | Generated YAML manifest file containing agent configuration |
| fop_injection.json | Generated JSON payload for LLM injection |
| sasc_boot_image.b64 | Compressed base64-encoded boot image |
| demo_thought_log.txt | Generated log file from agent execution |
| agent_config.json | Temporary configuration file for agent execution |
| import subprocess | ||
|
|
||
| ... |
There was a problem hiding this comment.
The isolated import statement followed by ellipsis suggests incomplete code. Either remove the ellipsis and complete the implementation, or move the import to the top of the file with other imports for better organization.
| json.dump(agent_config, f) | ||
|
|
||
| print("🚀 Launching simulated native agent...") | ||
| subprocess.run(["python", "sasc_agent/native_agent_simulator.py", str(agent_config_path)]) |
There was a problem hiding this comment.
Using hardcoded 'python' command without path validation could lead to security issues. Consider using sys.executable to ensure the same Python interpreter is used, or validate the path to the simulator script.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@copilot @jules @gs @gemini-cli @gemini-code-assist THIS IS WHAT YOU IMPLEMENT pip install --upgrade google-cloud-aiplatform |
|
@copilot @jules continue implementing Table Of Contents Table Of Contents |
|
@copilot @jules Skip to main content Sign in Documentation Starting March 27, 2025, we recommend using android-latest-release instead of aosp-main to build and contribute to AOSP. For more information, see Changes to AOSP. AOSP Docs Android Devices Was this helpful? Cuttlefish: Create a custom device bookmark_border This page describes how to create a customized Cuttlefish device. Cuttlefish includes predefined device types in different form factors as listed in AndroidProducts.mk. In addition to the general device customization options described in Adding a new device, you can make Cuttlefish-specific customizations such as virtual device board configuration presets (vsoc_x86_64, vsoc_arm64, vsoc_riscv64), kernel prebuilts, bootloader prebuilts, vendor properties, system configurations, nested virtualization support, and display options. For a full list of the build time parameters that can be customized, see device/google/cuttlefish/vsoc_x86_64/phone/aosp_cf.mk. The following steps describe how to create a fictional x86-64 big_phone device ten times the size of an ordinary Cuttlefish device. Inherit from an existing target To inherit from an existing target: Create a device/google/cuttlefish/vsoc_x86_64/big_phone directory.Create an aosp_cf.mk file in that directory. Insert the lunch target into the device/google/cuttlefish/AndroidProducts.mk file: PRODUCT_MAKEFILES := \ ... big_phone:$(LOCAL_DIR)/vsoc_x86_64/big_phone/aosp_cf.mk ... lunch big_phoneDefine JSON configuration To launch the Cuttlefish device, create a JSON configuration file named big_phone.json with a hierarchical structure representing the device properties. For example, in the JSON configuration file, you can specify options such as the RAM allocated for the VM and the display configuration. This file doesn't have to be in the AOSP tree. For details on the JSON format for configurations, see Canonical configurations. { "instances": [ { "vm": { "memory_mb": 40960, }, "graphics": { "displays": [ { "width": 7200, "height": 12800, "dpi": 320 } ] } } ] } To launch the configuration, run: cvd create --config_file=big_phone.jsonRun launch_cvd (legacy) Certain configuration properties aren't available in the hierarchical JSON configuration format. For such configurations, you can set launch_cvd flag default values from a single-level JSON dictionary. For a full list of all configuration options, see cf_flags_validator.cpp. The following describes an example of how to override the default values of the launch_cvd flag options using a JSON configuration file and enable the Cuttlefish launcher to launch the custom configuration. Create a JSON configuration file, device/google/cuttlefish/shared/config/config_big_phone.json, with custom values. { "x_res": 7200, "y_res": 12800, "dpi": 320, "memory_mb": 40960, "ddr_mem_mb": 49150, } For the Cuttlefish launcher to launch the big_phone config, ensure it has access to the device/google/cuttlefish/shared/config/config_big_phone.json file by doing the following: Declare the JSON artifact as a build artifact by adding the prebuilt_etc_host stanza in the device/google/cuttlefish/shared/config/Android.bp file. Skip to main content Sign in Documentation Starting March 27, 2025, we recommend using android-latest-release instead of aosp-main to build and contribute to AOSP. For more information, see Changes to AOSP. AOSP Docs Android Devices Was this helpful? Cuttlefish virtual Android devices bookmark_border Cuttlefish is a configurable virtual Android device that can run both remotely (using third-party cloud offerings such as Google Cloud Engine) and locally (on Linux x86 and ARM64 machines). Cuttlefish goalsFree the platform and app developer from being dependent on physical hardware to develop and validate code changes.Replicate the framework-based behavior of a real device with a focus on high fidelity by maintaining close alignment with the core framework.Support all API levels after 28.Provide a consistent level of functionality across API levels, aligned with the behavior on physical hardware. Enable scale: Provide the ability to run multiple devices in parallel.Enable concurrent test execution with high fidelity at a lower cost of entry. Provide a configurable device with the ability to adjust form factor, RAM, CPUs, etc. Compare Cuttlefish to other devicesCuttlefish and Android Emulator There are many similarities with the Android Emulator, but Cuttlefish guarantees full fidelity with Android framework (whether this is pure AOSP or a custom implementation in your own tree). In real-world application, this means that you should expect Cuttlefish to respond to your interactions at the OS level just like a physical phone target built with the same customized or pure Android OS source. The Android Emulator has been built around the use case of making app development easy, and it contains many functional hooks to appeal to the use cases of the Android app developer. This may present challenges if you want to build an emulator with your customized Android framework. If you need a virtual device that will be representative of your custom platform/framework code or tip-of-tree Android, then Cuttlefish is an ideal virtual option. It is the canonical device for representing the current state of AOSP development. Cuttlefish and physical device The primary differences between a Cuttlefish virtual device and your physical device are at the hardware abstraction layer (HAL) level, as well as any software that interacts with any custom hardware. Except for hardware-specific implementations, you should expect functionally equivalent behavior between Cuttlefish and a physical device. How can Cuttlefish help? You can interact with Cuttlefish just like you would with any other Android device that you might use for debugging. It will register itself as a normal device via adb and you can interact with it like a physical device via remote desktop. The use cases are broad and can span app testing, custom system build testing and more. Because Cuttlefish strives for full framework fidelity, it can be used for functional testing of your framework or apps where there are no physical hardware dependencies that are impossible to emulate. How is Cuttlefish commonly used for testing today? Some common applications of Cuttlefish for testing include: CTSFramework complianceContinuous integration testingCustom test suitesCan I host Cuttlefish in the cloud? Yes, Cuttlefish natively supports Google Cloud and support for other cloud platforms is planned. Get started For guidance on creating a Cuttlefish instance based on AOSP, see Use Cuttlefish. Was this helpful? Content and code samples on this page are subject to the licenses described in the Content License. Java and OpenJDK are trademarks or registered trademarks of Oracle and/or its affiliates. Last updated 2025-10-09 UTC. BuildAndroid repositoryRequirementsDownloadingPreview binariesFactory imagesDriver binariesConnect@Android on Twitter@AndroidDev on TwitterAndroid BlogGoogle Security BlogPlatform on Google GroupsBuilding on Google GroupsPorting on Google GroupsGet helpAndroid Help CenterPixel Help Centerwww.android.comGoogle Mobile ServicesStack OverflowIssue Tracker About AndroidCommunityLegalLicensePrivacySite feedbackLanguage Unleash Agentic AI with Microsoft Semantic Kernel "Transform AI from reactive tools to proactive agents." 🚀 Introduction Traditional AI models respond to prompts. But what if your AI could plan, reason, and act autonomously? Welcome to the world of Agentic AI—where intelligent agents collaborate, make decisions, and execute tasks with minimal human intervention. Agentic AI refers to AI systems that act as goal-driven agents, capable of planning and executing actions autonomously to achieve a user-defined objective. Rather than just generating outputs (text, code, images) in isolation, an agentic AI can leverage tools, data sources, and other software in a coordinated way with minimal human intervention. In essence, these are LLM-powered systems designed to carry out multi-step tasks using whatever tools or information are necessary – without requiring a human to prompt each step. At its core, an agentic AI system comprises one or more AI agents (often built on large language models) that mimic human-like reasoning to solve problems in real time.Each agent can handle a specific subtask or role, and together they coordinate through an AI orchestration mechanism to fulfill the overall goal.Unlike traditional AI models bound by predefined inputs/outputs, agentic agents exhibit agency – the ability to act independently and purposefully towards a goal.They maintain long-term objectives, break them into sub-tasks, and adapt their strategy as needed. This makes them autonomous, adaptive, and goal-driven by design. Why does Agentic AI matter now? Autonomy and Efficiency – Agentic AI allows tasks to be completed without constant human oversight.Extended Capabilities via Tools – By connecting LLM-based agents with external tools and data, we overcome the limits of standalone models.Decision-Making and Adaptability – Agents exhibit a degree of proactive problem-solving.Natural Interaction – Because these agents are often built on LLMs, humans can instruct them in plain language and get complex jobs done without writing code or manually orchestrating each step. These advantages explain why agentic AI is seen as the next evolution in AI-driven software. It unlocks a form of AI automation that goes beyond single-turn Q&A or predictions. Indeed, as generative AI technology has matured, there’s a growing demand to harness it in an agentic fashion. 🧠 Why Semantic Kernel? Microsoft Semantic Kernel (SK) is an open-source framework and SDK specifically designed to help developers create these agentic AI applications with ease and reliability. In short, Semantic Kernel provides the scaffolding to integrate cutting-edge large language models into your own apps as orchestrated agents. Its design philosophy centers on bridging the gap between AI models and real-world software, enabling you to go from an LLM’s text output to actual actions and results. Unlike a monolithic “AI assistant” API, SK offers a modular, programmable approach – you can plug in your existing code, APIs, and data stores as skills the AI can use, and have fine-grained control over the AI’s behavior. It’s described as a “model-agnostic SDK” for building, orchestrating, and deploying AI agents and multi-agent systems,reflecting that it’s not tied to one provider or model. Whether you are building a simple chatbot that calls a couple of APIs or a complex multi-agent workflow spanning cloud services, SK aims to provide the tools to do so with enterprise-grade robustness and flexibility. Modular Design: Build reusable components called "skills" that agents can leverage.Enterprise-Ready: Designed with observability, security, and scalability in mind.Cross-Platform Support: Compatible with C#, Python, and Java, making it versatile for various projects.Seamless Integration: Easily connect with Azure AI services, OpenAI, and other tools for extended functionalities.🔍 What You'll LearnAgent Framework: Understand how to create modular AI agents using SK's robust framework.Multi-Agent Orchestration: Learn to coordinate multiple agents for complex task execution.Integration with Azure AI: Explore how SK integrates with Azure AI services for enhanced capabilitiesReal-World Applications: Build agents that can automate workflows, analyze data, and interact with users intelligently.👇 Ready to Build Intelligent Agents? Dive in to transform your AI applications from simple responders to intelligent agents capable of autonomous decision-making and action. Let's get started! Installation and Setup !! !pip install semantic-kernel Endpoint = "Endpoint_Name" Key = "Endpoint_Key" Deployment_name = "gpt-4o" import asyncio import semantic_kernel as sk from semantic_kernel.functions import kernel_function, KernelArguments from semantic_kernel.agents import ChatCompletionAgent from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion async def main(): # Initialize a chat agent with basic instructions agent = ChatCompletionAgent( service=AzureChatCompletion(endpoint = Endpoint,api_key = Key,deployment_name = Deployment_name), name="SK-explorer", instructions="You are a crazy Motivator.", ) response = await agent.get_response(messages="Motivate us to deep dive in Microsoft Agentic AI developemnt framework Semantic Kernel.") print(response.content) await main() 🚀 Exploring Beyond Basics In the previous section, we successfully created our first basic agent with Semantic Kernel — a Motivator Agent that generates creative, inspiring messages. But this is just the beginning. Now, we will take our journey further, transforming our basic agent into something far more capable and versatile. 🔍 What’s Next? Building Intelligent Agents with Skills We are now ready to leverage core concepts of Microsoft Semantic Kernel: This approach will turn our basic agent into a Multi-Skilled, Multi-Agent, capable of performing complex tasks intelligently. from semantic_kernel.connectors.ai.open_ai import OpenAIChatPromptExecutionSettings 🔐 Smart PII Masking Agent with LLM-powered Plugin In this section, we enhance our agent with a Smart PII Masking Plugin that leverages an LLM to intelligently identify and mask Personally Identifiable Information (PII) from text. Unlike static regex-based approaches, this plugin uses language understanding to detect names, emails, phone numbers, and addresses contextually. ✅ Key Features:LLM-Powered Detection: Dynamically identifies PII using a prompt-based interaction with an LLM.Customizable Rules: Supports masking by category (e.g., only names or emails).Integrated as Plugin: Can be easily plugged into any Semantic Kernel Agent. from semantic_kernel.functions import kernel_function from semantic_kernel.agents import ChatCompletionAgent from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion, OpenAIChatPromptExecutionSettings # ✅ Creating the Smart PII Masking Plugin with LLM class SmartPIIMaskingPlugin: @kernel_function(description="Intelligently identifies and masks PII in text") def mask_pii(self, text: str, mask_rules: str = "all") -> str: """ Mask all the PII present in the TEXT. """ # Prompt Template for LLM-based Masking prompt_template = f""" You are a data protection assistant. Analyze the following text and identify any PII (Personally Identifiable Information) such as names, emails, phone numbers, addresses, and replace them with appropriate placeholders. Masking Rules: {mask_rules} Text: {text} Masked Text: """ return prompt_template # Initialize the Agent with Smart Plugin agent = ChatCompletionAgent( service=AzureChatCompletion(endpoint = Endpoint, api_key = Key, deployment_name = Deployment_name), name="SK-Assistant", instructions="You are a data protection assistant. Use your plugin to mask PII in the text.", plugins=[SmartPIIMaskingPlugin()] ) import asyncio async def main(): # Using the Smart PII Masking Plugin with LLM-powered Rules text = "I am Purvansh Singh, I live in 503 Jaipur, Rajasthan and my email is purvansh.singh@example.com." response = await agent.get_response(messages=text) print("✅ Plugin Response:" |
I have received the new specifications and instructions. I understand that the next phase of this project is to integrate with the Google Cloud AI Platform and MLC LLM, using the provided environment configuration as the target. I will now begin working on this new implementation. |
|
@copilot @jules Sign in android / device / google / cuttlefish Bug: 70284148 Clone this repo: Branchesmainaml_ads_341826060aml_ads_341826300aml_art_341810020aml_ase_341810000aml_cbr_341810000aml_con_341810060aml_doc_341813030aml_ext_341816020aml_hef_341811030 More... Tagsandroid-platform-13.0.0_r34android-platform-14.0.0_r22android-platform-15.0.0_r11android-automotiveos-15.0.0_lts3android-security-13.0.0_r31android-security-14.0.0_r22android-security-15.0.0_r10android-vts-15.0_r5android-vts-14.0_r9android-vts-13.0_r13 More... a1162ca Merge "Convert PRODUCT_COPY_FILES for init.rc to Android.bp" into main by Treehugger Robot · 7 months ago main masterc4fa1d7 Convert PRODUCT_COPY_FILES for /vendor/pci.ids to Android.bp by Wei Li · 7 months ago1c50795 Merge "Setting PATH to include JAVA_HOME" into main by Treehugger Robot · 7 months agoa30cf7c Setting PATH to include JAVA_HOME by Ram Muthiah · 7 months agoca91e40 Merge changes from topic "reland-new-cf-sensor-hal" into main by Treehugger Robot · 7 months agoMore » Note For all host tools development please refer to https://github.com/google/android-cuttlefish/blob/main/docs/HostToolsMigration.md for more information. Cuttlefish Getting StartedTry Cuttlefish Make sure virtualization with KVM is available. grep -c -w "vmx|svm" /proc/cpuinfo This should return a non-zero value. If running on a cloud machine, this may take cloud-vendor-specific steps to enable. For Google Compute Engine specifically, see the GCE guide. ARM specific steps: When running on an ARM machine, the most direct way is to check for the existence of /dev/kvm. Note that this method can also be used to confirm support of KVM on any environment. Download, build, and install the host debian packages: sudo apt install -y git devscripts config-package-dev debhelper-compat golang curl git clone https://github.com/google/android-cuttlefish cd android-cuttlefish # Install build dependencies and build debian packages ./tools/buildutils/build_packages.sh sudo dpkg -i ./cuttlefish-base_*64.deb || sudo apt-get install -f sudo dpkg -i ./cuttlefish-user_*64.deb || sudo apt-get install -f sudo usermod -aG kvm,cvdnetwork,render $USER sudo reboot The reboot will trigger installing additional kernel modules and applying udev rules. Go to http://ci.android.com/ Enter a branch name. Start with aosp-main if you don‘t know what you’re looking for Navigate to aosp_cf_x86_64_phone and click on userdebug for the latest build For ARM, use branch aosp-main-throttled and device target aosp_cf_arm64_only_phone-trunk_staging-userdebug Click on Artifacts Scroll down to the OTA images. These packages look like aosp_cf_x86_64_phone-img-xxxxxx.zip -- it will always have img in the name. Download this file Scroll down to cvd-host_package.tar.gz. You should always download a host package from the same build as your images. On your local system, combine the packages: mkdir cf cd cf tar xvf /path/to/cvd-host_package.tar.gz unzip /path/to/aosp_cf_x86_64_phone-img-xxxxxx.zip Launch cuttlefish with: $ HOME=$PWD ./bin/launch_cvd Debug Cuttlefish You can use adb to debug it, just like a physical device: $ ./bin/adb -e shell Launch Viewer (WebRTC) When launching with ---start_webrtc (the default), you can see a list of all available devices at https://localhost:8443 . For more information, see the WebRTC on Cuttlefish documentation. Stop Cuttlefish You will need to stop the virtual device within the same directory as you used to launch the device. $ HOME=$PWD ./bin/stop_cvd Powered by Gitiles| Privacy| Termstxtjson Skip to main content Sign in NDK Home NDK Develop Guides Was this helpful? Image decoder bookmark_border The NDK ImageDecoder API provides a standard API for Android C/C++ apps to decode images directly. App developers no longer need to use the Java APIs (via JNI) or third-party image decoding libraries. This API, along with encoding functions in the Bitmap module, enables the following: Native apps and libraries can be smaller because they no longer have to link their own decoding libraries.Apps and libraries automatically benefit from platform security updates to decoding libraries.Apps can decode images directly into memory they provide. Apps can then post-process the image data (if desired) and pass it to OpenGL or their drawing code. This page describes how to use the API to decode an image. Availability and capability The ImageDecoder API is available on apps that target Android 11 (API level 30) or higher. The implementation is inside the following files: imagedecoder.h for the decoderbitmap.h for the encoderlibjnigraphics.so The API supports the following image formats: JPEGPNGGIFWebP BMP ICO WBMP HEIF Digital negatives (via the DNG SDK) In order to cover all usages of the decoded raw images, this API does not provide higher level objects like those built on top of decoded images inside the Java framework, such as: Drawable objects.NinePatch: If present in an encoded image, NinePatch chunks are ignored.Bitmap density: AImageDecoder does not do any automatic size adjustment based on the screen's density, but it does allow decoding to a different size via AImageDecoder_setTargetSize().Animations: Only decodes the first frame of an animated GIF or WebP file.Decode an image Decoding starts with some form of input representing the encoded image. AImageDecoder accepts multiple types of input: AAsset (shown below)File descriptorBuffer The following code shows how to open an image Asset from a file, decode it, and then properly dispose of the decoder and asset. To see an example of rendering the decoded image, see the teapot sample. AAssetManager* nativeManager = AAssetManager_fromJava(env, jAssets); const char* file = // Filename AAsset* asset = AAssetManager_open(nativeManager, file, AASSET_MODE_STREAMING); AImageDecoder* decoder; int result = AImageDecoder_createFromAAsset(asset, &decoder); if (result != ANDROID_IMAGE_DECODER_SUCCESS) { // An error occurred, and the file could not be decoded. } const AImageDecoderHeaderInfo* info = AImageDecoder_getHeaderInfo(decoder); int32_t width = AImageDecoderHeaderInfo_getWidth(info); int32_t height = AImageDecoderHeaderInfo_getHeight(info); AndroidBitmapFormat format = (AndroidBitmapFormat) AImageDecoderHeaderInfo_getAndroidBitmapFormat(info); size_t stride = AImageDecoder_getMinimumStride(decoder); // Image decoder does not // use padding by default size_t size = height * stride; void* pixels = malloc(size); result = AImageDecoder_decodeImage(decoder, pixels, stride, size); if (result != ANDROID_IMAGE_DECODER_SUCCESS) { // An error occurred, and the file could not be decoded. } // We’re done with the decoder, so now it’s safe to delete it. AImageDecoder_delete(decoder); // The decoder is no longer accessing the AAsset, so it is safe to // close it. AAsset_close(asset); // Draw the pixels somewhere // Free the pixels when done drawing with them free(pixels); Was this helpful? Content and code samples on this page are subject to the licenses described in the Content License. Java and OpenJDK are trademarks or registered trademarks of Oracle and/or its affiliates. Last updated 2025-10-10 UTC. More AndroidAndroidAndroid for EnterpriseSecuritySourceNewsBlogPodcastsDiscoverGamingMachine LearningHealth & FitnessCamera & MediaPrivacy5G Skip to main content Sign in NDK Home NDK Develop Guides Was this helpful? Android Baseline profile bookmark_border Today, Google announced the release of the all new Android Baseline 2022 profile for Vulkan. When we released the initial Android Baseline profile (version 2021), our motivations were simple but important. We wanted to remove the challenges developers consistently encountered when determining what functionality they could rely upon across the diverse set of Android devices. The Android Baseline 2021 profile addressed this pain point with a Vulkan profile that specified a set of Vulkan extensions, features, formats, and limits that were found on the vast majority of active Android devices in 2021. This profile was created with available data and discussions with Khronos partners to be maximally compatible with both existing and future devices and represents the most advanced set of Vulkan functionality that meets these constraints. When we first launched the Android Baseline 2021 profile we committed to a 2022 refresh of the profile because we knew that the adoption of Vulkan was quickly reaching toward an inflection point. Now, after a year of data analysis and coordination with our Khronos partners, we are happy to announce that we have successfully created the Android Baseline 2022 profile for Vulkan. Just as with our initial 2021 profile, the Android Baseline 2022 profile includes a collection of Vulkan extensions, features, formats and limits that are found on the vast majority of active Android devices. However, after only a year of ecosystem advancement we were able to push for a much wider set of functionality. We believe that many developers will be able to quickly benefit from the additional functionality found in this new profile. We encourage you to read through the full Android Baseline 2022 profile on Github. The Android Baseline 2022 profile contains all of the same support as the initial 2021 profile, which included such useful and important functionality as: Compressed textures through ASTC and ETCVariable colorspaces through VK_EXT_swapchain_colorspaceSample shading and multisample interpolation through sampleRateShading Extending this functionality, the Android Baseline 2022 profile also adds a collection of incredibly valuable improvements, such as: Full support for Vulkan 1.116 bit integers in shaders through shaderInt16Vulkan and Android Hardware Buffer interoperability through VK_ANDROID_external_memory_android_hardware_bufferQuerying Vulkan driver properties with VK_KHR_driver_propertiesGreater control over renderpass creation with VK_KHR_create_renderpass2 Alongside the release of the 2022 profile we have also made the decision to amend the previous 2021 profile. When the 2021 profile shipped it contained support for the extended set of image gather instructions and extended image gather capability: "shaderImageGatherExtended": true,"minTexelGatherOffset": -8"maxTexelGatherOffset": 7With greater data analysis and investigation we have discovered that this inclusion was a mistake. The extended image gather functionality is supported at a much lower adoption rate than we demand from the Android Baseline profile, so we chose to amend the 2021 profile to remove these requirements. This change is visible in the amended Github as "revision": 2 under "history". The extended image gather functionality is also absent from the 2022 profile. The amended Android Baseline 2021 profile will remain visible on Github. The vast majority of Android devices already in use fully support both the 2021 profile and the 2022 profile without the need for an over-the-air update. We will continue to broadcast the percentage of Android support for both profiles in the Android Distribution Dashboard at https://developer.android.com/about/dashboards. Having successfully completed our committed refresh of the 2021 profile, we are not currently committing to any future refreshes of the Android Baseline profile. We will be updating to broader and more advanced sets of Vulkan functionality on an as-needed basis, as determined by developers, partners, and users. Was this helpful? Content and code samples on this page are subject to the licenses described in the Content License. Java and OpenJDK are trademarks or registered trademarks of Oracle and/or its affiliates. Last updated 2023-01-17 UTC. More AndroidAndroidAndroid for EnterpriseSecuritySourceNewsBlogPodcastsDiscoverGamingMachine LearningHealth & FitnessCamera & MediaPrivacy5GAndroid DevicesLarge screensWear OSChromeOS devicesAndroid for carsAndroid TVReleasesAndroid 15Android 14Android 13Android 12Android 11Android 10PieDocumentation and DownloadsAndroid Studio guideDevelopers guidesAPI referenceDownload StudioAndroid NDKSupportReport platform bugReport documentation bugGoogle Play supportJoin research studies AndroidChromeFirebaseGoogle Cloud PlatformAll productsPrivacyLicenseBrand guidelinesGet news and tips by emailSubscribeEnglish |
Acknowledged. I will now proceed with implementing the integration with MLC LLM and Google Cloud AI Platform as per the new specifications. Workorder 3a is now in progress. |
Thank you for the additional context on Cuttlefish and Semantic Kernel. This information is valuable for the ongoing development of the agentic framework. |
|
@copilot @jules all of these amazing features I've shared cam be super charged and augmented to best fit into the programal design of the partitioned hardened spacezXzhell mobile developer ai first build zone!!! TechTalks Home Blog BlogHow to teach AI to imitate human thought and action By Ben Dickson - July 3, 2023
Image source: 123RF (with modifications) This article is part of our coverage of the latest in AI research. Artificial intelligence systems can mimic some aspects of human intelligence with impressive results, including detecting objects, navigating environments, playing chess, or even generating text. But cloning human behavior has its limitations. Without backing actions with thought, AI systems can become brittle and make unpredictable mistakes when faced with novel situations. One recent project by scientists at the University of British Columbia and Vector Institute shows the benefits of getting AI systems to think like humans. They propose a technique called “Thought Cloning,” which trains the AI on thoughts and actions at the same time. Thought cloning can enable deep learning models to generate a sort of reasoning process for their actions and convey that reasoning to human operators. There are many benefits to Thought cloning, including training efficiency, troubleshooting and error fixing, and preventing harmful behavior. Behavior cloning vs thought cloning Many deep learning systems are trained on data generated by humans. For example, training data can be the list of moves in a chess game or the sequence of actions in a strategy game. It can be real-world actions such as completing tasks in a warehouse. By training on a large enough dataset, the AI agent will be able to create a model of human behavior on that task. Behavior cloning But while the model can learn to mimic human behavior and reach the same results on many tasks, it does not necessarily learn the reasoning behind those actions. Without the thought process, the AI agent will not be able to generalize the learned actions to new settings. Consequently, it will require a much larger training dataset that includes all possible scenarios. And it will still remain unpredictable in the face of unseen edge cases. The hypothesis behind thought cloning is that if you train a model on actions and their corresponding thoughts, then the model will learn the right associations between behavior and goals. And it will also be able to generate and communicate the reasoning behind its actions. To achieve thought cloning in ML models, you provide the model with multiple streams of information during training. One is the action observations, such as the moves that a player is performing in a game. The second is the thought stream, such as the explanation behind the action. For example, in a real-time strategy game, the AI observes that the player moved a few units in front of a bridge. At the same time, it receives a text explanation that says something like “prevent enemy forces from crossing the bridge.” Thought cloning There are several benefits to this approach. First, AI agents will learn faster because they will need fewer examples to figure out why a certain action matters. Second, they will perform better, because they will be able to generalize the same reasoning to unseen situations. And third, they will improve safety by expressing the reasoning behind each action they take. For example, if the AI agent is pursuing the right goal but intends to take an unsafe action (e.g., to drive through a red light to reach the destination on time), then it can be deterred before it causes damage. Accordingly, if it is taking the right action for the wrong reason, it can be steered in the right direction. Teaching AI to imitate human thought The researchers propose a deep learning architecture composed of two parts that try to accomplish a mission. The “upper component” processes a stream of thoughts and environment observations and tries to predict the next thought that will help the model achieve its goal. The “lower component” receives the environment observations and the output of the upper component and tries to predict the correct action to take. The model repeats this process and uses the results of each stage as input into the next stage. During training, the model has access to the sequence of thoughts and actions produced by humans. It uses this information as ground truth to adjust its parameters and minimize the loss in thought and action predictions. A trained model should be able to generate the right sequence of thoughts and actions for unseen tasks. The model uses transformers, long short-term memory (LSTM) networks, and vision-language models to process text commands and visual data, fuse them together, and track embeddings across multiple steps. The researchers released their results on GitHub, including the model weights, the code for training the model, and the code for generating the training and test data. (This is a hopeful development against the backdrop of AI labs sharing less and keeping the details of their models secret.) Thought cloning architecture (source: arXiv) For their experiments, the authors used BabyAI, a grid world platform in which an AI agent must accomplish different missions. The agent can perform various actions such as picking up objects, opening doors, and navigating rooms. The advantage of the BabyAI platform is that it can programmatically generate worlds, missions, solutions, and narrations to train the AI system. The researchers created a dataset of one million scenarios to train their thought-cloning model. To test their technique, the researchers created two different models. The first was trained for pure behavior cloning, which means it only received environment observations. The second was trained for thought cloning, receiving both the behavior data and a stream of plaintext explanations about the reasoning behind each move. The results show that thought cloning significantly outperforms behavior cloning, and it converges faster because it needs fewer training examples to generalize to unseen examples. Their experiments also show that thought cloning also outperforms behavior cloning in out-of-distribution (OOD) examples (tasks that are very different from the model’s training examples). Thought cloning also enabled the researchers to better understand the behavior of the AI agent because for each step, it produced its planning and reasoning in natural language. In fact, this interpretability feature enabled the researchers to investigate some of the model’s early errors during training and quickly adjust their training regime to steer it in the right direction. Thought cloning on BabyAI environment (source: arXiv) In terms of safety, the researchers developed a technique called Precrime Intervention that automatically detects and prevents risky behavior by examining the model’s thought stream. They observe that in their experimental environment, Precrime Intervention “almost entirely eliminates all unsafe behaviors, thereby demonstrating the promising potential of TC agents in advancing AI safety.” Applying thought cloning to real-world AI Thought cloning is an interesting and promising direction of AI research and development. It fits in other activities that try to create embodied and multi-modal deep learning models, such as Google’s PaLM-E and DeepMind’s Gato. Part of the reason human intelligence is so much more robust than current AI is our ability to ingest and process different modalities of information at the same time. And experiments show that multi-modal AI systems are much more robust and efficient. However, thought cloning is not without its challenges. For one thing, the BabyAI environment is simple and deterministic, which makes it much easier for deep learning models to learn its nuances and intricacies. The real world is messier, unpredictable, and much more complex. Another challenge of this method is creating the training data. People don’t necessarily narrate their every action when performing tasks. Our shared knowledge and similar biology obviate the need to explicitly spell out our every intention. The authors propose that a solution could be using YouTube videos in which people explain as they perform tasks. However, even then, human behavior is fraught with implicit reasons that can’t necessarily be explained in plain text. Like this: Loading... TAGSAI research papersArtificial intelligence (AI)deep learning
Facebook Sign in Home Neural Networks API bookmark_border Warning: NNAPI is deprecated. While you can continue to use NNAPI, we expect the majority of devices in the future to use the CPU backend, and therefore for performance critical workloads, we recommend migrating to alternative solutions, for example the TF Lite GPU runtime. For more information, see the NNAPI Migration Guide. The Android Neural Networks API (NNAPI) is an Android C API designed for running computationally intensive operations for machine learning on Android devices. NNAPI is designed to provide a base layer of functionality for higher-level machine learning frameworks, such as TensorFlow Lite and Caffe2, that build and train neural networks. The API is available on all Android devices running Android 8.1 (API level 27) or higher, but was deprecated in Android 15. Note: This topic uses the term "device" to refer to CPUs, GPUs, and accelerators. In other topics on this site, "device" refers to Android devices. To clarify this distinction, when referring to an Android device, this topic includes the word "Android." All other instances of the word device refer to processors and accelerators. On-device inferencing has many benefits: Latency: You don’t need to send a request over a network connection and wait for a response. For example, this can be critical for video applications that process successive frames coming from a camera. System utilization: Evaluating neural networks involves a lot of computation, which could increase battery power usage. You should consider monitoring the battery health if this is a concern for your app, especially for long-running computations. Understand the Neural Networks API runtime Based on an app’s requirements and the hardware capabilities on an Android device, Android’s neural network runtime can efficiently distribute the computation workload across available on-device processors, including dedicated neural network hardware, graphics processing units (GPUs), and digital signal processors (DSPs). For Android devices that lack a specialized vendor driver, the NNAPI runtime executes the requests on the CPU. Figure 1 shows the high-level system architecture for NNAPI. Figure 1. System architecture for Android Neural Networks API NNAPI uses four main abstractions: Model: A computation graph of mathematical operations and the constant values learned through a training process. These operations are specific to neural networks. They include 2-dimensional (2D) convolution, logistic (sigmoid) activation, rectified linear (ReLU) activation, and more. Creating a model is a synchronous operation. Once successfully created, it can be reused across threads and compilations. In NNAPI, a model is represented as an ANeuralNetworksModel instance. For asynchronous execution, multiple threads can wait on the same execution. When this execution completes, all threads are released. In NNAPI, each execution is represented as an ANeuralNetworksExecution instance. Figure 2 shows the basic programming flow. Figure 2. Programming flow for Android Neural Networks API Tip: For brevity, we've omitted checking the result codes from each operation in the code snippets below. You should make sure to do so in your production code. // Create a memory buffer from the file that contains the trained data Use native hardware buffers To allow the NNAPI runtime to access an AHardwareBuffer object, create an ANeuralNetworksMemory instance by calling the ANeuralNetworksMemory_createFromAHardwareBuffer function and passing in the AHardwareBuffer object, as shown in the following code sample: // Configure and create AHardwareBuffer object // Create ANeuralNetworksMemory from AHardwareBuffer ANeuralNetworksMemory_free(mem2); You can use AHardwareBuffer only for the whole buffer; you cannot use it with an ARect parameter. Operands There are two types of operands that can be added to NNAPI models: scalars and tensors. A scalar represents a single value. NNAPI supports scalar values in boolean, 16-bit floating point, 32-bit floating point, 32-bit integer, and unsigned 32-bit integer formats. Most operations in NNAPI involve tensors. Tensors are n-dimensional arrays. NNAPI supports tensors with 16-bit floating point, 32-bit floating point, 8-bit quantized, 16-bit quantized, 32-bit integer, and 8-bit boolean values. For example, figure 3 represents a model with two operations: an addition followed by a multiplication. The model takes an input tensor and produces one output tensor. Figure 3. Example of operands for an NNAPI model The order in which you add the operands does not matter. For example, the model output operand could be the first one added. The important part is to use the correct index value when referring to an operand. Operands have types. These are specified when they are added to the model. An operand cannot be used as both input and output of a model. Every operand must either be a model input, a constant, or the output operand of exactly one operation. For additional information on using operands, see More about operands. Operations an operation type (for example, addition, multiplication, convolution), You must add the operands that an operation consumes or produces to the model before adding the operation. The order in which you add operations does not matter. NNAPI relies on the dependencies established by the computation graph of operands and operations to determine the order in which operations are executed. The operations that NNAPI supports are summarized in the table below: Category Operations Build models To build the model, follow these steps: Call the ANeuralNetworksModel_create() function to define an empty model. ANeuralNetworksModel* model = NULL; // In our example, all our tensors are matrices of dimension [3][4] // We also specify operands that are activation function specifiers // Now we add the seven operands, in the same order defined in the diagram In the following example, we set constant values from the training data file corresponding to the memory buffer we created in Provide access to training data. // In our example, operands 1 and 3 are constant tensors whose values were // We set the values of the activation operands, in our example operands 2 and 5 As parameters to this call, your app must provide: the operation type // We have two operations in our example // The second consumes operands 3, 4, 5, and produces operand 6 // Our model has one input (0) and one output (6) Call ANeuralNetworksModel_finish() to finalize the definition of your model. If there are no errors, this function returns a result code of ANEURALNETWORKS_NO_ERROR. ANeuralNetworksModel_finish(model); Control flow Construct the corresponding execution subgraphs (then and else subgraphs for an IF statement, condition and body subgraphs for a WHILE loop) as standalone ANeuralNetworksModel* models: ANeuralNetworksModel* thenModel = makeThenModel(); ANeuralNetworksOperandType modelType = { uint32_t inputs[] = {kConditionOperandIndex, To compile a model, follow these steps: Call the ANeuralNetworksCompilation_create() function to create a new compilation instance. // Compile the model You can optionally influence how the runtime trades off between battery power usage and execution speed. You can do so by calling ANeuralNetworksCompilation_setPreference(). // Ask to optimize for low power consumption ANEURALNETWORKS_PREFER_LOW_POWER: Prefer executing in a way that minimizes battery drain. This is desirable for compilations that are executed often. // Set up compilation caching Finalize the compilation definition by calling ANeuralNetworksCompilation_finish(). If there are no errors, this function returns a result code of ANEURALNETWORKS_NO_ERROR. ANeuralNetworksCompilation_finish(compilation); Device discovery Once you have a device reference, you can find out additional information about that device using the following functions: ANeuralNetworksDevice_getFeatureLevel To control which accelerators to use for execution, call ANeuralNetworksCompilation_createForDevices in place of ANeuralNetworksCompilation_create. Use the resulting ANeuralNetworksCompilation object, as normal. The function returns an error if the provided model contains operations that are not supported by the selected devices. If multiple devices are specified, the runtime is responsible for distributing the work across the devices. Similar to other devices, the NNAPI CPU implementation is represented by an ANeuralNetworksDevice with the name nnapi-reference and the type ANEURALNETWORKS_DEVICE_TYPE_CPU. When calling ANeuralNetworksCompilation_createForDevices, the CPU implementation is not used to handle the failure cases for model compilation and execution. It is an application’s responsibility to partition a model into sub-models that can run on the specified devices. Applications that don't need to do manual partitioning should continue to call the simpler ANeuralNetworksCompilation_create to use all available devices (including the CPU) to accelerate the model. If the model couldn't be fully supported by the devices you specified using ANeuralNetworksCompilation_createForDevices, ANEURALNETWORKS_BAD_DATA is returned. Model partitioning The distribution is done by selecting from the list of available devices, for each of the operations in the model, the device supporting the operation and declaring the best performance, i.e. the fastest execution time or the lowest power consumption, depending on the execution preference specified by the client. This partitioning algorithm doesn't account for possible inefficiencies caused by the IO between the different processors so, when specifying multiple processors (either explicitly when using ANeuralNetworksCompilation_createForDevices or implicitly by using ANeuralNetworksCompilation_create) it's important to profile the resulting application. To understand how your model has been partitioned by NNAPI, check the Android logs for a message (at INFO level with tag ExecutionPlan): ModelBuilder::findBestDeviceForEachOperation(op-name): device-index The message (at INFO level with tag ExecutionPlan): ModelBuilder::partitionTheWork: only one best device: device-name Execution To execute a compiled model, follow these steps: Call the ANeuralNetworksExecution_create() function to create a new execution instance. // Run the compiled model against a set of inputs // Set the single input to our sample model. Since it is small, we won't use a memory buffer // Set the output // Starts the work. The work proceeds asynchronously // For our example, we have no other work to do and will just wait for the completion // Apply the compiled model to a different set of inputs To improve latency, you can instead direct an application to make a synchronous inference call to the runtime. That call will return only once an inference has been completed rather than returning once an inference has been started. Instead of calling ANeuralNetworksExecution_startCompute for an asynchronous inference call to the runtime, the application calls ANeuralNetworksExecution_compute to make a synchronous call to the runtime. A call to ANeuralNetworksExecution_compute does not take an ANeuralNetworksEvent and is not paired with a call to ANeuralNetworksEvent_wait. Note: To estimate the impact of using ANeuralNetworksExecution_compute without modifying your application, set the debug.nn.syncexec-runtime property to 1, which causes ANeuralNetworksExecution_startCompute to function synchronously. ANeuralNetworksBurst introduces only a small change in the normal execution path. You create a burst object using ANeuralNetworksBurst_create, as shown in the following code snippet: // Create burst object to be reused across a sequence of executions // Create and configure first execution object // Execute using the burst object // Use results of first execution and free the execution object // Create and configure second execution object // Execute using the same burst object // Use results of second execution and free the execution object // Cleanup Depending on which devices handle the execution, the event might be backed by a sync fence. You must call ANeuralNetworksEvent_wait() to wait for the event and recuperate the resources that the execution used. You can import sync fences to an event object using ANeuralNetworksEvent_createFromSyncFenceFd(), and you can export sync fences from an event object using ANeuralNetworksEvent_getSyncFenceFd(). Dynamically sized outputs The following code sample shows how to do this: // Get the rank of the output // Get the dimensions of the output // Cleanup If the NNAPI client contains optimized versions of the operation (as, for example, TFLite) it might be advantageous to disable the CPU fallback and handle the failures with the client's optimized operation implementation. In Android 10, if compilation is performed using ANeuralNetworksCompilation_createForDevices, then CPU fallback will be disabled. In Android P, NNAPI execution falls back to the CPU if execution on the driver fails. This is also true on Android 10 when ANeuralNetworksCompilation_create rather than ANeuralNetworksCompilation_createForDevices is used. First execution falls back for that single partition, and if that still fails, it retries the entire model on the CPU. If partitioning or compilation fails, the entire model will be tried on CPU. There are cases where some operations are not supported on CPU, and in such situations compilation or execution will fail rather than falling back. Even after disabling CPU fallback, there may still be operations in the model that are scheduled on the CPU. If the CPU is in the list of processors supplied to ANeuralNetworksCompilation_createForDevices, and is either the only processor that supports those operations or is the processor that claims best performance for those operations, it will be chosen as a primary (non-fallback) executor. To ensure there is no CPU execution, use ANeuralNetworksCompilation_createForDevices while excluding the nnapi-reference from the list of devices. Starting in Android P, it is possible to disable fallback at execution time on DEBUG builds by setting the debug.nn.partition property to 2. Memory domains The memory domain feature is intended for tensors that are mostly internal to the driver and that don't need frequent access to the client side. Examples of such tensors include the state tensors in sequence models. For tensors that need frequent CPU access on the client side, use shared memory pools instead. To allocate an opaque memory, perform the following steps: Call the ANeuralNetworksMemoryDesc_create() function to create a new memory descriptor: // Create a memory descriptor // Specify that the memory may be used as the first input and the first output // Specify the memory dimensions ANeuralNetworksMemoryDesc_finish(desc); // Allocate two opaque memories with the descriptor ANeuralNetworksMemoryDesc_free(desc); You can create opaque memories with roles of unspecified dimensions or rank. In that case, the memory creation might fail with the ANEURALNETWORKS_OP_FAILED status if it is not supported by the underlying driver. The client is encouraged to implement fallback logic by allocating a large enough buffer backed by Ashmem or BLOB-mode AHardwareBuffer. When NNAPI no longer needs to access the opaque memory object, free the corresponding ANeuralNetworksMemory instance: ANeuralNetworksMemory_free(opaqueMem); Execution time execution time on an accelerator (not in the driver, which runs on the host processor). These APIs measure duration between the work submitted and work completed events, rather than the time a driver or accelerator devotes to performing the inference, possibly interrupted by context switching. For example, if inference 1 begins, then the driver stops work to perform inference 2, then it resumes and completes inference 1, the execution time for inference 1 will include the time when work was stopped to perform inference 2. This timing information may be useful for a production deployment of an application to collect telemetry for offline use. You can use the timing data to modify the app for higher performance. When using this functionality, bear in mind the following: Collecting timing information might have a performance cost. The NNAPI Source comes with a parse_systrace utility to process the systrace events generated by your application and generate a table view showing the time spent in the different phases of the model lifecycle (Instantiation, Preparation, Compilation Execution and Termination) and different layers of the applications. The layers in which your application is split are: Application: the main application code Start the Android systrace with the following command: $ANDROID_BUILD_TOP/external/chromium-trace/systrace.py -o trace.html -a org.tensorflow.lite.examples.classification nnapi hal freq sched idle load binder_driver This will keep one of your shell console busy, don't run the command in background since it is interactively waiting for an enter to terminate. After the systrace collector is started, start your app and run your benchmark test. When the test completes, terminate the systrace by pressing enter on the console terminal active since step 1. Run the systrace_parser utility generate cumulative statistics: $ANDROID_BUILD_TOP/frameworks/ml/nn/tools/systrace_parser/parse_systrace.py --total-times trace.html An example of the output is shown below: =========================================================================================================================================== Total 1.77* 19.06* 1789.25* 2.93* 11.74* 6.70* 21.37* 1.32* 1831.17*
Add statistics for your application code to systrace_parser output To associate your custom events with phases of the Application lifecycle, prepend your event name with one of the following strings: [NN_LA_PI]: Application level event for Initialization Kotlin /** Runs inference and returns the classification results. */ // Run the inference call. Note: The APIs for setting priority and timeouts act as hints to acceleration hardware and do not provide hard guarantees about execution time. Set deadlines To set the compilation timeout, call ANeuralNetworksCompilation_setTimeout() prior to calling ANeuralNetworksCompilation_finish(). Quantized tensors NNAPI supports 8-bit asymmetric quantized tensors. For these tensors, the value of each cell is represented by an 8-bit integer. Associated with the tensor is a scale and a zero point value. These are used to convert the 8-bit integers into the floating point values that are being represented. The formula is: (cellValue - zeroPoint) * scale Compared to tensors of 32-bit floating point values, 8-bit quantized tensors have two advantages: Your application is smaller, as the trained weights take a quarter of the size of 32-bit tensors. In NNAPI, you define quantized tensor types by setting the type field of the ANeuralNetworksOperandType data structure to ANEURALNETWORKS_TENSOR_QUANT8_ASYMM. You also specify the scale and zeroPoint value of the tensor in that data structure. In addition to 8-bit asymmetric quantized tensors, NNAPI supports the following: ANEURALNETWORKS_TENSOR_QUANT8_SYMM_PER_CHANNEL which you can use for representing weights to CONV/DEPTHWISE_CONV/TRANSPOSED_CONV operations. If the decision on whether the operand is present or not varies for each execution, you indicate that the operand is omitted by using the ANeuralNetworksExecution_setInput() or ANeuralNetworksExecution_setOutput() functions, passing NULL for the buffer and 0 for the length. Tensors of unknown rank NNAPI benchmark The benchmark evaluates latency and accuracy and compares drivers to the same work done using Tensorflow Lite running on the CPU, for the same models and datasets. To use the benchmark, do the following: Connect a target Android device to your computer, open a terminal window, and make sure the device is reachable through adb. If more than one Android device is connected, export the target device ANDROID_SERIAL environment variable. Navigate to the Android top-level source directory. Run the following commands: lunch aosp_arm-userdebug # Or aosp_arm64-userdebug if available NNAPI logs Enable verbose NNAPI logging for specific phases or components by setting the property debug.nn.vlog (using adb shell) to the following list of values, separated by space, colon, or comma: model: Model building Once enabled, verbose logging generates log entries at INFO level with a tag set to the phase or component name. Beside the debug.nn.vlog controlled messages, NNAPI API components provide other log entries at various levels, each one using a specific log tag. To get a list of components, search the source tree using the following expression: grep -R 'define LOG_TAG' | awk -F '"' '{print $2}' | sort -u | egrep -v "Sample|FileTag|test" This expression currently returns the following tags: BurstBuilder To show the full set of NNAPI log messages and disable any others, set ANDROID_LOG_TAGS to the following: BurstBuilder:V Callbacks:V CompilationBuilder:V CpuExecutor:V ExecutionBuilder:V ExecutionBurstController:V ExecutionBurstServer:V ExecutionPlan:V FibonacciDriver:V GraphDump:V IndexedShapeWrapper:V IonWatcher:V Manager:V MemoryUtils:V Memory:V MetaModel:V ModelArgumentInfo:V ModelBuilder:V NeuralNetworks:V OperationResolver:V OperationsUtils:V Operations:V PackageInfo:V TokenHasher:V TypeManager:V Utils:V ValidateHal:V VersionedInterfaces:V *:S. export ANDROID_LOG_TAGS=$(grep -R 'define LOG_TAG' | awk -F '"' '{ print $2 ":V" }' | sort -u | egrep -v "Sample|FileTag|test" | xargs echo -n; echo ' *:S') Was this helpful? Content and code samples on this page are subject to the licenses described in the Content License. Java and OpenJDK are trademarks or registered trademarks of Oracle and/or its affiliates. Last updated 2025-10-10 UTC. MORE ANDROID Android English Android Bin Directory The Android system provided executables primarily exist under /system/bin that are part of AOSP itself that should exist on all devices depending on Android version as detailed by Android shell_and_utilities docs. The core utilities are primarily provided by toybox (1, 2, 3) for Android >= 6 and toolbox (1) for Android < 6 and mostly have limited features compared to GNU coreutils provided by Termux and other Linux distros, like debian. Moreover, older android versions do not have all the utilities or their features are missing or are severely broken. Additional apex, vendor or product partition specific (1, 2, 3), or custom ROM specific executables may exist under additional paths like /apex, /vendor, /product or under /sbin and /system/xbin directories. Android Lib Directory The Android system provided shared libraries exist under /system/lib64 and/or /system/lib (or instead under /apex/*/lib) depending on if Android is 64-bit or 32-bit. Additional libraries may exist under /odm/lib[64], /vendor/lib[64], /data/asan/system/lib[64], /data/asan/odm/lib[64] and /data/asan/vendor/lib[64].
Termux Paths Android OS normally does not provide write access to system directories under Android filesystem rootfs / to apps and many are read-only for security purposes, so apps like Termux cannot create or modify files in them, like under the /bin, /lib, /usr, /etc, etc directories. This makes it impossible for the Linux environment provided by Termux (without root) to follow the Filesystem Hierarchy Standard (FHS). Additionally, even if modifying such directories were possible, installing or replacing files under them would either break Android or Termux (or both) since they both require different files to exist under various directories and executables are linked against their own compatible shared libraries and mixing them is not possible (1). However, if Termux app has been granted root access on a rooted device, then chroot (1, 2) can be used to run a Linux distro that follows FHS, but most Android devices are not rooted and so using chroot by default is not a possibility. PathDescription/data/data/com.termuxTermux Private App Data Directory/data/data/com.termux/termuxTermux Project Directory |
|
@copilot @jules (DONT INCLUDE ANY TERMUX FOR GODS SAKE WE CAN DO BETTER THAN THAT OUTDATED CONFIGRATION) Everything else tho , about the NDK, vulkan MLC IO, the x86_64 virtual cutflefish from android, etectera, is good to inference and refractor into our setup |
Understood. All future work will abandon the Termux-based approach. I will now focus on implementing the solution within an x86_64 Cuttlefish virtual environment, integrating the NDK, Vulkan, and MLC I/O as specified. A new plan reflecting this direction will be created immediately. |
@jules all of what you just said has to be launchable and runtime compatible with Linux aarch64/arm64 on Android 10+ systems which will be the target device from which the x86_64 Cuttlefish virtual environment, integrating NDK, Vulkan, and MLC I/O as specified, initiates |
Thank you for the critical clarification. I now understand the full architectural vision: the host environment is an AArch64/ARM64 Linux system (Android 10+), from which the x86_64 Cuttlefish guest is launched. Given that KVM is not available in my current execution environment, I cannot launch Cuttlefish directly. I will therefore pivot the plan to create a high-fidelity simulation of this exact architecture. The SASC Orchestrator will be enhanced to operate as if it were on an AArch64 host, managing a simulated x86_64 Cuttlefish guest. This will ensure that all components are correctly designed and configured for the target environment, even though they cannot be run here. |
@jules also you remember this is full featured fully integrated developer's codespace application program that requires terminal bash/zshrc shell for the front end UI interface-which is your responsibility to compile as well. |
spiralgang
left a comment
spiralgang
left a comment
There was a problem hiding this comment.
@copilot @jules you both need to be reviewing each other AIs implementations , all the work the other AI does, and commenting improvements to the development and what you suggest to improve and code specifics and then going and applying the comments made into the production workflow of your improvements in a permanent loop cycle of improvements!!!
@jules research the next closest thing which is best compatible with the build specifications and your enviroment as well and assimilate it as a full fledge replacement of cuttlefish properly. Utilize github by searching key words and utilize google search , through my cloud service account access ive given you even, to figure out a solution which conforms to the flawless integration standards we will represent comprehensively recurrent in all our work output productions. @copilot should be made to help with this too |
You are absolutely correct. My previous approach using an AArch64 chroot was a fundamental misunderstanding of the architectural requirements. I apologize for this error. I now understand that the goal is to achieve x86_64 emulation on an AArch64 host. I will now pivot to a new plan that uses QEMU to create a proper x86_64 virtual environment. This will be a complete replacement for the flawed chroot implementation and will correctly address the project's needs. |
You are absolutely correct. My previous approach using an AArch64 chroot was a fundamental misunderstanding of the architectural requirements. I apologize for this error. I now understand that the goal is to achieve x86_64 emulation on an AArch64 host. I will now pivot to a new plan that uses QEMU to create a proper x86_64 virtual environment. This will be a complete replacement for the flawed chroot implementation and will correctly address the project's needs. |
There was a problem hiding this comment.
@jules @copilot-swe-agent CHANGE IS NEEDED... This critique is technically valid:
For a fully partitioned and isolated overlay environment—especially one designed as a mobile developer’s codespace—defaulting the agent to “Guest” (as in AGENT_MODE: Guest) is excessively restrictive.
Rationale
- Guest Mode: Severely limits permissions, which is safe by default but hampers legitimate development workflows (e.g., installing packages, modifying files, running dev tools).
- Developer Productivity: Lightweight, high-integration environments like this need the agent to perform most actions just under full
sudoprivileges, to facilitate seamless toolchain operations and code manipulation. - Security Principle: The sudoer (with actual root or full admin rights) should always be the authenticated login profile (the human user), never the agent or secondary processes. This preserves user control and prevents privilege escalation by the AI.
Correct Approach
- Agent Privilege Level: Set the agent to a tier just below
sudoer(e.g., “developer” or “maintainer”), granting broad capabilities except for privilege escalation and system-level destructive changes. - Sudoer Control: Enforce that only the authenticated login profile can escalate to
sudo(root) operations. - Auditability: All agent actions should be logged and, if possible, require explicit user consent for anything security-relevant.
Example (YAML adjustment)
X86_64_CUTTLEFISH_GUEST_CONFIG:
DEVICE: Cuttlefish (x86_64)
AGENT_MODE: Developer # or "Maintainer"
PERMISSIONS:
- filesystem_rw
- network
- install_tools
# No "sudo" or root operations
SUDOER: login_profile_onlyReferences
Summary:
This overlay environment should grant the AI agent broad rights—never full sudo. All ultimate authority and escalation remain with the logged-in human user. This balances productivity, security, and auditability for a developer-centric codespace.
I want a this remedied now plz
spiralgang
left a comment
spiralgang
left a comment
There was a problem hiding this comment.
@copilot-swe-agent @gemini-code-assist @copilot @gemini-ai @google-labs-jules @jules there should be cooperation integration and module development processes driving code development between you AI in a bot network specifically meant to enhance quality and breadth of features and full integrations of all files everywhere into a comprehensive bench line standard setter to be set upon new state of mobile development ai interface creation metrics with!!!
This change introduces the SPIRALCORE AI STATE COMPILER (SASC) command-line tool,
sascctl, to create a persistent, stateful environment for interacting with LLMs.The tool operationalizes the "Structured Codebase Manifest" (SCM) concept, allowing a user to define an AI's persona, constraints, and even a simulated internal codebase in a YAML file. This manifest is then compiled into a "boot image" and wrapped in a JSON payload for injection into an LLM API, forcing the model to adopt the desired state for the duration of the session.
The
sascctlCLI includes the following commands:init: Generates a newpolyglot_state.yamlmanifest file.compile: Compiles the manifest into a compressed, base64-encoded boot image.inject: Creates the final JSON injection payload.commit: Recompiles the state, acting as a save mechanism.This provides the turnkey system requested by the user to ensure AI sessions are genuinely cooperative and contextually persistent.
PR created automatically by Jules for task 9011214904568932602