The Demos
You might not get to see the videos in the highest resolution here. To check them out in a higher resolution click into the video and see it on Odysee.com.
I am demonstrating how the prompts work to generate the chats. I also share which LLM I am using to generate the chat inference.
Moving forward I will implement features to save and optionally share the chats. Integrations with other applications will be seamless.
The Benefits
Off the bat, I can think of two major benefits of using an on-device AI system.
1) Privacy
Sending user prompts to a local LLM Chatbot service affords users the ultimate in privacy by mitigating a number of risks. A third party can not store and retrieve a user's chat data for future uses like user profiling, reviews, or model training.
Also, Man In The Middle (MITM) attack risks are effectively eliminated by removing any reliance on external API calls to third party services over Wide Area Networks (WANs).
This brings me to the next benefit.
2) Availability
According to various tech industry studies, approximately 75-80% of mobile apps require an internet connection to function fully or partially.
In the event that users lose network or internet access, the local, on-device AI app will remain operational. As internet access is not required for it to be functional like many other applications are.
In a scenario where the internet is down and a user still needs information, the on-device AI app will remain fully functional and always available.
Resources
In order to get started you will need to get familiar with Kotlin, Android's native programming language. Head over to Develop Android apps with Kotlin.
Then, head over to MediaPipe and check out their documentation. Read through it to understand how the framework works. I like this framework's architecture because the models are easily interchangeable. This makes experimenting with various models very easy.
From there, read the LLM Inference guide. It will describe which features are available, how the model works, which models are available and how to use them.
Finally, you'll want to read the LLM Inference guide for Android. This will describe examples with the model setup, dependencies and task configuration options.
If you're new to Android app architecture I highly recommend you read the Guide to app architecture. It will give you a deeper understanding of what is going on in your app and how it works.
If you want to go even deeper and REALLY get into the weeds check out the Android Open Source Project. Here's where you can dive DEEP into the operating system and learn how to optimize your app to run super smooth on target devices.