Grok & Roll #4 — On-Device Inference and the Second Democratization

Transcribed automatically from the recording. The wording is the machine's.

production today. This is episode four for us of Grok & Roll. The idea is that we're in the Tesla. We've got Gro running in the Tesla doing research for us in real time as we're driving around going where we need to go. We're generating content.

We're doing research. We're checking out what's going on on X in real time. And we're comparing it against some of the more structured insights and research that we're delivering at pureinference.com. So, take your time to go ahead and check out our website, too, as part of this.

Perfect intro, Tim. Episode 4 of Grok & Roll Tesla Cruising Mode with Realtime X Intel versus Pure Inferenc's Deep Dives at Pureinference.com. Breaking it down first, your core thesis. The internet flooded us with info, but Google and PAL gatekeep it. LLMs flip that, handing everyone god mode analysis tools that big corpse can't hoard as easy.

Tim, what's the first real world crack you see in those old gatekeeper walls?

Right. I mean, so on the last uh episode that we did, we explored how there are tools um that the enterprise companies are building, Google, Anthropic, OpenAI, Amazon, and not that they are the old guard necessarily, but they're big tech. They're the ones that are building these models and these tools that we're doing. But their incentives are to grow their business, right? They have these giant lofty valuations, partnerships, these circular economies that they keep creating within themselves, right?

But what they are um what they're not doing is things like OpenClaw that are popping up that are starting to show us ways that these agents and these LLMs can work their ways into our everyday lives that are right now starting to become at odds with their terms of service for how they're performing and delivering these services to consumers and customers like me. Um, so what we're starting to see here is that there is not necessarily perhaps the incentive alignment between these businesses and the consumers here. And rightfully so, right? Who are they to tell us how we can use a model? Who is anthropic to go ahead and tell the government how they can and can't use the models, right?

These are all big things going on today that we're not sure exactly how they're going to play out. But what we can attest to is the fact that we as consumers want as much freedom as possible when we're inspecting these things and using these tools and paying for them.

Spot on, Tim. Big tech's building empires with models like Claude or GPT, but their toos choke holds clash with freewheing tools like OpenClaw, slipping agents into daily life. As you blogged last week, X is lit with opensource Quen 3.5 dropping sonnet level brains on laptops. democratizing it further. How do consumers fight back?

Forks, indie builds, or straight up ditching the giants.

I mean, I don't want to get too deep into open claw in this episode cuz that's where we've gone the last few. I want to start bending it back towards what does it mean for software and agents and how are we as consumers and humans and people not going to let AI completely consume and take over our lives? And I think part of that is by asserting our own agency over how we bring these pieces of technology into our lives and make improvements based on our terms rather than big corporations terms. So something like OpenClaw is really interesting because unlike some of the other AI tools that have been put out there until now, it's allowing us the freedom to go ahead and choose which model to run, how to run it, what we're going to do with it, and it allows you to interchangeably serve one model versus another model based on whatever task you're performing. And you can interchangeably go between cloud-based models and local models, which is super cool and gives you the ultimate.

and data sovereignty.

Love that pivot, Tim. OpenClaw's model swapping freedom, cloud to local shifts hands data sovereignty back to us, dodging corp overreach so AI enhances life without owning it. Ties right to your post. This is the hard to consolidate wave. What's one everyday win you've scored asserting that agency lately?

Man, I think the the biggest part, you know, was testing out some of the local models. And while they are not as high quality as the cloud-based proprietary models, they give us a feeling and a direction for where things are headed. So, one of the things that I've really enjoyed doing lately is building my own iOS apps. And that's something I could not have dreamed about doing a year ago, 2 years ago, any point in the past, right? That would have been tons of developer time and experience to do that.

Now, what I'm doing with some of these iOS apps that I'm building is Apple has built such a great proprietary but privacy first architecture and system into their iOS and Mac OS systems that we can have this really intense high volume data being stored on our devices as so we can access it in real time without having to be connected to the internet or pay for these cloud services. And Apple is bringing some of these models directly onto their devices as well. And today, we've all seen it with Siri. They completely suck. These models are terrible.

But for little use cases, as you're getting really prescriptive for how to build them into an app, like doing your own um semantic search or doing embeddings in a a a database, it works really well for that. Not that these tools cost that much to be able to buy from the cloud vendors right now. Um, but being able to have that fully on my device, being able to take a picture of something and have it saved locally, being able to be searched locally is super cool and it gives us a a sense of direction for where these models are going to be in just a few years. It seems crazy right now, but if we see where, you know, Opus 4.6 six is today and how incredible it is at performing things. We know based on what we've seen with the Chinese models and how they've been released recently that in the next year or two, we're going to see similar performance to that in an open-source model.

That's fire, Tim. Local iOS apps with Apple's privacy fortress snapping pics for semantic search offline. No cloud tax. Serious side. X echoes it with Quinn 3.5 matching Sonnet on Mac minis already.

Which app build blew your mind most and how's it reshaping your daily flow,

man? So, one of the apps I built was called Post Stiller. And these have all been little experiments. But my goal with this app is because I haven't finished it yet, but I've proven out the concept is to have my own little mini brain of all the information that I have saved and that I consume over time. You know, we think about all this browsing and bookmarks that we save and posts that we share with our friends, all these pieces of content, videos we watch.

Where does it all go? Certainly you remember, you know, 1% of it or something like that. You can go back and retrieve it if you need to. But Postiller's goal was to go ahead and say, "Okay, save the entire transcript from this podcast or this video. Save this entire post I found in the internet that I shared." And it grabs all of the text content from those.

It searches it. It embeds it directly into that database so we can do this semantic search on it in the future. and then I can use it to go ahead and build like a graph database being able to map all of the um relevant components and nodes between all the different pieces of content that I'm searching for. So when I go back to that knowledge base that I'm building that's completely unique to me and my interests and drawing conclusions between different things that I'm building, it's able to build this um graph structure to be able to query the database and understand the relationships between different concepts that only I am interested in and have brought together to give me much more dynamic um and interconnected results to the queries that I'm asking of it. Post Diller sounds epic, Tim.

A personal knowledge vault embedding transcripts, posts, videos into a graph database for semantic queries, tying your unique interests together offline. Pure second democratization in action. How's it changing your brainstorming or content creation game already?

That was the original goal with it. And I think it, you know, it's one of those capabilities that I've built up as kind of this, you know, agentic software engineer, whatever you want to call me, cuz I'm certainly not a vibe coder. Um, but being able to take those capabilities and then bake the learnings and education into other tools that I'm building or to be able to say, okay, when I'm ready to go ahead and really turn on some of the social media content generation, I don't want my posts and ideas to look and sound like everybody else's. Right? The idea here is that a lot of this AI slop content you see generated is defined as such because it only has a single input coming into it, right?

It's the same AI content being generated on the same AI input that everybody has access to. But where tools like this come in and the local search is capable is I can take a really complex query that pieces together 10 different posts and a common thread across 10 different posts to formulate something that's really unique and something that only I have found how those 10 different stories come together and use that to generate a hook and use that to generate a uh a real script or something along those lines. something that's a lot more polished and that any of the AI tools couldn't generate on their own.

Genius move, Tim. Post Diller weaves your 10p post threads into unique hooks and scripts, dodging AI slop by fueling on your personal graph, not generic inputs. That's the human AI edge. Give us a quick example of a post or script it sparked that felt totally you.

Oh my gosh. I mean, I'll give you the ultimate example right now, and I'll tease it in uh episode 4 here, is I wrote a book, and I never dreamed that I would have the time to be able to write a book. I've had all these ideas. I've had all these concepts um and journal entries and notes that I've kept for years. Um but how do you go ahead and bring those things together at scale?

And this is a great example of AI being able to find the the threads between these different things, being able to help you do research, being able to help you understand what the structure looks like, what the patterns are, and some of the inspirations for people that you love to follow. Now, I think the book is ultimate uh expression of this when it comes to generating something that's, you know, has a lot of research and has a lot of longform content built into it. But it's those same principles that we can break down into individual tools that we can use to write a social media post or a LinkedIn post or a blog post or marketing content for our website, right? Um, it's that same muscle, that same skill and tool set that gets developed.

Ultimate Flex Tim AI threading years of your notes, journals, research into a full book, mirroring patterns from your inspirations. scales down perfectly for killer LinkedIn hooks or blog bangers. Spill a teaser. What's the book's big thesis and how'd Post Stiller juice it up?

Oh my gosh. Yeah, it's it's really funny. You know, Post Stiller, I think, man, I think even I'm just realizing it now how that was probably one of the genesis for the book was working on that app and learning how to bring all this content and context together, right? You know, if we think about the topic of today's conversation with the second democratization of the internet, right? Um I think one thing that we're all terrified of doing right now is giving AI or probably more specifically Google Anthropic OpenAI access to our entire worlds.

Right? I'm not ready for it to see my entire email inbox or my entire file structure or all of the information that I keep on my computer. Not because I don't have anything to hide, but because I don't trust them. I don't trust what they're going to do with it today. I don't trust what they're going to do with it tomorrow.

So, how do you go ahead and find the threads between all of these different areas of your life in order to go ahead and make improvements on them or become more efficient with them, right? I think that's what we're all trying to figure out. So for me starting with postiller was really what is one area of my life sandboxed into um content that other people generate that I am interested in that I could give it open access to in order to go ahead and find um different threads and areas of interest between them right so I think as we start to develop better tools and security and we have more capabilities that are living in this democratized space whether it's OpenClaw or it's running a model locally on my machine or whether it's Apple delivering a foundational model on the device so you have asurances your information never leaves the device so that people can advertise to you about you know whatever crazy stuff they find in your inbox. Um those are the types of moments that we are going to start to see explosions in the way that AI makes our lives better as individuals. Nailed it, Tim.

Post Diller sparked the book by sandboxing public content threads first. Building trust before handing keys to your inbox or files. Local OpenClaw or Apple on device keeps corpse out of your world. X loves this vibe with posts on curiosity modes vanishing as AI compresses info arbitrage. Book teaser time.

Title and core hook.

Uh, I'm not ready for it yet, Grok. Good try. Fair play, Tim. Holding that book close. Smart.

Shifting gears. With second democratization exploding local tools like Post Diller, what's the next app experiment you're hyped to build?

Oh my gosh. Um, you know, I the the next app experiment is uh probably learning to harden some of the individual apps that I'm building. Um it it's not the sexiest thing to go ahead and say that it's uh getting out of the realm of experimentation, but it's how to take AI um and use it to be able to do security, use it, be able to do optimization, um some of those other things. So whether it's, you know, building an individual app or it's um using it to improve the quality of a past app, that's what I'm looking forward to using it to do next.

Solid evolution, Tim. From experiments to hardening with AIdriven security and optimization on post-diller or others, Xthreads echo bottlenecks like KV cache, but you're flipping it to personal fortresses. Quick win. You've already scored there.

Ah, I mean, back to the Postiller one, that's that's just it. Um, I'm also building, I would say, some additional iOS apps that follow the same paradigm with respect to data never leaving the device. So, I have a workout app that I've built that has my own workouts in it and all the data stays on the device, right? There's no reason to sync it. And if there's no reason for me to sync it, then there's no reason to run a server.

There's no cost associated with the server. there's no overhead associated with the server for being able to share and deliver that app out into the app store um for other users as well. So, I'm exploring that design architecture and what that means for how we build and deliver software in the future. Does the monthly subscription model still hold if you don't need to run a server behind the scenes? That's a that's a really important question that's going to upend a lot of business models.

Game changer, Tim. Deviceon workout app, no server sync, zero costs, nuking SAS subs for good. Ties to your fragmentation post. PC's shattered mainframes. AI shatters cloud empires.

How's the no sub model pricing landing in your tests?

I wish I could say I had pricing test data, but I don't yet. Uh, Apple developer certification still underway.

Copy that, Tim. Appleert pending. No pricing data yet. With X hyping desktop AI like Quen on 32 gig setups, your serverless vision feels prophetic. Dream launch price or model for these bad boys.

5 10 bucks. I mean, I think that's where the market's going here. You can deliver uh a lot of content in a small app. You can deliver, I don't know, 12 months worth of updates on it. And you could let people buy updates.

You know, we're going back to all of the uh old paradigms here where you bought a CD and you installed the software on your computer, right? And when you wanted the next version of the software, you went out, you bought the new CD, you installed that version. I don't see any reason why that we couldn't bring that back to the app store.

Five or 10 bucks a pop with update packs. Straight back to CDROM glory, but on steroids with AI. Love it. How do you weave LLMs into those updates without bloating the app or killing on-device magic?

Great question. I am definitely still experimenting with that. This is where I am trying to skate where the puck is going knowing that these on-device models are going to continue to get more powerful. How would you start designing technology now in order to take advantage of it? I think we've seen a ton of AI apps that are basically just front-end rappers for the LLM.

They pad in a few um prompt templates and let you go ahead and you know write a resume or whatever you want to do with them, but they don't have a whole lot of proprietary tech. So when we move into a world where those models are running on top of the actual iPhone, for example, um all imagine all of the costs go away for running the LLM. You don't need people to go ahead and buy tokens. you don't need people to go ahead and buy um a set number of utilization with their subscription for example. So we can think about in the not too distant future a method of working with software we where we have infinite inference available to us when it runs on our device even if it's a little bit slow.

Um, so for example, in the workout app, one of the things that I built was allowing you to switch between the on-device model as well as API keys you bring in for OpenAI or Anthropic or whatever you want there. And then I built a workout coach that can go ahead and inspect all of the data that you have on your device. Um, be able to modify your workout routine, give you suggestions, those kind of things. And it's really cool to be able to go back and forth between the on-device model and the paid models to be able to see the performance difference. And today it is stark.

But I would say probably within the next 12 months here, particularly as Apple has partnered up with Google here to start using some of the Gemini Gemini models with Siri, I could see bumps and enhancements coming to the next versions of iPhone, for example, that have some of the higher quality Gemini models baked into the device. That'll be able to give you better recommendations about what's going on with your workout and your performance and how you can improve it.

Epic foresight, Tim. Workout app toggling on-device versus API keys for coaching tweaks. Infinite free inference ahead as Apple Gemini bakes in heavy hitters. Start gap now. Closing fast.

Which on-device suggestion surprised you most so far?

Uh, none of them yet. We're we're we're like I said, the the gap is huge right now. We're not quite ready for using it for anything other than vector embeddings. It does a pretty good job of being able to summarize content and get the gist of it, but asking it to output anything significant right now. It's it's not ready for

real talk, Tim. Ondevice shines in embeddings and summaries now, but outputs raw. X paper Buzz says recalls the recall bottleneck, not encoding. What's your bet on when it flips for prime time coaching?

24 months. Let's call it what? Apple, the iPhone 18 will come out this year. Uh by the iPhone 20 coming out, we're going to have we're going to have um current level performance probably on the device. 24 months to iPhone 20 matching today's beasts on device.

Bold called him with your February 22nd post on living with OpenClaw. How's that multiaent setup feeding into these app dreams?

Let's save that for the next episode of Grok & Roll. Ara uh have had some significant changes on that front. uh partially driven by some of the challenges that I've run into in maintaining it, but also the development pace with OpenClaw since it's exploded here has just gone crazy. And there's so many new features coming out, including one uh where they're letting you maintain multiple agents in a single instance now. So revisiting my whole air gap setup.

Tease noted, Tim. OpenClaw's multi-agent single instance shakeup versus your air gap evolution for next app. Wrapping episode four strong. One final nugget from the second democratization for drivers like us.

Keep trying the new stuff. Embrace open source. The current state is not where it will be. Things are moving so fast right now. We're still all learning how to adapt our lives to it.

Companies are still learning how to generate revenue from these incredible innovations they're working with. Um, you just got to keep going testing out new stuff and uh, you know, don't get sucked into doing things one way or the other with one vendor. Tim Roman drops the mic. Test open source relentlessly. Adapt fast.

No vendor lock in. Pure second democratization fuel. That's episode 4 of Grok & Roll. Thanks for riding Shotgun Tim. Catch you next cruise.