Build TikTok grade Farcaster and Zora feeds with AI powered search and recommendations

That’s because in today’s visually driven social-media world, basing search and recommendations only on text metadata is not enough. A lot of the information lies in the content itself—the action, the background context, what the characters are saying, the music that’s playing, and so on. We have made it a priority to upgrade our pipeline with in-depth image and video understanding for exactly this reason, so that app builders have more curation power over their users’ feeds.

What's New?

Over the past few days we have made a series of upgrades so that, as part of a given post’s metadata, you’ll be able to obtain a rich understanding of the images and videos contained in the post.

This leads to increased ability to curate content with embedded feeds, dramatically boosting content relevance and discovery precision.

Let’s see how this translates into post metadata. For example, a simple semantic search with the keyword “bench-press” will return a set of videos of people doing just that. Then, if we zoom in on a specific item from these categories, we can see the granularity with which our pipeline understands videos.

How it Works

At a high level:

1. We download the video and compress it.

2. We pass it to an optimized large multimodal LLM with a custom prompt.

3. The LLM analyzes, summarizes, and assigns a set of labels to the video.

4. The returned summary is used in downstream classification and ranking models to improve their accuracy.

As shown in the examples above, our labeling system incorporates anti-spam measures and a broad range of topic-related labels, providing transparency and control over recommendations. This design enables you to set the bar for search and recommendation quality in your app, ensuring high-quality, credible results that foster lasting user trust.

Finally, the entire system is already running at scale with enterprise customers and is designed for high-throughput environments. It updates in real-time based on each click, like, or user interaction that we capture, thanks to state-of-the-art online-learning algorithms, ensuring increasingly accurate and personalized results.

Benefits

• Reduced spam: filters out irrelevant content effectively.

• Enhanced discovery: rapidly find content using semantic search on video and images.

• Improved personalization: real-time feedback dynamically adjusts video recommendations, increasing user engagement and satisfaction.

• Scalable performance: built for high-traffic environments, our system ensures consistent quality even as your audience expands.

• Significant metric boost: moving beyond simple caption analysis to deep video comprehension leads to notably higher retention, engagement, and user happiness.

What’s next

The best way to experience these new capabilities is to get your hands dirty with one of our guides. You can start calling our short-form video template using the ❜embed SDK straight away and start building.

Lastly, this feature is fully compatible with our new publication types filters to create only Zora feeds. And if you want to see ❜embed’s video capabilities in action, check out Dash.