Jekyll2023-05-06T08:31:07+00:00https://www.kix.in/feed.xmlkixTalks & RamblingsAnant NarayananBuilding a Hacker News ChatGPT Plugin2023-05-05T00:00:00+00:002023-05-05T00:00:00+00:00https://www.kix.in/2023/05/05/hacker-news-chatgpt-plugin<p>I recently received access to develop and use ChatGPT plugins, and embarked on a project to build a Hacker News integration as a learning exercise. My goal was to enable retrieval of content from HN to answer questions and produce insights in conversations with ChatGPT.</p>
<p>You can experience the plugin in one of three ways:</p>
<ul>
<li>Check out the <a href="https://hn.kix.in">simple demo</a> which approximates parts of the plugin.</li>
<li>(or) add ‘hn.kix.in’ as a plugin — if you have ChatGPT plugin access.</li>
<li>(or) watch this short video:</li>
</ul>
<video controls="" src="https://user-images.githubusercontent.com/37190/236521903-da8eb5a6-3b8e-4125-a8c0-64b869d47f55.mp4"></video>
<p><button class="pure-button pure-button-accent" onclick="window.location.href='https://hn.kix.in'">
<ion-icon name="bulb"></ion-icon>
<b>Simple Demo</b>
</button>
<button class="pure-button pure-button-accent" onclick="window.location.href='https://github.com/anantn/hn-chatgpt-plugin'">
<ion-icon name="logo-github"></ion-icon>
<b>Source Code</b>
</button></p>
<p>In this blog post I’ll cover the process of building this plugin. If you are interested in learning about how ChatGPT plugins work, the Hacker News API and dataset, or building a semantic search index through use of embeddings — read on!</p>
<h2 id="what-are-chatgpt-plugins">What are ChatGPT plugins?</h2>
<p><a href="https://openai.com/blog/chatgpt-plugins">Plugins are a new feature</a> announced by OpenAI that allows ChatGPT to extend its functionality by calling external APIs. This unlocks key capabilities such as web browser and code execution, but also allows for bringing various data sources into the large language model. No more “knowledge cutoff” problems!</p>
<p>The <a href="https://platform.openai.com/docs/plugins/introduction">official documentation</a> goes into the process of building a plugin in much more detail, but at a high level, you can build a ChatGPT plugin by:</p>
<ul>
<li><strong>Describing an existing (or new) API</strong> to ChatGPT in plain english. The specific format is the “OpenAPI” (formerly Swagger) spec, but the most important fields in the spec are the description fields which ChatGPT will read to understand your API.</li>
<li><strong>Processing API calls</strong> when ChatGPT calls them. The system will decide when to invoke your API given your description and the user’s utterance, it generally does a pretty good job at this. Data returned by your API will then be processed by ChatGPT as part of the “prompt” in order to do whatever the user is asking - whether it is taking an action or answering a question.</li>
</ul>
<p>There <a href="https://openai.com/waitlist/plugins">is a waitlist</a> for both using and creating plugins, if you aren’t signed up already.</p>
<h2 id="tips-for-plugin-development">Tips for plugin development</h2>
<p>While in theory you can just “plug and play” an existing API by providing an OpenAPI specification for it, to get the most out of the integration, I’ve found that creating something more bespoke to the specific style in which ChatGPT invokes them is useful. A few things I’ve learned:</p>
<ul>
<li><strong>Fewer calls with more arguments are better than many calls with fewer arguments</strong>. My initial design for the hacker news API involved individual endpoints for stories, comments, polls, etc. Simplifying it to just <code class="language-plaintext highlighter-rouge">/items</code> and <code class="language-plaintext highlighter-rouge">/users</code> with many query parameters to further control the output worked much better.</li>
<li><strong>Learn what functionality to add by iteration</strong>. I found that ChatGPT would sometimes hallucinate parameters for your API that don’t exist. My initial API did not have a <code class="language-plaintext highlighter-rouge">sort_order</code> parameter, but I kept seeing ChatGPT add it for certain types of queries. That was a good hint for me to just implement it! You can (and should) <a href="https://platform.openai.com/docs/plugins/getting-started/running-a-plugin">run a plugin API on localhost</a> first which makes iteration fairly quick and easy.</li>
<li><strong>Be as terse as possible</strong>. This holds true for both your OpenAPI specification and the actual API responses. You do need to be descriptive but short and to-the-point descriptions actually stuck more than lengthy flowery language. I’ve noticed that if your actual API responses are too long, it increases chances of hallucinations or the model just ignoring your response. This is likely related to context window limits for the GPT models.
<ul>
<li>The official documentation states the limit for API respones is 100,000 characters - in practice you’ll want to be well below it.</li>
<li>Some plugin authors have found a trick by forgoing JSON as an output format altogether, plain text responses work just as well and saves quite a few characters!</li>
</ul>
</li>
<li><strong>Be tolerant of inputs, more than usual</strong>. ChatGPT is a very language driven model and is not as precise when it comes to numbers. Avoid use of things like UNIX timestamps in your APIs, it’s often better to receive standardized date formats like ISO8601, and even better to accept natural language.
<ul>
<li>Using parsers like <a href="https://github.com/scrapinghub/dateparser"><code class="language-plaintext highlighter-rouge">dateparser</code></a> in python for processing natural language dates and times can be helpful.</li>
<li>ChatGPT often inserts comments into its <code class="language-plaintext highlighter-rouge">POST</code> requests. If you handle JSON as payload, use a parser like <a href="https://json5.org/"><code class="language-plaintext highlighter-rouge">json5</code></a> to be tolerant of this.</li>
</ul>
</li>
<li><strong>Set reasonable defaults</strong>. I’ve fluctuated on the default <code class="language-plaintext highlighter-rouge">limit</code> value for the <code class="language-plaintext highlighter-rouge">/items</code> endpoint, from 1 to 10 and back to 5. I’ve found that 3 was the magic number that allowed the response to be as long as possible without throwing ChatGPT off the rails while still being useful enough to summarize any given topic.</li>
<li><strong>Use ChatGPT itself to help you</strong>! Not only can ChatGPT write code for the implementation of your API, it’s also very good at creating terse descriptions of APIs from lengthy documentation. That’s often a great starting point - I started my project by throwing the <a href="https://github.com/HackerNews/API">Hacker News Firebase API documentation</a> at it.</li>
</ul>
<p>With these guidelines in mind, here is a rough sketch of what we want a Hacker News API to look like:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">/items</span> <span class="c1"># find and retrieve stories, comments, polls, or jobs</span>
<span class="s">query</span> <span class="c1"># search for items matching this text</span>
<span class="s">type</span> <span class="c1"># story, comment, poll, job</span>
<span class="s">by</span> <span class="c1"># filter by author</span>
<span class="s">after_time</span> <span class="c1"># content submitted after this (natural language ok)</span>
<span class="s">before_time</span>
<span class="s">min_comment</span> <span class="c1"># minimum number of comments for a story</span>
<span class="s">max_comment</span>
<span class="s">min_score</span> <span class="c1"># minimum score for a story</span>
<span class="s">max_score</span>
<span class="s">sort_by</span> <span class="c1"># relevance, score, time, or number of comments</span>
<span class="s">sort_order</span> <span class="c1"># asc or desc</span>
<span class="s">limit</span> <span class="c1"># maximum number of items to return</span>
<span class="s">offset</span> <span class="c1"># offset into the results to page through</span>
<span class="s">/users</span> <span class="c1"># find and retrieve users</span>
<span class="s">...</span> <span class="c1"># similar API as above</span>
</code></pre></div></div>
<p>You can see the full API I <a href="https://hn.kix.in/docs">ended up with here</a>. The directions you give the plugin in the <code class="language-plaintext highlighter-rouge">ai-plugin.json</code> manifest file through the <code class="language-plaintext highlighter-rouge">description_for_model</code> are even more important than the individual <code class="language-plaintext highlighter-rouge">description</code> lines you put in your OpenAPI schema. This part will likely take a lot of tweaking for you to find something that works optimally. For the hacker news plugin, here is the prompt I ended up using:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Retrieve stories, comments, polls, and jobs from the Hacker News (HN) community in real-time. Follow these guidelines:
General rules:
1. You MAY provide natural language for dates, but ONLY after converting spelled-out numbers into their numerical equivalents. For instance, 'a couple of days' should become '2 days' and 'few weeks later' should become '3 weeks later'.
2. ALWAYS attempt to provide the hacker news URL (hn_url) and original URL (url) in your response.
3. ONLY incorporate API response data in your output.
4. Utilize the 'text' and 'top_comments' fields from API responses to answer questions, provide insights, and generate summaries.
Using find_items:
1. Search for user requested topics with find_items.
2. Remove 'Ask HN' prefix from user queries when providing them as the 'query' argument.
3. Use 'text' and 'top_comments' fields to answer questions or provide summaries.
4. Request a minimum of 3 stories for summarizing or searching a topic.
Using get_item:
1. Obtain more comments for any story using this endpoint.
2. Provide an ID obtained from find_items.
Using get_user and find_users:
1. Use get_user to access detailed information about a single user.
2. Employ find_users to search for users based on specific criteria.
</code></pre></div></div>
<p>Now let’s talk about the best way to implement this API!</p>
<h2 id="search-index-considerations">Search index considerations</h2>
<p>There are really two main pieces to our API. The first is to retrieve content matching a certain set of filters, which feels like a straightforward mapping to a SQLite database or even directly with the Hacker News Firebase API.</p>
<p>The second, more interesting part is implementing the <code class="language-plaintext highlighter-rouge">query</code> argument on the <code class="language-plaintext highlighter-rouge">/items</code> endpoint. Plugin users are likely to want to retrieve many kinds of content from hacker news using natural language.</p>
<p>Hacker News already has a <a href="https://github.com/HackerNews/API">Firebase API</a> to retrieve the raw data, but this by itself is insufficient, as you need a search index in order to properly rank and retrieve only a <em>subset</em> of documents for any given user query.</p>
<p>There are basically two options for building such a search index:</p>
<ol>
<li><strong>Traditional keyword search</strong>. This is the classic information retrieval technique refined over a couple of decades, and services like ElasticSearch and Algolia make it easy to create such indexes. Algolia already has a <a href="https://hn.algolia.com">great HN search index</a> that can “plug and play” with ChatGPT plugins for the most part.</li>
<li><strong>Semantic search</strong>. With all the attention on AI recently, a fairly old technique called “embeddings” has received renewed interest and enthusiasm. Embeddings are a way to generate an n-dimensional vector for any input content, such that content “similar” to each other will be near each other in this n-dimensional space.</li>
</ol>
<p>I first built a plugin with the <a href="https://github.com/anantn/hn-chatgpt-plugin/tree/main/algolia">Algolia search API</a>. It performed relatively well, especially when the questions were “keyword-y” in nature, like asking for more information about a specific a project or person. However, there was room for improvement on questions that were more generic in nature or long queries of a conversational style. There is no publicly available API or dataset for embeddings on the HN corpus, so time to roll up my sleeves and build one!</p>
<h2 id="downloading-the-dataset">Downloading the dataset</h2>
<p>⬇️ <a href="https://huggingface.co/datasets/anantn/hacker-news/tree/main">Download the SQLite DB from HuggingFace</a> 🤗</p>
<p>The first step was to download the Hacker News corpus onto my computer. As of April 2023, HN contained just under 36 million items (an item can be a story, comment, job, or poll) and just under 900k users. That’s small enough to download and process on a single computer but large enough to make it a non-trivial and interesting exercise!</p>
<p>I wrote implementation in <a href="https://github.com/anantn/hn-chatgpt-plugin/tree/main/hn-to-sqlite/node">node</a>, <a href="https://github.com/anantn/hn-chatgpt-plugin/tree/main/hn-to-sqlite/go">go</a>, and <a href="https://github.com/anantn/hn-chatgpt-plugin/tree/main/hn-to-sqlite/python">python</a> to see which one would perform best. Node turned out to be the most reliable because it uses the Firebase SDK while the go and python versions used the REST API. I ended up using the node version, taking the performance hit for better reliability. To make the download faster, I simply parallelized it over 32 AWS spot instances:</p>
<ul>
<li><a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/hn-to-sqlite/node/fetch.js">fetch.js</a> is the core download script.</li>
<li><a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/hn-to-sqlite/node/run.sh">run.sh</a> is a quick-and-dirty user-script to parallelize the download on AWS EC2. Note the hard-coded number of machines.</li>
<li><a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/hn-to-sqlite/node/fetch-users.js">fetch-users.js</a> is a script to fetch user data profiles, can be done on a single machine and is fairly quick.</li>
<li><a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/hn-to-sqlite/python/merge.py">merge.py</a> can be used to merge each partition into a single sqlite file.</li>
</ul>
<p>This cost around $4 and took an hour to run. The final DB is around 32GB on disk, but compresses down to 6.5GB. Check out <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/playground.ipynb">this python notebook</a> for quick and dirty ways to visualize data from this SQLite table.</p>
<aside>
<p>⚠️ Note: the database on hugging face includes indexes on various columns for efficient queries. I added these indexes after bulk download & inserts which is much more efficient. Incremental inserts are now slower due to these indexes, but the volume is low enough to not matter.</p>
</aside>
<h2 id="constructing-embeddings">Constructing embeddings</h2>
<p>Next step is to take all this content and generate embeddings from them. To do this, we have three main questions to answer: what embedder to use, how to structure the input content, and where will we store the embeddings for retrieval.</p>
<h3 id="embedder-options">Embedder options</h3>
<p>There are many ways to construct embeddings from all kinds of data (text, images, even video). Our focus is on text, so a good place to start is by looking at the <a href="https://huggingface.co/spaces/mteb/leaderboard">“Massive Text Embedding Benchmark” (MTEB)</a>. You can filter the leaderboard by various criteria to find the right embedder for your use-case.</p>
<p>Note that some embedding services run in the cloud behind an API call, such as OpenAI’s <a href="https://beta.openai.com/docs/guides/embeddings/types-of-embedding-models">ada-002</a> or <a href="https://docs.cohere.com/docs/embeddings">Cohere</a>. Most of them can be downloaded and run locally though, normally in python. <a href="https://python.langchain.com/en/latest/index.html">LangChain</a> is a good way to quickly experiment with different embedders with small datasets.</p>
<p>After a bunch of experimentation, I decided to pick <a href="https://github.com/HKUNLP/instructor-embedding"><code class="language-plaintext highlighter-rouge">instructor-large</code></a> as it gave me a good balance of quality and speed of generation, plus the ability to run locally and leverage <a href="https://twitter.com/anantn/status/1641672926687801344">my new NVIDIA GPU</a>.</p>
<p>In your selection of the embedder, also keep in mind that whatever you choose to embed your corpus will also be the one you will need to use at runtime when processing user queries!</p>
<h3 id="structuring-input-documents">Structuring input documents</h3>
<p>Most embedding models have a maximum token length they will accept as input, so we need to think about how to represent our data. The default for <code class="language-plaintext highlighter-rouge">instructor-large</code> is <code class="language-plaintext highlighter-rouge">512</code> tokens, but can be extended to around <code class="language-plaintext highlighter-rouge">1024</code> with only a slight dip in quality.</p>
<p>The naive approach would be to simply embed each item in our table (story or comment), giving us around ~35 million embeddings. But most of these items are much smaller than 512 tokens, not to mention, not every piece of content is worth embedding because it might be spam or of low relevance.</p>
<p>Since comments on a story are usually pretty related to the story itself, a smarter way would be to group all the comments for a story along with the story text and treat it as a single “document”. By adding an additional filter for stories that have at-least 20 upvotes and 3 comments to make each document meaningful (and weed out spam), we get:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sqlite</span><span class="o">></span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">items</span> <span class="k">where</span> <span class="n">score</span> <span class="o">>=</span> <span class="mi">20</span> <span class="k">and</span> <span class="n">descendants</span> <span class="o">>=</span> <span class="mi">3</span><span class="p">;</span>
<span class="mi">402007</span>
</code></pre></div></div>
<p>That’s a much more manageable ~400k documents. Keep in mind that since we are grouping comments together with the story, a document can get much longer than 1024 tokens. To solve this, we will chunk each document into “pages” of up to 1024 tokens each. It seems on average, every document is around 7.5 pages:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sqlite</span><span class="o">></span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">embeddings</span><span class="p">;</span>
<span class="mi">3050324</span>
</code></pre></div></div>
<h3 id="storing-embeddings-for-retrieval">Storing embeddings for retrieval</h3>
<p>Embeddings are just an array of floats. The length of this array is known as the <em>dimensionality</em> of our vector. instructor-large produces embeddings of <code class="language-plaintext highlighter-rouge">768</code> dimensions. A floating point number can be represented in 32 or 64 bits. Assuming we use 32-bit floats, one embedding would be just over <code class="language-plaintext highlighter-rouge">3kb</code> (768x4 bytes).</p>
<p>Note that the size of the output embedding is fixed for any input size. This is one of the reasons we tried to maximize the number of tokens per embedding when generating our input document. For 3 million embeddings that’s around 10GB of data. Nothing a sqlite table can’t handle so let’s just store it there.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sqlite> select * from sqlite_schema;
table|embeddings|embeddings|2|CREATE TABLE embeddings (
id INTEGER PRIMARY KEY AUTOINCREMENT,
story INTEGER,
part_index INTEGER,
embedding BLOB,
UNIQUE (story, part_index)
)
</code></pre></div></div>
<p>Generating 3 million embeddings took almost an entire day on my RTX 4090. I let this job run overnight. Each embedding itself takes a fraction of a second to generate, but is not parallelizable on my Gaming GPU as there is not enough VRAM to load multiple model instances.</p>
<h2 id="vector-search--indexing">Vector search & indexing</h2>
<p>Now that we have generated embeddings for the most interesting stories and comments, we can use it as the basis of a semantic search engine. This process boils down to:</p>
<ol>
<li>Generate an embedding for the query.
<ul>
<li><code class="language-plaintext highlighter-rouge">instructor-large</code> accepts an instruction argument while generating an embedding, note that we give <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/embeddings/embedder.py#L13">different instructions</a> for query embedding than we do for document embedding.</li>
</ul>
</li>
<li>Find <code class="language-plaintext highlighter-rouge">k</code> vectors nearest to the query embedding.</li>
<li>Rank these k vectors to obtain results with the highest relevance to your input query.
<ul>
<li>Limit this list to the top <code class="language-plaintext highlighter-rouge">n</code> results, retrieve supporting metadata for each item and return them.</li>
</ul>
</li>
</ol>
<p>Step 2 is the most interesting part of this flow. Generally, if you have less than a million embeddings, the naive approach of comparing your query embedding to <em>every</em> vector in your dataset is <a href="https://twitter.com/anantn/status/1647048626752090114">quite feasible</a>. When comparing two vectors you can quickly compute the distance between them. Sorting by distance in ascending order is an easy way to find the most relevant documents for a query. This approach is known as k-NN (k-nearest neighbors).</p>
<p>With more than a million embeddings the brute force approach breaks down and starts taking too long. We have to find some way to reduce the number of vectors we have to compare with. There are a few strategies to do this:</p>
<ul>
<li><strong>Compress your embedding</strong> down to fewer dimensions. There are a handful of smart ways to employ lossy compression while minimizing a drop in accuracy. In this approach you don’t reduce the number of embeddings to compare against per-se, but rather reduce the size of each vector so as to reduce the time taken for each comparison.
<ul>
<li>As an example, if you compress <code class="language-plaintext highlighter-rouge">768</code> dimensions down to <code class="language-plaintext highlighter-rouge">384</code>, you can now do 2 million vector comparisons by brute force in a reasonable amount of time. <strong>Quantization</strong> is one common way by which you can reduce the dimensionality of a vector. Google’s <a href="https://github.com/google-research/google-research/tree/master/scann">ScaNN library</a> is a popular choice.</li>
</ul>
</li>
<li><strong>Cluster your embeddings</strong>. You can pre-process your dataset into <code class="language-plaintext highlighter-rouge">n</code> clusters, compare the query embedding to the centroid of every cluster. Then you only have to compare the query embedding to vectors in cluster that was closest.
<ul>
<li>There are small variations of this where you can have a large number of smaller clusters and you compare the vector to everything from a few adjacent clusters. Facebook’s <a href="https://github.com/facebookresearch/faiss">FAISS library</a> has a few implementations of this general type of technique.</li>
</ul>
</li>
<li><strong>Small world graphs</strong>. This is another way to partition your dataset following the intuition that vectors based on real data will follow “small world” clustering rules similar to the real world (e.g. <a href="https://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon">Six Degrees of Kevin Bacon</a>).
<ul>
<li>In this technique we navigate the graph finding “small world” clusters. More complex implementations (like HNSW - hierarchical navigable small world) add other techniques to make this more robust. The FAISS library mentioned above also has an HNSW implementation.</li>
</ul>
</li>
<li><strong>Partitioning using trees</strong>. One technique to partition your vectors are to pick two random vectors and split by a plane equidistant between them. This is effectively a random split and can be repeated multiple times until the number of vectors in each leaf node is low enough. One might also construct a “forest” of binary trees with different random split paths taken.
<ul>
<li>In practice, this works very well when you have a small number of dimensions (less than 100). Spotify’s <a href="https://github.com/spotify/annoy">Annoy library</a> is a popular implementation of this technique.</li>
</ul>
</li>
</ul>
<p>These strategies are known as “Approximate Nearest Neighbors” or “ANN”. I <a href="https://towardsdatascience.com/comprehensive-guide-to-approximate-nearest-neighbors-algorithms-8b94f057d6b6">recommend this guide</a> if you want to dive deeper on any of these.</p>
<p>One of the primary benefits of a ChatGPT plugin is the ability to access real-time data, so for our use-case we need to consider the ability to update the embedding index with new data periodically. I settled on a simple <code class="language-plaintext highlighter-rouge">IndexIVFFlate</code> implementation using FAISS. This is a type of clustering based on assigning vectors to a <a href="https://en.wikipedia.org/wiki/Voronoi_diagram">voronoi cell</a>. The cells are determined once at boot based on the initial set of embeddings, new embeddings inserted are assigned to an existing cell.</p>
<p>All embeddings are loaded in-memory, for around 3M embeddings this takes around 16GB of RAM (there is some overhead due to clustering and metadata). FAISS has an option to use a disk-based index, but this was small enough to fit on my 32GB machine.</p>
<p>The full implementation of this is a very short <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/embeddings/search.py">80-line python program</a>!</p>
<h3 id="ranking">Ranking</h3>
<p>FAISS will return <code class="language-plaintext highlighter-rouge">k</code> embeddings nearest to your query ranked by distance. For our Hacker News plugin, story upvotes and time of submission are also pretty important factors. Relying only on distance would often surface stories with low scores or very old submissions at the very top which was undesirable.</p>
<p>Semantic search based on embeddings also has the drawback of not being great at exact keyword match, particularly when that word doesn’t occur often in the corpus. To make up for this slightly, I also introduced the notion of “topicality” where we boost stories whose title has words matching the query.</p>
<p>Once you normalize these four values on a 0-1 scale, you can pick weights to associate with each attribute. Through trial and error, I landed on something like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Compute topicality
</span><span class="n">query_words</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">word</span><span class="p">.</span><span class="n">lower</span><span class="p">()</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">query</span><span class="p">.</span><span class="n">split</span><span class="p">())</span>
<span class="n">title_words</span> <span class="o">=</span> <span class="p">[</span><span class="n">word</span><span class="p">.</span><span class="n">lower</span><span class="p">()</span> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">title</span><span class="p">.</span><span class="n">split</span><span class="p">()]</span>
<span class="n">topicality</span> <span class="o">=</span> <span class="n">calculate_topicality</span><span class="p">(</span><span class="n">query_words</span><span class="p">,</span> <span class="n">title_words</span><span class="p">)</span>
<span class="c1"># Weights for score, distance, story age, and topicality
</span><span class="n">w1</span><span class="p">,</span> <span class="n">w2</span><span class="p">,</span> <span class="n">w3</span><span class="p">,</span> <span class="n">w4</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span> <span class="mf">0.25</span><span class="p">,</span> <span class="mf">0.35</span><span class="p">,</span> <span class="mf">0.2</span>
<span class="n">score_rank</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">w1</span> <span class="o">*</span> <span class="n">normalized_scores</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="o">+</span> <span class="n">w2</span> <span class="o">*</span> <span class="n">normalized_distances</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="o">+</span> <span class="n">w3</span> <span class="o">*</span> <span class="n">normalized_ages</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="o">+</span> <span class="n">w4</span> <span class="o">*</span> <span class="n">topicality</span>
<span class="p">)</span>
</code></pre></div></div>
<p>You can see we give the most importance to the story age, followed by the vector distance, and finally account for story upvotes and topicality. The full ranker implementation <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/api-server/search.py#LL163C12">can be found here</a>.</p>
<h2 id="keeping-the-data--index-updated">Keeping the data & index updated</h2>
<p>Moving onto our next piece of the puzzle — keeping the data updated. Luckily for us, the Firebase API helps us keep things real-time, simply by subscribing to the <a href="https://github.com/HackerNews/API#changed-items-and-profiles">changes endpoint</a>. This endpoint is updated roughly every 15 to 30 seconds and typically has a dozen item and user profile changes.</p>
<p>Fetching these items and profiles on every update, then inserting them into the SQLite table was <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/embeddings/updater.py#L253">fairly straightforward</a>. What’s more complex is what we do with our embeddings index — simply adding to the items to the table isn’t enough since the API won’t be able to find it through a text search.</p>
<p>I refactored the code that did the initial embedding pass to also run on individual documents. Recall that generating an embedding is a single-threaded sequential process (because of my limited VRAM). Generating embeddings every time a story was updated had the chance to completely starve incoming queries from being embedded which would be bad.</p>
<p>To solve this problem, I employed two techniques:</p>
<ul>
<li>While the data updates are processed in real-time, we batch the embedding updates <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/embeddings/updater.py#L17">every 15 minutes</a>. This allows us to collect a bunch of changes to an active story (comments are added rapidly and upvoted) and process them together.</li>
<li>Implemented a <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/embeddings/embedder.py#L24">priority queue</a> in the embedder service such that we would always process embedding an incoming query over embedding an updated document.</li>
</ul>
<p>This gave us a good balance between keeping our data updates while not compromising on the machine’s ability to respond to incoming queries.</p>
<h2 id="api-server--qa">API server + Q&A</h2>
<p>All of this is brought together by a <a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/api-server/main.py#L54">FastAPI server</a> to implement the API spec we defined earlier. Doing this was pretty easy through use of SQAlchemy.</p>
<p>We run two independent processes, one for the data update and embedder service, and another for the FastAPI server. The update/embedder service has write locks on the SQLite databases while the FastAPI opens the db in readonly mode.</p>
<p>The first time our embedding server starts, we do a quick “catchup” on any missed stories or embedding updates to keep the database fresh even if the server was offline for any reason.</p>
<p>The final step is to take the text and comments from returned results in the <code class="language-plaintext highlighter-rouge">/items</code> API call — and optionally generate an answer using ChatGPT-3.5-turbo. This is pretty simple to do, we just take the text and prompt the model with something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Given the following hacker news discussions:
<story titles>
<comments>
Answer the question: {user-query}
</code></pre></div></div>
<p><a href="https://github.com/anantn/hn-chatgpt-plugin/blob/main/api-server/utils.py#L114">This functionality</a> powers what you see on the simple demo page and is an approximation of the plugin experience right in ChatGPT.</p>
<h2 id="closing-thoughts">Closing thoughts</h2>
<p>Hope this was a useful tutorial on building a non-trivial ChatGPT plugin and helped with your understanding of embeddings and semantic search. My advice for anyone dipping their toes in this space is to:</p>
<ul>
<li><strong>Focus on the first principles of what you are building.</strong> There is a lot of buzz around embeddings and AI, but having a conceptual understanding of these tools will help you navigate the landscape. You don’t need to know <em>how</em> these tools work as long you know <em>what</em> they do, and <em>why</em> you need them.</li>
<li><strong>Keep things simple and beware premature optimization!</strong> I’ve seen a few examples that are built for hyper-scale from day one, but it’s usually a better idea to start small and only add layers of complexity as you need them. The entirety of this particular project is around 2500 lines of python code, including boilerplate.</li>
<li><strong>Use ChatGPT liberally.</strong> You’d be surprised at how much this tool can you help you, right from writing API descriptions and specs, to full fledged server code, to helping debug issues when they occur. $20/mo for ChatGPT plus is an absolute bargain.</li>
</ul>
<p>Happy hacking!</p>anantI recently received access to develop and use ChatGPT plugins, and embarked on a project to build a Hacker News integration as a learning exercise. My goal was to enable retrieval of content from HN to answer questions and produce insights in conversations with ChatGPT.Fine-tuning with LoRA: create your own avatars & styles!2023-04-07T00:00:00+00:002023-04-07T00:00:00+00:00https://www.kix.in/2023/04/07/sd-lora-finetuning<p>Remember <a href="https://land.prisma-ai.com/magic-avatars/">Magic Avatars</a> in the Lensa app that were all the rage a few months ago? The custom AI generated avatars from just a few photos of your face were a huge hit!</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_reference.jpg"><img src="/images/2023/anant_reference.jpg" alt="Reference Portrait Image" /></a>
<em>Reference portrait image of me</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/lensa_avatar.jpg"><img src="/images/2023/lensa_avatar.jpg" alt="Example Lensa Magic Avatars" /></a>
<em>One of my Lensa “Magic Avatars”</em></p>
</div>
</div>
<p>The technology behind this product is the open source <a href="https://en.wikipedia.org/wiki/Stable_Diffusion">Stable Diffusion</a> image generation model. You can run this model on your own computer, on a cloud GPU instance, or even a <a href="https://colab.research.google.com/">free colab notebook</a> — to generate your own avatars. In this post I’ll walk you through the process of doing exactly that!</p>
<p>Before diving in, it would help to review the <a href="/2023/04/01/txt2img">AI Primer: Image Models 101</a> post if you are new to the world of image models in general. I also wrote a quick guide on getting Stable Diffusion <a href="/2023/04/05/quick-sd-install-guide">set up on your computer</a> — I’ll assume you’ve already done that.</p>
<h2 id="what-is-fine-tuning">What is fine-tuning?</h2>
<p>I previously argued that one of the main advantages Stable Diffusion has over a proprietary model like Midjourney is the ability to customize it. While Midjourney produces stunning imagery with very little effort, it is going to have a difficult time producing photos of your likeness or in a niche style. This is getting better with their image-to-image features, but for any work requiring a high degree of flexibility and control, it is hard to beat Stable Diffusion’s capabilities.</p>
<p><a href="/images/2023/lora_collage.jpg"><img src="/images/2023/lora_collage.jpg" alt="Example of AI-generated characters and style" /></a>
<em>Examples of AI-generated images with Stable Diffusion, after fine-tuning</em></p>
<p>This level of customization is unlocked by the concept of <em>fine-tuning</em>. At a high level, fine-tuning is the process of taking a large pre-trained model and training it on your own data to achieve a specific result. This process is becoming increasingly popular in the ML community as the pre-trained models get larger and much more capable. It provides the last mile tuning you need to get dramatically improved performance on your specific problem — with much less effort than it would take to build a whole new model from scratch.</p>
<p>Fine-tuning has been successfully applied in many realms such as <a href="https://ai.stackexchange.com/questions/39023/are-gpt-3-5-series-models-based-on-gpt-3">ChatGPT</a> and <a href="https://github.com/tatsu-lab/stanford_alpaca">Alpaca</a> for text. In the image generation space, it is typically used to teach models to generate images featuring custom characters, objects, or specific styles — especially those that the large pre-trained model has not encountered before.</p>
<h2 id="types-of-fine-tuning">Types of fine-tuning</h2>
<p>The <em>classic</em> way to fine-tune image models is conceptually simple: provide a large dataset of labeled image and caption pairs; then re-run training using the existing model weights as a prior. This process is very similar to how the pre-trained model learned concepts in the first place. Lambda Labs wrote <a href="https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusion-how-we-made-the-text-to-pokemon-model-at-lambda">very good article</a> on how they produced the “text-to-pokemon” model by doing this.</p>
<p>This type of fine-tuning, while substantially cheaper and easier to do than building a new model from scratch, still requires a lot of data (on the order of hundreds of images) and GPU compute time. Since then, there have been many innovative techniques published by researchers on how to make this process even more effective and efficient. I’ll briefly touch on the three most popular ones:</p>
<h3 id="textual-inversion">Textual inversion</h3>
<p><a href="https://textual-inversion.github.io/">This paper</a> from researchers at Tel Aviv University and NVIDIA proposed a way to learn a new concept from as little as 3-5 example images, and notably does <em>not</em> require changing the base pre-trained model in any way. I won’t go into the details of how this works here, but paperspace has a <a href="https://blog.paperspace.com/dreambooth-stable-diffusion-tutorial-part-2-textual-inversion/">good tutorial</a> on this process, and there is a page maintained in the Automatic1111 UI Wiki on how to use and train using <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Textual-Inversion">textual inversion</a>.</p>
<p>The ultimate output of this process is a very small “embeddings” file that is typically less than 100 kilobytes. This embedding can be used in tandem with the base model to generate the learned concepts in new ways.</p>
<h3 id="dreambooth">Dreambooth</h3>
<p><a href="https://dreambooth.github.io/">This paper</a> from researchers at Google also only requires 3-5 example images and is able to learn a new character or style. However, Dreambooth does this by directly modifying the pre-trained models and updates the weights. This is a more powerful technique, but the output of the process is a whole new model that is roughly the same size as the pre-trained one. For Stable Diffusion 1.5 that means your newly produced model will be around 4.5 gigabytes!</p>
<p>There is evidence to suggest that commercial products like Lensa’s Magic Avatars uses this technique with great results. Replicate has a <a href="https://replicate.com/blog/dreambooth-api">good blog post</a> on how to train a model using Dreambooth, and their service makes it easy to make one if you don’t have access to a powerful GPU.</p>
<h3 id="lora">LoRA</h3>
<p>This type of fine-tuning is based on the paper <a href="https://arxiv.org/abs/2106.09685">“Low-Rank Adaptation of Large Language Models”</a> — which as the name suggests — was originally a technique used to fine-tune large language models like GPT-3.</p>
<p>While the general technique predates both Textual Inversion and Dreambooth, its application to diffusion models for image generation is very new, kicked off by early this year by <a href="https://github.com/cloneofsimo/lora">cloneofsimo</a>. This method produces an output that is between 50 and 200 megabytes in size, and does not require modifying the pre-trained model.</p>
<h3 id="pros--cons">Pros & Cons</h3>
<p>Let’s summarize these three techniques.</p>
<table width="100%" class="pure-table pure-table-bordered">
<thead>
<tr>
<th>Type</th>
<th># of Examples</th>
<th>Output Size</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>Textual Inversion</td>
<td>3-5 minimum, ideally 20-30</td>
<td>< 100 KB embeddings file</td>
<td><b>OK quality</b>, works better for remixing of existing concepts in the base model.<br /><br />You can combine multiple embeddings at runtime to generate multiple concepts, a single embedding usually represents a single concept or style.</td>
</tr>
<tr>
<td>Dreambooth</td>
<td>3-5 minimum, ideally 20-30</td>
<td>~4.5 GB full model</td>
<td><b>Great quality</b>, works well for both characters and styles.<br /><br />Since it produces a new full model, you have to train multiple characters in a single training. Mixing styles can be tricky to accomplish.</td>
</tr>
<tr>
<td>LoRA</td>
<td>20-50 for characters, 50-200 for styles</td>
<td>~50-200 MB tensor files</td>
<td><b>Good quality</b>, somewhere between Textual Inversion and Embeddings. You can apply multiple LoRAs at runtime (just like embeddings) and are very flexible to mix and match.</td>
</tr>
</tbody>
</table>
<p>Note: I won’t discuss another technique called <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2670">“hypernetworks”</a> here (this is <em>NOT</em> the same as the technique popularized by <a href="https://arxiv.org/abs/1609.09106">this 2016 paper</a>), primarily because vetted knowledge of how to make one optimally is hard to come by.</p>
<p>In this tutorial, I will <strong>focus on the LoRA fine-tuning technique</strong>. In my own experimentation, I’ve found it gave me results that were higher quality than textual inversion, but with an output not as heavyweight as Dreambooth. This let me build a number of characters and styles fairly quickly.</p>
<h2 id="basics-of-a-lora-setup">Basics of a LoRA setup</h2>
<p>Training a LoRA model itself takes only around 10 minutes, but expect the whole process including setting up and preparing training data to take around 2 hours.</p>
<p>There are two main options you have for LoRA training. The original implementation by <a href="https://github.com/cloneofsimo/lora">cloneofsimo</a> seems to have worked well for many, however, I had two issues with it:</p>
<ul>
<li>The default parameters did not work well with my own training data.</li>
<li>The output of this implementation is not compatible with the Automatic1111 UI (that I recommended in my post on <a href="/2023/04/05/quick-sd-install-guide">Installing Stable Diffusion</a>). If you do want to try out this original implementation, replicate once again has an <a href="https://replicate.com/blog/lora-faster-fine-tuning-of-stable-diffusion">easy step-by-step tutorial</a> on how to do it.</li>
</ul>
<p>I opted for <a href="https://github.com/kohya-ss/sd-scripts">kohya’s implementation</a> because it produces outputs compatible with the Automatic1111 UI and offers additional options for adjusting the fine-tuning process. I ran this locally on my (<a href="https://twitter.com/anantn/status/1641672926687801344">brand new!</a>) RTX 4090, but the process can be run on a machine with as little as 6 GB of VRAM. This guide is focused on fine-tuning locally with an NVIDIA card. If you don’t have one, you can use <a href="https://colab.research.google.com/">Google Colab</a> to train your models, which has a generous free tier - <a href="https://github.com/Linaqruf/kohya-trainer">this notebook</a> uses much of the same techniques I’ll talk about here.</p>
<p>There is very handy UI wrapper on top of kohya’s training scripts called <a href="https://github.com/bmaltais/kohya_ss"><code class="language-plaintext highlighter-rouge">kohya_ss</code></a> that we will be using. It makes it easier to manage various configurations and has some nice utilities like auto-captioning that will come in handy.</p>
<p>We’ll use <code class="language-plaintext highlighter-rouge">pyenv</code> again to manage our Python environment. Let’s make a new one for the <code class="language-plaintext highlighter-rouge">kohya_ss</code> GUI, as the requirements here differ slightly from the ones needed for the Automatic1111 UI.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pyenv <span class="nb">install </span>3.10.10
<span class="nv">$ </span>git clone git@github.com:bmaltais/kohya_ss.git
<span class="nv">$ </span><span class="nb">cd </span>kohya_ss
<span class="nv">$ </span>pyenv <span class="nb">local </span>3.10.10
</code></pre></div></div>
<p>Now, let’s install the requirements for the GUI and training scripts. Execute in this specific order, so you have the most optimized version of pytorch (you basically want the one with CUDA 11.8 support):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt
<span class="nv">$ </span>pip <span class="nb">install </span>torch torchvision <span class="nt">--extra-index-url</span> https://download.pytorch.org/whl/cu118
<span class="nv">$ </span>pip <span class="nb">install</span> <span class="nt">-U</span> xformers
</code></pre></div></div>
<p>Finally, let’s configure <code class="language-plaintext highlighter-rouge">accelerate</code>. The defaults work just fine, the only setting that I changed was to use <code class="language-plaintext highlighter-rouge">bf16</code> for mixed precision. All Ampere architecture cards (RTX 3000 or higher) support this format, which results in speedier training runs. Use <code class="language-plaintext highlighter-rouge">fp16</code> if you care about backward compatibility, e.g. you want to run inference on your fine-tuned model on cards that don’t support <code class="language-plaintext highlighter-rouge">bf16</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ accelerate config
In which compute environment are you running?
This machine
Which type of machine are you using?
No distributed training
Do you want to run your training on CPU only (even if a GPU is available)? [yes/NO]:
NO
Do you wish to optimize your script with torch dynamo?[yes/NO]:
NO
Do you want to use DeepSpeed? [yes/NO]:
NO
What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]:
all
Do you wish to use FP16 or BF16 (mixed precision)?
bf16
</code></pre></div></div>
<p>Let’s test that everything worked by starting the GUI:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># From the kohya_ss directory</span>
<span class="nv">$ </span>python kohya_gui.py
Load CSS...
Running on <span class="nb">local </span>URL: http://127.0.0.1:7860
To create a public <span class="nb">link</span>, <span class="nb">set</span> <span class="sb">`</span><span class="nv">share</span><span class="o">=</span>True<span class="sb">`</span> <span class="k">in</span> <span class="sb">`</span>launch<span class="o">()</span><span class="sb">`</span><span class="nb">.</span>
</code></pre></div></div>
<p>Open up the printed URL in your browser. If that worked, we are ready to begin the process of fine-tuning. I’ll walk you through what I did to train a LoRA for my own face.</p>
<h2 id="training-data-preparation">Training data preparation</h2>
<p>Machine learning makes the age-old computer science concept of <a href="https://en.wikipedia.org/wiki/Garbage_in,_garbage_out">“Garbage in, garbage out”</a> painfully obvious. Expect to spend a good chunk of time preparing your training data, this is time well-invested towards ensuring a good quality output! I can imagine this workflow getting more and more automated over time, but for now, we’ll have to do it manually.</p>
<h3 id="image-selection">Image selection</h3>
<p>My first stop was Google Photos, where I grabbed the 50 most recent images that included my face. A few guidelines to keep in mind:</p>
<ul>
<li>Focus on <strong>high resolution, high quality</strong> images. If a photo is blurry or low resolution, err on the side of not including it in the training set.</li>
<li>You need to be able to <strong>crop the photo</strong> such that you are the only face in the photo after doing the cropping.</li>
<li>A good variety of <strong>close-up photos</strong> and some full-body shots are ideal. Avoid photos where you are really far away, there’s not much for the model to learn from these.</li>
<li>Try to shoot for <strong>variety</strong> in terms of lighting, poses, backgrounds, and facial expressions. The greater the diversity, the more flexible your model will be.</li>
</ul>
<aside>
<p>💡Tip: Sometimes blurry or low-res images can be salvaged through the process of upscaling. There is a built-in upscaler in the Automatic1111 UI, <a href="https://www.reddit.com/r/StableDiffusion/comments/xkjjf9/upscale_to_huge_sizes_and_add_detail_with_sd/">here is a tutorial</a> on how to use it. I got good results using the <a href="https://github.com/JingyunLiang/SwinIR">SwinIR upscaling model</a>.</p>
</aside>
<p>Applying these guidelines, I got a set of 25 usable images. I then proceeded to crop them — the default Windows Photos tool worked great, <a href="https://www.birme.net/">birme</a> is another useful online tool. Don’t worry about cropping to a specific size or aspect ratio, just crop so your face and/or body are the predominant entity in the image with some background detail but ideally no other people. Make sure all your cropped images are in a single directory and have unique names (excluding the file extension).</p>
<h3 id="captioning">Captioning</h3>
<p>The next step in the process is to caption each image. In my experiments, there was a marked difference in quality when training with captions compared to without, so I highly recommend doing this.</p>
<p>Fortunately, as I described in my image models primer post, you can use automated techniques to generate image descriptions, which can significantly reduce your workload! One of the reasons we are using the <code class="language-plaintext highlighter-rouge">kohya_ss</code> GUI instead of the original training scripts directly is for the captioning feature, so let’s fire that up.</p>
<p>Open up the printed URL in your browser, head over to the <code class="language-plaintext highlighter-rouge">Utilities</code> tab and <code class="language-plaintext highlighter-rouge">BLIP Captioning</code>. Use the following settings:</p>
<ul>
<li>Image folder to caption: should point to the directory containing the cropped images from the previous step.</li>
<li>Caption file extension: the convention is to have a <code class="language-plaintext highlighter-rouge">.txt</code> file next to each image file containing the description for it.</li>
<li>Prefix to add to BLIP caption: it helps to prefix each description with <code class="language-plaintext highlighter-rouge">photo of <token>,</code> — so the model learns to associate the token with your likeness.</li>
<li>Increase “Number of beams” to 10, and set “Min length” to 25, so you get captions with at-least two sentences.</li>
</ul>
<p>The page should look something like this:</p>
<p><a href="/images/2023/kohya_ss_captioning.jpg"><img src="/images/2023/kohya_ss_captioning.jpg" alt="BLIP captioning" /></a></p>
<aside>
<p>⚠️ Caution: when selecting an appropriate token to prefix your captions with, consider how common your name is. In my case I chose <code class="language-plaintext highlighter-rouge">anantn</code> because it is a fairly uncommon token. If you pick something like <code class="language-plaintext highlighter-rouge">john</code> or <code class="language-plaintext highlighter-rouge">mary</code> your results will be diluted by what the model has already learned about these names, mainly from celebrities and public figures.</p>
<p>You can review the <a href="https://huggingface.co/runwayml/stable-diffusion-v1-5/raw/main/tokenizer/vocab.json">frequency of tokens</a> by their appearance in the base Stable Diffusion 1.5 model, ideally you should pick one that isn’t on that list.</p>
</aside>
<p>Click “Caption images” and let it run! Shouldn’t take more than a few minutes, you can review the progress on the command line where you launched <code class="language-plaintext highlighter-rouge">kohya_gui.py</code>.</p>
<p>Once the process is complete, open up the folder of training images where you should now see a <code class="language-plaintext highlighter-rouge">.txt</code> file with the same filename as each image. I would do a quick once over to make sure the captions are sensible, feel free to make edits and save them back to the same text file. Here are some general guidelines for good captions:</p>
<ul>
<li>Your goal is to associate your likeness with the token you chose. Characteristics that are the same in every photo (in my case, “black hair” and “brown skin”) should not be in the caption.</li>
<li>Explicitly caption things that are different in each photo, e.g. “wearing sunglasses” or “wearing a white shirt”.</li>
<li>Describe the type of photo, e.g. “close up” or “full body”.</li>
<li>If the photo contains other elements such as a background, describe them as well, e.g. “in a kitchen” or “in a park”.</li>
<li>If the photo was taken in the iPhone portrait mode, call it out, and include tags like “blurry background”.</li>
</ul>
<p>Remember, the more accurate and descriptive your captions are, the easier it will be for the model to be flexible in image generation (e.g., swapping out black hair for purple hair).</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_captioned.jpg"><img src="/images/2023/anant_captioned.jpg" alt="Example caption" /></a>
<em>Example auto-generated caption: “photo of anantn, a man in an orange shirt and sunglasses sitting on a rock in the middle of the desert”</em></p>
</div>
</div>
<h2 id="hyperparameter-selection">Hyperparameter selection</h2>
<p>I spent a lot of time playing with the knobs you have at your disposal when fine-tuning. I’ll discuss the relevant hyperparameters in a more detail below, but if you’re just interested in the optimal configuration I found, jump ahead to the <a href="#training">training section</a>. In fact, I’d recommend you do that first, and then come back here to read about the hyperparameters to further refine your LoRA.</p>
<h3 id="training-steps">Training steps</h3>
<p>The total number of training steps your fine-tuning run will take is dependent on 4 variables:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>total_steps = (num_images * repeats * max_train_epochs) / train_batch_size
</code></pre></div></div>
<p>Your goal is to end up with a step count between 1500 and 2000 for character training. The number you can pick for <code class="language-plaintext highlighter-rouge">train_batch_size</code> is dependent on how much VRAM your GPU has, and the higher the number the faster your training goes. However, I wouldn’t pick a number higher than 2 — and for most cases a default of 1 works just fine; since a higher <code class="language-plaintext highlighter-rouge">train_batch_size</code> means you need more training images, and the training time is pretty fast as it is.</p>
<p>Generally, you also want more repeats than epochs — since there is the option to checkpoint your fine-tuning every epoch — and you’ll want to make use of that to see how learning is progressing. Epochs in the 5 to 20 range are reasonable, adjust your repeats accordingly.</p>
<p>In my case, recall that I had 25 example images. I went with:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">train_batch_size</code> = 1</li>
<li><code class="language-plaintext highlighter-rouge">repeats</code> = 15</li>
<li><code class="language-plaintext highlighter-rouge">max_train_epochs</code> = 5</li>
</ul>
<p>These values impute a step count of <strong>1875</strong>.</p>
<aside>
<p>❗NOTE: In the <code class="language-plaintext highlighter-rouge">kohya_ss</code> GUI, you can only specify the <code class="language-plaintext highlighter-rouge">batch_size</code> and <code class="language-plaintext highlighter-rouge">num_epochs</code> parameters. The <code class="language-plaintext highlighter-rouge">num_repeats</code> parameter is implicitely specified by naming your training images folder a certain way: <code class="language-plaintext highlighter-rouge"><repeats>_<token></code>. Thus, in my case, I renamed my folder to <code class="language-plaintext highlighter-rouge">15_anantn</code>.</p>
</aside>
<h3 id="learning-rates">Learning rates</h3>
<p>The learning rate hyperparameter controls how quickly the model absorbs changes from the training images. Under the hood, there are really two components to learning, the “text encoder” and “UNET”. To oversimplify their roles:</p>
<ul>
<li>The “text encoder” learning rate (<code class="language-plaintext highlighter-rouge">text_encoder_lr</code>) controls how quickly the captions are absorbed.</li>
<li>The “UNET” leaning rate (<code class="language-plaintext highlighter-rouge">unet_lr</code>) controls how quickly the visual artifacts are absorbed.</li>
</ul>
<p>Through repeated experimentation, the community has concluded that it is better to learn these at different rates — “text encoder” should be learned at a slower rate than the “UNET”.</p>
<p>The default values here are pretty sane, set <code class="language-plaintext highlighter-rouge">text_encoder_lr</code> to <code class="language-plaintext highlighter-rouge">5e-5</code> and <code class="language-plaintext highlighter-rouge">unet_lr</code> to <code class="language-plaintext highlighter-rouge">1e-4</code>. You can specify them in scientific notation or written out in decimal like <code class="language-plaintext highlighter-rouge">0.00005</code> and <code class="language-plaintext highlighter-rouge">0.0001</code> respectively.</p>
<p>Note that if you specify learning rates for the text encoder and UNET separately as suggested above, the global <code class="language-plaintext highlighter-rouge">learning_rate</code> parameter is ignored.</p>
<h3 id="scheduler--optimizer">Scheduler & Optimizer</h3>
<p>The next option you have to tweak is the <code class="language-plaintext highlighter-rouge">lr_scheduler_type</code>. There are basically three good options here:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">constant</code>: the learning rate is constant throughout the training process.</li>
<li><code class="language-plaintext highlighter-rouge">cosine_with_restarts</code>: the learning rate oscillates between the initial value and 0.</li>
<li><code class="language-plaintext highlighter-rouge">polynomial</code>: the learning rate polynomial decreases from the initial value to 0.</li>
</ul>
<p>Empirically, <code class="language-plaintext highlighter-rouge">polynomial</code> worked best for me, but many in the community swear by <code class="language-plaintext highlighter-rouge">cosine_with_restarts</code>. If you find the model isn’t really learning as quickly as you’d like, <code class="language-plaintext highlighter-rouge">constant</code> is worth a try. It’s hard to be prescriptive about the right option here as it seems to be very dependent on the shape, size, and quality of your training data. Since each training run only takes on the order of 10 minutes, this is one of the settings that’s worth experimenting with and seeing what works best for you.</p>
<p>A related setting is the <code class="language-plaintext highlighter-rouge">optimizer_type</code>. The default is <code class="language-plaintext highlighter-rouge">Adam8bit</code> which works wonderfully for most cases. In theory, the <code class="language-plaintext highlighter-rouge">Adam</code> optimizer works with higher precision but requires more VRAM and is only supported by the newer GPUs; in practice I didn’t find the improvement to be noticeable.</p>
<h3 id="network-rank--alpha">Network Rank & Alpha</h3>
<p>This is likely the most controversial setting. The “network rank” (interchangeably called “network dimensions”, represented as the <code class="language-plaintext highlighter-rouge">network_dim</code> parameter) is a proxy for how detailed your fine-tuning model can get. More dimensions mean more layers available to absorb the finer details of your training set. Be warned, too many layers without enough quantity or diversity in your training data could lead to bad results. Network alpha (<code class="language-plaintext highlighter-rouge">network_alpha</code>) is a dampening effect that controls how quickly the layers absorb new information.</p>
<p>This is where the controversy arises. ML theory suggests that you’d normally need only 8 dimensions (small number of layers) with an alpha of 1 (heavy dampening) to achieve good results, and these are in fact the defaults in both the <code class="language-plaintext highlighter-rouge">cloneofsimo</code> and <code class="language-plaintext highlighter-rouge">kohya</code> LoRA implementations. These values result in a further dampening effect on the learnings rates we chose above, on the order of <code class="language-plaintext highlighter-rouge">1 / 8 = 0.125</code>.</p>
<p>In practice, I and others have found these settings to result in extremely weak learning rates resulting in fine-tuned models that don’t produce images resembling the training data at all. You could, in theory, counteract this by increasing the learning rates themselves. What I’ve found works better is instead to crank up the number of dimensions and set an alpha to be <em>equal</em> to this number. This results in <em>NO</em> dampening effect, but it works since our learning rates were already conservative to begin with. I settled on:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">network_dim</code> = 128</li>
<li><code class="language-plaintext highlighter-rouge">network_alpha</code> = 128</li>
</ul>
<p>I’m no ML expert by any means, but can say these values empirically worked much better than the defaults for my training set. If any knowledgeable folks can provide insights on the theory behind these values, that would be very valuable for the community!</p>
<h3 id="adaptive-optimizers">Adaptive optimizers</h3>
<p>There are two very interesting and new “adaptive” optimizers available. While they didn’t work as well for me, they are worth a discussion. Both <code class="language-plaintext highlighter-rouge">DAdaptation</code> and <code class="language-plaintext highlighter-rouge">AdaFactor</code> optimizers find the best learning rate automatically through the learning process!</p>
<p>In my experimentation, I found that they settled on a rather low learning rate, producing models that made high quality images which showed no signs of overfitting at all — however — the likeness of my face just never got as close as with the other optimizers. You might have better luck with them, and I could be missing some other key factor. For instance, some have suggested that for AdaFactor to work properly you need a lot more training steps and epochs than usual.</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/dadaptation_example.jpg"><img src="/images/2023/dadaptation_example.jpg" alt="DAdaptation" /></a>
<em>DAdaptation output was pretty underfit</em></p>
</div>
</div>
<p>Both these optimizers work best with a high network rank, and an alpha equal to rank. However, it seems that optimal settings for these two optimizers require a lot more training steps than a classic <code class="language-plaintext highlighter-rouge">AdamW8bit</code> optimizer would. This makes a 10-minute training run into a 20 or 30-minute operation, and I just couldn’t justify the additional time relative to the quality improvement.</p>
<p>It’s very likely I am missing something though. If you have any insights on getting better results from these optimizers, please share them in the comments! If you want to try either of these optimizers out, keep the following caveats in mind:</p>
<h4 id="dadaptation">DAdaptation</h4>
<p><a href="https://github.com/facebookresearch/dadaptation">This optimizer</a> was developed by Meta and is intended as a drop-in replacement for <code class="language-plaintext highlighter-rouge">Adam</code>, Just <code class="language-plaintext highlighter-rouge">pip install dadaptation</code>, and for best results couple it with the following settings:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">scheduler</code> must be <code class="language-plaintext highlighter-rouge">constant</code>.</li>
<li><code class="language-plaintext highlighter-rouge">learning_rate</code> must be <code class="language-plaintext highlighter-rouge">1</code>.</li>
<li><code class="language-plaintext highlighter-rouge">text_encoder_lr</code> must be <code class="language-plaintext highlighter-rouge">0.5</code>.</li>
<li>You have to provide the following “Optimizer extra arguments”: <code class="language-plaintext highlighter-rouge">decouple=True</code> which instructs the optimizer to learn UNET and text encoder at different rates (text at half the rate of UNET since you provided the above values). You can also optionally provide the <code class="language-plaintext highlighter-rouge">weight_decay=0.05</code> argument, but I couldn’t really tell if this made a difference.</li>
</ul>
<h4 id="adafactor">AdaFactor</h4>
<p><a href="https://arxiv.org/abs/1804.04235">This optimizer</a> has shown very promising results in the language model community. It comes with its own scheduler that you must use:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">scheduler</code> must be set to <code class="language-plaintext highlighter-rouge">adafactor</code>.</li>
<li><code class="language-plaintext highlighter-rouge">learning_rate</code> must be <code class="language-plaintext highlighter-rouge">1</code>.</li>
<li><code class="language-plaintext highlighter-rouge">text_encoder_lr</code> must be <code class="language-plaintext highlighter-rouge">0.5</code>.</li>
<li>You have to provide the following “Optimizer extra arguments”: <code class="language-plaintext highlighter-rouge">relative_step=True scale_parameter=True warmup_init=False</code>. You can set <code class="language-plaintext highlighter-rouge">warmup_init=True</code> for smoother learning rate convergence, however, I’ve found you need a <em>lot</em> more training steps and epochs with this setting.</li>
</ul>
<h2 id="training">Training</h2>
<p>To recap, here are the hyperparameters I finally settled on. I suggest you start your first run with these, and go back to the previous sections for other options if you’re not happy with the results. The values below are for a training set of 25 images, adjust the number of repeats and epochs accordingly if you have more or less images.</p>
<table width="100%" class="pure-table pure-table-bordered">
<thead>
<tr>
<th>Hyperparameter</th>
<th>Recommended value</th>
</tr>
</thead>
<tbody>
<tr>
<td>batch_size</td>
<td>1</td>
</tr>
<tr>
<td>repeats</td>
<td>15</td>
</tr>
<tr>
<td>max_train_epochs</td>
<td>5</td>
</tr>
<tr>
<td>text_encoder_lr</td>
<td>5e-5</td>
</tr>
<tr>
<td>unet_lr</td>
<td>1e-4</td>
</tr>
<tr>
<td>lr_scheduler_type</td>
<td>polynomial</td>
</tr>
<tr>
<td>optimizer_type</td>
<td>AdamW8bit</td>
</tr>
<tr>
<td>network_dim</td>
<td>128</td>
</tr>
<tr>
<td>network_alpha</td>
<td>128</td>
</tr>
</tbody>
</table>
<p>Recall that you specify the number of <code class="language-plaintext highlighter-rouge">repeats</code> implicitly by naming your training images folder of the form <code class="language-plaintext highlighter-rouge"><repeats>_<token></code>. I recommend sharing the same training data and logs directory across multiple training runs, while creating a new directory for each new configuration of hyperparameters you want to experiment with. Something like this could work, for example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>|- lora
| |- training
| | |- 15_anantn
| | | |- 1.jpg
| | | |- 1.txt
| | | |- ...
| | | |- 25.jpg
| | | |- 25.txt
| |- outputs
| | |- v1
| | | |- config.json
| | |- v2
| | | |- config.json
| | |- ...
| |- logs
</code></pre></div></div>
<p>For the base model, I recommend sticking with Stable Diffusion v1.5 (v2.0+ still isn’t widely adopted by the community yet). Even if you plan on doing inference on models that are derived from v1.5, it is still beneficial to perform your initial on just the vanilla model for maximum flexibility with a variety of downstream models. You should already have a copy of this model if you <a href="/2023/04/05/quick-sd-install-guide">installed the Automatic1111 Stable Diffusion UI</a>.</p>
<p>Now all we have to do is plug these folder paths and hyperparameters into the UI Make sure to select the (confusingly named) <strong>Dreambooth LoRA</strong> tab at the very top!</p>
<p><a href="/images/2023/kohya_ss_folders.jpg"><img src="/images/2023/kohya_ss_folders.jpg" alt="Kohya GUI folders" /></a>
<em>Folder configuration</em></p>
<p><a href="/images/2023/kohya_ss_config.jpg"><img src="/images/2023/kohya_ss_config.jpg" alt="Kohya Configuration" /></a>
<em>Hyperparameter configuration</em></p>
<p>Double check all the values again. Couple of more settings to keep in mind:</p>
<ul>
<li>Don’t forget to set the “Caption extension” value to <code class="language-plaintext highlighter-rouge">.txt</code>!</li>
<li>For max resolution, if most of your images are larger than 768x768, then you can set <code class="language-plaintext highlighter-rouge">768,768</code> as the value. If not, leave it at the default <code class="language-plaintext highlighter-rouge">512,512</code>.</li>
</ul>
<p>Click the “Train model” button, and wait for the training to complete. You can follow progress on the terminal where you started <code class="language-plaintext highlighter-rouge">kohya_gui.py</code>. For 1875 steps on my RTX 4090, this took less than 10 minutes.</p>
<aside>
<p>💡Tip: You can run <code class="language-plaintext highlighter-rouge">tensorboard --logdir /path/to/logs</code> to pull up useful graphs on how your training went. If your <code class="language-plaintext highlighter-rouge">loss</code> ever hit <code class="language-plaintext highlighter-rouge">NaN</code>, it means your fine-tune was burned and you should double check your hyperparameters.</p>
</aside>
<h2 id="testing">Testing</h2>
<p>Now comes the fun part! Let’s test our model to see how we did.</p>
<p>For testing, I used <a href="https://huggingface.co/darkstorm2150/Protogen_x5.3_Official_Release">Protogen x5.3</a>, which is a fine-tuned derivation of Stable Diffusion v1.5. You can certainly use the plain v1.5 model, but models like Protogen give you a lot more creative tools and prompting options.</p>
<p>To use a model like Protogen, just download the safetensors file from HuggingFace and place it in the <code class="language-plaintext highlighter-rouge">models/Stable-diffusion/</code> directory in your Automatic1111 installation. At this point, also copy (or symlink) the output of your LoRA fine-tuning run into <code class="language-plaintext highlighter-rouge">models/Lora/</code>. The directory structure inside <code class="language-plaintext highlighter-rouge">stable-diffusion-webui</code> should now look something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>|- models
| |- Lora
| | |- last-000001.safetensors
| | |- last-000002.safetensors
| | |- last-000003.safetensors
| | |- last-000004.safetensors
| | |- last.safetensors
| |- Stable-diffusion
| | |- ProtoGen_X5.3.safetensors
</code></pre></div></div>
<p>Fire up launch.py (see my installation post for the best command-line arguments) and think of your test prompt. I chose something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a portrait of anantn, red hair, handsome face, oil painting, artstation, <lora:last:1.0>
Negative prompt: ugly, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, extra limbs, disfigured, deformed, watermark, signature, bad art, blurry, blurred
Steps: 35, Sampler: Euler a, CFG scale: 7, Seed: 363423421, Size: 512x512
</code></pre></div></div>
<p>You’ll obviously want to swap out the token <code class="language-plaintext highlighter-rouge">anantn</code> for whatever value you used during training. Click generate! If all went well you should see an oil painting of someone with your likeness with red hair 😊</p>
<aside>
<p>💡Tip: Copy the prompt above into the text box in your UI, and click the blue arrow under “Generate” to auto-populate the negative prompt and other fields.</p>
</aside>
<p><a href="/images/2023/automatic_ui_first_test.jpg"><img src="/images/2023/automatic_ui_first_test.jpg" alt="First test" /></a></p>
<h3 id="testing-tips">Testing tips</h3>
<p>Here are a few guidelines to keep in mind as you test your newly baked LoRA!</p>
<h4 id="xyz-plots">X/Y/Z Plots</h4>
<p>This is a very handy feature in Automatic1111 that lets you generate batches of images varying some aspect of your prompt. In the prompt we used above, the token <code class="language-plaintext highlighter-rouge"><lora:last:1.0></code> referred to only the last epoch at full strength. Varying this value through an X/Y plot can help you understand how training progressed and if a different epoch at a different strength produced more desirable results.</p>
<p>To generate something like that, just select “X/Y/Z plot” from the drop-down in “Script”. For “X type”, and “Y type” select “Prompt S/R”, which stands for Prompt Search/Replace.</p>
<ul>
<li>Set “X values” to <code class="language-plaintext highlighter-rouge">last, last-000001, last-000002, last-000003, last-000004</code></li>
<li>Set “Y values” to <code class="language-plaintext highlighter-rouge">1.0, 0.9, 0.8, 0.7, 0.6, 0.5</code></li>
</ul>
<p>This will cycle through all epochs you trained as well as give you a sense of how applying the LoRA at different strengths affects the output. I used X/Y/Z plots extensively to compare outputs not just between epochs, but also between different hyperparameter configurations. For example, here is an X/Y plot that compares the <code class="language-plaintext highlighter-rouge">polynomial</code>, <code class="language-plaintext highlighter-rouge">cosine_with_restarts</code>, and <code class="language-plaintext highlighter-rouge">constant</code> scheduler type at various strengths:</p>
<p><a href="/images/2023/automatic_ui_xy_plot.jpg"><img src="/images/2023/automatic_ui_xy_plot.jpg" alt="X/Y/Z plot" /></a></p>
<h4 id="underfit-or-overfit">Underfit or overfit?</h4>
<p>What you look for in a good LoRA can depend a lot on your particular training set and your own artistic sensibility. In my specific case, I was really looking for two main things:</p>
<ul>
<li>Likeness: the output should look like me! I comb my hair somewhat unconventionally, from right to left, the model picking up on that was a good sign that it picked up my likeness. Note how in the grid above, the combing direction changes at strength <code class="language-plaintext highlighter-rouge">1.0</code> for <code class="language-plaintext highlighter-rouge">polynomial</code> and <code class="language-plaintext highlighter-rouge">0.8</code> for <code class="language-plaintext highlighter-rouge">constant</code>.</li>
<li>Adaptability: an easy test was to see if the model could turn my (naturally black) hair into a bright red color. Hair turning blackish even though my prompt says red hair is a sign of overfitting. There can be other signs too, such as at strength <code class="language-plaintext highlighter-rouge">1.0</code> in <code class="language-plaintext highlighter-rouge">constant</code>, you can see elements in the background that are from my training set but not my prompt.</li>
</ul>
<p>It might be helpful for you to identify your own criteria based on details of your likeness and face, both on the low end (turning a stock image into your likeness) and on the high end (outputs looking too similar to training data, losing prompting flexibility).</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/underfit_example.jpg"><img src="/images/2023/underfit_example.jpg" alt="Example of underfit" /></a>
<em>AdaFactor produced underfit: likeness not strong enough, beard still present</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/overfit_example.jpg"><img src="/images/2023/overfit_example.jpg" alt="Example of overfit" /></a>
<em>Constant with high LR produced overfit: background is smudged</em></p>
</div>
</div>
<h4 id="generalizability">Generalizability</h4>
<p>Once you have found a particular epoch and strength that you like on a simple prompt like the one above — you can move onto to testing the flexibility and generalizability of your LoRA.</p>
<p>A good LoRA will give you good results in a variety of conditions; I encourage you to experiment with things like varying hair colors, adding accessories, changing outfits, and trying out both artistic and realistic environments and backgrounds. Here are a few that I tried:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_lora_1.jpg"><img src="/images/2023/anant_lora_1.jpg" alt="LoRA Anant Example 1" /></a>
<em>In military uniform, steampunk style</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_lora_2.jpg"><img src="/images/2023/anant_lora_2.jpg" alt="LoRA Anant Example 2" /></a>
<em>Realistic, jester clothes, swiss town</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_lora_3.jpg"><img src="/images/2023/anant_lora_3.jpg" alt="LoRA Anant Example 2" /></a>
<em>Working in a ramen shop, as a boy</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/anant_lora_4.jpg"><img src="/images/2023/anant_lora_4.jpg" alt="LoRA Anant Example 3" /></a>
<em>In an astronaut suit</em></p>
</div>
</div>
<p>I’m pretty pleased with the results - this feels like a solid, baked, generalizable LoRA 👍 While I was a happy “Magic Avatars” customer, these results far surpass what I got back from most commercial services offering AI-generated avatars! And I can generate a near-infinite number of these, limited only by my imagination and prompt-writing ability.</p>
<aside>
<p>💡Tip: If your images come out generally well, except for just one or two elements (like the face), you can fix this using inpainting rather than generating a whole new image with a different seed. Just click the “Send to img2img” button to use the <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#inpainting">inpainting feature</a>.</p>
</aside>
<p>Keep in mind the work shown in this post took me several tries over several days to achieve successful results. If you didn’t get great results on your first attempt, go back to the section on <a href="#hyperparameter-selection">hyperparameter selection</a> and see if tweaking these values helps. I’d also encourage keeping a training diary to keep track of your experiments and results.</p>
<h2 id="objects--styles">Objects & Styles</h2>
<p>Now that you have a repeatable process for creating new LoRAs, you can try them for different characters, objects, and styles. Applying the same process to a training set consisting of photos of my dog, I got impressive results:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/samahan_lora_1.jpg"><img src="/images/2023/samahan_lora_1.jpg" alt="LoRA Samahan Example 1" /></a>
<em>Wearing metallic armor</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/samahan_lora_2.jpg"><img src="/images/2023/samahan_lora_2.jpg" alt="LoRA Samahan Example 2" /></a>
<em>Ghibli style</em></p>
</div>
</div>
<p>My friend <a href="https://twitter.com/vikrum5000">Vikrum</a> had an awesome idea to create art resembling the <a href="https://hyperallergic.com/649642/india-vibrant-idiosyncratic-truck-art/">style found on Indian trucks</a>. This art style is quite unique, but very niche and hence not present in any mainstream image generation model. The ubiquitous phrase <a href="https://www.atlasobscura.com/articles/the-origins-of-horn-ok-please-indias-most-ubiquitous-phrase">“Horn OK Please”</a> is painted on the back of nearly every truck in India, surrounded by art in a distinctive style.</p>
<p><a href="/images/2023/horn_ok_please.jpg"><img src="/images/2023/horn_ok_please.jpg" alt="Indian truck art" /></a>
<em>Typical art style found on the back of Indian trucks</em></p>
<p>We had trouble reproducing this style in Midjourney, but that makes it a perfect use-case for LoRAs with Stable Diffusion. Once again, following the process above, we produced a LoRA that can represent any concepts in the style of Indian truck art:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/lora_eagle_truck.jpg"><img src="/images/2023/lora_eagle_truck.jpg" alt="Eagle in Indian truck style" /></a>
<em>American bald eagle in Indian truck style</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/lora_mona_truck.jpg"><img src="/images/2023/lora_mona_truck.jpg" alt="Mona Lisa in Indian truck style" /></a>
<em>Mona Lisa in Indian truck style</em></p>
</div>
<p>This LoRA was trained with only 30 training images; I suspect we can do substantially better with more training data. Anecdotally, styles transfer best at 100+ training images. Here is <a href="https://www.reddit.com/r/StableDiffusion/comments/11r2shu/i_made_a_style_lora_from_a_photoshop_action_i/">a reddit post</a> on creating a style LoRA based on the Artist Photoshop effect, which could be another good resource.</p>
<h2 id="lycoris-locon--loha">LyCORIS (LoCon / LoHa)</h2>
<p>There have been some extensions to the core LoRA algorithm, called “LoCon” and “LoHA”, which you might see in the dropdown options in the <code class="language-plaintext highlighter-rouge">kohya_ss</code> GUI. These are built on the learning algorithms <a href="https://github.com/KohakuBlueleaf/LyCORIS">in this repository</a>.</p>
<p>I gave these a try but was unable to reproduce results that got anywhere close to the quality of the original LoRA — even with the suggested parameters of <32 dimensions and low alpha (1 or lower). It’s worth keeping an eye on these methods as they evolve, but for now I suggest sticking with conventional LoRA.</p>
<h2 id="summary">Summary</h2>
<p>LoRA is a powerful and versatile fine-tuning method to create custom characters, objects, or styles. While the training data and captioning process is rather cumbersome today, I imagine large parts of this process will be automated to a great degree in the coming months. It wouldn’t surprise me to see many LoRA-based commercial applications pop up in the near future.</p>
<p>Let me know how this method worked for you, or if you have any questions or comments on the process!</p>
</div>anantRemember Magic Avatars in the Lensa app that were all the rage a few months ago? The custom AI generated avatars from just a few photos of your face were a huge hit!Quick guide to installing Stable Diffusion2023-04-05T00:00:00+00:002023-04-05T00:00:00+00:00https://www.kix.in/2023/04/05/quick-sd-install-guide<p>In my last post <a href="/2023/04/01/txt2img/">“AI primer: Image Models 101”</a>, I recommended Midjourney and Stable Diffusion as my top two choices. If your goal is to make beautiful images and art, Midjourney is the best choice, as they have a very easy to use service and can produce high quality output with minimal effort.</p>
<p>However, if you are a hobbyist and tinkerer, or just want to invest a bit of time to go deeper in this space, Stable Diffusion is a great choice. It’s a bit more involved to set up, but has several advantages:</p>
<ul>
<li>It is one of the few image generation models that you can run fully offline on your own computer - it’s free and no cloud service required!</li>
<li>You can generate images that you have a hard time generating with models like Midjourney, such as images with an obscure or very specific style.</li>
<li>The model weights are open to inspect, and you can fine-tune these weights to achieve various effects (which we will talk about in a future post).</li>
<li>It has a powerful suite of image to image models — such as for upscaling, inpainting, and outpainting.</li>
<li>There’s a large community of open source developers and hobbyists who work on a wide range of tools and applications around it.</li>
</ul>
<p>Sold? Let’s get started!</p>
<h2 id="easy-1-click-options">Easy 1-click options</h2>
<p>The open source community has made it easy to set up a basic Stable Diffusion setup in just a few clicks. You can use these options if you want something quick and easy to work with.</p>
<ul>
<li><a href="https://nmkd.itch.io/t2i-gui">NMKD UI</a> is the easiest 1-click option for Windows users.</li>
<li><a href="https://diffusionbee.com/">Diffusion Bee</a> is a popular option for Mac users.</li>
<li><a href="https://stable-diffusion-ui.github.io/">Easy Diffusion</a> has installers for Windows and Linux that are fairly easy to use.</li>
</ul>
<p>The option I would most recommend however, is the <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui">Automatic1111 Web UI</a>. This UI has the most functionality and is actively used by the community for the most cutting-edge work in this space. Installing and using it is a little more daunting compared to the other three options above, but if ease of use were your goal, you might be better off with something like Midjourney anyway.</p>
<p>So, if working with the command line and getting a bit in the weeds doesn’t faze you, read on!</p>
<h2 id="pre-requisites">Pre-requisites</h2>
<p>Large AI models typically require immense compute capability, particularly in the form of GPUs. Even though Stable Diffusion is light enough to run on your own computer, you’ll still get the best results from having a fairly beefy setup:</p>
<ul>
<li>PC with 8 GB or more RAM, and at least 10 GB of free disk space.</li>
<li>A good GPU with at least 6 GB of VRAM. NVIDIA is preferred though AMD can work too.</li>
<li>If you are on a Mac, then any of the Apple Silicon (M1 or higher) laptops will work, thanks to <a href="https://machinelearning.apple.com/research/stable-diffusion-coreml-apple-silicon">CoreML optimizations</a> made by the Apple ML research team!</li>
</ul>
<p>If you are on a Windows PC, I recommend installing WSL2 first. WSL2 is basically an easy way to run a full Linux installation within Windows. While Stable Diffusion is fully supported on Windows natively, I’ve only set it up in WSL2 and there are a lot of nice Unix tools that make my workflows easier.</p>
<p>Installing WSL is <a href="https://learn.microsoft.com/en-us/windows/wsl/install">fairly straightforward</a>, open a Windows terminal or PowerShell and type:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span> wsl <span class="nt">--install</span>
</code></pre></div></div>
<p>This installs the latest version of Ubuntu, which is what I use. Most of the instructions that follow are common to Windows, Linux, and Mac.</p>
<p>Python is the <em>lingua franca</em> of all things AI, so we’ll need an installation. Python is notorious for a very messy package management system that is rife with version conflicts, specifically so for commonly used ML packages that aren’t always compatible with every python version. I don’t recommend using the standard system version for this reason.</p>
<p>A lot of projects suggest using python virtual environments (<a href="https://docs.python.org/3/library/venv.html">venv</a>), but I go a different route. venvs are annoying because you have to manually activate them, and sometimes I do want two projects to use the same set of python packages. So I went with <a href="https://github.com/pyenv/pyenv">pyenv</a> which is a lightweight way to manage multiple python versions. Installing pyenv is easy:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl https://pyenv.run | bash
</code></pre></div></div>
<p>Make sure to add the required lines to your<code class="language-plaintext highlighter-rouge">~/.bashrc</code> or <code class="language-plaintext highlighter-rouge">~/.zshrc</code> file as instructed. For bash this looks something like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'export PYENV_ROOT="$HOME/.pyenv"'</span> <span class="o">>></span> ~/.bashrc
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"'</span> <span class="o">>></span> ~/.bashrc
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'eval "$(pyenv init -)"'</span> <span class="o">>></span> ~/.bashrc
</code></pre></div></div>
<p>You can test this worked by opening a new terminal and typing:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pyenv versions
</code></pre></div></div>
<p>Don’t worry about installing a python version for now, we’ll do it while installing the Stable Diffusion UI. Now is a good time to install <code class="language-plaintext highlighter-rouge">git</code> if you haven’t already:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>git
</code></pre></div></div>
<p>On the Mac, git should already be installed if you have ever installed Xcode. If you didn’t, I recommend you install Homebrew first, and then install git:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>brew <span class="nb">install </span>git
</code></pre></div></div>
<p>You should now have all the prerequisites!</p>
<h2 id="installing-stable-diffusion-webui">Installing stable-diffusion-webui</h2>
<p>Automatic1111’s UI has everything you need to get going. I recommend installing it by cloning the Github repository as it makes keeping up to date with changes easy:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
</code></pre></div></div>
<p>Because we are using pyenv, the installation instructions will differ somewhat compared to the official ones. But feel free to use the official installer scripts if you prefer using the <code class="language-plaintext highlighter-rouge">venv</code> setup, they work just as well.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Make sure you are in the webui folder</span>
<span class="nv">$ </span><span class="nb">cd </span>stable-diffusion-webui
<span class="c"># Python version 3.10.6 works best</span>
<span class="nv">$ </span>pyenv <span class="nb">install </span>3.10.6
<span class="c"># We set this specific version to activate whenever we are in this directory</span>
<span class="nv">$ </span>pyenv <span class="nb">local </span>3.10.6
</code></pre></div></div>
<p>Now we can install the UI:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># IGNORE if you are on WSL or Linux, only do this on Mac:</span>
<span class="nv">$ </span><span class="nb">source </span>webui-macos-env.sh
<span class="c"># launch.py also intalls any dependencies the first time you use it</span>
<span class="nv">$ </span>python launch.py
</code></pre></div></div>
<p>This will take some time the first time you run it, as it will also install all your python dependencies into pyenv, as well as fetch the Stable Diffusion 1.5 model weights. Once it’s done, you should see something like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Startup time: 8.2s (import torch: 1.0s, import gradio: 0.9s, import ldm: 0.3s, other imports: 1.0s, load scripts: 0.3s, load SD checkpoint: 4.6s, create ui: 0.1s).
</code></pre></div></div>
<p>Launch that URL in a browser and you should see something like this:</p>
<p><a href="/images/2023/sd-default-ui.jpg"><img src="/images/2023/sd-default-ui.jpg" alt="Default stable-diffusion-webui" /></a>
<em>Default stable-diffusion-webui</em></p>
<h2 id="generating-your-first-image">Generating your first image</h2>
<p>The UI can be a bit overwhelming at first, but you only need a few key inputs to get started. On the top left, the UI should have automatically selected the “v1-5-pruned-emaonly.safetensors” checkpoint. This is the default stable diffusion v1.5 image generation model. The real power of Stable Diffusion comes from community-generated variants of this base model, which you can download and select here. However, for now, let’s stick with the default.</p>
<p>The two big text boxes are where you will type your prompt and negative prompt. Stable diffusion is unique in its ability to receive negative prompts to suppress undesirable elements in an image. AI models specifically struggle with drawing accurate human hands, faces, and limbs, which can sometimes lead to deformed images — negative prompts are a great way to help with this.</p>
<p>For now, type this into your prompt text box:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>masterpiece, best quality, cartoon illustration of a corgi, happy, running through a beautiful green field, flowers, sunrise in the background
</code></pre></div></div>
<p>and in the negative prompt:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>low quality, worst quality, bad anatomy, bad composition, poor, low effort
</code></pre></div></div>
<p>Make sure “Sampling method” is left at <code class="language-plaintext highlighter-rouge">Euler a</code>. Set “Sampling steps” to <code class="language-plaintext highlighter-rouge">40</code>.
Set “Seed” to <code class="language-plaintext highlighter-rouge">1481517414</code>. Click the big “Generate” button!</p>
<p><a href="/images/2023/sd-first-image.jpg"><img src="/images/2023/sd-first-image.jpg" alt="Your first SD image" /></a></p>
<p>If all went well you should see an image that’s almost exactly or as close to the one in picture above! That’s because we used the same base model, sampler, sampling steps and most importantly the <code class="language-plaintext highlighter-rouge">seed</code> number.</p>
<p>Congratulations on generating your first image with Stable Diffusion!</p>
<h2 id="tips-for-nvidia-gpus">Tips for NVIDIA GPUs</h2>
<p>If you have any NVIDIA GPU, you will get better performance by starting the UI with the <code class="language-plaintext highlighter-rouge">xformers</code> argument:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python launch.py <span class="nt">--xformers</span>
</code></pre></div></div>
<p>For this to work you have to install the proper NVIDIA video drivers. If you are on WSL, make sure to install the drivers on <em>Windows</em> and not Linux. There is an annoying bug in WSL currently that <em>also</em> requires you to create a few symbolic links for <code class="language-plaintext highlighter-rouge">libcuda.so</code> - <a href="https://github.com/microsoft/WSL/issues/5548#issuecomment-1363676648">the workaround is described here</a>, essentially you run this in a Windows terminal or PowerShell:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">></span> C:
<span class="o">></span> <span class="nb">cd</span> <span class="se">\W</span>indows<span class="se">\S</span>ystem32<span class="se">\l</span>xss<span class="se">\l</span>ib
<span class="o">></span> del libcuda.so
<span class="o">></span> del libcuda.so.1
<span class="o">></span> mklink libcuda.so libcuda.so.1.1
<span class="o">></span> mklink libcuda.so.1 libcuda.so.1.1
</code></pre></div></div>
<p>You’ll have to do this every time you update your NVIDIA drivers until Microsoft fixes the bug in WSL.</p>
<p>Confusingly, if you have an NVIDIA RTX 40 series card, my observation is that you will get the best performance by <strong>not using</strong> <code class="language-plaintext highlighter-rouge">xformers</code> - but instead updating to the latest version of <code class="language-plaintext highlighter-rouge">torch</code> with CUDA 11.8 support:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pip <span class="nb">install </span><span class="nv">torch</span><span class="o">==</span>2.0.0 torchvision <span class="nt">--extra-index-url</span> https://download.pytorch.org/whl/cu118
</code></pre></div></div>
<p>Then launch stable-diffusion-webui with the following arguments:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python launch.py <span class="nt">--opt-sdp-no-mem-attention</span> <span class="nt">--opt-channelslast</span>
</code></pre></div></div>
<p>You can substitute <code class="language-plaintext highlighter-rouge">--opt-sdp-no-mem-attention</code> with just <code class="language-plaintext highlighter-rouge">--opt-sdp-attention</code> for even faster performance at the cost of some non-determinism (you may not be able to recreate the exact image even with the same seed).</p>
<p>To squeeze even more performance out of Stable Diffusion, if you are on WSL, consider disabling <a href="https://www.majorgeeks.com/content/page/hardware_accelerated_gpu_scheduling.html">“Hardware-accelerated GPU scheduling”</a> from your Windows Settings. This may impact performance of games running natively on Windows though, YMMV.</p>
<h2 id="things-to-try">Things to try</h2>
<p>Now that you have a basic set up, you can start to explore all the knobs presented to you in the UI. Here are some things to try:</p>
<ul>
<li>The dice icon next to “Seed” sets the value to -1 which will use a random number every time you click generate. Try this a few times to see how you can generate an almost infinite variety of images from the same prompt.</li>
<li>The “CFG” Scale slide above it sets how directive you want your prompt to be. Low values will give the model freedom to be much more creative in its interpretation of your prompt, while high values will make it more likely to generate images that stick very closely with your prompt.
<ul>
<li>At first glance, it might seem like a good idea to set this to higher values, but I’ve realized I’m not very good at writing detailed descriptions of images. Lower values can be quite helpful to fill in details you didn’t think of, but are natural in the output 😉</li>
</ul>
</li>
<li>Setting “Batch size” to 4 will generate a set of four images for every prompt, so you can generate more variations in parallel (pair with Seed of -1).</li>
<li>The “Sampling methods” are fun to play with. Almost all of them will converge on the same image given enough steps. Euler converges to a good image in a reasonably few number of steps (30-ish), others can generate higher quality images but need more steps to fully realize.
<ul>
<li>The exception to convergence are the methods suffixed with <code class="language-plaintext highlighter-rouge">a</code>, such as <code class="language-plaintext highlighter-rouge">Euler a</code> or <code class="language-plaintext highlighter-rouge">DPM 2 a</code>. These samplers will converge on a different image that are a bit more random than the ones without. It’s great for creativity and variety, but if you want determinism then models like <code class="language-plaintext highlighter-rouge">Euler</code> or <code class="language-plaintext highlighter-rouge">DDIM</code> are reliable and fast. Check out <a href="https://www.reddit.com/r/StableDiffusion/comments/x4zs1r/comparison_between_different_samplers_in_stable/">this reddit post</a> on a comparison between samplers!</li>
</ul>
</li>
<li>Of course, the prompt and negative prompts are the most important input into this process. You can get help from <a href="https://chat.openai.com/chat">ChatGPT</a> to help you create prompts or get inspiration from other creators (googling for “stable diffusion prompts” can get you started).</li>
</ul>
<p>That covers most of the basic functionality of text-to-image generation with Stable Diffusion. There’s a lot of power to uncover through the various tabs at the top and custom scripts or extensions, but we’ll leave that for another time.</p>
<p>Have fun and let your creativity loose!</p>anantIn my last post “AI primer: Image Models 101”, I recommended Midjourney and Stable Diffusion as my top two choices. If your goal is to make beautiful images and art, Midjourney is the best choice, as they have a very easy to use service and can produce high quality output with minimal effort.AI Primer: Image Models 1012023-04-01T00:00:00+00:002023-04-01T00:00:00+00:00https://www.kix.in/2023/04/01/txt2img<p>It’s spring of 2023, and if you aren’t paying attention to what’s happening in the world of AI, you really should. The world is about to change in big ways!</p>
<p>I’ll save the general discussion on AI for another post — I’m hoping for this article to be one of a multipart series where we dive into the practical aspects of each major area of advancement. My goal is to give readers a taste of what’s going on, what the capabilities of these AI models are, and why you should care.</p>
<p>Let’s start with image generation models, as they’re the oldest in the recent wave of innovation. Skip ahead to the <a href="#model-comparison">model comparison and summary section</a> if you are already familiar with text-to-image models!</p>
<h2 id="text-to-image">Text-to-image</h2>
<p>The most basic capability of image generation models is to convert a plain language description into a picture. Let’s start there:</p>
<blockquote>
<p>“the indian warrior arjuna, riding a golden chariot, pulled by four white horses, in a battlefield, highly detailed, digital art”</p>
</blockquote>
<p>This type of text is called a “prompt” in the world of text-to-image models. There are a number of these models commercially available today, and here are a few results from the ones I tried:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/01_runwayml.jpg"><img src="/images/2023/01_runwayml.jpg" alt="Runway ML" /></a>
<em>RunwayML</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/02_dalle.jpg"><img src="/images/2023/02_dalle.jpg" alt="Dall-E 2" /></a>
<em>Dall-E 2</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/03_firefly.jpg"><img src="/images/2023/03_firefly.jpg" alt="Adobe Firefly" /></a>
<em>Firefly</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/04_bing_dalle2.jpg"><img src="/images/2023/04_bing_dalle2.jpg" alt="Bing Dall-E 2 Exp" /></a>
<em>Bing (Dall-E 2 Exp)</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/05_diffusion.jpg"><img src="/images/2023/05_diffusion.jpg" alt="Stable Diffusion" /></a>
<em>Stable Diffusion</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/06_midjourney.jpg"><img src="/images/2023/06_midjourney.jpg" alt="Midjourney" /></a>
<em>Midjourney</em></p>
</div>
</div>
<p>Well, isn’t that neat? It’s remarkable that these images have never been created until now. The AI made them on-the-fly based on my prompt, and drawing from its vast knowledge of all images it has “seen”.</p>
<p>A discerning viewer might look at this and not be impressed - you can easily spot a number of defects: the horses aren’t quite lined up right, one of them painted only one horse instead of four, the human faces are pretty blurry and sometimes creepy looking, and so on.</p>
<p>However, I attempted to list the images in order of how impressive they are (yes, I know, art is subjective, bear with me). They come from:</p>
<ul>
<li><a href="https://runwayml.com/">RunwayML</a>, which seems to be running a very early version of the Stable Diffusion open source generation models (also listed below).</li>
<li><a href="https://openai.com/product/dall-e-2">Dall-E 2</a>, which is a commercial model from OpenAI that kick-started the modern age of image generation models.</li>
<li><a href="https://www.adobe.com/sensei/generative-ai/firefly.html">Firefly</a>, the most recent entrant from Adobe that focuses on commercial use and sourcing training material from licensed images.</li>
<li><a href="https://en.wikipedia.org/wiki/Stable_Diffusion">Stable Diffusion</a>, an open source model that kicked off a whole revolution in image generation we will cover in the next post.</li>
<li><a href="https://www.bing.com/create">Bing Create</a>, a partnership between OpenAI and Microsoft on the next version of “Dall-E 2”, lovingly called “Dall-E 2 Exp”.</li>
<li><a href="https://midjourney.com/">Midjourney v5</a>, the latest commercial model from their research lab, perhaps the most impressive of the bunch with a very active and fast-growing community.</li>
</ul>
<p>Even if you aren’t impressed by any of these individual images, I instead direct your attention how quickly the rate of improvement has been. All the services above have a free tier, feel free to try them out with your own prompts!</p>
<p>One model not listed above but worth keeping an eye on is <a href="https://imagen.research.google/">ImageN from Google</a>. You can see a few examples of what it’s capable of on their website. It remains to be seen if Google will eventually release a version of this publicly.</p>
<h3 id="prompt-engineering">Prompt Engineering</h3>
<p>Another reason this capability is impressive, is that the images shown above came from a prompt I didn’t spend very much time thinking about. It was just the first sentence that popped in my head. People have been able to achieve vastly superior quality by really crafting their prompt to better direct the AI.</p>
<p>Here’s a couple of <em>photo-realistic</em> AI-generated images that have been doing the rounds lately:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/fake-pope.jpg"><img src="/images/2023/fake-pope.jpg" alt="Fake Pope" /></a></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/fake-trump.jpg"><img src="/images/2023/fake-trump.jpg" alt="Fake Trump" /></a></p>
</div>
</div>
<p>Do I have your attention yet? These are <strong>FAKE</strong> AI-generated images, but it takes quite a bit of scrutiny to find out. In the <a href="https://www.buzzfeednews.com/article/chrisstokelwalker/pope-puffy-jacket-ai-midjourney-image-creator-interview">Pope’s image</a>, note how his hand that is holding what seems to be a Starbucks cup appears mangled. In <a href="https://www.bbc.com/news/world-us-canada-65069316">Trump’s case</a>, the photo is missing a finger.</p>
<p>Both of these images were made by Midjourney v5, and I suspect by the time v6 rolls out, it will be increasingly hard to tell which images are fake and which are real. We’ll have to rely on the sources they come from and other ways of confirming visual evidence instead.</p>
<p>Here are a couple examples of beautiful artistic images that further showcase how high quality image generation can get:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/good_dragon-mj5.jpg"><img src="/images/2023/good_dragon-mj5.jpg" alt="Enchanted Dragon" /></a>
<em>“character concept, cute baby dragon in the woods, dungeon and dragons, fantasy, medieval”, from Midjourney</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/elephant_tux-bing.jpg"><img src="/images/2023/elephant_tux-bing.jpg" alt="Elephant in a Tux" /></a>
<em>“a renaissance painting of an elephant in a tuxedo”, from Bing Create</em></p>
</div>
</div>
<p>So, what’s the takeaway? With sufficient effort put into your prompt, you can create very high quality <strong>photo-realistic</strong> as well as <strong>artistic</strong> images!</p>
<p>The process of writing great prompts is often referred to as “prompt engineering”. While over time we expect image generation models to produce fantastic outputs just from simple descriptions, their usage today predominantly relies on creative use of prompts to obtain the desired results.</p>
<p>It’s also note-worthy that prompts that work well in one model don’t necessarily translate to another. I recommend sticking with a single tool and refining your prompts there, learning through iteration. Midjourney’s interface is particularly good for this - the main way to make images with them is by joining their Discord. There you can watch <em>other</em> users go through the process of creating images, which is sure to inspire you. And you will learn a few tricks to make great prompts along the way!</p>
<h2 id="image-to-image">Image-to-image</h2>
<p>Turning text into pictures is the most common use of image generation models. However, they can also be applied to existing images. This is a very useful capability to further refine images you made through prompting, or to enhance photos or art that already exist.</p>
<p>There are four main ways you can use this capability.</p>
<h3 id="upscaling">Upscaling</h3>
<p>This is the simplest application of image-to-image models. You can take an image that is of low resolution (say 512x512 pixels) and upscale it to a higher resolution (say 1024x1024 pixels). This isn’t just resizing the image to make it bigger, that would just result in a blurry and ugly mess.</p>
<p>What the model does is repaint the image at the higher resolution, adding detail and clarity along the way to make it look good at a bigger size. It can only do this by accurately predicting what detail should be added - given the context of the image. Because modern image generation models have “seen” so many images, they are able to do this very well.</p>
<p>I made a favicon for this site a long time ago and still only have the low resolution 64x64 version of it. It looks pretty blurry when scaled up to 256x256, let’s see how the upscaler does:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2">
<p><a href="/images/2023/favicon_small.jpg"><img src="/images/2023/favicon_small.jpg" alt="Low res favicon" /></a></p>
</div>
<div class="pure-u-1 pure-u-md-1-2">
<p><a href="/images/2023/favicon_upscaled.jpg"><img src="/images/2023/favicon_upscaled.jpg" alt="High res favicon" /></a></p>
</div>
</div>
<p>Nice! Very crisp, and it picked up on the grunge font details.</p>
<h3 id="style-transfer">Style transfer</h3>
<p>This process can take the style from one image and apply it to another. The classic example of this process is the “Mona Lisa” in the style of “Starry Night”:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/mona-lisa_starry-night.jpg"><img src="/images/2023/mona-lisa_starry-night.jpg" alt="Mona Lisa in the style of Starry Night" /></a></p>
</div>
</div>
<p>Style transfer technology (known as neural style transfer) has actually existed for a while, and predates modern text-to-image models. These are basically “filters” that Instagram, Snapchat, and TikTok have made so popular. However, advancements in text-to-image models have made image-to-image style transfers much more versatile and flexible.</p>
<p>Previously available techniques required a model to be pre-trained for a specific style. Present-day style transfer can be done much more fluidly by first generating an image (e.g. by using prompting as we discussed above) in a particular style, then mixing it with an image of a subject. This is a much more flexible approach, and opens the door to mixing and matching any style with any subject. Remixing is now limited only by your imagination!</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/mona-lisa_new-yorker.jpg"><img src="/images/2023/mona-lisa_new-yorker.jpg" alt="Mona Lisa in the style of New Yorker" /></a>
<em>Mona Lisa in the style of a New Yorker front page caricature, illustration</em></p>
</div>
</div>
<h3 id="inpainting">Inpainting</h3>
<p>Inpainting is the process of replacing a portion of an image with something else. This process works similarly to the text-to-image process we talked about earlier. The only additional complexity is that the generated image is aware of the image content surrounding it and tries to blend in well.</p>
<p>Like style transfer, inpainting as a concept and technology has existed for a while and also predates modern text-to-image models by several years. However, because of these new image generation capabilities, you are able to go far beyond the classical use-cases of inpainting, in a much more flexible and generalizable way.</p>
<h4 id="fixing-defects">Fixing defects</h4>
<p>The simplest application of inpainting is used to restore damaged parts of images. The canonical example here is to remove scratches from scanned analog pictures. This capability has been available from even early versions of <a href="https://opencv.org/">OpenCV</a>:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/inp_damaged.jpg"><img src="/images/2023/inp_damaged.jpg" alt="Damaged Photo" /></a>
<em>Damaged Version</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/inp_restored.jpg"><img src="/images/2023/inp_restored.jpg" alt="Restored Photo" /></a>
<em>Restored Version</em></p>
</div>
</div>
<h4 id="removal">Removal</h4>
<p>Modern inpainting can do a lot more than that. Google recently launched <a href="https://blog.google/products/photos/magic-eraser-android-ios-google-one/">“Magic Eraser”</a> in Google Photos, which can remove arbitrary subjects or objects from images, which is a potent example of this capability.</p>
<blockquote class="twitter-tweet">
<p lang="en" dir="ltr">Pixel 6 Magic Eraser tool at work <a href="https://t.co/dyeUa76HYe">pic.twitter.com/dyeUa76HYe</a></p>
<p>— Shevon Salmon (@its_shevi) <a href="https://twitter.com/its_shevi/status/1452687247313100808?ref_src=twsrc%5Etfw">October 25, 2021</a>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></p>
</blockquote>
<p>Check out this page of <a href="https://www.androidpolice.com/google-pixel-magic-eraser-list/">real world examples</a> of magic eraser at work - you can remove not just people, but almost every object in the frame.</p>
<h4 id="replacement">Replacement</h4>
<p>An even more powerful application is replacing subjects or objects with something else. Note how text-to-image models we used above can sometimes struggle with generating detailed human faces that look natural. Well, inpainting just the face and iterating through a few more generations with models optimized to generating good human faces can help fix that!</p>
<p>Let’s try this technique on the image we generated earlier of Arjuna on a horse using Stable Diffusion:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/inp_face_mark.jpg"><img src="/images/2023/inp_face_mark.jpg" alt="Mark inpainting area" /></a>
<em>Mark portion of image to replace</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/inp_face_final.jpg"><img src="/images/2023/inp_face_final.jpg" alt="Final output" /></a>
<em>Newly generated face blended in!</em></p>
</div>
</div>
<p>Now we’re really getting somewhere! We can use the same inpainting techniques to fix other issues with the image, such as the hand holding the reins.</p>
<h3 id="outpainting">Outpainting</h3>
<p>OpenAI was first to apply the inpainting technique in a creative way — to extend the boundaries of an existing image. Outpainting can be thought of as inpainting, but with a blank canvas extending beyond the original boundaries. This lets you dream up how an image might be extended in a way that matches the original style and blends in.</p>
<p>The artist August Kamp collaborated with OpenAI and used Dall-E to outpaint the classic Dutch painting “Girl with a Pearl Earring” by Johannes Vermeer:</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/outp_pearl.jpg"><img src="/images/2023/outp_pearl.jpg" alt="Girl with a Pearl Earring" /></a>
<em>Original Painting by Johannes Vermeer</em></p>
</div>
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/outp_pearl_final.jpg"><img src="/images/2023/outp_pearl_final.jpg" alt="Outpainted version by August Kamp" /></a>
<em>Imaginative Expansion by August Kamp</em></p>
</div>
</div>
<p>You can watch the full process of outpainting in action <a href="https://openai.com/blog/dall-e-introducing-outpainting">in the original blog post from OpenAI</a>.</p>
<h2 id="image-to-text">Image-to-text</h2>
<p>One interesting possibility is that of applying these image models — <em>in reverse</em> — to generate descriptive text of any picture.</p>
<p>This technique is not to be confused with Optical Character Recognition (OCR for short), which is a method to extract any text found in pictures into a digital form — for example scanning the phone number from a photo of a receipt.</p>
<p>Instead, I’m talking about using these models to <em>describe</em> an image in natural language, essentially captioning them. As an example, let’s take the following image and feed it to <a href="https://github.com/pharmapsychotic/clip-interrogator">“clip-interrogator”</a>, a popular python library that combines the two most-used image captioning models (<a href="https://openai.com/research/clip">CLIP from OpenAI</a> and <a href="https://github.com/salesforce/BLIP">BLIP from Salesforce</a>).</p>
<div class="pure-g">
<div class="pure-u-1 pure-u-md-1-2 imgholder">
<p><a href="/images/2023/gorillaz.jpg"><img src="/images/2023/gorillaz.jpg" alt="Gorillaz music video still" /></a>
<em>Still from a Gorillaz music video</em></p>
</div>
</div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span>
<span class="kn">from</span> <span class="nn">clip_interrogator</span> <span class="kn">import</span> <span class="n">Config</span><span class="p">,</span> <span class="n">Interrogator</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">Image</span><span class="p">.</span><span class="nb">open</span><span class="p">(</span><span class="s">'gorillaz.jpg'</span><span class="p">).</span><span class="n">convert</span><span class="p">(</span><span class="s">'RGB'</span><span class="p">)</span>
<span class="n">ci</span> <span class="o">=</span> <span class="n">Interrogator</span><span class="p">(</span><span class="n">Config</span><span class="p">(</span><span class="n">clip_model_name</span><span class="o">=</span><span class="s">"ViT-L-14/openai"</span><span class="p">))</span>
<span class="k">print</span><span class="p">(</span><span class="n">ci</span><span class="p">.</span><span class="n">interrogate</span><span class="p">(</span><span class="n">image</span><span class="p">))</span>
</code></pre></div></div>
<p>Running this program prints:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cartoonish illustration of a man in front of a table with a tablecloth,
2d gorillaz,
winning awards,
handsome hip hop young black man,
excited facial expression,
goro and kunkle,
victorious on a hill,
dvd cover,
table in front with a cup
</code></pre></div></div>
<p>Cool! You can imagine feeding text similar to this as a prompt to various text-to-image models to generate creative variations. Besides the obvious use-cases around accessibility, there are many interesting applications of this technology that revolve around prompt engineering and creating your own custom image models that we will discuss in future posts.</p>
<h2 id="text-to-video">Text-to-video</h2>
<p>Ok, we’ve already covered a lot! But, stay with me, we have one more potential application to cover.</p>
<p>Generating full-fledged videos from a plain language description is a rapidly emerging field. While there are no readily available tools to do this today like there are for image generation, three key players to watch are:</p>
<ul>
<li>Meta AI kick-started innovation in this area and published impressive demos of their <a href="https://makeavideo.studio/">“make a video” research</a> late in 2022.</li>
<li>This was shortly followed by Google’s <a href="https://imagen.research.google/video/">ImageN video generation</a> demos about a month later, which are equally impressive.</li>
<li>RunwayML seems closest to providing a usable commercial tool. Their <a href="https://twitter.com/runwayml/status/1640337292542844928?s=20">“Gen1” product</a> allows video editing of existing videos through text prompts, while <a href="https://research.runwayml.com/gen2">“Gen2” promises</a> full video generation capabilities like the Meta and Google demos showed above.</li>
</ul>
<p>Some community members have hacked together a poor man’s version of text-to-video, by stitching together multiple generated images using a tool called <a href="https://github.com/lllyasviel/ControlNet">ControlNet</a> to more finely control the output. This is an advanced technique that is out of scope for this post, but we may discuss it in the future.</p>
<h2 id="ethical-considerations">Ethical considerations</h2>
<blockquote>
<p>“With great power comes great responsibility.”
<br />
-Uncle Ben</p>
</blockquote>
<p>We just walked through a lot of awesome capabilities and there is a lot to be optimistic about. Just imagine the reaction of a renaissance-era painter upon hearing that one day there will be a machine that can produce any imaginable art from just a plain language description of it!</p>
<p>But, as with any major technological leap, there are a myriad number of ways to misuse these tools. We’ve seen this happen with PCs, the internet, mobile phones, and social media — the potential for abuse with AI is going to be much larger because these capabilities are so much more powerful. I believe the potential for abuse with image generation models is particularly high because pictures and video tend to be much more evocative and believable.</p>
<ul>
<li><strong>Impact on artists</strong>. These models gained their capabilities by learning from the vast amount of images generated and cataloged by artists and photographers who published their work on the internet. Some model creators were careful to only source licensed content (e.g. <a href="https://news.adobe.com/news/news-details/2023/Adobe-Unveils-Firefly-a-Family-of-new-Creative-Generative-AI/default.aspx">Adobe Firefly</a>) and give artists tools to exclude their work from training sets. Others (e.g. <a href="https://www.theverge.com/2023/1/17/23558516/ai-art-copyright-stable-diffusion-getty-images-lawsuit">Stable Diffusion</a>) were a bit more fast and loose in their image acquisition strategy.
<ul>
<li>The debate is nuanced, as human artists also learn from other works they have observed throughout their lifetime. There is an ongoing philosophical question on what is truly original work, and what is merely remixing past observations. Society must contend with how we can sustainably compensate artists, particularly for work that eventually leads to commercial outcomes for people and teams using models to generate art based on their work.</li>
<li>There is the question of how disruptive this technology will be to the livelihood of artists and illustrators. These tools certainly make them all much more productive, but the fact remains that as productivity rises, industries will need fewer people to produce the same amount of work. My suggestion for both aspiring and established creatives is to start mastering these tools.</li>
<li>The very top artists of the world will be in greater demand to create truly unique and creative work that the world has not seen before. This is already happening in the modern art world. Scarcity will generate value, but this value is likely to be captured by a handful of the world’s best.</li>
</ul>
</li>
<li><strong>Impact on society</strong>. This technology will very likely be used to generate fake news and misinformation, as we showed with the images of Trump and the Pope above. We’ve seen this happen with <a href="https://en.wikipedia.org/wiki/Deepfake">“deepfakes”</a> — fabricated images or videos portraying individuals saying things they never said, or in situations that never occurred in reality. Deepfakes were challenging and laborious to make by hand, until now. This will become a trivial process going forward, something that can be done in large quantities even by non-technical people. The “evidence of your eyes” will require much more scrutiny in this new era.
<ul>
<li>Abuse such as blackmail or <a href="https://en.wikipedia.org/wiki/Revenge_porn">revenge porn</a> based on fabricated images are highly likely to become more commonplace. Governments should move quickly to enact and enforce stringent laws to protect individuals from this type of misuse.</li>
<li>These models have shown to reflect bias that exists in the world and their training data. There is potential for unintentionally reinforcing harmful stereotypes. Indeed, in my own experience, I’ve noticed these models tend to generate humans with fair skin, light eyes, and blonde hair by default. Generating a diverse set of images of various body types, skin tones, and hair colors requires a lot more intentional effort.</li>
<li>On a larger scale, this technology will be weaponized to spread propaganda and influence politics worldwide. In the long run, society as a whole will need to adapt as a response to the inundation of fake content, as we grow to rely more on authoritative and trustworthy sources for information.</li>
</ul>
</li>
</ul>
<p>I urge you to keep these ethical considerations in mind as you wield these extremely powerful tools to create content, but also in consuming content that will increasingly be AI-generated.</p>
<p>Make sure to read and adhere to the licenses that you agree to when using these models, as they also reinforce the ethical considerations above. Be mindful of the impact that your creations will have on society and other individuals.</p>
<h2 id="model-comparison">Model comparison</h2>
<p>Ok, now that you’ve seen what the breadth of capabilities are, let’s summarize the most talked about tools available today!</p>
<table width="100%" class="pure-table pure-table-bordered">
<thead>
<tr>
<th>Model</th>
<th>Features</th>
<th>Price</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://runwayml.com/">RunwayML</a></td>
<td>
✅ Inpainting <br />
✅ Outpainting <br />
✅ Customization <br /><br />
See <a href="https://runwayml.com/ai-magic-tools/">full list</a> of their AI magic tools.
</td>
<td>
Free trial for the first 25 images.<br /><br />
<a href="https://runwayml.com/pricing/">$12/month</a> for 125 images/month thereafter.
</td>
<td>
Runway's claim to fame is their easy to use video editing tools. Their AI magic tools cover a wide range of capabilities, some of which are newer and less mature.
</td>
</tr>
<tr>
<td><a href="https://openai.com/product/dall-e-2">Dall-E 2</a></td>
<td>
✅ Inpainting <br />
✅ Outpainting <br />
❌ Customization <br />
</td>
<td>
Free for 15 images per month.<br /><br />
<a href="https://openai.com/pricing#image-models">$0.016-$0.020</a> per image thereafter.
</td>
<td>
Dall-E 2 is useful for generating abstract or artistic images. It is less competitive at photorealism.
</td>
</tr>
<tr>
<td><a href="https://firefly.adobe.com/">Adobe Firefly</a></td>
<td>
❌ Inpainting <br />
❌ Outpainting <br />
❌ Customization <br />
</td>
<td>
Free during the beta.
</td>
<td>
Adobe's focus is on generating content that is safe to use commercially. However, during the beta, no commercial use is allowed, and additional features such as inpainting and customization are still in the works.
</td>
</tr>
<tr>
<td><a href="https://www.bing.com/create">Dall-E 2 Exp (Bing Create)</a></td>
<td>
❌ Inpainting <br />
❌ Outpainting <br />
❌ Customization <br />
</td>
<td>
First 25 images are created fast.<br /><br />
Subsequent images are still free, but slower to generate.
</td>
<td>
Bing hosts an improved version of Dall-E 2 on their website. It produces higher quality images than stock Dall-E 2 but cannot be customized, and does not support inpainting.
</td>
</tr>
<tr>
<td><a href="https://stability.ai/blog/stable-diffusion-announcement">Stable Diffusion</a></td>
<td>
✅ Inpainting <br />
✅ Outpainting <br />
✅ Customization <br />
</td>
<td>
Free to run on your own computer.<br /><br />
Cloud versions offered by multiple providers cost in the range of $0.02-$0.06 per image.
</td>
<td>
Stable diffusion is an open image generation model that can be run locally on your computer with no restrictions and infinite usage.<br /><br />
<a href="https://stability.ai/">Stability AI</a> also offers a paid hosted version that runs in the cloud and is more user-friendly for non-technical audiences.
</td>
</tr>
<tr>
<td><a href="https://www.midjourney.com/">Midjourney</a></td>
<td>
❌ Inpainting <br />
❌ Outpainting <br />
⚠️ Customization* <br /><br />
* You can customize to a limited extent by uploading <a href="https://docs.midjourney.com/docs/image-prompts">your own image as part of the prompt</a> to do style transfers or redraws. Midjourney also launched
<code>/describe</code> which can <a href="https://twitter.com/midjourney/status/1643053450501169157">caption an image</a> (image-to-text).
</td>
<td>
Free trial discontinued <a href="https://decrypt.co/124972/midjourney-free-ai-image-generation-stopped-over-deepfakes">due to abuse</a>.<br /><br />
$10/month for ~200 images (varies by quality and operation).
</td>
<td>
Midjourney has the most unique and curated art style of all these models.
<br /><br />
They also boast a very active community and unique user interface through use of Discord.
</td>
</tr>
</tbody>
</table>
<p>My recommendations:</p>
<ul>
<li><strong>If you are just getting started</strong> in the world of AI image generation, start with the <em>Bing Create</em> tool for your first few images. It is free and easy to use, although it won’t produce the highest quality images and has no customization options.</li>
<li>If you want to <strong>increase the quality of your images with minimal effort</strong>, I recommend signing up for <em>Midjourney</em>. They have a very active community, and their models produce the best-looking images with minimal tweaking to prompts!</li>
<li>If you are <strong>fully invested in this space</strong> and don’t mind committing time to installing software and tweaking your prompts, I highly recommend <em>Stable Diffusion</em>. You can get the highest quality outputs with sufficient prompt engineering, and the customization options are unparalleled.</li>
</ul>
<p>I’ve been personally spending the most time with Stable Diffusion (and a little on Midjourney) — my next post will cover some ways in which you can fine-tune your own models with Stable diffusion to include your own styles and subjects!</p>
<p>I hope you enjoyed this introduction, and that you’ll unleash your creativity with these newfound superpowers. Please do so responsibly ❤️</p>anantIt’s spring of 2023, and if you aren’t paying attention to what’s happening in the world of AI, you really should. The world is about to change in big ways!Migrating a G Suite Legacy Account2022-03-20T00:00:00+00:002022-03-20T00:00:00+00:00https://www.kix.in/2022/03/20/gsuite-migration<p>Google <a href="https://9to5google.com/2022/01/19/g-suite-legacy-free-edition/">recently announced</a>
that all “G Suite legacy free edition” (formerly known as “Google Apps”, currently known as “Google Workspace”)
accounts will need to transition to their paid workspace plans starting July 1, 2022. Legacy users will get access
to a discounted rate of $3/user/month, which will turn into $6/user/month starting July 2023 at the lowest tier.</p>
<p>I’m the sole user on my G Suite account, so the new rates aren’t a big issue per se. I’ve been getting a ton of
value from this service for over a decade — namely the ability to use Gmail and other Google services but
with my own custom domain.</p>
<p>The bigger issue is that Google Workspace accounts have long been denied access to several products.
You <a href="https://support.google.com/googlepay/answer/10191029?hl=en">can’t use the new GPay</a>, you cannot sign up for
<a href="https://one.google.com/about">Google One</a>, and you cannot sign up for a <a href="https://support.google.com/store/answer/11201976?hl=en">Pixel Pass</a>.
For some time you couldn’t sign up for Google Fi either, though <a href="https://workspaceupdates.googleblog.com/2017/06/project-fi-now-available-for-g-suite.html">that changed</a> a few years ago.</p>
<p>So with this news, I decided to bite the bullet and transition my account to a plain old consumer Google account.
Google already has a mechanism to <a href="https://support.google.com/a/answer/6364687?hl=en">transition educational accounts into personal ones</a>,
and it appears they might be working on a <a href="https://bit.ly/3qoBaNh">solution for all accounts “soon”</a>. However, I
didn’t want to wait for that solution, and I don’t have a ton of payments or purchases on my GSuite account. It’s
mostly the data that needed to be moved, and I figured this would be a good exercise to see how much of my life
exactly depends on my Google account.</p>
<p>I came across this <a href="https://www.39digits.com/migrate-g-suite-account-to-a-personal-google-account">excellent article</a> detailing the steps. I had to tweak some of the instructions
to fit my use-cases, and in some cases found simpler ways to migrate, which I’ll write about here.</p>
<h2 id="setup">Setup</h2>
<p>Before you begin, it is important to note what you <em>cannot</em> transfer from a G Suite account through a manual
migration:</p>
<ul>
<li>Any purchases or subscriptions made through Google Play (Apps, music, games).</li>
<li>Google pay subscriptions and payment methods.</li>
<li>Accounts on external websites that you signed into via Google.</li>
</ul>
<p>That last point varies from provider to provider, some will let you relink by verifying your email address, but
others won’t work at all. You can review all third-party apps you <a href="https://myaccount.
google.com/permissions">logged into via Google here</a>.</p>
<p>I’m probably missing other things — but if any of these are important to you then this migration is not suitable.
You’re better off upgrading to the paid plan, or waiting for Google to roll out their migration tool.</p>
<p>I’ll refer to the old G Suite account as a “workspace” account here on out. The new plain old consumer Google
account will be called the “personal” account.</p>
<ol>
<li>
<p><strong>Start by backing up everything in your workspace account.</strong> You can do this via <a href="https://takeout.google.com/settings/takeout">Google Takeout</a>. It may take a day or two for them to generate the data depending on how much you have. Recommend storing these zip files somewhere safe.</p>
</li>
<li>
<p><strong>Review the services and data</strong> stored on your workspace account <a href="https://myaccount.google.com/dashboard">via this dashboard</a>. It gives you a nice overview of all your data and services used, which will come in handy as you decide how to migrate each one.</p>
</li>
<li>
<p><strong>Create a new browser profile</strong>, and sign up for a “plain” Google account, one with a @gmail.com suffix. Having both accounts logged in on two Chrome profiles is pretty handy as you follow the steps below.</p>
</li>
</ol>
<p>Now, we’ll start migrating data for each service one-by-one.</p>
<h2 id="gmail">Gmail</h2>
<p>This is arguably the most important service — and was the primary reason I signed for a workspace account all
those years ago.</p>
<p>We’ll do this process in three phases:</p>
<h3 id="redirect-incoming-mail">Redirect incoming mail</h3>
<p>If you use Google domains to manage your custom domain, this part is relatively easy. You can use the Google
domains <a href="https://support.google.com/domains/answer/3251241?hl=en">email forwarding feature</a> to redirect all mail
from your workspace email address into your personal one.
If you don’t use Google domains built-in DNS service, forwarding won’t work —
you will have to <a href="https://support.google.com/domains/answer/9428703?hl=en">update MX records at your DNS host</a>.
My DNS is run by Cloudflare, and updating my MX records to the ones listed on that support page worked well.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Name # Type # Priority # Value
@ MX 5 gmr-smtp-in.l.google.com
@ MX 10 alt1.gmr-smtp-in.l.google.com
@ MX 20 alt2.gmr-smtp-in.l.google.com
@ MX 30 alt3.gmr-smtp-in.l.google.com
@ MX 40 alt4.gmr-smtp-in.l.google.com
</code></pre></div></div>
<p>Send a test email from somewhere else to make sure you are able to receive email for your custom domain in the inbox of your personal account before proceeding.</p>
<h3 id="configure-outgoing-mail">Configure outgoing mail</h3>
<p>Now let’s make it so you can send email for your custom domain but from your personal account.</p>
<ol>
<li>On the gmail screen in your personal email, go to “Settings”, then “Accounts and Import”.</li>
<li>Click on “Add another email address” under “Send mail as”.</li>
<li>Put in the email of your workspace account, with “Treat as an alias” checked.</li>
<li>Enter <code class="language-plaintext highlighter-rouge">smtp.gmail.com</code> as the SMTP server, set port to <code class="language-plaintext highlighter-rouge">465</code>, and put in the credentials to your <em>personal</em> account in username and password.</li>
<li>This won’t work the first time, and you’ll be directed to enable “Less-secure” mode for your personal Google account, which <a href="https://myaccount.google.com/lesssecureapps">you’ll have to enable</a>.</li>
<li>Retry with your credentials again and it should work this time. You can now set this email as the default through the “make default” link.</li>
</ol>
<p>Try sending an email and ensure everything is working properly.</p>
<h3 id="migrate-all-your-old-mail">Migrate all your old mail</h3>
<p>There are a few ways to move all your email: using POP/IMAP is a popular option but has a drawback that you can’t migrate labels if you made heavy use of them.</p>
<p>I decided to use a custom tool — <a href="https://github.com/GAM-team/got-your-back">“Got Your Back”</a> — which uses the Gmail API and preserves labels.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># This is generally a pretty terrible way to install scripts, be warned</span>
bash <<span class="o">(</span>curl <span class="nt">-s</span> <span class="nt">-S</span> <span class="nt">-L</span> https://git.io/gyb-install<span class="o">)</span>
</code></pre></div></div>
<p>Start by downloading all your email from the workspace account:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># gyb installs into ~/bin by default</span>
~/bin/gyb <span class="nt">--email</span> <your-workspace-account@custom.domain>
</code></pre></div></div>
<p>This process will first ask for authorization to manage your Google cloud account (instructions provided when you run the command).
Once you’ve created the app, make sure to go into “APIs & Services” > “Oauth consent screen” and click “Make external”. You can make it external for just one “Test user”,
enter the email for your personal account here. This will be necessary later when you import your emails into the personal account.</p>
<p>After the cloud app is created and you’ve pasted in the requisite client ID & secret, GYB will request authorization to read your email. On this screen, it is sufficient
to grant just “readonly access”. GYB will then begin downloading your email — this process took around 2 hours for me, YMMV.</p>
<p>Once this is done, you can then upload the email to your new account:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/bin/gyb <span class="nt">--email</span> <your-personal-account@gmail.com> <span class="nt">--action</span> restore <span class="se">\</span>
<span class="nt">--local-folder</span> GYB-GMail-Backup-<your-workspace-account@custom.domain>
</code></pre></div></div>
<p>This process takes much longer (took around 14 hours for me). You can tend to moving your other services while this is happening, but you should see all your old email
start appearing in your personal inbox.</p>
<p>You can delete GYB from your cloud console when the process is complete.</p>
<h3 id="filters">Filters</h3>
<p>If you have a lot of filters, you can export them from the “Filters and Blocked Addresses” settings page. Select all the filters — or just ones you want to move — and click export.
You can then import this file into your personal account. I recommend doing this after GYB has finished uploading your new email and labels, as the filters will rely on them.</p>
<h2 id="calendar">Calendar</h2>
<p>Migrating this involves a simple export & import of your calendar events through a single file.</p>
<ol>
<li>From the workspace calendar, go to Settings (gear in the top-right), then “Import & Export”.</li>
<li>Click on “Export”, which will download a zip file.</li>
<li>Unzip the file and upload it in the “Import” section of settings on your personal account.</li>
</ol>
<p>Note that this will only copy the events, but not any linked users. You’ll also have to re-share any calendars imported if you had shared them previously.</p>
<h2 id="photos">Photos</h2>
<p>Your takeout zip file should contain all your photos which you can upload again. However, I found an easier way to accomplish this, by using the <a href="https://support.google.com/photos/answer/7378858">“partner sharing”</a> feature. On your workspace account, initiate a partner share from settings to your personal account and accept it on the other end. On the personal account, select the option to save all photos from your partner account into your library.</p>
<p>Once I did this, face recognition on my family and pets didn’t work out of the box. I had to disable and re-enable the feature from my personal account to get this to work. After about a day, the timeline and photos all appeared exactly as they did in the workspace account. You may disable partner sharing once all photos have been saved to the library on your personal account.</p>
<p>Keep in mind that any albums shared with your workspace account must also be shared to your personal account — manually and one at a time. This can be a time consuming process depending on how many albums were shared with you, but I found no way to automate this.</p>
<h2 id="drive">Drive</h2>
<p>I didn’t have much stuff in Google drive, so I ended up just uploading everything again manually. For any Google sheets, docs, or slides — just share them directly with your personal account. Google Drive does offer a <a href="https://www.google.com/drive/download/">desktop app</a> to make this process a bit easier.</p>
<h2 id="youtube">YouTube</h2>
<p>I was rather lucky in this regard. Back when Google asked all YouTube accounts to be migrated to a Google Plus account, the backlash was so immense that they quickly offered an option to keep your YouTube account separate (but linked) to your Google account. I remember taking advantage of this option, which has since come to be known as a <a href="https://support.google.com/youtube/answer/9367690?hl=en">“brand” account</a>.</p>
<p>If you’re in a similar situation as me — transferring your YouTube uploads, watch history, and playlists becomes somewhat simple. You need to add your personal account as an admin to your channel — doing this is not obvious. Go to <a href="https://studio.youtube.com/">studio.youtube.com</a>, click “Settings” in the left pane, then “Permissions” > “Manage Permissions”. This takes you to a page where you can invite your personal account as an “Owner” to your channel.</p>
<p>After 7 days, you will be able to switch the personal account from just “Owner” to “Primary Owner”. At this point, you can remove the workspace account, and still retain your YouTube account.</p>
<h2 id="google-fi">Google Fi</h2>
<p>Moving a Fi mobile subscription to a personal Google account is thankfully a <a href="https://support.google.com/fi/answer/6201840#zippy=%2Cwhat-to-do-if-your-admin-turns-off-google-fi">documented and supported</a> process. Painful, but doable.</p>
<p>Contact <a href="https://support.google.com/fi/gethelp">Fi support</a> and they’ll walk you through the steps. It generally involves verifying both your workspace and personal accounts. You can only do this if you have fully paid off your phone. If you have additional friends or family on your account, you’ll have to remove them from your plan (temporarily). This was the most painful part of the process as additional Fi subscribers on my account basically lost cell service for around 2 hours.</p>
<p>Once it is moved over, you keep your original phone number, billing, and service. At this point you can re-add your friends & family, everything should be back to normal.</p>
<h2 id="analytics">Analytics</h2>
<p>If you use Google analytics, you can add your personal account as an admin for any properties you created. By giving this account full admin privileges, you retain basically the same functionality as before.</p>
<h2 id="cloud">Cloud</h2>
<p>In case you use Google Cloud or Firebase, these services are also tied to your workspace account. Similar to Analytics, adding your personal account as admin on projects you wanted to keep was a simple way of retaining access.</p>
<h2 id="alerts-groups-keep">Alerts, Groups, Keep</h2>
<p>I found no way to migrate these services to the new account. I manually recreated alerts from my personal account, and re-subbed to the groups I was interested in. For Keep, the notes you had are available in the takeout file as plain text.</p>
<h2 id="android-phone">Android Phone</h2>
<p>The final step was to switch my Android phone to my personal account. You can login with <a href="https://support.google.com/googleplay/answer/2521798?hl=en">multiple Google accounts</a> on Android which was helpful. I first moved my WhatsApp backup, by uploading it to my personal account (you can also use <a href="https://faq.whatsapp.com/android/chats/how-to-restore-your-chat-history">local backup</a>).</p>
<p>After about a week of using my phone this way to make sure all data was moved over, I did a factory reset and logged back in with only my personal account. It’s been a couple of weeks using it that way and I haven’t had to go back to my workspace account for anything!</p>
<h2 id="closing-thoughts">Closing Thoughts</h2>
<p>Switching from workspace to a personal account was a time consuming yet insightful process. As more and more of your personal data moves to the cloud, you end up being beholden to a single company for your whole digital life. This can be scary, and using a custom domain is one of the important ways in which I’m able to retain some control over my digital identity. Going through this process made me build some confidence in my ability to move things over to a new provider should the need arise in the future.</p>
<p>While being able to download your data through services like Takeout is a helpful start, we are still a long way from true data portability. As the process above has outlined, it’s not just about access to your <strong>raw data</strong> but <strong>metadata</strong> that may be provider specific — such as comments on photos, filters & labels on your email, and your video uploads and watch history. I dream of a future where you are able to seamlessly store, control, and and move not just your raw data but also all digital interactions that you have had or others have had with content you create.</p>
<p>Google has been working on the <a href="https://datatransferproject.dev/">Data Transfer Project</a>, which Apple recently joined and includes contributions from Facebook, Microsoft, and Twitter. The project has similar goals but currently only works for moving Photos between a select few services. We shall see if this initiative will expand to more types of data in the future!</p>anantGoogle recently announced that all “G Suite legacy free edition” (formerly known as “Google Apps”, currently known as “Google Workspace”) accounts will need to transition to their paid workspace plans starting July 1, 2022. Legacy users will get access to a discounted rate of $3/user/month, which will turn into $6/user/month starting July 2023 at the lowest tier.Project Assemble Redux2020-05-24T00:00:00+00:002020-05-24T00:00:00+00:00https://www.kix.in/2020/05/24/project-assemble-redux<p>Last time I <a href="/2011/02/02/project-assemble/">posted</a> about building PCs was in 2011. <a href="https://proness.kix.in/misc/dream_comp.html">That PC</a> lasted me quite a while - 6 years - at which point it got an upgrade that I didn’t <a href="https://proness.kix.in/misc/dream_comp2.html">write about</a> (Intel 6700k on an Asus Z170, with a GTX 970). That build certainly held its own and even ran <a href="https://www.half-life.com/en/alyx/">Half-Life: Alyx</a> on a Rift just fine. But, the graphics card in particular is starting to show its age, and hey, with everyone stuck at home I figured it was time for another upgrade.</p>
<p>I haven’t really stopped using Macbook Pros for work - so my PC mostly gets used for gaming (and occassionally reusing my mouse/keyboard/monitor with a docked MBP, via <a href="https://symless.com/synergy">Synergy</a> for working on side projects). Gaming is usually synonymous with an Intel-based build (and all my builds so far have used their chips), however, I was pleasantly surprised to discover that AMD has been giving Intel a run for their money of late. Hooray for competition!</p>
<p>The <a href="https://www.theverge.com/circuitbreaker/2019/5/28/18642251/amd-ryzen-3000-cpus-3900x-3800x-3700x-3600x-3600-price-release-date-specs">Ryzen 3rd gen CPUs</a> are without a doubt the best bang for buck in the consumer CPU market. Even with <a href="https://www.anandtech.com/show/15758/intels-10th-gen-comet-lake-desktop">Intel’s 10th generation chips</a> launching, which bring market-leading raw clock speed performance, they still can’t match AMD’s price point to core count ratio. While Intel still dominates the highest end gaming segment, doing almost anything else on your computer (like streaming, multi tasking in your browser, or writing code) means that AMD pulls ahead of Intel quite handily at the cost of slightly lower gaming performance.</p>
<p>On the graphics card end, NVIDIA is still king of the hill. Wish there were more competition here, but the <a href="https://en.wikipedia.org/wiki/GeForce_20_series">Turing RTX 20 series</a> are the best consumer cards in market, and with the new Ampere architecture expected to launch with the <a href="https://www.tomshardware.com/news/nvidia-rtx-3080-ampere-all-we-know">RTX 30 series cards</a> later this year, there are no signs of them slowing down.</p>
<p>After a couple days of research, I settled on the final <a href="https://proness.kix.in/misc/dream_comp3.html">build spec</a>.</p>
<p><a href="/images/2020/pc-parts.jpg"><img src="/images/2020/pc-parts.jpg" alt="PC Parts" /></a></p>
<h2 id="cpu">CPU</h2>
<p><a href="https://www.amazon.com/gp/product/B07SXMZLPK/">Ryzen 3700x</a>. This is an 8-core/16-thread processor at a base clock of 3.6GHz but almost always runs at 4GHz by default, and can boost up to 4.4GHz occasionally. I probably could have gotten away with a 3600x just fine, but this buys me a smidge of future proofing given the graphics card I was going to pair with.</p>
<h2 id="motherboard">Motherboard</h2>
<p><a href="https://www.amazon.com/gp/product/B07SVRZGMX/">Gigabyte x570 Aorus Elite</a>. Again, probably could have gotten away with the older 450-series motherboard, but the 570 comes with PCIe Gen 4 and USB 3.2 support. Not that much more expensive either.</p>
<p><a href="/images/2020/motherboard-and-cpu.jpg"><img src="/images/2020/motherboard-and-cpu.jpg" alt="Motherboard & CPU" /></a></p>
<h2 id="ram">RAM</h2>
<p><a href="https://www.amazon.com/gp/product/B07WTS8T2W/">G.Skill 2x16GB @ 3600MHz</a>. AMD builds have a reputation for being somewhat finicky with RAM setup. The Ryzen 7 is basically built to take advantage of 3600Mhz DDR4 and this kit is widely used with no known compatibility problems. 3200Mhz will also work just fine if you want to save a little.</p>
<p><a href="/images/2020/ram.jpg"><img src="/images/2020/ram.jpg" alt="RAM" /></a></p>
<h2 id="gpu">GPU</h2>
<p><a href="https://www.amazon.com/gp/product/B07VFKM4VQ/">Asus ROG Strix RTX 2080 Super</a>. The RTX 20 “Super” series is great value for money and beats every non “super” card (except for the very high-end 2080Ti). If you’re gaming at 1080p, a 2070 super is of great value, but I was going for a 1440p monitor (more on that below) and felt the 2080 would last me longer at that resolution.</p>
<p><a href="/images/2020/gpu.jpg"><img src="/images/2020/gpu.jpg" alt="GPU" /></a></p>
<h2 id="power-supply">Power Supply</h2>
<p><a href="https://www.amazon.com/gp/product/B07WDLTKNM/">EVGA Supernova 650 G5</a>. Something is off with the PSU supply chain, with most units sold out at all major retailers. Not sure if this is COVID-19 related or otherwise, but buying a PSU right now is mostly just a function of what you can get. My first choice was a Corsair, but this EVGA had “OK” reviews and was available. A few days after I ordered, a bunch of Seasonic units were back in stock which might have been my second choice. One interesting thing about power draw over the years is that they have reduced: SLI is no longer in vogue and every component has just gotten more efficient. The units are also “modular” now, which means you only use the cables you actually need (this is a bigger deal than average for me, my previous full size “high air flow” Cooler Master tower with a non-modular power supply had a really bad dust problem).</p>
<h2 id="hard-drive">Hard Drive</h2>
<p><a href="https://www.amazon.com/gp/product/B07TJX83W2/">Aorus NVMe Gen4 M.2 2TB</a>. Ok, I’ll admit this drive is somewhat of an overkill, but if I’m getting a Gen 4 compatible CPU and Motherboard, why not? These are some blazing fast read/write speeds, that you’ll likely only notice with some very disk heavy workload (like compiling Firefox?). This is my first M.2 drive, and I quite like that it fits right on your motherboard. Very happily kissed my old set of SSDs and spinning drives goodbye!</p>
<p><a href="/images/2020/m2.jpg"><img src="/images/2020/m2.jpg" alt="M2 SSD" /></a></p>
<h2 id="tower">Tower</h2>
<p><a href="https://www.amazon.com/gp/product/B074DQVB97/">Fractal Meshify C</a>. Given nobody uses optical drives anymore, it just makes sense to go for a compact mid-tower case unless you’re planning to go crazy with expansion and storage. I liked the look and feel of the Meshify series, and Fractal are known for their great cable management and generally high quality cases. I’m not crazy about RGB in my build, so the dark tinted tempered glass it comes with works pretty well.</p>
<h2 id="monitor">Monitor</h2>
<p><a href="https://www.microcenter.com/product/484980/dell-alienware-aw3418dw-341-uw-qhd-120hz-hdmi-dp-g-sync-curved-ips-led-gaming-monitor">Alienware AW3418DW</a>. This monitor is what really started the whole upgrade idea in my head, as I happened to find a really good deal at $750 (it usually retails for around $999). Figured it was time to embrace the ultrawide 1440p experience!</p>
<h2 id="peripherals">Peripherals</h2>
<p><a href="https://www.amazon.com/gp/product/B07SXX7P6D/">Kinesis Gaming Freestyle</a> keyboard, and <a href="https://www.amazon.com/gp/product/B07BMGTR6D/">Kinesis Gaming Vektor</a> mouse. I use a <a href="https://kinesis-ergo.com/shop/advantage2/">Kinesis Advantage</a> for work, and really love their gaming products too. High quality, reliable hardware, what more to say?</p>
<h2 id="assembly">Assembly</h2>
<p>Actual assembly was easier than ever, with everything living on the board itself, cable management was trivial and the whole thing was up and running in just a couple of hours. I made one small change, which was to swap out the Wrath Prism cooler included with the CPU for a <a href="https://www.amazon.com/gp/product/B07H25DYM3">Hyper 212 (Black Edition)</a>. The prism cooler worked fine from a thermal perspective, it was just too noisy for my taste.</p>
<p><a href="/images/2020/final-build.jpg"><img src="/images/2020/final-build.jpg" alt="Final Build" /></a></p>
<p>Final touches were on the actual desk setup. I had to figure out how to make use of my two old monitors alongside the new ultrawide. Decided to change out my desk to one <a href="https://www.amazon.com/gp/product/B000W8I1D8">that would fit</a> all the monitors side-by-side with a couple of monitor arms. This is how it all came together:</p>
<p><a href="/images/2020/final-setup.jpg"><img src="/images/2020/final-setup.jpg" alt="Final Setup" /></a></p>
<p>The PC’s been running smoothly and performed as expected on the benchmarks, though it runs a little hotter than I’m used to. Over a sustained gaming session, both the CPU and GPU stay just a little below 70°C which feels nominal, so I’m not too worried.</p>
<p>It felt like the whole PC building process has gotten much smoother over time, and the average consumer is lot more informed (YouTube has thousands of videos on the topic these days). Definitely a great time to get into it as a hobby if that’s something you’ve always wanted to try!</p>anantLast time I posted about building PCs was in 2011. That PC lasted me quite a while - 6 years - at which point it got an upgrade that I didn’t write about (Intel 6700k on an Asus Z170, with a GTX 970). That build certainly held its own and even ran Half-Life: Alyx on a Rift just fine. But, the graphics card in particular is starting to show its age, and hey, with everyone stuck at home I figured it was time for another upgrade.Introducing ThinMusic2018-12-29T00:00:00+00:002018-12-29T00:00:00+00:00https://www.kix.in/2018/12/29/introducing-thinmusic<p>At the peak of my career as a software engineer, I spent most of my free time either playing video games or reading books about engineering management. These days, my day job is mostly engineering management, and so I find myself carving out play-time to write some code (and of course, still indulge in video games).</p>
<p>A result of that play-time over this winter break merits broader sharing than my usual side project. I built a web player for Apple Music, <a href="https://www.thinmusic.com">called ThinMusic</a>, to scratch two of my itches:</p>
<ol>
<li>As an Apple Music subscriber, I had no way to play songs on my Linux desktop.</li>
<li>Scrobbling my play history on Apple Music to <a href="http://last.fm">last.fm</a> has never worked reliably.</li>
</ol>
<p>The latter point irked me quite a bit (given I’ve been scrobbling consistently since 2006 or so), but not enough to switch to Spotify which supports scrobbling natively (worth mentioning that Apple Music’s family plan is best in the industry especially with international family members, adding additional inertia).</p>
<p>However, at this year’s <a href="https://developer.apple.com/videos/wwdc2018/">WWDC</a>, Apple announced <a href="https://developer.apple.com/documentation/musickitjs">MusicKit JS</a> which was quite intriguing on its own, but also opened the doors to kill these two birds with one stone. Thus, ThinMusic was born:</p>
<p><a href="/images/2018/thinmusic.png"><img src="/images/2018/thinmusic.png" alt="ThinMusic Screenshot" /></a></p>
<p>ThinMusic requires an Apple Music subscription (and a Facebook account so it can store the authentication tokens securely). It supports all the basic features of a music player and works on any modern browser. It is also optimized for desktop use, since on mobile devices you’re probably better off with Apple Music’s native app (available on both iOS and Android). You can use it on mobile if you really want, but be warned the experience is not as good (mostly due to my laziness to optimize the layout and make a real responsive design).</p>
<p>As an added bonus, it appears this might be a good way to play songs on the <a href="https://portal.facebook.com">Portal</a>, since the <a href="https://support.apple.com/en-in/HT209250">Apple Music skill for Alexa</a> doesn’t work on it yet:</p>
<p><a href="/images/2018/thinmusic-portal.png"><img src="/images/2018/thinmusic-portal.png" alt="ThinMusic on Portal" /></a></p>
<p>Just open the browser app, navigate to <a href="https://www.thinmusic.co">thinmusic.com</a> and login. Since it is running inside the browser, there is no support for voice control (and who wants to type for extended periods on the Portal), but if you just want to queue up a playlist quickly, this setup can work pretty well.</p>
<p>If you’re an Apple Music subscriber, give ThinMusic a whirl and email <a href="mailto:support@thinmusic.com">support@thinmusic.com</a> with your questions or suggestions!</p>Anant NarayananAt the peak of my career as a software engineer, I spent most of my free time either playing video games or reading books about engineering management. These days, my day job is mostly engineering management, and so I find myself carving out play-time to write some code (and of course, still indulge in video games).Teaching Ozlo about Pokémon GO2016-08-04T00:00:00+00:002016-08-04T00:00:00+00:00https://www.kix.in/2016/08/04/ozlo-pokemon-go<p>Pokémon GO is all the rage these days. <a href="https://www.ozlo.com">Ozlo</a>, your friendly AI sidekick,
would be remiss if he didn’t help you catch them all!</p>
<p><a href="/images/2016/ozlo-pokemon.png"><img src="/images/2016/ozlo-pokemon.png" alt="Ozlo learns about Pokémon GO" /></a></p>
<p>Thanks to Ozlo’s unique, knowledge-based approach to the world, we were able to teach him about
Pokémon in just under a week, including how to find PokéStops and Pokémon Gyms near places you
might be going. In this blog post, we’ll take a look at some of Ozlo’s inner workings,
what goes into teaching him a completely new concept, and why his ability to learn quickly matters.</p>
<p>The process involves three high-level steps:</p>
<ul>
<li>Feeding Ozlo data about the new concept</li>
<li>Teaching Ozlo to understand how people talk about the concept</li>
<li>Teaching Ozlo how to talk to people about what it knows</li>
</ul>
<p>We’ll cover each of these steps one-by-one and then discuss why it’s important we do things this way —
and why that makes Ozlo fundamentally different than many other chatbots and AI assistants out there.</p>
<h2 id="data">Data</h2>
<p>Ozlo’s view of the world consists of <em>entities</em> (people, places, or things) and relationships among them.
Teaching Ozlo about something new begins with acquiring data about the subject that so we can augment
his knowledge of the world. This can happen by several means — crawling the web, hitting APIs,
and obtaining data from partners, for example.</p>
<p>In Pokémon GO’s case, we decided to focus on a use-case that helps you play the game effectively but
doesn’t break it or cheat in any way: finding PokéStops. PokéStops are places all around the world,
and they have certain attributes that identify them: coordinates, a name, picture and sometimes a description.</p>
<p>Once we found all the PokéStops in the US, we turned them into entities and started creating relationships.
Ozlo already knows about all the cities in the US as well as what landmarks and restaurants exist in each city.
With this knowledge, he can perform reasoning to know that if a given place is inside the polygon for a given
city’s boundary, then the place must be in the city (and so on…)</p>
<p>When this process concludes, Ozlo has a mental map of where all the PokéStops in the US are located,
which of them are “gyms”, what cities they are in, and which landmarks and restaurants they are near.</p>
<h2 id="understanding">Understanding</h2>
<p>Next, we had to teach Ozlo some of the common ways in which humans might ask him about PokéStops.
In the beginning that involves just writing out some examples and telling Ozlo what each of them mean.</p>
<p>Consider the following sentence, resembling something a human might ask Ozlo:</p>
<blockquote>
<p>“Show me pokemon gyms near the ferry building”</p>
</blockquote>
<p>There’s a lot in that sentence that Ozlo can already understand! He has a basic understanding of the English language,
but also knows how people talk about restaurants and landmarks (since we taught him that earlier). What does Ozlo see
in that sentence?</p>
<blockquote>
<p><em>“show me”</em>: Here’s a hint that the answer to this question requires some sort of visual presentation.</p>
</blockquote>
<blockquote>
<p><em>“near”</em>: I’ve seen this word many times before and when it is followed by a name of a place, I know what that means.</p>
</blockquote>
<blockquote>
<p><em>“ferry building”</em>: Looks like I have many entities that match this name. But, I can rank all the places with this
name by their popularity and distance from where the user currently location to narrow down a likely candidate.</p>
</blockquote>
<p>The only part of that sentence Ozlo didn’t quite understand was “pokemon gyms”. This is where we step in and give him
some examples along with what they mean:</p>
<blockquote>
<p><em>“pokestops”</em>: This means entities that are PokéStops</p>
</blockquote>
<blockquote>
<p><em>“pokemon gyms”</em>: This means entities that are PokéStops of type “gym”</p>
</blockquote>
<p>We also added many more variations of the above to give him a basic understanding of PokéStops. And don’t forget —
Ozlo also keeps learning as you use him — so he’ll collect a lot more examples over time than what we just start
him off with!</p>
<h2 id="presentation">Presentation</h2>
<p>The final step was to teach Ozlo how to turn his answer into words and interactions that humans can understand.
In many ways this is exactly the reverse of Ozlo trying to translate what a human said into terms he can understand.</p>
<p>Ozlo already has a good knowledge of English, so he can mostly construct the sentence on his own. We just need to give
him a few hints and we get:</p>
<blockquote>
<p>“There are many Pokémon Gyms around Ferry Building”</p>
</blockquote>
<p>Then we construct the visual format of the response. In our iOS app we settled on using the “multi-pin map” element,
which is an easy way to view several points of interest in a given geographic area. For now, we just tell Ozlo what
type of visual result format to use, based on the user’s device.</p>
<p><a href="/images/2016/ozlo-pokemon-mpm.png"><img src="/images/2016/ozlo-pokemon-mpm.png" alt="Ozlo learns about Pokémon GO" /></a></p>
<p>Ozlo’s capabilities aren’t limited to just rendering maps though - he can choose between a variety of output formats -
and we pick the one that’s best suited to the medium you’re using to communicate with him.</p>
<h2 id="why-this-matters">Why This Matters</h2>
<p>Why go to all this effort to actually teach Ozlo about PokéStops instead of just having Ozlo redirect your question
to some other service? We’ve talked about the <a href="http://venturebeat.com/2016/07/17/personal-assistant-bots-like-siri-and-cortana-have-a-serious-problem/">multi-agent problem</a> before — and we believe there is a fundamental
difference between bots that <strong>know</strong> things and bots that <strong>guess</strong> what other services might know about things.</p>
<p>As Ozlo’s knowledge of the world grows, adding more data to it enriches his entire world view. There’s a network effect
between entities — because these entities have relationships with each other — adding new entities has an exponential
effect on Ozlo’s understanding of the world. This is what lets us leverage the fact that Ozlo already knows about
“the Ferry Building” to help you find out what PokéStops are near it with only a minimal amount of effort.</p>
<p>We can’t wait for the day where we’re not the only ones teaching Ozlo about new concepts! In the meantime, please keep
using Ozlo and giving him feedback to help him continue to learn more about the world.</p>Anant NarayananPokémon GO is all the rage these days. Ozlo, your friendly AI sidekick, would be remiss if he didn’t help you catch them all!Meet Ozlo2016-05-12T00:00:00+00:002016-05-12T00:00:00+00:00https://www.kix.in/2016/05/12/meet-ozlo<p>Two days ago, a project I’ve been working on for a little over two years was unveiled to the world.
Meet Ozlo, your friendly AI sidekick!</p>
<p><a href="https://dribbble.com/shots/2704505-Meet-Ozlo"><img src="/images/2016/ozlo.gif" alt="Ozlo" /></a></p>
<p>First things first: if you haven’t signed up yet, hit up <a href="https://www.ozlo.com/?vip=ANANT">this link</a> which
includes a VIP code to fast-track you into our invite-only app.</p>
<p>A lot has been said about Ozlo already: <a href="https://medium.com/teamozlo/introducing-ozlo-d5cce73d7ba5">Charles Jolley</a> (co-founder),
<a href="https://news.greylock.com/our-investment-in-ozlo-a7f6eb9f61eb#.r1j0eeu8k">John Lilly</a> (investor), <a href="http://lloyd.io/meet-ozlo">Lloyd Hilaiel</a> (friend & colleague),
<a href="http://todd.agulnick.com/2016/05/11/what-ive-been-up-to-ozlo/">Todd Agulnick</a> (friend & colleague)
and even <a href="https://www.buzzfeed.com/alexkantrowitz/ozlo-the-ai-chatbot-wants-to-help-you-find-coffee-and-food">Buzzfeed</a>!
Here’s my perspective…</p>
<h2 id="why">Why</h2>
<p>It didn’t take me very long since I first heard the idea for a better mobile search experience from
<a href="https://twitter.com/michaelrhanson">Mike</a> <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> and <a href="https://twitter.com/okito">Charles</a>
to stop what I was doing and jump on board.</p>
<p>The fundamental problem we’re trying to solve is that even though our smart phones enable us to do a lot more than we could before,
the process of finding people, places and things on them is not very different from how you would do it on a desktop.</p>
<p>That’s usually the natural course for any technology to take.<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> The first application on any new platform usually is a v1 – “available here too” – product. This first version often under-utilizes
the platform’s true capabilities and its creators can quickly be lulled into thinking that they’ve created the optimal experience
for the consumer.</p>
<p>What’s v2 for search on mobile devices? To answer this question is why we created Ozlo.</p>
<h2 id="what">What</h2>
<p>In attempting to answer this question, we built something that we thought might work. It didn’t work quite as well
as we’d have liked. So we did it again. And again. Fast-forward two years and you arrive at Ozlo: a <strong>personal</strong> and <strong>intelligent</strong>
companion that <em>helps you find things</em>.</p>
<p>The first manifestation of that idea is an <a href="http://ozlo.com/download">iOS application</a> that can help you find food.
In the app, you interact with Ozlo via a chat-like interface. Here I am trying to find that place that I can’t quite
remember the name of:</p>
<video controls="" autoplay="" loop="">
<source src="/images/2016/indian-pizza.mp4" type="video/mp4" />
</video>
<p>This iteration of the app is purposely focused on one goal – finding you food. But there are several underlying themes
that have the potential to pave Ozlo’s way to something grander:</p>
<h3 id="conversational">Conversational</h3>
<p>Searching for something is usually not a one-shot type of activity. Humans don’t work that way. We ask a question,
and often follow up with more questions; until we’ve refined our own thoughts to ultimately get the answer we’re
looking for. It’s exciting that Ozlo has the potential to participate in this back-and-forth.</p>
<h3 id="personal">Personal</h3>
<p>Ozlo has the potential to know you over time, learn about your preferences and interests in a meaningful way.
To me, this brings a face to the otherwise utilitarian search box that feels disconnected and impersonal.</p>
<p>As a vegetarian, I can already appreciate Ozlo helping me find hidden gems at restaurants I’d usually dismiss.
What if Ozlo could also recommend movies for me to watch, grab that hard to get restaurant reservation
and help me find the perfect anniversary gift?</p>
<h3 id="intelligent">Intelligent</h3>
<p>In the past few years, technology seems like it’s finally getting to the point where building an agent that can
really <em>understand</em> what humans say is tantalizingly close to being possible.<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup></p>
<p>Ozlo is different from usual search engines, the ones that return results with the same words as your query,
without knowing what the words mean. Ozlo tries to understand what you said and then tries to arrive at an answer.
To me, that makes Ozlo intelligent.</p>
<p>Training Ozlo to understand the nuances of human language is going to be a very difficult task. But it is by no
means impossible, given the resources we (as computer engineers and scientists) have at our disposal these days.</p>
<h2 id="how">How</h2>
<p>The really interesting bits are in the technology behind Ozlo and how we built it. This is some of the deepest
technology I’ve ever had a part in building and I’m extremely proud of it. To make Ozlo work, we’ve had to write
several pieces of software from scratch.</p>
<p>On the backend:</p>
<ul>
<li><em>Data Pipeline</em>:<br />to ingest, dedupe and glean structure from the mess of data we find; at scale; with speed.</li>
<li><em>Search Engine</em>:<br />to index the facts our data pipeline emits and allow us to efficiently query it; at scale, with speed.</li>
<li><em>Query Understanding</em>:<br />to turn human language into a series of structured queries machines can understand.</li>
<li><em>Dialog System</em>:<br />to keep track of the high-level structure of the conversation you’re having with Ozlo.</li>
</ul>
<p>On the frontend:</p>
<ul>
<li><em>Language Synthesis</em>:<br />to turn structured results back into friendly text humans can understand.</li>
<li><em>Layout Language</em>:<br />to efficiently and generatively render results as a graphical layout.</li>
<li><em>View Synthesis</em>:<br />to aggregate, refine and generate the final layout humans will see.</li>
<li><em>iOS App</em>:<br />to turn that layout back into pixels that are delightful to look at and interact with.</li>
</ul>
<p>We built most of our backend in Go. It’s no secret that I’ve been a <a href="https://www.kix.in/2009/11/11/go-why-i-e29da4-google/">fan of Go</a>
since its inception, primarily because of my affinity to Plan 9; but this is the first time I’ve been able to observe it being used at a
large scale for a production-quality project. I couldn’t be happier with our choice, and I’ll admit that I’ve had some days where I
get into work only because I’m excited by the prospect of writing some Go.</p>
<p>We built most of our frontend in NodeJS (and ObjC for the iOS app, of course). It’s also no secret that I’ve been a huge
<a href="https://www.google.com/#safe=off&q=site:kix.in+javascript">proponent of Javascript</a> and our frontend has been chugging along
happily (we’ve had a few refactorings, but really, what JS code base doesn’t go through atleast two?). Say what you will
about JS & NPM, especially in the recent past, one cannot deny the convenience and speed of development that is offered by the
JS runtime.</p>
<p><a href="/images/2016/ozlo-cityscape.png"><img src="/images/2016/ozlo-cityscape.png" alt="Ozlo Cityscape" /></a></p>
<p>I hope you share my excitement about Ozlo. Looking forward to whatever comes next!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>John calls Mike an <a href="http://techcrunch.com/2012/08/09/mike-hanson-joins-greylock-as-eir/">“anytime, anything, anywhere”</a> person, and it couldn’t be truer. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Take publishing for instance – when tablets were first introduced – a publication’s first instinct was to just take what they had on paper and turn it into pixels. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>We’ve observed the resurrection of the term “AI” to refer to this sort of thing. It’s often an overloaded term, but there is no doubt that the industry as a whole has made big technological strides in deep learning and machine intelligence. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Anant NarayananTwo days ago, a project I’ve been working on for a little over two years was unveiled to the world. Meet Ozlo, your friendly AI sidekick!Amazon Echo2015-04-26T00:00:00+00:002015-04-26T00:00:00+00:00https://www.kix.in/2015/04/26/amazon-echo<p>I received my <a href="http://www.amazon.com/oc/echo/">Amazon Echo</a> recently. I ordered it merely as a curiosity and to generally stay aware of industry trends. But after just a few days of using it at home; I love it enough to prompt dusting off this blog after almost two years of no posts!</p>
<p>The first thing that caught my attention was the sheer accuracy of its voice recognition. The state of the art is already pretty good; the dual pillars of cheap, persistent computing power (i.e., “the cloud”) and renewed interest in machine learning brought us Siri (and their Microsoft & Google equivalents). In my experience, these have been accurate more than 95% of the time, a long way ahead from the days of offline speech recognition (à la <a href="http://www.nuance.com/dragon/index.htm">Dragon Naturally Speaking</a>). The bar is already high, but I’m comfortable saying Amazon Echo’s voice recognition is definitely better than Siri’s or Google’s.</p>
<p>I’m not sure how they pull it off. Maybe it’s not because of better software, but simply better hardware. The Echo has an array of 7 microphones that are always listening and wake up as soon as you say “Alexa”…</p>
<p>There is an element of genius in packaging it as a standalone cylinder that sits in a somewhat central location in your home. The experience of being able to talk to it, hands-free, from a wide range of places in your home sounds like it wouldn’t be that big a deal… until you actually do it. Then it makes having to find your phone, pick it up, and push a button to make it do something seem archaic and boring.</p>
<p>Now, the only problem is that even though Echo understands what I’m saying, it doesn’t always know how to respond. The companion app often shows an accurate transcription of what I said, but since it did not fall into one of the categories it’s designed to handle at the moment, I don’t get a satisfactory response.</p>
<p>But there are relatively easy ways to fix that. The <a href="https://developer.amazon.com/public/solutions/devices/echo">Echo API</a> is currently invite-only, like the ability to purchase the device itself. As more developers get their hands on it, we’ll start to see many interesting things happen. Some developers already <a href="http://hackaday.com/2014/12/24/home-automation-with-the-amazon-echo/">hacked their way</a> into <a href="http://blog.zfeldman.com/2014-12-28-using-amazon-echo-to-control-lights-and-temperature/">controlling their home lighting</a> (Amazon now officially supports integrations with Philips Hue). I’m itching to make it control my home theater system, currently riddled with half a dozen remotes.</p>
<p>The idea of having a single hub in your home that’s always listening and can control all your other devices is incredibly appealing to me. For one, it means that your TV, Xbox, Nest, and other home devices don’t have to build in (often bad) voice recognition systems themselves.</p>
<p>Finally, I was impressed by how handily the Echo passed Larry Page’s “<a href="http://www.businessinsider.com/larry-page-toothbrush-test-google-acquisitions-2014-8">toothbrush test</a>”. Even with its somewhat limited functionality, my wife and I have already used it several times every day since we received it — for things ranging from compiling a shared grocery list, adding reminders, setting alarms, and playing music.</p>
<p>I think Amazon is onto something big. Priced at an aggressive $99, the Echo has the potential to make it into the living room of a large majority of households in the US (and the world, eventually). Will we now see a flurry of competitors from tech giants and startups?</p>
<p>Exciting times!</p>Anant NarayananI received my Amazon Echo recently. I ordered it merely as a curiosity and to generally stay aware of industry trends. But after just a few days of using it at home; I love it enough to prompt dusting off this blog after almost two years of no posts!