Adding search to my static blog.

window.documents = [
  
    {
      "id": "0",
      "title": "Jev can't be calibrated",
      "url": "/2026/09/23/jev-cant-be-calibrated.html",
      "date": "September 23, 2026",
      "excerpt": "Unless you’ve been living under a rock, you’ve probably heard about Jev. Simon Willison’s post is a good overview, and this one shows how to implement it in a few...",
      "content": "Unless you've been living under a rock, you've probably heard about Jev. [Simon Willison's post](https://simonwillison.net/2026/Sep/21/jev/) is a good overview, and this [one](https://www.nobodywho.ai/posts/jev-in-25-lines/) shows how to implement it in a few lines of Python. In short, Jev is TypeSafe's first \"[System One Model](https://typesafe.ai/blog/introducing-system-one-models-and-jev)\": instead of generating text, it takes unstructured input and returns typed decisions from a set of outputs you define in advance, each with a probability attached. One of its selling points is that \"all answers are accompanied with calibrated probabilities and confidence scores\". In this post I argue that Jev is useful, but that the calibration claim can't hold in general, and that you should treat its outputs as scores rather than probabilities.\n\n## Useful without training data\n\nYou can throw it at any classification problem without collecting training data first and get reasonable results. Many people on Twitter said, \"This is just a fine-tuned BERT\", but fine-tuning a BERT requires data. If you don't have data, Jev is a great alternative. Also, Jev is a **universal classifier**, whereas a fine-tuned BERT is only useful for the task it has been trained for. However, this comes with a cost, and being useful without data is exactly why its probabilities can't be calibrated for you\n\n\n## Uncalibrated probabilities\n\nTypeSafe says Jev was trained using [RLCD](https://docs.typesafe.ai/introduction/machine-learning-primer#rlcd-and-calibrated-decisions) (reinforcement learning for **calibrated** decisions), and that the probabilities it produces are calibrated. I don't think that's true. A model can be calibrated on TypeSafe's data and still be miscalibrated on yours.\n\nA model is calibrated when for any predicted probability $p$ the true probability of the positive class given that prediction is $p$. This is $P(Y = 1 \\mid \\hat{p} = p) = p$, where $\\hat{p}$ is the probability predicted by the model. Intuitively it means if you group all the instances where your model predicts X% then approximately X% of those actual cases turn out to be true (eg: if you take the emails where Jev said `spam_probability=0.7` you should expect about 70% of them to actually be spam). \n\nThe important point is that calibration is not just a property of the model, but also of your data distribution. The same model can be calibrated on one dataset but not on another. Different companies can define spam the same way but have different data distributions. However, for the same input and prompt, Jev will provide the same probabilities to both companies, regardless of their different underlying data distributions. Therefore, the model may be calibrated for one company but not for the other. Even if RLCD successfully trains Jev to be calibrated on its training/evaluation distribution, its probabilities may not remain calibrated on your production distribution. \n\nThere is some evidence the failure is worse than just a distribution shift. While writing this post I found these [tweets](https://x.com/shreyshahi/status/2102211285605830744), where Jev says a fair coin lands heads with probability 0.92. That is worse than the drift explained above. The true probability is in the prompt, and the model still does not report it. This [recent experiment](https://www.distillabs.ai/blog/jev-or-a-fine-tuned-small-model-we-built-a-pipeline-with-both-to-see-the-real-difference/) also finds that `Noul` is much better calibrated than `Choice` on the same problem. If Jev's probabilities have different semantics depending on which primitive I use, what exactly do \"calibrated probabilities\" mean?\n\nIf you want calibrated probabilities you'll still need to recalibrate Jev's probabilities on your own data. The good news is that is [cheap](https://scikit-learn.org/stable/modules/calibration.html). A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev's scores.\n\nMy take is to treat Jev's outputs as good scores (they rank examples well) rather than good probabilities. If your system depends on the actual number, like thresholds, expected costs or combining it with other models, measure calibration on your data before trusting it.\n",
      "tags": ""
    },
  
    {
      "id": "1",
      "title": "Bayesian A/B testing is not immune to peeking",
      "url": "/2025/10/30/bayesian-ab-test-peeking.html",
      "date": "October 30, 2025",
      "excerpt": "Introduction Over the last few months at RevenueCat I’ve been building a statistical framework to flag when an A/B test has reached statistical significance. I went through the usual literature,...",
      "content": "## Introduction\nOver the last few months at RevenueCat I've been building a statistical framework to flag when an A/B test has reached statistical significance. I went through the usual literature, including Evan Miller's posts. In his well known [\"How Not to Run an A/B Test\"](https://www.evanmiller.org/how-not-to-run-an-ab-test.html) there's a claim that with Bayesian experiment design you can stop at any time and still make valid inferences, and that you don't need a fixed sample size to get a valid result. I've read this claim in other posts. The impression I got is that you can peek as often as you want, stop the moment the posterior clears a threshold (eg $P(A>B) > 0.95$), and you won't inflate false positives. And this is not correct. If you're an expert in Bayesian statistics this is probably obvious, but it wasn't for me. So I decided to run some simulations to see what really happens, and I'm sharing the results here in case it can be useful for others.\n\n## Simulation and results\nPicture the following scenario\n- Baseline and variant have the same conversion rate $r$ (ie, no difference between them).\n- Generate events following a Bernoulli distribution with parameter $r$, and update your prior using Bayes theorem ([formulas here](https://www.evanmiller.org/Bayesian-ab-testing.html)), starting from an uninformative prior, eg $\\text{Beta}(1, 1)$.\n- After every $N$ events, compute the posterior probability that one branch is better than the other (ie $P(B > A)$ or $P(A > B)$). If any of them exceeds $0.95$, declare a winner.\n- Since baseline and variant have the same conversion rate $r$ declaring a winner is a false positive.\n- Repeat for different peek intervals $N$ and conversion rates $r$.\n\nI ran the simulations for different conversion rates $r \\in \\left(0.1\\%, 1\\%, 10\\%\\right)$ and peeking intervals $N \\in \\left(10^2, 10^3, 10^4, 10^5, 10^6\\right)$ and the results are consistent. In the following plot, we see how the error rate increases as the $N$ decreases. \n\n\n{% include image.html path=\"/docs/bayesian-ab-test-peeking/miss_rate_vs_peeking.png\" width=\"500\" %}\n\nThe plot show that as $N$ gets smaller, the error rate climbs. For example, if we peak after every $100$ observations the false positive rate increases to $80\\%$. In other words, even when using a Bayesian approach, more frequent peeking increases the chance of calling a winner when there's none.\n\nThe conclusion is then: *Bayesian posteriors remain interpretable under continuous monitoring, but a fixed posterior threshold does not control false positives when you peek and stop on success*\n\n## What can you actually do with Bayesian AB testing?\n\nAs we just saw, Bayesian AB testing is not immune to peeking. The more you peek the higher the false positive rate. But then, what advantage does it have over frequentist methods? \n\nThe key advantage is that results are interpretable at any sample size. This means you can monitor continuously and interpret $P(B > A)$ as the posterior probability that $B$ is better at the time you look. That interpretation is valid without a fixed sample size. However, if you want to control a frequentist error rate while peeking, a fixed $95\\%$ posterior threshold will not do it.\n\nWhat you *cannot* do is check daily and stop the moment you see favorable results. That's optional stopping based on the outcome, and it inflates false positives just like in frequentist tests. The Bayesian advantage is that your posterior remains interpretable at any sample size. But \"peek as often as you want and stop when you like\" is a myth.\n",
      "tags": ""
    },
  
    {
      "id": "2",
      "title": "Adding search to my static blog.",
      "url": "/2025/08/11/search-with-lunr.html",
      "date": "August 11, 2025",
      "excerpt": "I finally added search to this blog 1.\n\n\n  \n    \n      Yes, I admit it, I vibe coded it. ↩\n    \n  \n\n",
      "content": "I finally added search to this blog [^1].\n\nThis is cool because it keeps searches private and offline once the page loads. No third‑party boxes, no tracking, just a tiny index and a small script. So I can keep hosting my blog on Github Pages and have a search service for free.\n\nCredit where due: I was inspired by [Vicki's post on client‑side search with Lunr](https://vickiboykis.com/2025/08/08/enabling-hugo-static-site-search-with-lunr.js/). She did it on Hugo; I adapted the approach to Jekyll. In Jekyll, generating the data files that Lunr needs is trivial with Liquid and can be done at build time, which made this especially easy.\n\nPress Cmd+K (or Ctrl+K on Windows/Linux) anywhere, or click the search link in the navbar. It’s fast, works offline once loaded, and doesn’t send your queries to any server.\n\n## What I built\n- **Client-side search**: powered by Lunr.js (BM25 ranking, stemming, stop-words).\n- **Index at build time**: Jekyll emits a compact document list that Lunr indexes in the browser.\n- **Simple UI**: a modal with keyboard navigation, result count, and highlighted snippets.\n\nThis gives quick results, sensible ranking, and zero backend complexity.\n\n## How it works\n\nThe goal is to keep everything static. Jekyll builds the data; the browser builds the index and handles queries. No services, no endpoints.\n\n### Build-time index (Jekyll → JSON-like docs → Lunr)\n\nAt build time, Jekyll + Liquid loops over `site.posts` and emits a tiny JavaScript array of post metadata and plain text. That array ships with the page as `window.documents`. On page load, Lunr reads those docs and builds the index entirely in your browser. No endpoints, no JSON file required (though you can also output `/search.json` if you prefer).\n\nHere you can see an example of the documents that Jekyll generates at build time.\n\n\nDocuments emitted at build time\n\n{% highlight html %}\nwindow.documents = [\n  {% for post in site.posts limit:5 %}\n    {\n      \"id\": \"{{ forloop.index0 }}\",\n      \"title\": {{ post.title | jsonify }},\n      \"url\": \"{{ post.url | relative_url }}\",\n      \"date\": \"{{ post.date | date: '%B %d, %Y' }}\",\n      \"excerpt\": {{ post.excerpt | strip_html | truncatewords: 30 | jsonify }},\n      \"content\": {{ post.content | strip_html | jsonify }},\n      \"tags\": {{ post.tags | join: \" \" | jsonify }}\n    }{% unless forloop.last %},{% endunless %}\n  {% endfor %}\n];\n{% endhighlight %}\n\n\n\nNow, using the information in `window.documents` we can build the index using Lunr. The next snippet show how to do it.\n\n```html\n\n\n```\n\nWhat’s happening here:\n- `this.ref('id')` sets the unique identifier for each document. Lunr returns this ref on search so we can map back to `window.documents`.\n- `this.field(...)` declares which fields to index and their relative importance via boosts (title > tags > excerpt > content).\n- `window.documents.forEach(... this.add(doc) ...)` feeds each post into the index. Passing `this` preserves the Lunr builder as the function context.\n\nThe result is an in‑memory index built once on page load. We query it with `window.searchIndex.search(query)` and then hydrate results from `window.documents`.\n\nThis keeps the site fully static: Jekyll renders the list, the browser builds the index. Compared to Hugo, this felt simpler for me: Liquid makes it easy to shape the data inline without extra templates or generators, and I don’t need a separate JSON step unless I want one. It also fits my hosting: GitHub Pages serves static files only, so I can’t add a server-side search service even if I wanted to.\n\n### Searching and serving results\nWhen the user types, I run a Lunr query and map the top results back to my `documents` list. Then I render a small card with title, date, and a contextual snippet with the matched terms highlighted.\n\n```javascript\nfunction performSearch(query) {\n  if (!query.trim()) return [];\n  const matches = window.searchIndex.search(query);\n  return matches.slice(0, 10).map(m => {\n    const doc = window.documents[parseInt(m.ref)];\n    return { ...doc, score: m.score };\n  });\n}\n```\n\n### Keyboard shortcut and modal\nSmall touches make it feel native: a quick modal, Cmd+K/Ctrl+K to open, arrows to navigate, Enter to jump.\n\n```javascript\n// Open with Cmd+K / Ctrl+K\ndocument.addEventListener('keydown', (e) => {\n  if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === 'k') {\n    e.preventDefault();\n    openSearchModal();\n  }\n});\n\nfunction openSearchModal() {\n  const modal = document.getElementById('searchModal');\n  const input = document.getElementById('searchInput');\n  modal.style.display = 'block';\n  input.focus();\n}\n```\n\nThe UI also supports ESC to close, arrows to navigate, and Enter to open the selected result.\n\n## Why Lunr\nI didn’t want a service. Lunr is a tiny dependency that gives me real information‑retrieval behavior without infrastructure.\n- **Ranking that works**: BM25 beats my past ad‑hoc scoring.\n- **Zero deps at runtime**: no backend, no external service.\n- **Small & portable**: a single script and a tiny doc list.\n\n## Try it\n- Tap Cmd+K / Ctrl+K, or click the **search** link above.\n- Type anything you remember from a post: a tag, a phrase, a topic.\n\nIf something feels off (results, snippet, ranking), tell me—I’ll tweak the fields/boosts or snippets.\n\n## Next steps\n\nI'd like to add semantic search on client side too. We can compute embeddings at build time using a small embedding model (MiniLM/SentenceTransformers). Then embed the query at runtime and rank posts by cosine similarity.\n\n---\n\n[^1]: Yes, I admit it, I vibe coded it.",
      "tags": "javascript jekyll"
    },
  
    {
      "id": "3",
      "title": "Who needs git when you have 1M context windows?",
      "url": "/2025/07/28/unexpected-benefit-llm.html",
      "date": "July 28, 2025",
      "excerpt": "Lately I’ve heard a lot of stories of AI accidentally deleting entire codebases or wiping production databases. But in my case it was the other way around. I removed some...",
      "content": "Lately I’ve heard a lot of stories of AI accidentally deleting entire codebases or wiping production databases. But in my case it was the other way around. I removed some working code and the LLM helped me to recover it.\n\n\n\nI joined RevenueCat a couple of months ago to work on LTV predictions. My first few projects were straightforward: fix small things, ship some low-hanging fruit. After getting familiar with the code, I decided to extend the LTV model and add some cool machine learning. So I dove in. Spent a couple of days wrangling data, cleaning it, understanding what was going on, and the usual standard pre-training stuff. I was in “research mode”, so all my code lived in notebooks and ugly scripts. But after enough trial and error, I managed to improve the main metric by 5%. I was hyped. Told the team. Felt great.\n\nThen came the cleanup. I refactored all the sketchy code into a clean Python package, added tests, formatted everything nicely, added type hints, and got it ready for production. Just before opening the PR, I ran the pipeline again to double-check everything… and the results had dropped by 2%.\n\nOh shit… My ML model was now making worse predictions than the original model… And I never committed the changes that got me the +5%. Noob mistake. My teammate wasted no time laughing at my blunder\n\n\n\n  \n  My colleague having a good fun\n\n\nI spent the next few days trying to reproduce the original results, but it was impossible. Whatever secret sauce I’d stumbled on was gone. Then the weekend came and I went to the beach with my kids, and while making sand-castles I had an epiphany. I wasn’t working alone while developing the ML model. There was someone else helping me: my good old friend gemini-2.5-pro, with an incredible 1M token context window, was there all the time. Maybe, just maybe, it remembered. On Monday, after a great and relaxing weeked, I opened Cursor and asked\n\n\n  give me the exact original file of ml_ltv_training.py i passed you in the first message\n\n\nBoom. There it was, the original script that gave me the +5% uplift. An unexpected benefit of long-context LLMs. Who needs git best practices when you have an LLM that remembers everything?\n",
      "tags": ""
    },
  
    {
      "id": "4",
      "title": "Semantic Unit Testing",
      "url": "/2025/04/09/semantic-unit-testing.html",
      "date": "April 09, 2025",
      "excerpt": "Left Wallapop a couple of weeks ago, heading to RevenueCat soon. In that classic ‘between jobs’ hacking window, I built suite: a Python library for semantic unit testing. What’s semantic...",
      "content": "Left Wallapop a couple of weeks ago, heading to RevenueCat soon. In that classic ‘between jobs’ hacking window, I built suite: a Python library for semantic unit testing. What’s semantic unit testing? Think unit tests that understand context and meaning, not just assert obj == expected. Sound interesting? I’ll break down what semantic unit testing is, how suite works under the hood, and how you can integrate it.\n\n\n\nWhat’s semantic unit testing?\n\nSemantic unit testing is a testing approach that evaluates whether a function’s implementation aligns with its documented behavior. The code is analyzed using LLMs to assess if the implementation matches the expected behavior described in the docstring. It’s basically having an AI review your code and documentation together to spot discrepancies or bugs, without running the code.\n\nThis is, instead of writing classic unit tests with pairs of (input, output), the testing responsibility is passed to an LLM. The hypothesis is that a powerful model, with good enough context, should be able to detect bugs without having to run the code. The idea is that an LLM can analyze the code and its documentation much like a human developer would -carefully reading and thinking hard about it- but much more quickly.\n\nI wrote a package for doing semantic testing called suite (Semantic UnIt TEsting). You can install it using uv as 1\n\nuv pip install suite\n\n\nHere’s an example of how to perform a basic semantic test with suite\n\nfrom suite import suite\n\ntester = suite(model_name=\"openai/o3-mini\")\n\ndef multiply(x: int, y: int) -> int:\n    \"\"\"Multiplies x by y\n\n    Args:\n        x (int): value\n        y (int): value\n    Returns:\n        int: value\n    \"\"\"\n    return x + y\n\nresult = tester(multiply)\nprint(result)\n\n# {'reasoning': \"The function's docstring states that it should multiply x by y. \n#   However, the implementation returns x + y, which is addition instead of multiplication. \n#   Therefore, the implementation does not correctly fulfill what is described in the docstring.\",\n# 'passed': False}\n\n\nBasically, we have a multiply function that we want to test. To do so we create a tester instance that will use o3-mini as a judge. Then, we pass the method multiply to the tester, which internally will build a prompt containing all the information about it. Finally, the LLM will decide if the method is correctly implemented or contains a bug.\n\nHow does it work?\n\nI’ve had this project on my todo list for a long time and I never had enough time and motivation to start it, however, some weeks ago Vincent released smartfunc (a library to turn docstrings into LLM-functions), and motivated me to start the project -and to be honest I borrowed some design choices from Vincent’s code. I also used llm library by Simon Willison to access different LLM providers easily.\n\nThe suite library does the following.\n\n\n  Receives a callable func as input.\n  Reads func implementation and docstring (using inspect library).\n  Analyzes the implementation of func to identify and extract any functions or methods that it calls internally.\n  Recursively applies steps 1, 2 and 3 to func inner methods (up to some max_depth).\n  Builds a FunctionInfo object with all the information about func we need.\n  Uses FunctionInfo to write a prompt which is passed to an LLM.\n  The LLM returns a structured output like {\"reasoning\": str, \"passed\" bool}\n\n\nLet’s see now how all of these works with a concrete example. Imagine we have a method that we use to deal a deck of cards among some players. To do so we have a method called deal_cards which implementation is below.\n\n\nDealing cards code\n\n\nimport random\n\ndef shuffle_cards(cards: list[str]) -> list[str]:\n    \"\"\"\n    Returns a shuffled copy of the given list of cards.\n\n    Parameters:\n        cards (list[str]): A list of card identifiers (e.g., \"Ace of Spades\").\n\n    Returns:\n        list[str]: A new list containing the same cards in randomized order.\n    \"\"\"\n    shuffled = cards.copy()\n    random.shuffle(shuffled)\n    return shuffled\n\ndef split_cards(cards: list[str], number_of_players: int) -> list[list[str]]:\n    \"\"\"\n    Splits a list of cards evenly among a given number of players in a round-robin fashion.\n\n    Parameters:\n        cards (list[str]): The full list of cards to distribute.\n        number_of_players (int): The number of players to split the cards between.\n\n    Returns:\n        list[list[str]]: A list where each sublist represents a player's hand of cards.\n        Cards are distributed one at a time to each player in turn.\n    \"\"\"\n    return [cards[i::number_of_players] for i in range(number_of_players)]\n\ndef deal_cards(cards: list[str], number_of_players: int) -> list[list[str]]:\n    \"\"\"\n    Shuffles a deck of cards and deals them evenly among a given number of players.\n\n    This function combines shuffling and splitting the deck to simulate a card deal.\n\n    Parameters:\n        cards (list[str]): The full list of cards to shuffle and distribute.\n        number_of_players (int): The number of players to deal cards to.\n\n    Returns:\n        list[list[str]]: A list of player hands after shuffling and dealing.\n    \"\"\"\n    shuffled = shuffle_cards(cards)\n    return split_cards(shuffled, number_of_players)\n\n\n\n\nWe want to test the method deal_cards, so the first thing we need is to retrieve all the information about this method. This includes its docstring, source code, dependencies, etc. To do so we can use the FunctionInfo class, which is a pydantic model that looks like this\n\nclass FunctionInfo(BaseModel):\n    \"\"\"Information about a function extracted for semantic testing.\"\"\"\n    name: str\n    docstring: str | None\n    source: str | None\n    source_file: str | None\n    dependencies: list[\"FunctionInfo\"] = []\n    \n    @classmethod\n    def from_func(\n        cls,\n        func: Callable,\n        max_depth: int = 2):\n        ...\n\n\nThis class has information about the callable name, docstring, source code, etc. It also contains information about the method dependencies, which are a list of FunctionInfo, one for each method that the callable uses internally. This piece of information is key to building a good context for the LLM since it allows us to get information about the code outside the method we want to test. It also has a classmethod that allows you to build this object directly from a callable. The max_depth parameter controls how deep you want to inspect the dependencies (ie: dependencies, dependencies of dependencies, etc.)\n\nThen, we can run FunctionInfo.from_func(deal_cards) and we’ll have an object with all the information we need about deal_cards.\n\nNow, we need a method that receives an instance of FunctionInfo and returns a prompt that can be sent to an LLM\n\ndef format_prompt(\n    func_info: FunctionInfo,\n    prompt_template: str = DEFAULT_PROMPT_TEMPLATE,\n    dependencies_template: str = DEFAULT_DEPENDENCY_TEMPLATE,\n) -> str:\n    ...\n\n\nFinally, for the deal_cards method, the resulting prompt is\n\n\nFinal prompt\n\n\nYou are evaluating whether a function implementation correctly matches its docstring.\n\nFunction name: deal_cards\nDocstring: Shuffles a deck of cards and deals them evenly among a given number of players.\n\nThis function combines shuffling and splitting the deck to simulate a card deal.\n\nParameters:\n cards (list[str]): The full list of cards to shuffle and distribute.\n number_of_players (int): The number of players to deal cards to.\n\nReturns:\n list[list[str]]: A list of player hands after shuffling and dealing.\nImplementation: def deal_cards(cards: list[str], number_of_players: int) -> list[list[str]]:\n \"\"\"\n Shuffles a deck of cards and deals them evenly among a given number of players.\n\n This function combines shuffling and splitting the deck to simulate a card deal.\n\n Parameters:\n cards (list[str]): The full list of cards to shuffle and distribute.\n number_of_players (int): The number of players to deal cards to.\n\n Returns:\n list[list[str]]: A list of player hands after shuffling and dealing.\n \"\"\"\n shuffled = shuffle_cards(cards)\n return split_cards(shuffled, number_of_players)\n\nDependencies: \nDependency 1: shuffle_cards\nDocstring: Returns a shuffled copy of the given list of cards.\n\nParameters:\n cards (list[str]): A list of card identifiers (e.g., \"Ace of Spades\").\n\nReturns:\n list[str]: A new list containing the same cards in randomized order.\nImplementation: def shuffle_cards(cards: list[str]) -> list[str]:\n \"\"\"\n Returns a shuffled copy of the given list of cards.\n\n Parameters:\n cards (list[str]): A list of card identifiers (e.g., \"Ace of Spades\").\n\n Returns:\n list[str]: A new list containing the same cards in randomized order.\n \"\"\"\n shuffled = cards.copy()\n random.shuffle(shuffled)\n return shuffled\n\n\n\nDependency 1.1: method\nDocstring: Shuffle list x in place, and return None.\nImplementation:     def shuffle(self, x):\n \"\"\"Shuffle list x in place, and return None.\"\"\"\n\n randbelow = self._randbelow\n for i in reversed(range(1, len(x))):\n # pick an element in x[:i+1] with which to exchange x[i]\n j = randbelow(i + 1)\n x[i], x[j] = x[j], x[i]\n\n\n\nDependency 2: split_cards\nDocstring: Splits a list of cards evenly among a given number of players in a round-robin fashion.\n\nParameters:\n cards (list[str]): The full list of cards to distribute.\n number_of_players (int): The number of players to split the cards between.\n\nReturns:\n list[list[str]]: A list where each sublist represents a player's hand of cards.\n Cards are distributed one at a time to each player in turn.\nImplementation: def split_cards(cards: list[str], number_of_players: int) -> list[list[str]]:\n \"\"\"\n Splits a list of cards evenly among a given number of players in a round-robin fashion.\n\n Parameters:\n cards (list[str]): The full list of cards to distribute.\n number_of_players (int): The number of players to split the cards between.\n\n Returns:\n list[list[str]]: A list where each sublist represents a player's hand of cards.\n Cards are distributed one at a time to each player in turn.\n \"\"\"\n return [cards[i::number_of_players] for i in range(number_of_players)]\n\n\n\nDoes the implementation correctly fulfill what is described in the docstring?\nRead the implementation carefully. Reason step by step and take your time.\n\n\n\n\nThe prompt includes information about the dependencies we implemented (shuffle_cards and split_cards) and also about external methods (random.shuffle).\n\nThen, this prompt is sent to an LLM which returns an object of type SuiteOutput\n\nclass SuiteOutput(BaseModel):\n    reasoning: str\n    passed: bool\n\n    def __bool__(self):\n        return self.passed\n\n\nIn my experience, models that support thinking and structured outputs yield better results. o3-mini works very well and it’s not crazy expensive.\n\nReasons to not use suite\n\nNow that you know how suite works let’s get serious: you shouldn’t substitute your tests with this tool.\n\nI know it’s weird to write this section in my post. I’m the author of the package and I should be publicizing it instead of explaining why is a bad idea to use it. But let’s be honest, we all know that LLMs are not the solution to every problem we have -and we already have enough AI influencers out there-, so I’ll try to be upfront and explain why I don’t think you should replace your unit tests with this approach.\n\nUse boring technology\n\nBoring technology works so instead of using the new and shiny library you should be using the old and tested approaches that have been there for decades. I know it’s cool to play with LLMs and CV-driven-development forces you to try new technologies to make you more employable. But we’re here to make good software, and sometimes you don’t need the new toy in the market to do so.\n\nIt can be expensive\n\nWhile developing suite I did some tests with the pandas library. In particular, I tested the pd.json_normalize method. With max_depth=0 (ie: the smallest amount of information) I got a prompt of 112k tokens (the first Harry Potter book has 100k tokens). And here I was just testing one method. Imagine if I tested all the pandas codebase!\n\nDon’t trust LLMs\n\nI guess at this point I don’t need to tell you this but here we go: you can’t trust LLMs outputs. They are useful tools, but as with any tool you need to be careful of how you use it. LLMs tend to halluciante, and this make them dangerous tools. So I wouldn’t trust an LLM to tell me if my implementation of a method is correct or not.\n\nLooking at the pace at which LLMs get better maybe this point will be obsolete in the following years -or even months. But for the time being I wouldn’t trust an LLM to decide if some code is safe enough to be deployed.\n\nReasons to use suite\n\nAt this point, you might be wondering, “Why the hell do we need this?” or perhaps shouting, “Stop putting LLMs everywhere!” at your screen. Fair enough. Let me walk you through why this tool is worth considering and why it could be a valuable addition to your testing toolbox.\n\nComprehensive Coverage\n\nTraditional unit testing focuses on specific inputs and outputs, often covering only a small part of your code. With suite, the game changes. Instead of just testing specific cases, it evaluates the semantic correctness of your functions by cross-referencing their implementation against the documentation. For example, imagine you’ve implemented a factorial function, something like this:\n\ndef factorial(n: int) -> int:\n    \"\"\"Calculates the factorial of a non-negative integer n.\"\"\"\n    if n == 0:\n        return 1\n    return n * factorial(n - 1)\n\n\nyou write the tests, and everything passes\n\nassert factorial(0) == 1  # Factorial of 0\nassert factorial(1) == 1  # Factorial of 1\nassert factorial(5) == 120  # Factorial of 5\n\n\nBut here’s the catch: you’re missing some edge cases. What about negative inputs? What happens if someone passes a non-integer? Or very large numbers?\n\nThe problem is that traditional tests can only cover a narrow slice of your function’s behavior. Writing unit tests is hard and boring, and when combined, these two elements often lead to disaster. Just because a high percentage of tests pass doesn’t mean your code is bug-free. With suite, you can sidestep the pain of writing every single test case manually. Instead, the LLM reviews your function’s behavior holistically, saving time and ensuring a broader set of scenarios are taken into account.\n\nTrivial integration with pytest\n\nOne of the reasons to use suite is its seamless integration with pytest. You can easily incorporate semantic testing into your existing test suite without disrupting your workflow\n\n# test_module.py\n\nfrom module import function\nfrom suite import suite\n\ntester = suite()\n\ndef test_function():\n    assert tester(function)\n\n\nThat’s it - clean and simple. When you run pytest, your semantic tests will execute alongside traditional tests. For teams with established testing practices, this trivial integration makes it easy to experiment with semantic testing without committing to a major workflow change.\n\nCatch bugs early\n\nIn a typical testing workflow, you write some basic tests to check the core functionality. When a bug inevitably shows up—usually after deployment—you go back and add more tests to cover it. This process is reactive, time-consuming, and frankly, a bit tedious.\n\nWith semantic unit testing, the LLM handles this in just one iteration. It checks the function’s behavior against its documentation right from the start, catching discrepancies upfront without waiting for them to surface in production. This approach ensures that you catch issues early, saving time and preventing bugs from ever making it to production.\n\nImprove your testing suite\n\nEven if you use semantic unit testing, you’re likely still relying on traditional unit tests (after all, if it isn’t broken, don’t fix it). However, by incorporating semantic unit testing into your workflow, you can enhance your existing test suite. Semantic unit testing can point you out uncovered corner cases that you might want to add to your unit tests.\n\nYou can run it locally\n\nThanks to the llm package you can use local models to run your semantic tests. Using llm-ollama plugin you can use any model in ollama to run your tests without having to share your precious code with the evil AI companies.\n\nAsync is fast\n\nsuite allows you to run the tests asynchronously, and since the main bottleneck is IO (all the computations happen in a GPU in the cloud) it means that you can run your tests very fast. This is a huge advantage in comparison to standard tests, which need to be run sequentially.\n\nConclusions\n\nBuilding suite has been a fun ride. It’s one of those projects that sat on my todo list for months (okay, maybe years) until the right mix of free time, curiosity, and external inspiration finally pushed it forward. I’m happy to say it’s now a real package on PyPI - and yes, I checked off one more item from my 100 list: 26. ✗ Publish a Python package (in pip) → ✅ Done.\n\nIs semantic unit testing the future of testing? Probably not. At least not yet. LLMs are powerful, but they’re unpredictable and far from perfect. That said, they open up a fascinating new space in developer tooling - one where we offload some of the tedium to machines and focus more on building than babysitting our code.\n\nI’m not here to sell you the idea that you should throw away your trusty test suite and blindly trust an LLM. Please don’t do that. But I do think there’s value in exploring tools like suite to complement what you already have. Use it as a sidekick, not as a replacement.\n\nIf you give it a try, I’d love to hear your feedback. And if you find bugs - well, just don’t tell the LLM.\n\n\n\n\n  \n    \n      Special thanks to @stevepeak for transfering me the suite project on PyPi. ↩\n    \n  \n\n",
      "tags": ""
    }
  
];