Prompt Engineering with Prompt Improvement Tools
Recently, we’ve been working on AI assistants to help users get started with the OpenFn platform quicker. Creating customer-facing chatbots isn’t just a technical task – to improve the user experience, you need to draw on the expertise of someone who works with clients directly in the relevant domain.
One way to do this is to work on the LLM prompts across teams. LLM providers such as Anthropic and OpenAI offer no-code UIs to refine and test prompts, and these can be a useful tool to draw on the experience of not just a product engineering team, but also a client-facing team when creating AI solutions.
This post will give an overview of how to use a prompt improvement tool. We’ll use Anthropic’s prompt improver, but OpenAI also has a very similar prompt playground. To use these, you need to have an account for the platform console to pay for tokens, and not just a chat subscription to chat to Claude or ChatGPT.
How to use Anthropic’s Prompt Improver
1. Access the Prompt Improver and Create Your Initial Prompt
To get to the Anthropic prompt improver, go to your console dashboard at https://console.anthropic.com/dashboard and click on “Improve an existing prompt”. You can fill in a first version of your prompt in template form. It needs to have a variable in double curly braces. In my example, the variable represents an artwork that is to be analysed according to the prompt instructions.
2. Review and Refine Claude’s Improvements
After filling in your draft prompt, Claude will have a first pass at improving it. In my example, the output format clarification seems useful, but the prompt seems rather long. I would probably click on “Improve prompt” and tell Claude to be more concise for the next version.
3. Add Examples for Better Output Formatting
To further clarify what type of output format and style you are expecting, you can add examples. You can directly generate an output, which can be helpful as a starting point, if only to see how your current prompt could be misunderstood by the model.
4. Test Your Prompts with Multiple Inputs
You should also test your prompts on as many inputs as possible. In the evaluate tab, you can add test cases to run through with each new prompt version. You can add the “ideal” answers and score the results, which could be used to collaborate between teams, or even score future prompt versions automatically with an LLM. The model settings and prompt versions are recorded for future reference. You can also generate more test cases, which can help speed up the process of creating a test set.
That’s all you need to get started with Anthropic’s prompt improver. Let us know if you have any questions or experiences to share about prompt engineering!