jpf

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 3, 2026 License: MIT Imports: 25 Imported by: 2

README

Go Report Card Go Ref

Providing essential building blocks and robust LLM interaction interfaces, jpf enables you to craft custom AI solutions without the bloat.

Features

  • Retry and Feedback Handling: Resilient mechanisms for retrying tasks and incorporating feedback into interactions.
  • Customizable Models: Seamlessly integrate LLMs from multiple providers using unified interfaces.
  • Token Usage Tracking: Stay informed of API token consumption for cost-effective development.
  • Stream Responses: Keep your users engaged with responses that are streamed back as they are generated.
  • Easy-to-use Caching: Reduce the calls made to models by composing a caching layer onto an existing model.
  • Out-of-the-box Logging: Simply add logging messages to your models, helping you track down issues.
  • Industry Standard Context Management: All potentially slow interfaces support Go's context.Context for timeouts and cancellation.
  • Rate Limit Management: Compose models together to set local rate limits to prevent API errors.
  • MIT License: Use the code for anything, anywhere, for free.

Installation

Install jpf in your Go project via:

go get github.com/JoshPattman/jpf

Learn more about JPF in the Core Concepts section.

Examples

There are multiple examples available in the examples directory.

Core Concepts

  • jpf aims to separate the various components of building a robust interaction with an LLM for three main reasons:
    • Reusability: Build up a set of components you find useful, and write less repeated code.
    • Flexibility: Write code in a way that easily allows you to extend the LLM's capabilities - for example you can add cache to an LLM without changing a single line of business logic.
    • Testability: Each component being an atomic piece of logic allows you to unit test and mock each and every piece of logic in isolation.
  • Below are the core components you will need to understand to write code with jpf:
Model
  • Models are the core component of jpf - they wrap an LLM with some additional logic in a consistent interface.
// Model defines an interface to an LLM.
type Model interface {
	// Responds to a set of input messages.
	Respond(ctx context.Context, messages []Message) (ModelResponse, error)
}

type ModelResponse struct {
	// Extra messages that are not the final response,
	// but were used to build up the final response.
	// For example, reasoning messages.
	AuxiliaryMessages []Message
	// The primary response to the users query.
	// Usually the only response that matters.
	PrimaryMessage Message
	// The usage of making this call.
	// This may be the sum of multiple LLM calls.
	Usage Usage
}

// Message defines a text message to/from an LLM.
type Message struct {
	Role    Role
	Content string
	Images  []ImageAttachment
}
  • Models are built using composition - you can produce a very powerful model by stacking up multiple less powerful models together.
    • The power with this approach is you can abstract away a lot of the complexity from your client code, allowing it to focus primarily on business logic.
// All model constructors in jpf return the Model interface,
// we can re-use our variable as we build it up.
var model jpf.Model

// Switch, based on a boolean variable, if we should use Gemini or OpenAI.
// If using Gemini, we will scale the temperature down a bit (NOT useful - just for demonstration).
if useGemini {
    model = jpf.NewGeminiModel(apiKey, modelName, jpf.WithTemperature{X: temperature*0.8})
} else {
    model = jpf.NewOpenAIModel(apiKey, modelName, jpf.WithTemperature{X: temperature})
}

// Add retrying on API failures to the model.
// This will retry calling the child model multiple times upon an error.
if retries > 0 {
    model = jpf.NewRetryModel(model, retries, jpf.WithDelay{X: time.Second})
}

// Add cache to the model.
// This will skip calling out to the model if the same messages are requested a second time.
if cache != nil {
    model = jpf.NewCachedModel(model, cache)
}

// We now have a model that may/may not be gemini / openai, with retrying and cache.
// However, the client code does not need to know about any of this - to it we are still just calling a model!
  • Note that even though models can stream back text, it is only intended as a temporary and unreliable way to distract users while waiting for requests.
    • You should always aim to make your code work without streaming, and add it in as an add-in later on to improve the UX - this is more robust.
Encoder
  • An Encoder provides an interface to take a specific typed object and produce some messages for the LLM.
    • It does not actually make a call to the Model, and it does not decode the response.
// Encoder encodes a structured piece of data into a set of messages for an LLM.
type Encoder[T any] interface {
	BuildInputMessages(T) ([]Message, error)
}
  • For more complex tasks, you may choose to implement this yourself, however there are some useful encoders built in.
Parser
  • A Parser parses the output of an LLM into structured data.
  • As with encoders, they do not make any LLM calls.
// Parser converts the LLM response into a structured piece of output data.
// When the LLM response is invalid, it should return [ErrInvalidResponse] (or an error joined on that).
type Parser[U any] interface {
	ParseResponseText(string) (U, error)
}
  • You may choose to implement your own parser, however in my experience a JSON object is usually sufficient output.
  • When an error in response format is detected, the response decoder must return an error that, at some point in its chain, is an ErrInvalidResponse (this will be explained in the pipeline section).
Validator
  • A Validator checks the parsed output of an LLM against the input to validate it.
  • These are optional in all pipelines (can be passed as nil if no further validation is required).
// Validator takes a parsed LLM response and validates it against the input.
// When the LLM response is invalid, it should return [ErrInvalidResponse] (or an error joined on that).
type Validator[T, U any] interface {
	ValidateParsedResponse(T, U) error
}
  • There are no implementations of this in jpf due to how usage-specific the validation would be - you should impolement your own.
  • As with the above, validation errors should return an ErrInvalidResponse.
Pipeline
  • A Pipeline is a collection of a Encoder, Parser, Validator (optional), Model, and some additional logic.
  • Your business logic should only ever be interacting with LLMs through a pipeline.
  • It is a very generic interface, but it is intended to only ever be used for LLM-based functionality.
// Pipeline transforms input of type T into output of type U using an LLM.
// It handles the encoding of input, interaction with the LLM, and decoding of output.
type Pipeline[T, U any] interface {
	Call(context.Context, T) (U, Usage, error)
}
  • It is not really expected that users will implement their own pipelines, but that is absolutely possible.
  • jpf ships with three built-in pipelines:
    • NewOneShotPipeline: No retries on validation fails, return errors immediately.
    • NewFeedbackPipeline: On ErrInvalidResponse, add the error to the conversation and try again.
    • NewFallbackPipeline: On ErrInvalidResponse, try again with the next model option.
  • Notice in the above, we have introduced a second place for retries to occur - this is intentional.
    • API-level errors should be retried at the Model level - these are errors that are not the fault of the LLM.
    • LLM response errors should be retried at the Pipeline level - these are errors where the LLM has responded with an invalid response, and we would like to tell it what it did wrong and ask again.
  • However, if you choose not to use these higher-level retries, you can simply use the one-shot pipeline.

FAQ

  • I want to change my model's temperature/structured output/output tokens/... after I have built it!
    • The intention is to provide functions that need to use an LLM with a builder function instead of a built object. This way, you can use the builder function multiple times with different parameters.
    • Take a look at the examples to see this concept.
    • This design decision was made as it prevents you from injecting unnecessary LLM-related data into business logic.
  • Where are the agents?
    • Agents are built on top of LLMs, but this package is designed for LLM handling, so it lives at the level below agents.
    • Take a look at JChat or react to see how you can build an agent on top of JPF.
  • Why does this not support MCP tools on the OpenAI API / Tool calling / Other advanced API features?
    • Relying on API features like tool calling, MCP tools, or vector stores is not ideal for two reasons: (a) it makes it harder to move between API/model providers (b) it gives you less flexibility and control.
    • These features are not particularly hard to add locally, so you should aim to do so to ensure your application is as robust as possible to API change.

Author

Developed by Josh Pattman. Learn more at GitHub.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidResponse = errors.New("llm produced an invalid response")
)

Functions

func HashMessages added in v0.6.0 

func HashMessages(salt string, inputs []Message) string

func TransformByPrefix added in v0.8.0 

func TransformByPrefix(prefix string) func(string) string

Types

type CachedModelOpt added in v0.8.2 

type CachedModelOpt interface {
	// contains filtered or unexported methods
}

type ConcurrentLimiter added in v0.7.0 

type ConcurrentLimiter chan struct{}

func NewMaxConcurrentLimiter added in v0.7.0 

func NewMaxConcurrentLimiter(n int) ConcurrentLimiter

NewMaxConcurrentLimiter creates a ConcurrentLimiter that allows up to n concurrent operations. The limiter is implemented as a buffered channel with capacity n.

func NewOneConcurrentLimiter added in v0.7.0 

func NewOneConcurrentLimiter() ConcurrentLimiter

NewOneConcurrentLimiter creates a ConcurrentLimiter that allows only one operation at a time. This is a convenience function equivalent to NewMaxConcurrentLimiter(1).

type Encoder added in v0.9.0 

type Encoder[T any] interface {
	BuildInputMessages(T) ([]Message, error)
}

Encoder encodes a structured piece of data into a set of messages for an LLM.

func NewFixedEncoder added in v0.9.0 

func NewFixedEncoder(systemPrompt string) Encoder[string]

NewFixedEncoder creates an Encoder that encodes a static system prompt and raw user input as messages.

func NewTemplateEncoder added in v0.9.0 

func NewTemplateEncoder[T any](systemTemplate, userTemplate string) Encoder[T]

NewTemplateEncoder creates a Encoder that uses Go's text/template for formatting messages. It accepts templates for both system and user messages, allowing dynamic content insertion. The data parameter to BuildInputMessages should be a struct or map with fields accessible to the template. If either systemTemplate or userTemplate is an empty string, that message will be skipped.

type FakeReasoningModelOpt added in v0.8.0 

type FakeReasoningModelOpt interface {
	// contains filtered or unexported methods
}

type FeedbackGenerator added in v0.6.0 

type FeedbackGenerator interface {
	FormatFeedback(Message, error) string
}

FeedbackGenerator takes an error and converts it to a piece of text feedback to send to the LLM.

func NewRawMessageFeedbackGenerator added in v0.6.0 

func NewRawMessageFeedbackGenerator() FeedbackGenerator

NewRawMessageFeedbackGenerator creates a FeedbackGenerator that formats feedback by returning the error message as a string.

type GeminiModelOpt added in v0.8.0 

type GeminiModelOpt interface {
	// contains filtered or unexported methods
}

type ImageAttachment added in v0.7.0 

type ImageAttachment struct {
	Source image.Image
}

func (*ImageAttachment) ToBase64Encoded added in v0.7.0 

func (i *ImageAttachment) ToBase64Encoded(useCompression bool) (string, error)

type Message 

type Message struct {
	Role    Role
	Content string
	Images  []ImageAttachment
}

Message defines a text message to/from an LLM.

type Model 

type Model interface {
	// Responds to a set of input messages.
	Respond(context.Context, []Message) (ModelResponse, error)
}

Model defines an interface to an LLM.

func NewCachedModel added in v0.7.0 

func NewCachedModel(model Model, cache ModelResponseCache, opts ...CachedModelOpt) Model

NewCachedModel wraps a Model with response caching functionality. It stores responses in the provided ModelResponseCache implementation, returning cached results for identical input messages and salts to avoid redundant model calls.

func NewConcurrentLimitedModel added in v0.7.0 

func NewConcurrentLimitedModel(model Model, limiter ConcurrentLimiter) Model

NewConcurrentLimitedModel wraps a Model with concurrency control. It ensures that only a limited number of concurrent calls can be made to the underlying model, using the provided ConcurrentLimiter to manage access.

func NewFakeReasoningModel 

func NewFakeReasoningModel(reasoner Model, answerer Model, opts ...FakeReasoningModelOpt) Model

NewFakeReasoningModel creates a model that uses two underlying models to simulate reasoning. It first calls the reasoner model to generate reasoning about the input messages, then passes that reasoning along with the original messages to the answerer model. The reasoning is included as a ReasoningRole message in the auxiliary messages output. Optional parameters allow customization of the reasoning prompt.

func NewGeminiModel added in v0.8.0 

func NewGeminiModel(key, modelName string, opts ...GeminiModelOpt) Model

NewGeminiModel creates a Model that uses the Google Gemini API. It requires an API key and model name, with optional configuration via variadic options.

func NewLoggingModel added in v0.7.0 

func NewLoggingModel(model Model, logger ModelLogger) Model

NewLoggingModel wraps a Model with logging functionality. It logs all interactions with the model using the provided ModelLogger. Each model call is logged with input messages, output messages, usage statistics, and timing information.

func NewOpenAIModel added in v0.7.0 

func NewOpenAIModel(key, modelName string, opts ...OpenAIModelOpt) Model

NewOpenAIModel creates a Model that uses the OpenAI API. It requires an API key and model name, with optional configuration via variadic options.

func NewRateLimitedModel added in v0.9.0 

func NewRateLimitedModel(model Model, limiter *rate.Limiter) Model

func NewRetryChainModel added in v0.8.2 

func NewRetryChainModel(models []Model) Model

NewRetryChainModel creates a Model that tries a list of models in order, returning the result from the first one that doesn't fail. If all models fail, it returns a joined error containing all the errors.

func NewRetryModel 

func NewRetryModel(model Model, maxRetries int, opts ...RetryModelOpt) Model

NewRetryModel wraps a Model with retry functionality. If the underlying model returns an error, this wrapper will retry the operation up to a configurable number of times with an optional delay between retries.

func NewTimeoutModel added in v0.9.0 

func NewTimeoutModel(model Model, timeout time.Duration) Model

NewTimeoutModel creates a new model that will cause the context of the child to timeout, a specified duration after Respond is called. Caution: It only tells the context to timeout - it will not forecfully stop the child model if it does not respect the context.

func NewUsageCountingModel added in v0.7.0 

func NewUsageCountingModel(model Model, counter *UsageCounter) Model

NewUsageCountingModel wraps a Model with token usage tracking functionality. It aggregates token usage statistics in the provided UsageCounter, which allows monitoring total token consumption across multiple model calls.

type ModelLogger added in v0.7.0 

type ModelLogger interface {
	ModelLog(ModelLoggingInfo) error
}

ModelLogger specifies a method of logging a call to a model.

func NewJsonModelLogger added in v0.7.0 

func NewJsonModelLogger(to io.Writer) ModelLogger

NewJsonModelLogger creates a ModelLogger that outputs logs in JSON format. The logs are written to the provided io.Writer, with each log entry being a JSON object containing the model interaction details.

func NewSlogModelLogger added in v0.8.0 

func NewSlogModelLogger(logFunc func(string, ...any), logMessages bool) ModelLogger

Logs calls made to the model to a slog-style logging function. Can optionally log the model messages too (this is very spammy).

type ModelLoggingInfo added in v0.6.0 

type ModelLoggingInfo struct {
	Messages             []Message
	ResponseAuxMessages  []Message
	ResponseFinalMessage Message
	Usage                Usage
	Err                  error
	Duration             time.Duration
}

ModelLoggingInfo contains all information about a model interaction to be logged. It includes input messages, output messages, usage statistics, and any error that occurred.

type ModelResponse added in v0.8.0 

type ModelResponse struct {
	// Extra messages that are not the final response,
	// but were used to build up the final response.
	// For example, reasoning messages.
	AuxiliaryMessages []Message
	// The primary response to the users query.
	// Usually the only response that matters.
	PrimaryMessage Message
	// The usage of making this call.
	// This may be the sum of multiple LLM calls.
	Usage Usage
}

func (ModelResponse) IncludingUsage added in v0.8.0 

func (r ModelResponse) IncludingUsage(u Usage) ModelResponse

Utility to include another usage object in this response object

func (ModelResponse) OnlyUsage added in v0.8.0 

func (r ModelResponse) OnlyUsage() ModelResponse

Utility to allow you to return the usage but 0 value messages when an error occurs.

type ModelResponseCache added in v0.6.0 

type ModelResponseCache interface {
	GetCachedResponse(ctx context.Context, salt string, inputs []Message) (bool, []Message, Message, error)
	SetCachedResponse(ctx context.Context, salt string, inputs []Message, aux []Message, out Message) error
}

func NewFilePersistCache added in v0.9.0 

func NewFilePersistCache(filename string) (ModelResponseCache, error)

NewFilePersistCache creates an in-memory cache that persists to the given filename. On creation, it loads the cache from the file (if it exists). Whenever SetCachedResponse is called, the entire cache is saved back to the file.

func NewInMemoryCache added in v0.6.0 

func NewInMemoryCache() ModelResponseCache

NewInMemoryCache creates an in-memory implementation of ModelResponseCache. It stores model responses in memory using a hash of the input messages as a key.

func NewSQLCache added in v0.7.3 

func NewSQLCache(ctx context.Context, db *sql.DB) (ModelResponseCache, error)

type OpenAIModelOpt added in v0.8.0 

type OpenAIModelOpt interface {
	// contains filtered or unexported methods
}

type Parser added in v0.9.0 

type Parser[U any] interface {
	ParseResponseText(string) (U, error)
}

Parser converts the LLM response into a structured piece of output data. When the LLM response is invalid, it should return ErrInvalidResponse (or an error joined on that).

func NewJsonParser added in v0.9.0 

func NewJsonParser[T any]() Parser[T]

NewJsonParser creates a Parser that tries to parse a json object from the response. It can ONLY parse json objects with an OBJECT as top level (i.e. it cannot parse a list directly).

func NewStringParser added in v0.9.0 

func NewStringParser() Parser[string]

NewStringParser creates a Parser that returns the response as a raw string without modification.

func NewSubstringAfterParser added in v0.9.0 

func NewSubstringAfterParser[T any](decoder Parser[T], separator string) Parser[T]

Wrap an existing Parser with one that takes only part of the response after the separator into account. If an error is detected when getting the substring, ErrInvalidResponse is raised.

func NewSubstringParser added in v0.9.0 

func NewSubstringParser[T any](decoder Parser[T], substring func(string) (string, error)) Parser[T]

Wrap an existing Parser with one that takes only the part of interest of the response into account. The part of interest is determined by the substring function. If an error is detected when getting the substring, ErrInvalidResponse is raised.

type Pipeline added in v0.9.0 

type Pipeline[T, U any] interface {
	Call(context.Context, T) (U, Usage, error)
}

Pipeline transforms input of type T into output of type U using an LLM. It handles the encoding of input, interaction with the LLM, and decoding of output.

func NewFallbackPipeline added in v0.9.0 

func NewFallbackPipeline[T, U any](
	encoder Encoder[T],
	parser Parser[U],
	validator Validator[T, U],
	models ...Model,
) Pipeline[T, U]

Creates a Pipeline that first tries to ask the first model, and if that produces an invalid format will try to ask the next models until a valid format is found. This is useful, for example, to try a second time with a model that overwrites the cache.

func NewFeedbackPipeline added in v0.9.0 

func NewFeedbackPipeline[T, U any](
	encoder Encoder[T],
	parser Parser[U],
	validator Validator[T, U],
	feedbackGenerator FeedbackGenerator,
	model Model,
	feedbackRole Role,
	maxRetries int,
) Pipeline[T, U]

NewFeedbackPipeline creates a Pipeline that first runs the encoder, then the model, finally parsing the response with the decoder. However, it adds feedback to the conversation when errors are detected. It will only add to the conversation if the error returned from the parser is an ErrInvalidResponse (using errors.Is).

func NewOneShotPipeline added in v0.9.0 

func NewOneShotPipeline[T, U any](
	encoder Encoder[T],
	parser Parser[U],
	validator Validator[T, U],
	model Model,
) Pipeline[T, U]

NewOneShotPipeline creates a Pipeline that runs without retries. The validator may be nil.

type ReasoningEffort 

type ReasoningEffort uint8

ReasoningEffort defines how hard a reasoning model should think. 

const (
	LowReasoning ReasoningEffort = iota
	MediumReasoning
	HighReasoning
)

type RetryModelOpt added in v0.8.0 

type RetryModelOpt interface {
	// contains filtered or unexported methods
}

type Role 

type Role uint8

Role is an enum specifying a role for a message. It is not 1:1 with openai roles (i.e. there is a reasoning role here). 

const (
	SystemRole Role = iota
	UserRole
	AssistantRole
	ReasoningRole
	DeveloperRole
)

func (Role) String added in v0.6.0 

func (r Role) String() string

type Usage 

type Usage struct {
	InputTokens     int
	OutputTokens    int
	SuccessfulCalls int
	FailedCalls     int
}

Usage defines how many tokens were used when making calls to LLMs.

func (Usage) Add 

func (u Usage) Add(u2 Usage) Usage

type UsageCounter added in v0.6.0 

type UsageCounter struct {
	// contains filtered or unexported fields
}

Counts up the sum usage. Is completely concurrent-safe.

func NewUsageCounter added in v0.6.0 

func NewUsageCounter() *UsageCounter

NewUsageCounter creates a new UsageCounter with zero initial usage. The counter is safe for concurrent use across multiple goroutines.

func (*UsageCounter) Add added in v0.6.0 

func (u *UsageCounter) Add(usage Usage)

Add the given usage to the counter.

func (*UsageCounter) Get added in v0.6.0 

func (u *UsageCounter) Get() Usage

Get the current usage in the counter.

type Validator added in v0.9.0 

type Validator[T, U any] interface {
	ValidateParsedResponse(T, U) error
}

Validator takes a parsed LLM response and validates it against the input. When the LLM response is invalid, it should return ErrInvalidResponse (or an error joined on that).

type Verbosity added in v0.8.0 

type Verbosity uint8
const (
	LowVerbosity Verbosity = iota
	MediumVerbosity
	HighVerbosity
)

type WithDelay added in v0.7.0 

type WithDelay struct{ X time.Duration }

type WithHTTPHeader added in v0.7.0 

type WithHTTPHeader struct {
	K string
	V string
}

type WithJsonSchema added in v0.8.0 

type WithJsonSchema struct{ X map[string]any }

type WithMaxOutputTokens added in v0.8.0 

type WithMaxOutputTokens struct{ X int }

type WithMessagePrefix added in v0.8.0 

type WithMessagePrefix struct{ X string }

type WithPrediction added in v0.8.0 

type WithPrediction struct{ X string }

type WithPresencePenalty added in v0.8.0 

type WithPresencePenalty struct{ X float64 }

type WithReasoningAs added in v0.8.0 

type WithReasoningAs struct {
	X                Role
	TransformContent func(string) string
}

type WithReasoningEffort added in v0.7.0 

type WithReasoningEffort struct{ X ReasoningEffort }

type WithReasoningPrompt added in v0.7.0 

type WithReasoningPrompt struct{ X string }

type WithSalt added in v0.8.2 

type WithSalt struct{ X string }

type WithStreamResponse added in v0.9.0 

type WithStreamResponse struct {
	// Called when the stream begins, may be called multiple times if retries occur.
	OnBegin func()
	// Called when a new chunk of text is received.
	OnText func(string)
}

type WithSystemAs added in v0.8.0 

type WithSystemAs struct {
	X                Role
	TransformContent func(string) string
}

type WithTemperature added in v0.7.0 

type WithTemperature struct{ X float64 }

type WithTopP added in v0.8.0 

type WithTopP struct{ X int }

type WithURL added in v0.7.0 

type WithURL struct{ X string }

type WithVerbosity added in v0.8.0 

type WithVerbosity struct{ X Verbosity }

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.