internal

package
v0.0.0-...-b4f74d8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 16, 2026 License: GPL-3.0 Imports: 67 Imported by: 0

Documentation

Overview

Package internal runs a read-through cache server.

Package internal is a generated GoMock package.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AuthorKey 

func AuthorKey(authorID int64) string

AuthorKey returns a cache key for an author ID.

func BookKey 

func BookKey(bookID int64) string

BookKey returns a cache key for a book (edition) ID.

func Log 

func Log(ctx context.Context) *slog.Logger

Log returns a logger scoped to the request ID if present in the context.

func NewBatchedGraphQLClient 

func NewBatchedGraphQLClient(url string, client *http.Client, every time.Duration, batchSize int, reg *prometheus.Registry) (graphql.Client, error)

NewBatchedGraphQLClient creates a batching GraphQL client. Queries are accumulated and executed regularly accurding to the given rate.

func NewGRGQL 

func NewGRGQL(_ context.Context, rate time.Duration, batchSize int, reg *prometheus.Registry) (graphql.Client, error)

NewGRGQL returns a new GraphQL client for use with GR. The provided http.Client must be non-nil and is used for issuing requests. If a non-empty cookie is given the requests are authorized and use are allowed more RPS.

func NewMetrics 

func NewMetrics() *prometheus.Registry

NewMetrics creates a new Rrometheus registry with default collectors already registered.

func NewMux 

func NewMux(h *Handler, reg *prometheus.Registry) http.Handler

NewMux registers a handler's routes on a new mux.

func NewUpstream 

func NewUpstream(host string, proxy string) (*http.Client, error)

NewUpstream creates a new http.Client with middleware appropriate for use with an upstream.

func SetLogLevel 

func SetLogLevel(l charm.Level)

SetLogLevel sets the log level.

func WorkKey 

func WorkKey(workID int64) string

WorkKey returns a cache key for a work ID.

Types

type AuthorResource 

type AuthorResource struct {
	ForeignID     int64   `json:"ForeignId"`
	Name          string  `json:"Name"`
	Description   string  `json:"Description"`
	ImageURL      string  `json:"ImageUrl"`
	URL           string  `json:"Url"`
	RatingCount   int64   `json:"RatingCount"`
	AverageRating float32 `json:"AverageRating"`

	// Relations.
	Works  []workResource   `json:"Works"`
	Series []SeriesResource `json:"Series"`

	// New fields.
	KCA string `json:"KCA"`
}

AuthorResource collects every edition of every work by an author.

type CloudflareCache 

type CloudflareCache struct {
	// contains filtered or unexported fields
}

CloudflareCache is a caching layer which no-ops except when new cache values are written, in which case a cache bust is queued with Cloudflare.

func NewCloudflareCache 

func NewCloudflareCache(apiKey string, zoneID string, pather func(string) string, reg *prometheus.Registry) (*CloudflareCache, error)

NewCloudflareCache creates a new CloudflareCache.

func (*CloudflareCache) Delete 

func (cc *CloudflareCache) Delete(ctx context.Context, key string) error

Delete is a no-op, since it's only used on our "refresh author" sentinel which doesn't impact users.

func (*CloudflareCache) Expire 

func (cc *CloudflareCache) Expire(ctx context.Context, key string) error

Expire busts the Cloudflare CDN by enqueuing the path to be busted.

func (*CloudflareCache) Get 

func (cc *CloudflareCache) Get(ctx context.Context, key string) ([]byte, bool)

Get is a no-op.

func (*CloudflareCache) GetWithTTL 

func (cc *CloudflareCache) GetWithTTL(ctx context.Context, key string) ([]byte, time.Duration, bool)

GetWithTTL is a no-op.

func (*CloudflareCache) Set 

func (cc *CloudflareCache) Set(ctx context.Context, key string, _ []byte, _ time.Duration)

Set queues a URL for busting.

type Controller 

type Controller struct {
	// contains filtered or unexported fields
}

Controller facilitates operations on our cache by scheduling background work and handling cache invalidation.

Most operations take place inside a singleflight group to prevent redundant work.

The request path is limited to Get methods which at worst perform only O(1) lookups. More expensive work, like denormalization, is handled in the background. The original metadata server likely does this work in the request path, hence why larger authors don't work -- it can't complete O(work * editions) within the lifespan of the request.

Another significant difference is that we cache data eagerly, when it is requested. We don't require a full database dump, so we're able to grab new works as soon as they're available.

func NewController 

func NewController(cache cache[[]byte], getter getter, persister persister, reg *prometheus.Registry) (*Controller, error)

NewController creates a new controller. Background jobs to load author works and editions is bounded to at most 10 concurrent tasks.

func (*Controller) GetASIN 

func (c *Controller) GetASIN(ctx context.Context, asin string) (int64, error)

GetASIN returns the best known edition ID for the given ASIN, or a not found error if there is none.

func (*Controller) GetAuthor 

func (c *Controller) GetAuthor(ctx context.Context, authorID int64) ([]byte, time.Duration, error)

GetAuthor loads an author or returns a cached value if one exists.

func (*Controller) GetBook 

func (c *Controller) GetBook(ctx context.Context, bookID int64) ([]byte, time.Duration, error)

GetBook loads a book (edition) or returns a cached value if one exists. TODO: This should only return a book!

func (*Controller) GetISBN 

func (c *Controller) GetISBN(ctx context.Context, isbn isbn.ISBN) (int64, error)

GetISBN returns the best known edition ID for the given ISBN13, or a not found error if there is none.

func (*Controller) GetSeries 

func (c *Controller) GetSeries(ctx context.Context, seriesID int64) ([]byte, error)

GetSeries returns a cached series if one exists.

func (*Controller) GetWork 

func (c *Controller) GetWork(ctx context.Context, workID int64) ([]byte, time.Duration, error)

GetWork loads a work or returns a cached value if one exists.

func (*Controller) Recommendations 

func (c *Controller) Recommendations(ctx context.Context, page int64) (RecommentationsResource, error)

Recommendations returns recommended work IDs.

func (*Controller) Run 

func (c *Controller) Run(ctx context.Context)

Run is responsible for denormalizing data and handling our worker pools.

func (*Controller) Search 

func (c *Controller) Search(ctx context.Context, query string) ([]SearchResource, error)

Search queries the metadata provider.

func (*Controller) Shutdown 

func (c *Controller) Shutdown(ctx context.Context)

Shutdown waits for all refresh and denormalization goroutines to finish submitting their work and then closes the denormalization channel. Run will run to completion after Shutdown is called.

type GRGetter 

type GRGetter struct {
	// contains filtered or unexported fields
}

GRGetter fetches information from a GR upstream.

func NewGRGetter 

func NewGRGetter(cache cache[[]byte], gql graphql.Client, upstream *http.Client) (*GRGetter, error)

NewGRGetter creates a new Getter backed by G——R——.

func (*GRGetter) GetAuthor 

func (g *GRGetter) GetAuthor(ctx context.Context, authorID int64) ([]byte, error)

GetAuthor returns an author with all of their works and respective editions. Due to the way R works, if a work isn't returned here it's not fetchable.

On an initial load we return only one work on the author. The controller handles asynchronously fetching all additional works.

func (*GRGetter) GetAuthorBooks 

func (g *GRGetter) GetAuthorBooks(ctx context.Context, authorID int64) iter.Seq[int64]

GetAuthorBooks enumerates all of the "best" editions for an author. This is how we load large authors.

func (*GRGetter) GetBook 

func (g *GRGetter) GetBook(ctx context.Context, bookID int64, saveEditions editionsCallback) (_ []byte, workID, authorID int64, _ error)

GetBook fetches a book (edition) from GR.

func (*GRGetter) GetSeries 

func (g *GRGetter) GetSeries(ctx context.Context, seriesID int64) (*SeriesResource, error)

GetSeries returns works belonging to the given series.

func (*GRGetter) GetWork 

func (g *GRGetter) GetWork(ctx context.Context, workID int64, saveEditions editionsCallback) (_ []byte, authorID int64, _ error)

GetWork returns a work with all known editions. Due to the way R—— works, if an edition is missing here (like a translated edition) it's not fetchable.

func (*GRGetter) Recommendations 

func (g *GRGetter) Recommendations(ctx context.Context, page int64) (RecommentationsResource, error)

Recommendations returns the trending works on the "explore" page.

func (*GRGetter) Search 

func (g *GRGetter) Search(ctx context.Context, query string) ([]SearchResource, error)

Search hits the auto_complete API that has been used historically, so it returns exactly the same results as legacy.

type HCGetter 

type HCGetter struct {
	// contains filtered or unexported fields
}

HCGetter implements a Getter using the Hardcover API as its source. It attempts to minimize upstream HEAD requests (to resolve book/work IDs) by relying on HC's raw external data.

func NewHardcoverGetter 

func NewHardcoverGetter(cache cache[[]byte], gql graphql.Client) (*HCGetter, error)

NewHardcoverGetter returns a new Getter backed by Hardcover.

func (*HCGetter) GetAuthor 

func (g *HCGetter) GetAuthor(ctx context.Context, authorID int64) ([]byte, error)

GetAuthor looks up an author on Hardcover.

func (*HCGetter) GetAuthorBooks 

func (g *HCGetter) GetAuthorBooks(ctx context.Context, authorID int64) iter.Seq[int64]

GetAuthorBooks returns all GR book (edition) IDs.

func (*HCGetter) GetBook 

func (g *HCGetter) GetBook(ctx context.Context, editionID int64, _ editionsCallback) ([]byte, int64, int64, error)

GetBook looks up a GR book (edition) in Hardcover's mappings.

func (*HCGetter) GetSeries 

func (g *HCGetter) GetSeries(ctx context.Context, seriesID int64) (*SeriesResource, error)

GetSeries isn't implemented yet.

func (*HCGetter) GetWork 

func (g *HCGetter) GetWork(ctx context.Context, workID int64, saveEditions editionsCallback) ([]byte, int64, error)

GetWork returns the canonical edition for a work.

func (*HCGetter) Recommendations 

func (g *HCGetter) Recommendations(ctx context.Context, page int64) (RecommentationsResource, error)

Recommendations returns trending work IDs from the past week.

func (*HCGetter) Search 

func (g *HCGetter) Search(ctx context.Context, query string) ([]SearchResource, error)

Search hits the GraphQL endpoint to fetch relevant work IDs and then fetches those in order to return the necessary edition and author IDs to the client.

type Handler 

type Handler struct {
	// contains filtered or unexported fields
}

Handler is our HTTP Handler. It handles muxing, response headers, etc. and offloads work to the controller.

func NewHandler 

func NewHandler(ctrl *Controller) *Handler

NewHandler creates a new handler.

type HeaderTransport 

type HeaderTransport struct {
	Key   string
	Value string
	http.RoundTripper
}

HeaderTransport adds a header to all requests. Best used with a scopedTransport.

func (*HeaderTransport) RoundTrip 

func (t *HeaderTransport) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip always sets the header on the request.

type LayeredCache 

type LayeredCache struct {
	// contains filtered or unexported fields
}

LayeredCache implements a simple tiered cache. In practice we use an in-memory cache backed by Postgres for persistent storage. Hits at lower layers are automatically percolated up. Values are compressed with gzip at rest.

cache.ChainCache has inconsistent marshaling behavior, so we use our own wrapper. Actually that package doesn't really buy us anything...

func NewCache 

func NewCache(ctx context.Context, dsn string, cf *CloudflareCache, reg *prometheus.Registry) (*LayeredCache, error)

NewCache constructs a new layered cache.

func (*LayeredCache) Delete 

func (c *LayeredCache) Delete(ctx context.Context, key string) error

Delete deletes a key from all layers of the cache. Expire should typically be used instead.

func (*LayeredCache) Expire 

func (c *LayeredCache) Expire(ctx context.Context, key string) error

Expire expires a key from all layers of the cache. This removes it from memory but keeps data persisted in Postgres without a TTL.

func (*LayeredCache) Get 

func (c *LayeredCache) Get(ctx context.Context, key string) ([]byte, bool)

Get returns a cache value, if it exists, and a boolean if a value was found.

func (*LayeredCache) GetWithTTL 

func (c *LayeredCache) GetWithTTL(ctx context.Context, key string) ([]byte, time.Duration, bool)

GetWithTTL returns the cached value and its TTL. The boolean returned is false if no value was found.

func (*LayeredCache) Set 

func (c *LayeredCache) Set(ctx context.Context, key string, val []byte, ttl time.Duration)

Set a key/value in all layers of the cache. TODO: Fuzz expiration

type Mockgetter 

type Mockgetter struct {
	// contains filtered or unexported fields
}

Mockgetter is a mock of getter interface.

func NewMockgetter 

func NewMockgetter(ctrl *gomock.Controller) *Mockgetter

NewMockgetter creates a new mock instance.

func (*Mockgetter) EXPECT 

func (m *Mockgetter) EXPECT() *MockgetterMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

func (*Mockgetter) GetAuthor 

func (m *Mockgetter) GetAuthor(ctx context.Context, authorID int64) ([]byte, error)

GetAuthor mocks base method.

func (*Mockgetter) GetAuthorBooks 

func (m *Mockgetter) GetAuthorBooks(ctx context.Context, authorID int64) iter.Seq[int64]

GetAuthorBooks mocks base method.

func (*Mockgetter) GetBook 

func (m *Mockgetter) GetBook(ctx context.Context, bookID int64, saveEditions editionsCallback) ([]byte, int64, int64, error)

GetBook mocks base method.

func (*Mockgetter) GetSeries 

func (m *Mockgetter) GetSeries(ctx context.Context, seriesID int64) (*SeriesResource, error)

GetSeries mocks base method.

func (*Mockgetter) GetWork 

func (m *Mockgetter) GetWork(ctx context.Context, workID int64, saveEditions editionsCallback) ([]byte, int64, error)

GetWork mocks base method.

func (*Mockgetter) Recommendations 

func (m *Mockgetter) Recommendations(ctx context.Context, page int64) (RecommentationsResource, error)

Recommendations mocks base method.

func (*Mockgetter) Search 

func (m *Mockgetter) Search(ctx context.Context, query string) ([]SearchResource, error)

Search mocks base method.

type MockgetterGetAuthorBooksCall 

type MockgetterGetAuthorBooksCall struct {
	*gomock.Call
}

MockgetterGetAuthorBooksCall wrap *gomock.Call

func (*MockgetterGetAuthorBooksCall) Do 

func (c *MockgetterGetAuthorBooksCall) Do(f func(context.Context, int64) iter.Seq[int64]) *MockgetterGetAuthorBooksCall

Do rewrite *gomock.Call.Do

func (*MockgetterGetAuthorBooksCall) DoAndReturn 

func (c *MockgetterGetAuthorBooksCall) DoAndReturn(f func(context.Context, int64) iter.Seq[int64]) *MockgetterGetAuthorBooksCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterGetAuthorBooksCall) Return 

func (c *MockgetterGetAuthorBooksCall) Return(arg0 iter.Seq[int64]) *MockgetterGetAuthorBooksCall

Return rewrite *gomock.Call.Return

type MockgetterGetAuthorCall 

type MockgetterGetAuthorCall struct {
	*gomock.Call
}

MockgetterGetAuthorCall wrap *gomock.Call

func (*MockgetterGetAuthorCall) Do 

func (c *MockgetterGetAuthorCall) Do(f func(context.Context, int64) ([]byte, error)) *MockgetterGetAuthorCall

Do rewrite *gomock.Call.Do

func (*MockgetterGetAuthorCall) DoAndReturn 

func (c *MockgetterGetAuthorCall) DoAndReturn(f func(context.Context, int64) ([]byte, error)) *MockgetterGetAuthorCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterGetAuthorCall) Return 

func (c *MockgetterGetAuthorCall) Return(arg0 []byte, arg1 error) *MockgetterGetAuthorCall

Return rewrite *gomock.Call.Return

type MockgetterGetBookCall 

type MockgetterGetBookCall struct {
	*gomock.Call
}

MockgetterGetBookCall wrap *gomock.Call

func (*MockgetterGetBookCall) Do 

func (c *MockgetterGetBookCall) Do(f func(context.Context, int64, editionsCallback) ([]byte, int64, int64, error)) *MockgetterGetBookCall

Do rewrite *gomock.Call.Do

func (*MockgetterGetBookCall) DoAndReturn 

func (c *MockgetterGetBookCall) DoAndReturn(f func(context.Context, int64, editionsCallback) ([]byte, int64, int64, error)) *MockgetterGetBookCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterGetBookCall) Return 

func (c *MockgetterGetBookCall) Return(arg0 []byte, workID, authorID int64, arg3 error) *MockgetterGetBookCall

Return rewrite *gomock.Call.Return

type MockgetterGetSeriesCall 

type MockgetterGetSeriesCall struct {
	*gomock.Call
}

MockgetterGetSeriesCall wrap *gomock.Call

func (*MockgetterGetSeriesCall) Do 

func (c *MockgetterGetSeriesCall) Do(f func(context.Context, int64) (*SeriesResource, error)) *MockgetterGetSeriesCall

Do rewrite *gomock.Call.Do

func (*MockgetterGetSeriesCall) DoAndReturn 

func (c *MockgetterGetSeriesCall) DoAndReturn(f func(context.Context, int64) (*SeriesResource, error)) *MockgetterGetSeriesCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterGetSeriesCall) Return 

func (c *MockgetterGetSeriesCall) Return(arg0 *SeriesResource, arg1 error) *MockgetterGetSeriesCall

Return rewrite *gomock.Call.Return

type MockgetterGetWorkCall 

type MockgetterGetWorkCall struct {
	*gomock.Call
}

MockgetterGetWorkCall wrap *gomock.Call

func (*MockgetterGetWorkCall) Do 

func (c *MockgetterGetWorkCall) Do(f func(context.Context, int64, editionsCallback) ([]byte, int64, error)) *MockgetterGetWorkCall

Do rewrite *gomock.Call.Do

func (*MockgetterGetWorkCall) DoAndReturn 

func (c *MockgetterGetWorkCall) DoAndReturn(f func(context.Context, int64, editionsCallback) ([]byte, int64, error)) *MockgetterGetWorkCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterGetWorkCall) Return 

func (c *MockgetterGetWorkCall) Return(arg0 []byte, authorID int64, arg2 error) *MockgetterGetWorkCall

Return rewrite *gomock.Call.Return

type MockgetterMockRecorder 

type MockgetterMockRecorder struct {
	// contains filtered or unexported fields
}

MockgetterMockRecorder is the mock recorder for Mockgetter.

func (*MockgetterMockRecorder) GetAuthor 

func (mr *MockgetterMockRecorder) GetAuthor(ctx, authorID any) *MockgetterGetAuthorCall

GetAuthor indicates an expected call of GetAuthor.

func (*MockgetterMockRecorder) GetAuthorBooks 

func (mr *MockgetterMockRecorder) GetAuthorBooks(ctx, authorID any) *MockgetterGetAuthorBooksCall

GetAuthorBooks indicates an expected call of GetAuthorBooks.

func (*MockgetterMockRecorder) GetBook 

func (mr *MockgetterMockRecorder) GetBook(ctx, bookID, saveEditions any) *MockgetterGetBookCall

GetBook indicates an expected call of GetBook.

func (*MockgetterMockRecorder) GetSeries 

func (mr *MockgetterMockRecorder) GetSeries(ctx, seriesID any) *MockgetterGetSeriesCall

GetSeries indicates an expected call of GetSeries.

func (*MockgetterMockRecorder) GetWork 

func (mr *MockgetterMockRecorder) GetWork(ctx, workID, saveEditions any) *MockgetterGetWorkCall

GetWork indicates an expected call of GetWork.

func (*MockgetterMockRecorder) Recommendations 

func (mr *MockgetterMockRecorder) Recommendations(ctx, page any) *MockgetterRecommendationsCall

Recommendations indicates an expected call of Recommendations.

func (*MockgetterMockRecorder) Search 

func (mr *MockgetterMockRecorder) Search(ctx, query any) *MockgetterSearchCall

Search indicates an expected call of Search.

type MockgetterRecommendationsCall 

type MockgetterRecommendationsCall struct {
	*gomock.Call
}

MockgetterRecommendationsCall wrap *gomock.Call

func (*MockgetterRecommendationsCall) Do 

func (c *MockgetterRecommendationsCall) Do(f func(context.Context, int64) (RecommentationsResource, error)) *MockgetterRecommendationsCall

Do rewrite *gomock.Call.Do

func (*MockgetterRecommendationsCall) DoAndReturn 

func (c *MockgetterRecommendationsCall) DoAndReturn(f func(context.Context, int64) (RecommentationsResource, error)) *MockgetterRecommendationsCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterRecommendationsCall) Return 

func (c *MockgetterRecommendationsCall) Return(arg0 RecommentationsResource, arg1 error) *MockgetterRecommendationsCall

Return rewrite *gomock.Call.Return

type MockgetterSearchCall 

type MockgetterSearchCall struct {
	*gomock.Call
}

MockgetterSearchCall wrap *gomock.Call

func (*MockgetterSearchCall) Do 

func (c *MockgetterSearchCall) Do(f func(context.Context, string) ([]SearchResource, error)) *MockgetterSearchCall

Do rewrite *gomock.Call.Do

func (*MockgetterSearchCall) DoAndReturn 

func (c *MockgetterSearchCall) DoAndReturn(f func(context.Context, string) ([]SearchResource, error)) *MockgetterSearchCall

DoAndReturn rewrite *gomock.Call.DoAndReturn

func (*MockgetterSearchCall) Return 

func (c *MockgetterSearchCall) Return(arg0 []SearchResource, arg1 error) *MockgetterSearchCall

Return rewrite *gomock.Call.Return

type Persister 

type Persister struct {
	// contains filtered or unexported fields
}

Persister tracks author refresh state across reboots.

func NewPersister 

func NewPersister(ctx context.Context, cache cache[[]byte], dsn string) (*Persister, error)

NewPersister creates a new Persister.

func (*Persister) Delete 

func (p *Persister) Delete(ctx context.Context, authorID int64) error

Delete records an in-flight refresh as completed.

func (*Persister) Persist 

func (p *Persister) Persist(ctx context.Context, authorID int64, bytes []byte) error

Persist records an author's refresh as in-flight.

func (*Persister) Persisted 

func (p *Persister) Persisted(ctx context.Context) ([]int64, error)

Persisted returns all in-flight author refreshes so they can be resumed. IDs are returned in FIFO order.

type RecommentationsResource 

type RecommentationsResource struct {
	WorkIDs []int64 `json:"workIds"`
}

RecommentationsResource contains recommended work IDs.

type Requestlogger 

type Requestlogger struct{}

Requestlogger logs some info about requests we handled.

func (Requestlogger) Wrap 

func (Requestlogger) Wrap(next http.Handler) http.Handler

Wrap applies middleware.

type ScopedTransport 

type ScopedTransport struct {
	Host string
	http.RoundTripper
}

ScopedTransport restricts requests to a particular host.

func (ScopedTransport) RoundTrip 

func (t ScopedTransport) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip forces the request to stick to the given host, so redirects can't send us elsewhere. Helpful to ensuring credentials don't leak to other domains.

type SearchResource 

type SearchResource struct {
	BookID int64                `json:"bookId"`
	WorkID int64                `json:"workId"`
	Author SearchResourceAuthor `json:"author"`
}

SearchResource represents a single search result.

type SearchResourceAuthor 

type SearchResourceAuthor struct {
	ID int64 `json:"id"`
}

SearchResourceAuthor is a nested field on SearchResource.

type SeriesResource 

type SeriesResource struct {
	ForeignID   int64  `json:"ForeignId"`
	Title       string `json:"Title"`
	Description string `json:"Description"`

	LinkItems []seriesWorkLinkResource `json:"LinkItems"`

	// New fields
	KCA string `json:"KCA"`
}

SeriesResource is a collection of works by one or more authors.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.