Changelog
0.1.9 — 2025-07-04
Section titled “0.1.9 — 2025-07-04”-
Added category-level publish functionality allowing atomic publishing of events to multiple streams within a category via
EventCategory.publish(). -
Added
StreamPublishDefinitionTypedDict for type-safe stream definitions in category publishes. -
Added
stream_publish_definitionfactory function to simplify creation of stream definitions for category-level publishes. -
Added
MultiLockutility class for coordinating multiple async locks in memory adapter. -
Added comprehensive test coverage for category publish functionality across all adapter implementations.
-
Added
EventStoreEventSourceFactoryforEventLogto enable log-level event processors.
Changed
Section titled “Changed”- Enhanced
EventStorageAdapterbase class with category-level save method supporting atomic multi-stream operations. - Implemented category-level save in
InMemoryEventStorageAdapterandPostgresEventStorageAdapterensuring atomicity and serialisation guarantee through appropriate locking.
0.1.8 — 2025-06-16
Section titled “0.1.8 — 2025-06-16”ContinueErrorHandlernow re-raises fatal exceptions and only continues on non-fatal exceptions.
0.1.7 — 2025-06-16
Section titled “0.1.7 — 2025-06-16”- The
make_event_brokerfactory was forcing anadapterofNoneto be passed despite the parameter being optional in the case of using Postgres storage. This has now been made optional.
0.1.6 — 2025-06-12
Section titled “0.1.6 — 2025-06-12”SingletonEventBrokerTypeType,DistributedEventBrokerTypeType,InMemoryEventBrokerStorageTypeTypeandPostgresEventBrokerStorageTypeTypehave been exported fromlogicblocks.event.processingso that they can be used wherever types are needed for theEventBrokerType.SingletonandEventBrokerType.Distributedbroker type specifiers.
Changed
Section titled “Changed”BrokerTypeandStorageType, used when invokingmake_event_broker, have been renamed toEventBrokerTypeandEventBrokerStorageTyperespectively.
0.1.5 — 2025-06-11
Section titled “0.1.5 — 2025-06-11”Removed
Section titled “Removed”- The
jsonmethods exposed by some types have been removed since they weren’t being used within the library. - The
NodeManager,NodeStateStore,InMemoryNodeStateStoreandPostgresNodeStateStoreclasses have been removed as they weren’t required by the subscription coordination algorithm. If you are creatingEventBrokerinstances using themake_*_event_brokerfactory functions, this will be invisible to you. - The
SubscriptionSourceMappingStoreandInMemorySubscriptionSourceMappingStoreclasses have been removed as they are no longer required by the subscription coordination algorithm. - Both
PostgresEventStoreEventSourceFactoryandInMemoryEventStoreEventSourceFactoryhave been removed - usages of both can be replaced withEventStoreEventSourceFactory.
- A
JsonValuetype alias type has been introduced which represents a datatype that can be converted to and from JSON. - Supporting
JsonObject,JsonArrayandJsonPrimitivetype alias types have been added for use when a subset ofJsonValueis required. - A
JsonValueTypetype has been introduced for situations where atypeis required, rather than a type alias type. - Type predicate functions,
is_json_value,is_json_object,is_json_arrayandis_json_primitivehave been added for use in type checking, with each function narrowing the type of its argument if the argument is of the requisite type. - Add a
@retry_on_unmet_condition_errorand@ignore_on_unmet_condition_errordecorators for handling common event processing scenarios. - Add more general
@retry_on_errordecorator retrying a function on a given error. - Add a
@event_store_transactiondecorator that wraps a function with arbitrary handling. - A new
StringPersistabletype has been added, along with correspondingStringSerialisable,StringDeserialisableandStringConvertibleprotocols representing the ability to serialise, deserialise and convert between custom types and strings. - Utility
serialise_to_stringanddeserialise_from_stringfunctions have been added for types adhering to theStringPersistabletype. EventStorecan now publish events into streams that have aJsonPersistablepayload and aStringPersistablename. This allows using custom types for each of these event fields so long as they can be appropriately serialised and deserialised. As yet, only writes support these types, with the primitive types being returned on read. However, a future change will add support for event schemas to allow typed reads.- When reading from the
EventStore, readers need to see events in a consistent order and should not skip any events being written concurrently. This requires serialising writes to the log such that all readers see the same event ordering all the time. However, sometimes it’s tolerable to only have consistent order without skips at category or stream level, e.g., when reading of events is only done at category or stream level respectively. To support this, thePostgresStorageAdapterandInMemoryStorageAdapternow allow the serialisation guarantee to be customised, by passing aEventSerialisationGuaranteeenum value (LOG,CATEGORY, orSTREAM) at construction time. - Added
ErrorHandlingServiceto allow for injection of error handling strategies in services. - Added
EventStore.log()that will return anEventLogclass which can read the entire log of events, unfiltered by category or stream.EventLoginherits fromEventSourcejust likeEventCategoryandEventStream, so it can be used in the same way as those classes, without thecategoryandstreamparameters. - The general purpose query package now supports
Functiontypes, currently only as part ofSortClause, but with the intention that these could additionally be supported inFilterClauseas well. The only function currently shipped with the library isSimilarity, which has been implemented using trigram matching. - As part of adding support for
Functions in the general purpose query language, the PostgreSQL and in-memory query conversion capabilities have been extended to support conversion of functions, allowing arbitraryFunctiontypes to be supported by the consumers of the conversion capability. - A new
ErrorHandlerabstraction has been introduced, withExitErrorHandler,RaiseErrorHandler,ContinueErrorHandler,RetryErrorHandlerandTypeMappingErrorHandlerimplementations. The existingErrorHandlingServicehas been extended to use this new abstraction, adding support for the various strategies for any delegate service.ExitErrorHandlerraises aSystemExitexception to exit the process. By default, it will exit with code 1, but this can be configured via theexit_codeconstructor parameter.RaiseErrorHandler, by default, raises the exception that was passed to it. This can be configured to raise a different exception via theexception_factoryconstructor parameter.ContinueErrorHandlerrequest that execution continues from the point that the error was handled. The decision included a value that can be used as the return value of the calling method if required. By default, the value isNone, but this can be configured via thevalue_factoryconstructor.RetryErrorHandlerrequests that the operation that caused the error be retried.TypeMappingErrorHandlerallows different error handling strategies for different exception types or their subclasses to be defined via thetype_mappingsconstructor parameter. By default, it will re-raise the handled exception, but this can be configured via thedefault_decision_factory. An example definition is:handler = TypeMappingErrorHandler(type_mappings=error_handler_type_mappings(exit_fatally=error_handler_type_mapping(types=[CompletelyFatalError],callback=log_exception),retry_execution=error_handler_type_mapping(types=[RetyableError],)))
- A new
ErrorHandlingServiceMixinhas been introduced to allow error handling to be mixed in toServiceclasses, rather than using theErrorHandlingServicedelegating approach. DistributedEventBrokernow automatically restarts on non-fatal exception. The error handling strategy can be configured via a newerror_handlerconstructor parameter, accepting anyErrorHandlerimplementation.- A new
SingletonEventBrokerhas been introduced, which is a much simplified event broker that assumes it is the only broker in the system and does no coordination or work distribution. In this model, the infrastructure is responsible for ensuring that only one instance of the broker is running at a time. - A new
make_event_brokerfunction has been added to thelogicblocks.event.processingpackage, which allows creation of different types of event brokers using different storage types. - Query now supports a
REGEX_MATCHESoperator, allowing regex matching on string fields. This uses the postgres ‘~’ operator for regex matching.
Changed
Section titled “Changed”- The
Codecprotocol has been split intoJsonValueSerialisableandJsonValueDeserialisable, with the composition of those being renamed toJsonValueConvertible. - The
CodecOrMappingtype alias has been renamed toPersistablesince that is what it is meant to represent. Similarly, aLoggabletype alias type has been introduced for those things that can be logged. - Rather than using
Mapping[str, Any]as the persistable type inProjectionStoreandEventStore,JsonValueis now used, which means implementations of theJsonValueConvertible(previouslyCodec) protocol must be changed to accept and produce aJsonValueinstead. - Types that previously supported a
dictmethod (e.g.,*Event,*Identifier) for rendering themselves to aMapping[str, Any] now have aserialisemethod which renders them to aJsonValue`. - Types that previously supported an
envelopermethod (e.g.,*Event,Projection) for rendering themselves to aMapping[str, Any] now have asummarisemethod which renders them to aJsonValue`. - The
NewEventandStoredEventclasses are now generic over their payload type, defaulting toJsonValue, rather thanMapping[str, Any]. - The
serialiseanddeserialisemethods, which are largely an internal detail of the library, but can be used by, for example, adapter implementers, have been made less strict in terms of the types they accept. - The
serialiseanddeserialisemethods now allow a fallback conversion callable to be provided in the case that they are unable to handle the provided values. By default, the fallback callables raiseValueError. - WriteConditions can now be combined using the
&and|operators- These can be chained together to create complex conditions, for example:
condition1 & condition2 | (condition3 & condition4)
- These can be chained together to create complex conditions, for example:
- Publish no longer takes a set of conditions, instead it accepts a single WriteCondition.
- “No conditions” is now represented by the
NoConditionsingleton rather than an empty set - The type parameters on
Projectorhave been reordered to better reflect the transformation taking place. So rather thanProjector[State, Identifier, Metadata]where identifier identifies the source of the events, the type signature is nowProjector[Identifier, State, Metadata] NewEventandStoredEventare now generic over their name type, in addition to the payload, i.e.,NewEvent[Name, Payload]andStoredEvent[Name, Payload]. This change is to allow methods/functions to target specific events, for example, asStoredEvent[Literal["profile-created"], ProfilePayload]. Previously, if two events had the same payload type but different names, it wasn’t possible to restrict a method/function signature to only one of them.- The
Persistabletype has been renamed toJsonPersistablefor parity with the addition ofStringPersistable. - The
Loggabletype has been renamed toJsonLoggablefor parity with the other JSON-related types. - The utility
serialiseanddeserialisefunctions have been renamed toserialise_to_json_valueanddeserialise_from_json_valuerespectively due to the addition ofserialise_to_stringanddeserialise_from_string. - The changes to how the
CoordinatorObserverEventBrokeroperates have resulted in breaking changes to the supporting table structure. Upgrading to this version requires migrations to drop thenodestable and to modify thesubscriberstable to have a JSONBsubscription_requestscolumn. Seesql/create_subscribers_table.sqlfor the latest table structure. - As part of adding support for
Functions in the general purpose query types,SortFieldandFilterClausehave had theirpathattribute renamed tofieldsince it now accepts, in the case ofSortFieldinitially, either aPathor aFunction. - The
EventBrokerSettingsclass has been renamed toDistributedEventBrokerSettingsto allow for other event broker implementations. - The
CoordinatorObserverEventBrokerclass has been renamed toDistributedEventBroker. ErrorHandlingServicehas been improved to allow for more error handling strategies such as exiting the process or retrying the operation. However, as part of this, a newErrorHandlertype has been introduced, which is expected to return a decision rather than raising or continuing directly.- The
DistributedEventBrokerhas undergone a number of changes in an effort to increase its robustness and reliability, specifically:- All long-running processes employed internally by the broker are now
executed within an
asyncio.TaskGroupto ensure each is correctly cancelled in case of an exception being thrown by any of them. The same approach has been used in those processes directly where they manage their own group of long-running processes. - The approach to acquiring the coordinator lock has been changed such that
now, rather than waiting for the lock to become available, the coordinator
attempts to acquire the lock and if it fails, it will retry on a schedule
controlled by the
coordinator_leadership_attempt_intervalsetting onDistributedEventBrokerSettings. - Rather than the leading coordinator continuing as leader until an error
occurs or the node is shut down, now the coordinator will abdicate after
some period of time, allowing another node to take over as leader. This
period is controlled by the
coordinator_leadership_max_durationsetting onDistributedEventBrokerSettings. - As mentioned in the “Added” section, the broker now automatically restarts on non-fatal exceptions, allowing it to recover from transient issues without manual intervention.
- All long-running processes employed internally by the broker are now
executed within an
- The
make_postgres_event_brokerandmake_in_memory_event_brokerfunctions have been deprecated in favour of the newmake_event_brokerfunction. EventStorageAdapteradapter argument can now optionally be provided to factories for building postgres event-brokers, in the case where you want to build a postgres backed event-broker but use an event-store with a different persistence backend.
Package Reorganisation
Section titled “Package Reorganisation”- The query types previously exported by the
logicblocks.event.projectionpackage have been moved to a dedicated package calledlogicblocks.event.querysince they are also used within theEventBrokerimplementation. This affects the following types:QueryClausePathOperatorSortFieldSortOrderLookupSearchFilterClauseSortClauseOffsetPagingClauseKeySetPagingClause
- The
logicblocks.event.dbpackage has been renamed tologicblocks.event.persistencesince it now houses persistence related abstractions for in-memory storage in addition to database-backed storage. - The
Postgres...prefixed types exported bylogicblocks.event.dbare now accessible without thePostgres...prefix fromlogicblocks.event.persistence.postgres. This affects the following types:PostgresConnectionSettingsPostgresConnectionSourcePostgresParameterisedQueryPostgresParameterisedQueryFragmentPostgresQueryPostgresConditionPostgresSqlFragment
- Some
Postgres...andInMemory...prefixed types exported bylogicblocks.projection.store.adaptersare now accessible without the prefixes fromlogicblocks.event.persistence.postgresandlogicblocks.event.persistence.memoryrespectively. This affects the following types:PostgresParameterisedQueryPostgresQueryPostgresQueryConverterPostgresTableSettingsInMemoryQueryConverterInMemoryClauseConverterInMemoryProjectionResultSetTransformer
PostgresTableSettingshas been generalised and renamed toTableSettingsand moved fromlogicblocks.event.store.adapters,logicblocks.projection.store.adapters,logicblocks.processing.broker.subscribers.stores.state.postgresandlogicblocks.processing.broker.subscriptions.stores.state.postresintologicblocks.event.persistence.postgressince it is now used in more contexts.
Behavioural Changes
Section titled “Behavioural Changes”- The
NoConditionvalue previously available inlogicblocks.event.store.conditionshas been changed to a class, such that it must be instantiated at the point of use.
Converter Abstraction
Section titled “Converter Abstraction”- The
PostgresQueryConverterabstraction previously utilised in the PostgreSQL projection storage adapter has been extracted and abstracted under thelogicblocks.event.persistencepackage. This approach was also copied in to thelogicblocks.event.processing.brokerpackage multiple times, and these copies have now been removed in favour of using the generalised abstraction. - The
QueryConverterabstraction has also been extended to the in-memory projection / subscriber / subscription storage cases for parity and to allow for further extension. - The
...Converterapproach to converting between generalised concepts and adapter-specific implementations has been also extended to the implementations ofQueryContstraints andWriteConditions in thelogicblocks.event.storepackage. - Previously,
QueryConstraints used a combination of a method on the constraint itself for in-memory storage adapter usage and asingledispatchfunction, declared in thelogicblocks.event.store.adapters.postgresmodule, for PostgreSQL storage adapter usage. The disparity by adapter type was confusing and inflexible and as such, thequery_constraint_to_sqlsingledispatchfunction and themet_bymethod on the constraint have been removed and a constraint converter has been introduced instead. The constraint converter is adapter-specific, allowing constraints to be converted into an appropriate form based on the nature of the storage mechanism. Seelogicblocks.event.store.adapters.postgres.convertersandlogicblocks.event.store.adapters.memory.convertersfor example implementations and extension points. - Similarly,
WriteConditions used a method on each condition which was passed the latest event in the stream, if it existed, to make an assessment of whether the condition was satisfied. Whilst this allowed for simplistic implementation of the condition check, it was not flexible enough to accommodate conditions which needed to be aware of properties of the whole stream or category. Instead, a condition converter has been introduced. Again, the condition converter is adapter-specific. For the PostgreSQL implementation, the condition enforcer receives the currently open connection, allowing for transactional checks to be performed against the database. For the in-memory implementation, the condition enforcer receives the currently open in-memory transaction allowing transaction checks against the in-memory database. Seelogicblocks.event.store.adapters.postgres.convertersandlogicblocks.event.store.adapters.memory.convertersfor example implementations and extension points.
- Fix race condition in coordinator.
- Fixed an issue in
make_subscriberso that a new subscriber ID is created by default each timemake_subscriberis called, if a value forsubscriber_idis not provided as an argument. - From time to time, the active
EventSubscriptionCoordinatorinside aCoordinatorObserverEventBrokerwould fail due to being unable to locate the subscription source mapping for a subscriber group that existed in the centralised subscriber state store. Upon failing, theEventBrokerwould become irrecoverable such that over time, all instances would fail and event processing would cease. To fix this, the sources requested by subscribers are now stored alongside the subscriber state in the centralised store.
- Allow for running
library:test:unit|integration|componentwith filter options to run only tests matching a specific pattern. Example usage:go "library:test:unit[TestAllTestsInFile]"orgo "library:test:component[test_a_specific_test]".