<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fred Feng on Medium]]></title>
        <description><![CDATA[Stories by Fred Feng on Medium]]></description>
        <link>https://medium.com/@paganini.fy?source=rss-c95d1b8ea67b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*G27gCh8gFq9tBsc_C0xPnQ.jpeg</url>
            <title>Stories by Fred Feng on Medium</title>
            <link>https://medium.com/@paganini.fy?source=rss-c95d1b8ea67b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:52:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@paganini.fy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kookaburra: an Embedded Object Database for Java, with Type-Safe Queries]]></title>
            <link>https://medium.com/@paganini.fy/kookaburra-an-embedded-object-database-for-java-with-type-safe-queries-0031d2f96f1b?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0031d2f96f1b</guid>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[javafx]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 10:07:56 GMT</pubDate>
            <atom:updated>2026-10-18T10:07:56.967Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An embedded object database for Java 17+: write queries like streams, and they execute like a database.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*4Qi7tbgIB7DN9XUZ" /></figure><h3>1. Overview</h3><p><strong>Kookaburra</strong>: <em>query Java objects like a database, right inside your application.</em></p><p>An embedded object database for Java 17+. Annotated classes in, type-safe chained queries or SQL-style OQL out, answered from indexes in milliseconds. Transactions at three isolation levels, unique and foreign-key constraints, exact decimals, online backup, and a desktop tool, Kookaburra Studio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yEHzI8TwRMkaVrtf" /></figure><p>Watch the 2:57 video tutorial (English subtitles):</p><p><a href="https://paganini2008.github.io/kookaburra/assets/kookaburra-tutorial.mp4">https://paganini2008.github.io/kookaburra/assets/kookaburra-tutorial.mp4</a></p><h3>2. What Problem Does It Solve</h3><p>Keeping your own objects usually means an ORM plus a relational database; the light alternative, stream().filter(), scans everything on every query. Kookaburra sits in between:</p><ul><li>No mapping layer: annotated objects are stored directly</li><li>No full scans: indexes answer in milliseconds on 10 million objects</li><li>No string field names: where(Book::getYear) or a generated Book_ metamodel</li><li>No duplicates or dangling references: @Unique on a field or a combination of fields, @ForeignKey with RESTRICT, CASCADE or SET_NULL</li><li>Safe multi-object changes: READ COMMITTED, SNAPSHOT and SERIALIZABLE transactions</li><li>Exact money: BigDecimal fields, compared by value</li><li>Backups while writing: db.backup(dir)</li><li>Not a black box: Kookaburra Studio browses, edits, queries and backs up any database</li></ul><h3>3. Quick Start</h3><p>Add the dependency:</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;com.github.paganini2008&lt;/groupId&gt;<br>    &lt;artifactId&gt;kookaburra-core&lt;/artifactId&gt;<br>    &lt;version&gt;1.0.0-SNAPSHOT&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>Store and query:</p><pre>try (ObjectDB db = ObjectDB.open(Path.of(&quot;data&quot;))) {<br>    db.batch()<br>      .insert(new Book(&quot;Dune&quot;, &quot;Frank Herbert&quot;, Genre.SCIFI, 1965, 9.99, &quot;classic&quot;, &quot;desert&quot;))<br>      .insert(new Book(&quot;Neuromancer&quot;, &quot;William Gibson&quot;, Genre.SCIFI, 1984, 12.50, &quot;cyberpunk&quot;))<br>      .insert(new Book(&quot;Hyperion&quot;, &quot;Dan Simmons&quot;, Genre.SCIFI, 1989, 14.20))<br>      .commit();<br><br>    List&lt;Book&gt; books = db.query(Book.class)<br>            .where(Book::getGenre).eq(Genre.SCIFI)<br>            .where(Book::getYear).ge(1960)<br>            .orderBy(Book::getYear).desc()<br>            .limit(2)<br>            .list();<br>    System.out.println(books.stream().map(Book::getTitle).toList());<br>}</pre><p>Output:</p><pre>[Hyperion, Neuromancer]</pre><h3>4. Requirements</h3><ul><li>Java 17+</li><li>Runtime dependencies: RoaringBitmap 1.6.22, the ANTLR 4.13.2 runtime, SLF4J API 2.0</li><li>Maven 3.8+ to build</li><li>Studio: Java 17+; one jar for Windows x64, Linux x64 and macOS Apple Silicon (JavaFX bundled)</li><li>Field types: boolean, all integer types, float / double, BigDecimal, String, enums, LocalDate / LocalDateTime / Instant, Ref&lt;T&gt;, lists and sets of these, @Embeddable classes (and lists / maps of them), Map, byte[], any Serializable</li></ul><h3>5. How It Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k_oIWsWnGvcW7vFS" /></figure><ul><li>You annotate plain classes; Kookaburra stores them in a data directory (or in memory). No server, no schema scripts.</li><li>You query with method references or OQL; indexes answer, and explain() shows which one.</li><li>You write objects, batches or transactions; commits are atomic and durable, constraints are checked at commit.</li><li>You open the directory in Studio, or call db.backup(dir) for a consistent copy while running.</li></ul><h3>6. Code Examples</h3><p><strong>Example 1</strong>: group and aggregate</p><p>Input: six books (4 sci-fi, 1 fantasy, 1 crime).</p><pre>db.query(Book.class).groupBy(where(Book::getGenre))<br>  .agg(Q.count().as(&quot;books&quot;), Q.avg(where(Book::getPrice)).as(&quot;avgPrice&quot;))<br>  .sortedBy(Q.count().as(&quot;books&quot;).desc())<br>  .toList();</pre><p>Output:</p><pre>SCIFI   books=4 avg=11.36<br>FANTASY books=1 avg=10.00<br>CRIME   books=1 avg=7.40</pre><p><strong>Example 2</strong>: OQL and a bulk update</p><pre>db.oql(&quot;SELECT b.author, b.year FROM Book b WHERE b.year &lt; :y ORDER BY b.year&quot;).param(&quot;y&quot;, 1950).rows();<br>db.oql(&quot;UPDATE Book b SET b.price = b.price * 0.9 WHERE b.genre = &#39;CRIME&#39;&quot;).execute();</pre><p>Output:</p><pre>J.R.R. Tolkien 1937<br>Raymond Chandler 1939<br>updated=1</pre><p><strong>Example 3</strong>: constraints, decimals and callbacks</p><pre>@Entity<br>public class Order {<br>    @Id long oid;<br>    @Unique String number;<br>    @ForeignKey(onDelete = OnDelete.CASCADE) Ref&lt;Customer&gt; customer;<br>    @Index BigDecimal total;<br>    Instant created;<br><br>    @PrePersist void stamp() { created = Instant.now(); }<br>}</pre><pre>insert a second &quot;A-1&quot;        → ConstraintViolationException: duplicate value &#39;A-1&#39; for unique Order.number<br>sum(total) of 19.90 + 10.15  → 30.05 (exact)<br>db.referrers(Order.class, ann) → [A-1, A-2]<br>delete customer Ann          → her orders go in the same commit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zlfT1D7z0JS7llJF" /></figure><p><strong>Example 4</strong>: SERIALIZABLE stops write skew</p><p>Two doctors are on call; each one goes off call if the other is still on call.</p><pre>db.inTransaction(TxOptions.defaults().isolation(Isolation.SERIALIZABLE), tx -&gt; {<br>    long onCall = tx.query(Doctor.class).where(Doctor::isOnCall).eq(true).count();<br>    if (onCall &gt; 1) {<br>        tx.update(tx.get(Doctor.class, me).offCall());<br>    }<br>});</pre><p>Under SNAPSHOT both commit and nobody is on call. Under SERIALIZABLE the second commit fails with a “serialization failure”, is retried, and one doctor stays on call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rh9KOkGo9Hh71D1j" /></figure><p><strong>Example 5</strong>: online backup</p><pre>BackupInfo info = db.backup(Path.of(&quot;backups/2026-10-17&quot;));   // writers keep going<br>try (ObjectDB copy = ObjectDB.open(info.path())) { … }        // restore = open</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hpXyXZe5I5YsjS3r" /></figure><p><strong>Example 6</strong>: nested objects and large values</p><pre>db.oql(&quot;SELECT s.code FROM Shipment s WHERE s.route.to.city = &#39;Perth&#39;&quot;).list();        // [S-1]<br>db.oql(&quot;SELECT s.code FROM Shipment s WHERE &#39;kiwi&#39; MEMBER OF s.items.sku&quot;).list();     // [S-2]<br>db.insert(new Document(&quot;scan&quot;, text, Files.readAllBytes(Path.of(&quot;scan.png&quot;))));        // MiB-sized values</pre><p>In Studio: the objects referencing the selected one, the constraints of every field, nested values as a tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8i0yFGrpR9q_ASOd" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N-R9Mtrb7WPibWhk" /></figure><h3>7. Configuration</h3><p>All settings live in one file, kookaburra.properties:</p><pre>kookaburra.durability=SYNC               # SYNC = on disk before a write returns, ASYNC = faster<br>kookaburra.transactions=false            # true enables db.begin() / db.inTransaction()<br>kookaburra.tx-isolation=SNAPSHOT         # READ_COMMITTED, SNAPSHOT or SERIALIZABLE<br>kookaburra.tx-timeout=60s                # open transactions older than this are rolled back<br>kookaburra.data=                         # Studio: default database (empty = data/ next to the jar)</pre><p>Tuning keys (memory, file sizes, background work) are commented in the file; the defaults suit most applications.</p><h3>8. Performance</h3><p>Environment: Apple M2 Max (12 cores), 32 GB, internal SSD, JDK 17, -Xmx8g, 10 million objects, measured with the kookaburra-bench harness (warm-up, p50, consistency checks after every phase).</p><pre>Operation (10 M objects)                           p50<br>-------------------------------------------------  -------------------<br>bulk insert, batches of 100 (ASYNC)                696 k objects/s<br>durable single inserts, 16 / 128 threads (SYNC)    3.2 k/s / 21.2 k/s<br>get(oid)                                           0.6 to 1.1 µs<br>count with an indexed filter                       0.24 ms<br>equality + range + limit 100                       0.21 ms<br>top-10 by indexed column, 2.3 M matches            0.39 ms<br>GROUP BY city COUNT(*)                             4.0 ms<br>unique-key lookup (OQL)                            0.12 ms<br>full scan, unindexed predicate                     0.81 s<br>transfers, 16 threads: SNAPSHOT / RC / SERIAL.     136 / 142 / 140 k/s<br>atomic increment of one hot counter, 16 threads    265 k/s, no lost updates<br>insert with @Unique / without (in memory)          235 / 258 k/s<br>online backup of 10.5 M objects                    25.4 s, writers not blocked<br>OQL DELETE of all objects / disk afterwards        1.32 M/s / 0 B left<br>reopen / heap after reopen                         2.0 ms / 6.5 MiB</pre><h3>9. Limitations &amp; Trade-offs</h3><ul><li>Each durable single write waits for the disk; batch or use ASYNC for bulk loads.</li><li>Single-collection queries only, no joins: use references with fetch, or the semi-join in(...).</li><li>Optimistic transactions: conflicts surface at commit and are retried, never blocked.</li><li>Not a fit for several processes writing one directory, for the SQL / JDBC / BI ecosystem, or for terabyte-scale data.</li></ul><h3>10. Summary</h3><ol><li>Stream-style code, database-style execution.</li><li>Type-safe fields from method references or a generated metamodel.</li><li>OQL covers the single-table SQL-92 core.</li><li>Three isolation levels; SERIALIZABLE closes write skew without locks.</li><li>@Unique and @ForeignKey with ON DELETE actions, checked atomically at commit.</li><li>Exact decimals, nested objects, large values.</li><li>Lifecycle callbacks and listeners, JPA style.</li><li>Online backup of a consistent snapshot; restore = open.</li><li>Sub-millisecond to a few milliseconds on 10 M objects; instant reopen, tiny heap.</li><li>One Studio jar; every feature tested in memory and on disk, coverage ≥ 90 %.</li></ol><p><em>Source and docs: </em><a href="https://github.com/paganini2008/kookaburra"><em>github.com/paganini2008/kookaburra</em></a><em> · </em><a href="https://paganini2008.github.io/kookaburra/"><em>paganini2008.github.io/kookaburra</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0031d2f96f1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SqlJam: Move Tables Across Databases with Schema, Data and All]]></title>
            <link>https://medium.com/@paganini.fy/sqljam-move-tables-across-databases-with-schema-data-and-all-03d2a339e73f?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/03d2a339e73f</guid>
            <category><![CDATA[jdbc]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[javafx]]></category>
            <category><![CDATA[oracle]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 05:10:38 GMT</pubDate>
            <atom:updated>2026-10-11T05:10:38.395Z</atom:updated>
            <content:encoded><![CDATA[<p>Tables, indexes, constraints, sequences, partitions and rows, in one go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k33iTEovYj6qaocY.png" /></figure><h3>Overview</h3><p>SqlJam is a Java 17 + JavaFX desktop tool that copies tables between relational and OLAP databases (MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse and more) in any direction. Import directly into a target database or into the same schema as copies, or save a self-describing export package of SQL or Parquet files and import it later.</p><p>Source: <a href="https://github.com/paganini2008/sqljam">https://github.com/paganini2008/sqljam</a></p><h3>What Problem Does It Solve?</h3><p>The hard part of a cross-database migration is not INSERT, it is the details: type names, identity columns, default value syntax, unsigned overflow, timestamps shifted by time zones, BLOBs inflating scripts, and old servers rejecting new syntax.</p><p>SqlJam reads one metadata tree and generates SQL with a dialect chosen by target type and version:</p><ul><li>Dialect differences → types, identifiers, defaults, identity and literals translated by the dialect of each target</li><li>Old versions → version subclasses: Oracle11gDialect, PostgreSQL9Dialect, MySQL56Dialect, SQLServer2008Dialect</li><li>Missing keys and indexes → primary keys, indexes, foreign keys, comments, sequences and partitions are copied</li><li>Huge scripts → data files split at 10 MB: orders.sql, orders_2.sql …</li><li>LOBs → stored as files and restored by primary key</li><li>Scripts without context → manifest.json with source, target, options, SHA-256 per file and row counts</li><li>Analytics needs Parquet → Parquet packages through an embedded DuckDB, Parquet files of other tools loaded into any table</li><li>Warehouses apart from OLTP → DuckDB and ClickHouse data sources, grouped as OLAP databases</li><li>Only part of a table → a data viewer query (columns, WHERE, GROUP BY, ORDER BY) and export of its rows</li><li>A copy next to the original → copies in the same schema with {table}_copy, keys and indexes renamed</li><li>Custom and modern types → PostgreSQL enums and domains, vectors, unions, maps, tuples and variants</li></ul><h3>Quick Start</h3><pre>git clone git@github.com:paganini2008/sqljam.git &amp;&amp; cd sqljam<br>./mvnw -DskipTests package<br>bin/sqljam.sh</pre><p>On Windows use mvnw.cmd and bin\sqljam.bat. The build writes everything to bin/: a runnable jar for each platform, the launchers, sqljam.properties and sqljam.vmoptions. Double click the jar of your platform, or run the launcher. Edit sqljam.properties or sqljam.vmoptions and restart, no rebuild needed.</p><p>A splash screen shows the loading of the configuration, the data sources and the JDBC drivers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*nK_Oe5Rtbs1tEHSV.png" /></figure><p>A short tutorial video walks through all of the steps below with the example database: <a href="https://github.com/paganini2008/sqljam/blob/master/docs/assets/sqljam-tutorial.mp4">https://github.com/paganini2008/sqljam/blob/master/docs/assets/sqljam-tutorial.mp4</a></p><p>Step 1. log in to a data source. The first start brings an Example Shop (H2) data source with a few e-commerce tables to try everything right away. Example on the login page or Help → Restore Example Database brings it back at any time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*spj3bk_8G8e_5I7Q.png" /></figure><p>Step 2. choose tables and a target in the export wizard. A target data source that does not exist yet is created from the + button next to it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*7YpbgA_4nVPBD4Sc.png" /></figure><p>Step 3. watch the progress:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*cfpnTiIYV6p7tx04.png" /></figure><p>Step 4. narrow the rows of a table on the Data tab and export just those rows. Right-click a connected data source to Disconnect when you are done, and Help → About SqlJam shows the version, the home page and the repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LZkk11SCum3JuXul.png" /></figure><p>Step 5. look inside an export package, Parquet files included:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qn6mxZzAlg-Wjtgr.png" /></figure><p>Step 6. load Parquet files of other tools into a table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*4K1PSC0zBQJae0n4.png" /></figure><h3>Requirements</h3><p>Only Java 17+ is needed. JavaFX and the JDBC drivers are inside the runnable jar, and the Maven Wrapper builds without a Maven installation.</p><p>Platforms:</p><ul><li>✅ Windows 10 / 11 (x64): sqljam — win.jar, sqljam.bat</li><li>✅ macOS Apple Silicon: sqljam — mac-aarch64.jar, sqljam.sh</li><li>✅ macOS Intel: sqljam — mac.jar, sqljam.sh</li><li>✅ Linux x64 (GTK 3): sqljam — linux.jar, sqljam.sh</li></ul><p>Databases:</p><ul><li>MySQL 5.5 to 9.x</li><li>PostgreSQL 9.x to 16</li><li>Oracle 11g to 23ai</li><li>SQL Server 2008 to 2022</li><li>MariaDB 10.3 to 11.x</li><li>H2 2.x, SQLite 3.x</li><li>DuckDB 1.x and ClickHouse 24.x to 26.x (OLAP)</li><li>All JDBC drivers are bundled</li></ul><h3>How It Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KDqAhg0O4HIUYZHF.png" /></figure><ul><li>Read metadata once: per-database operations complete comments, identity, generated columns, partitions and sequences.</li><li>Generate for the target: DbType.createDialect(major, minor) picks the version subclass.</li><li>Stream rows: count rows for percentage progress, page by primary key, normalize values, then batch insert or write SQL literals.</li><li>Parquet: rows pass an embedded DuckDB workspace, COPY TO writes Parquet files and read_parquet loads them back.</li></ul><h3>Code Examples</h3><h3>Example 1: export a MySQL database as a PostgreSQL package</h3><pre>ScriptExporter exporter = new ScriptExporter(new File(&quot;export&quot;),<br>        DataFileStrategy.FILE_PER_TABLE, 10 * 1024 * 1024);<br>Exporter.ExportConfiguration config = exporter.getConfiguration();<br>config.setDbType(DbType.MYSQL);<br>config.setUrl(DbType.MYSQL.getUrl(&quot;localhost&quot;, 3306, &quot;shop&quot;));<br>config.setUsername(&quot;root&quot;);<br>config.setPassword(&quot;secret&quot;);<br>config.setIdReused(true);<br>exporter.setTargetDbType(DbType.POSTGRESQL);<br>exporter.exportDdlAndData();</pre><p>Output:</p><pre>export/<br>  schema.sql<br>  data/orders.sql<br>  data/orders_2.sql<br>  lob/<br>  lob-manifest.json<br>  constraints.sql<br>  manifest.json</pre><p>The same tables as Parquet files:</p><pre>ParquetExporter parquet = new ParquetExporter(new File(&quot;export-parquet&quot;));<br>parquet.setTargetDbType(DbType.POSTGRESQL);<br>parquet.setCompression(&quot;ZSTD&quot;);<br>parquet.exportDdlAndData();</pre><h3>Example 2: Oracle → SQL Server directly</h3><pre>ImportExporter importer = new ImportExporter();<br>Exporter.ExportConfiguration source = importer.getExportConfiguration();<br>source.setDbType(DbType.ORACLE);<br>source.setUrl(DbType.ORACLE.getUrl(&quot;ora-host&quot;, 1521, &quot;ORCL&quot;));<br>source.setUsername(&quot;hr&quot;);<br>source.setPassword(&quot;secret&quot;);<br>source.setIncludedSchemaNames(new String[]{&quot;HR&quot;});<br>ImportExportHandler.ImportConfiguration target = importer.getImportConfiguration();<br>target.setDbType(DbType.SQLSERVER);<br>target.setUrl(DbType.SQLSERVER.getUrl(&quot;mssql-host&quot;, 1433, &quot;demo&quot;));<br>target.setUsername(&quot;sa&quot;);<br>target.setPassword(&quot;secret&quot;);<br>target.setTargetSchemaName(&quot;hr&quot;);<br>importer.exportDdlAndData();</pre><p>Output: tables, keys, indexes, foreign keys, comments and sequences in demo.hr. Identities continue from the imported max value.</p><p>Rows of a query and copies in the same schema:</p><pre>source.getTableQueries().put(&quot;EMPLOYEES&quot;, new TableQuery(List.of(&quot;EMPLOYEE_ID&quot;, &quot;NAME&quot;, &quot;SALARY&quot;),<br>        &quot;SALARY &gt; 1000&quot;, null, &quot;SALARY DESC&quot;));<br>target.setTableNamePattern(&quot;{table}_copy&quot;);</pre><p>Type translation examples:</p><ul><li>MySQL bigint unsigned → PostgreSQL numeric(20, 0) · Oracle NUMBER(20,0) · SQL Server decimal(20,0)</li><li>PostgreSQL jsonb → Oracle CLOB · SQL Server nvarchar(max) · MySQL json</li><li>SQL Server datetime2(7) → PostgreSQL timestamp · Oracle TIMESTAMP(7) · MySQL datetime(6)</li><li>PostgreSQL enum / domain → same type created in the target schema · varchar or the base type elsewhere</li><li>MySQL / Oracle VECTOR → PostgreSQL text · SQL Server nvarchar(max) · same type kept</li><li>ClickHouse Array(T) / Map(K, V) → PostgreSQL text · Oracle CLOB · SQL Server nvarchar(max)</li></ul><h3>Configuration</h3><p>All settings live in bin/sqljam.properties next to the runnable jar. Settings changed in the UI are saved to ~/.sqljam/sqljam.properties and win over the file.</p><ul><li>sqljam.export.max-file-size = 10485760, max bytes of a data file</li><li>sqljam.export.data-file-strategy = SINGLE_FILE, or FILE_PER_TABLE</li><li>sqljam.export.lob-separated = true, write LOBs as files</li><li>sqljam.export.page-size = 5000, rows per page, tuned by benchmark</li><li>sqljam.export.lob-page-size = 100, rows per page for tables with LOB columns</li><li>sqljam.import.batch-size = 1000, INSERT statements per batch when importing a package</li><li>sqljam.ui.theme = Primer Dark, 7 themes</li><li>sqljam.pool.maximum-size = 10, connection pool size</li></ul><h3>Performance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JfhC8KwCk3Uv4qn6.png" /></figure><p>200,000 rows × 6 columns, Apple M2 Max, 32 GB, JDK 17 (MySQL/PostgreSQL local, Oracle/SQL Server in Docker, default settings):</p><ul><li>H2 → Oracle 23ai: 2.1 s, 95.3k rows/s</li><li>H2 → export package (32 MB, 4 files): 2.9 s, 69.4k rows/s</li><li>H2 → PostgreSQL 16: 3.2 s, 62.4k rows/s</li><li>H2 → MySQL 9.6: 3.3 s, 60.7k rows/s</li><li>H2 → SQL Server 2022: 5.0 s, 40.1k rows/s</li><li>Export package → PostgreSQL: 4.5 s, 44.5k rows/s</li></ul><p>Quality: 578 tests (full cross-database import matrix with every column type of every database, export package and Parquet round trips, copies in the same schema, old-version SQL executed on real servers, UI and boundary tests), 91% line coverage.</p><h3>Design Trade-offs</h3><ul><li>Table-level focus: tables, keys, indexes, constraints, sequences, partitions and comments.</li><li>Rows paged by primary key: stable, deterministic reads on every database.</li><li>Tables copied one by one: predictable load on production servers.</li><li>Plain SQL in packages: readable and portable. Direct import uses batched inserts for speed.</li><li>Version-specific SQL: older servers get the syntax they understand.</li><li>Parquet through DuckDB: one embedded engine writes and reads Parquet, no Hadoop libraries.</li><li>Grouped rows are for viewing: exports carry real table rows.</li><li>Copies need their own names: a copy in the same schema takes a name such as {table}_copy.</li></ul><h3>Summary</h3><ol><li>Any-to-any copies. Every cross-database pair passes automated tests.</li><li>Dialects are chosen by type + version. Old versions are subclasses.</li><li>Tables, indexes, constraints, sequences, partitions and comments move together.</li><li>Export package = SQL + LOB files + manifest.json. Export and import are paired.</li><li>Data files split at 10 MB by default. LOBs stored separately.</li><li>Timestamps, unsigned numbers, bits, money, UUID and JSON survive the trip.</li><li>Pooled connections and batch inserts: 40k to 95k rows/s for direct imports.</li><li>A mainstream dark UI with 7 themes, percentage progress and cancel.</li><li>Parquet packages and Parquet files of other tools, plus DuckDB and ClickHouse as OLAP data sources.</li><li>Query the rows of a table, export just those rows, or copy tables inside the same schema.</li></ol><p>GitHub: <a href="https://github.com/paganini2008/sqljam">https://github.com/paganini2008/sqljam</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03d2a339e73f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vortex TSDB: Live Metrics in One Command]]></title>
            <link>https://medium.com/@paganini.fy/vortex-tsdb-live-metrics-in-one-command-668bbb731edd?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/668bbb731edd</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[time-series-database]]></category>
            <category><![CDATA[tsdb]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 11:40:07 GMT</pubDate>
            <atom:updated>2026-10-05T11:40:07.087Z</atom:updated>
            <content:encoded><![CDATA[<p>Lightweight. Replicated. Real-time. HTTP writes, minute aggregates, sliding windows — every node holds the whole dataset in memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Asl8-Ufr3CX7-YsT" /></figure><p>Version 1.0.0 · Images: fredfeng033/vortex-tsdb and ghcr.io/paganini2008/vortex-tsdb (amd64 · arm64) · Source: <a href="https://github.com/paganini2008/vortex">https://github.com/paganini2008/vortex</a> · Apache 2.0</p><h3>1. What Problem Does It Solve?</h3><p>Live metrics need exact counts from many writers, aggregation over any window, and a service that survives a dead node. General-purpose TSDBs bring a disk engine, a query language and several components. Vortex does only this layer, in memory, behind plain HTTP.</p><ul><li>Many components → one service + optional Redis</li><li>Counts drift under concurrency → writes serialised on the leader, then replicated</li><li>Windows need a query language → range · step · window · z, folded at read time</li><li>One node down, service down → full replica everywhere; new leader in ~5 s</li><li>Memory grows with every series → per-node key limit, overflow to Redis</li></ul><h3>2. Quick Start</h3><p>Free on Docker Hub and GHCR. No JDK, Node or build needed.</p><pre>curl -fsSLO https://raw.githubusercontent.com/paganini2008/vortex/main/deploy/docker-compose.yml<br>docker compose up -d<br><br>curl -X POST &#39;http://localhost:9080/tsd/push?t=long&amp;c=car&amp;d=speed&amp;v=44&#39;<br>curl &#39;http://localhost:9080/tsd/last?t=long&amp;c=car&amp;d=speed&#39;</pre><pre>{&quot;code&quot;:1,&quot;data&quot;:{&quot;value&quot;:44,&quot;timestamp&quot;:1791182928819},&quot;msg&quot;:&quot;ok&quot;, ...}</pre><ul><li>Web console: <a href="http://localhost:9080/">http://localhost:9080/</a></li><li>Swagger UI: <a href="http://localhost:9080/swagger-ui.html">http://localhost:9080/swagger-ui.html</a></li><li>From GHCR: VORTEX_REGISTRY=ghcr.io/paganini2008 docker compose up -d</li><li>Your Redis: VORTEX_REDIS_HOST=host.docker.internal VORTEX_REDIS_PASSWORD=secret</li><li>docker compose up -d</li></ul><p>Nodes take 20–30 s to start.</p><h3>3. Requirements</h3><ul><li>Run: Docker with compose v2</li><li>Overflow (optional): Redis, tested with 7.4 and 8.6</li><li>Build from source: JDK 17+, Maven 3.9, Node.js 20+</li><li>Stack: Spring Boot 4.1 · <a href="https://github.com/chaconne-ai/openspreader.git"><strong>Openspreader</strong></a> · Next.js 16 · Traefik v3</li></ul><h3>4. How It Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jm1_1oWLKLpkwUlC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KdUPEcJLj6izY1ry" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fMp_rGvygs3S-627" /></figure><ul><li>Minute bucket — max · min · sum · count per series per minute; retention as TTL</li><li>Latest value — one hash per category, read in one call</li><li>Catalog — every series by its last sample</li><li>Health — each node publishes its figures every 5 s</li></ul><h3>5. Code Examples</h3><p>Example 1 — push and read back</p><pre>for v in 44 67 112; do curl -X POST &quot;localhost:9080/tsd/push?t=long&amp;c=car&amp;d=speed&amp;v=$v&quot;; done<br>curl &#39;localhost:9080/tsd/retrieve?t=long&amp;c=car&amp;d=speed&amp;z=Asia/Shanghai&#39;</pre><pre>&quot;09:29:00&quot;: {&quot;count&quot;: 3, &quot;highestValue&quot;: 112, &quot;lowestValue&quot;: 44, &quot;totalValue&quot;: 223, &quot;averageValue&quot;: 74.3333}</pre><p>Example 2 — a sliding window</p><pre>curl &#39;localhost:9080/tsd/query?t=long&amp;c=car&amp;d=speed&amp;range=6h&amp;step=5&amp;window=15&#39;</pre><p>72 points, each over the 15 minutes before it. With window = step they are tumbling windows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zc0hfb15yeLbVy3u" /></figure><p>Example 3 — the whole cluster</p><pre>curl &#39;localhost:9080/tsd/categories&#39;<br>curl &#39;localhost:9080/tsd/health&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DPZlfbwa_45iX8py" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x3hwDt3wzI1AiJ8P" /></figure><h3>6. Configuration</h3><pre>VORTEX_RETENTION          24h     how long data is kept<br>VORTEX_SPAN_MINUTES       1       bucket width; divides 60<br>VORTEX_TIME_ZONE          UTC     zone when a request has no z<br>VORTEX_CACHE_MAX_KEYS     10000   keys per node in memory<br>VORTEX_REDIS_HOST         -       overflow Redis; blank deletes keys beyond the limit<br>VORTEX_SNAPSHOT_INTERVAL  5m      snapshot interval<br>VORTEX_GATEWAY_PORT       9080    host port</pre><h3>7. Performance &amp; Comparison</h3><p>Docker Desktop, 4 CPUs / 8 GB, 3 nodes, k6 through the gateway.</p><ul><li>826,781 writes at rising rates: 0 failures, identical totals on all nodes</li><li>Sustained: ~1,000 samples/s</li><li>Peak: 1,400–1,500 samples/s, leader ~2.3 CPUs, heap ≤ 310 MB</li><li>kill -9 the leader at 600/s: new leader in 4.7 s, 0.28% of requests failed</li><li>Graceful leader stop: immediate hand-over, 0 failures</li></ul><pre>             Vortex          Prometheus   InfluxDB      TimescaleDB<br>Writes       HTTP push       pull         HTTP push     SQL<br>Queries      HTTP params     PromQL       InfluxQL/SQL  SQL<br>Storage      memory+snap     disk         disk          PostgreSQL<br>HA           built in        extra setup  by edition    PG tooling<br>Best for     live metrics    monitoring   time series   analytics</pre><h3>8. Limitations &amp; Trade-offs</h3><ul><li>One leader applies every write — throughput does not grow with nodes</li><li>Data in memory — every node killed at once means back to the last snapshot</li><li>Async replication — writes in flight when the leader crashes fail; a few may be stored</li><li>Redis overflow — reads of spilled keys cost a round trip</li><li>No query language, alerting or auth — keep it on a private network</li></ul><p>Not for: ledgers · months of retention · tens of thousands of writes per second.</p><h3>9. Summary</h3><ol><li>One command: docker compose up -d, amd64 and arm64</li><li>Just HTTP: t/c/d/v to write, range/step/window/z to read</li><li>Exact: leader-serialised writes, no locks</li><li>Local reads: every node a full replica</li><li>Windows for free: folded from minute buckets</li><li>Self-healing: new leader in ~5 s</li><li>Durable enough: snapshots at stop and every 5 minutes</li><li>Bounded: key limit per node, optional Redis</li><li>Observable: /tsd/health and a web console</li><li>A clear niche: live metrics with days of retention</li></ol><p><strong>GitHub</strong>: <a href="https://github.com/paganini2008/vortex">https://github.com/paganini2008/vortex</a></p><p><strong>Website</strong>: <a href="https://paganini2008.github.io/vortex/">https://paganini2008.github.io/vortex/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=668bbb731edd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EasyJPA: Criteria API Power in Three Lines, Not Thirty]]></title>
            <link>https://medium.com/@paganini.fy/easyjpa-criteria-api-power-in-three-lines-not-thirty-f2aad7e4e8f9?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2aad7e4e8f9</guid>
            <category><![CDATA[hibernate-jpa]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[spring-data-jpa]]></category>
            <category><![CDATA[jpa]]></category>
            <category><![CDATA[jdbc]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 08:22:10 GMT</pubDate>
            <atom:updated>2026-10-01T08:22:10.862Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>EasyJPA</strong> is a Spring Boot starter that puts a fluent, lambda-driven API over the JPA Criteria API. Every condition is a method reference, every join has a name, and not a single JPQL string is written. Joins, subqueries, grouping, pagination, updates and native SQL all compose in one chain you read top to bottom.</p><p>Here is the entire pitch.</p><p>The Criteria API:</p><pre>CriteriaBuilder cb = em.getCriteriaBuilder();<br>CriteriaQuery&lt;User&gt; cq = cb.createQuery(User.class);<br>Root&lt;User&gt; root = cq.from(User.class);<br>List&lt;Predicate&gt; predicates = new ArrayList&lt;&gt;();<br>predicates.add(cb.equal(root.get(&quot;username&quot;), &quot;Jack&quot;));<br>predicates.add(cb.equal(root.get(&quot;password&quot;), &quot;123456&quot;));<br>cq.select(root).where(cb.and(predicates.toArray(new Predicate[0])));<br>User user = em.createQuery(cq).getSingleResult();</pre><p>EasyJPA:</p><pre>User user = userDao.query()<br>        .filter(new FilterList().eq(User::getUsername, &quot;Jack&quot;).eq(User::getPassword, &quot;123456&quot;))<br>        .selectThis().one();</pre><p>Same query, same type safety, same generated SQL.</p><h3>What Problem Does It Solve?</h3><p>The Criteria API is the only type-safe way to build a dynamic JPA query, and almost nobody enjoys using it. A three-condition filter costs ten lines of CriteriaBuilder plumbing; a multi-branch join means tracking Join&lt;?, ?&gt; variables by hand; a correlated subquery is close to unreadable. So teams fall back on JPQL strings or Spring Data method names — and lose type safety, or composability, or both.</p><p>The second problem is quieter and costs more. A paginated query is really <em>two</em> statements, the listing one and the counting one, and they are written separately. They drift. And when the query has a group by, the obvious count(*) counts rows rather than groups, so the total is simply wrong. EasyJPA derives both statements from one definition, and counts groups through a derived table.</p><p>It keeps Criteria’s type safety and dynamic composition.</p><p>It drops the CriteriaBuilder / Root / Predicate[] boilerplate.</p><p>It fixes pagination counts that drift, and group counts that count rows.</p><h3>Quick Start</h3><p>The current version is 2.0.0-SNAPSHOT:</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;com.github.paganini2008&lt;/groupId&gt;<br>    &lt;artifactId&gt;easyjpa-spring-boot-starter&lt;/artifactId&gt;<br>    &lt;version&gt;2.0.0-SNAPSHOT&lt;/version&gt;<br>&lt;/dependency&gt;<br><br>&lt;repositories&gt;<br>    &lt;repository&gt;<br>        &lt;id&gt;central-portal-snapshots&lt;/id&gt;<br>        &lt;url&gt;https://central.sonatype.com/repository/maven-snapshots/&lt;/url&gt;<br>        &lt;releases&gt;&lt;enabled&gt;false&lt;/enabled&gt;&lt;/releases&gt;<br>        &lt;snapshots&gt;&lt;enabled&gt;true&lt;/enabled&gt;&lt;/snapshots&gt;<br>    &lt;/repository&gt;<br>&lt;/repositories&gt;</pre><p>Name the provider:</p><pre>@EntityScan(basePackages = {&quot;com.example.entity&quot;})<br>@EnableJpaRepositories(repositoryFactoryBeanClass = HibernateEntityDaoFactoryBean.class,<br>        basePackages = {&quot;com.example.dao&quot;})<br>@Configuration(proxyBeanMethods = false)<br>public class JpaConfig {<br>}</pre><p>Extend EntityDao:</p><pre>public interface UserDao extends EntityDao&lt;User, Long&gt; {<br>}</pre><p>EntityDao is a JpaRepositoryImplementation, so save, findById and deleteAll are untouched. Query:</p><pre>List&lt;User&gt; users = userDao.query()<br>        .filter(Restrictions.eq(User::getVip, true))<br>        .sort(JpaSort.asc(User::getUsername))<br>        .selectThis().list();</pre><p>That is the whole setup. No properties to add, nothing to tune.</p><h3>Requirements</h3><p>Java — 17 or later.</p><p>Spring Boot — 4.0 or later. Spring Boot 3.1 to 3.5 takes the 1.0.x line instead.</p><p>Jakarta Persistence — 3.2.</p><p>JPA provider — Hibernate 7.4, or EclipseLink 5.0.1, or any Criteria API implementation.</p><p>Build — Maven 3.9 or later.</p><p>The version line follows the Spring Boot line, and the two are maintained apart. Spring Boot 4.0 and later takes the 2.0.x line, which is the one described here; Spring Boot 3.1 to 3.5 takes 1.0.x; Spring Boot 3.0 and earlier is not supported.</p><h3>How It Works</h3><pre>┌─────────────────────────────────────────────────────────────────────────────┐<br>│  YOUR CODE        userDao.query().filter(...).sort(...).selectThis().list()  │<br>└──────────────────────────────────┬──────────────────────────────────────────┘<br>                                   │  method references + aliases<br>┌──────────────────────────────────▼──────────────────────────────────────────┐<br>│  EASYJPA          Model ─────────── resolves an alias to a table            │<br>│                   Filter · Field · Column · JpaSort                         │<br>│                   JpaPageResultSet ─ one definition, two statements         │<br>└──────────────────────────────────┬──────────────────────────────────────────┘<br>                                   │  &quot;may I use a derived table here?&quot;<br>┌──────────────────────────────────▼──────────────────────────────────────────┐<br>│  JpaProvider        Hibernate   │   EclipseLink   │   Standard              │<br>│                     supportsDerivedTable() · supportsRightJoin() · …        │<br>└──────────────────────────────────┬──────────────────────────────────────────┘<br>                                   │<br>┌──────────────────────────────────▼──────────────────────────────────────────┐<br>│  JAKARTA PERSISTENCE   CriteriaBuilder · CriteriaQuery · Root · Predicate    │<br>└──────────────────────────────────┬──────────────────────────────────────────┘<br>                                   │<br>┌──────────────────────────────────▼──────────────────────────────────────────┐<br>│  DATABASE    H2 · PostgreSQL · MySQL · SQL Server · SQLite · Oracle         │<br>└─────────────────────────────────────────────────────────────────────────────┘</pre><p>Three mechanics are worth knowing, and none of them needs configuring.</p><p>Alias resolution. A lambda names an <em>entity</em>, not a table. The alias is looked up inside the statement being built and never outside it — so an outer query and its subquery keep their tables apart even when both query the same entity.</p><p>Two statements, one definition. JpaPageResultSet holds the listing query and derives the counting query from it, so the two cannot drift.</p><p>Providers are asked, not assumed. Every optional feature is a boolean on JpaProvider, readable at runtime.</p><p>The pagination split is the part worth seeing:</p><pre>      customPage().join(...).filter(...).groupBy(...).select(...)<br>                              one definition<br>                                    │<br>                 ┌──────────────────┴──────────────────┐<br>                 ▼                                     ▼<br>            list(10, 0)                           rowCount()<br>                 │                                     │<br>   select … offset ? rows                   no group by → select count(1) from (…) d<br>          fetch first ? rows only              group by → select count(1) from<br>                                                          ( &lt;the groups&gt; ) d</pre><h3>Code Examples</h3><p>Every SQL block below is what Hibernate actually emitted, captured from the test suite against H2. The model:</p><pre>User ──&lt; Order ──&lt; OrderProduct &gt;── Product        User.vip, Order.status, Order.totalPrice<br>                                        │          Product.name/price/location<br>                                      Stock        Stock.amount</pre><h3>Example 1 — Nested conditions</h3><p>Input: vip or (username in (&#39;Scott&#39;,&#39;Lee&#39;) and email is not null), by username.</p><p>Execution:</p><pre>List&lt;User&gt; users = userDao.query()<br>        .filter(Restrictions.eq(User::getVip, true)<br>                .or(new FilterList().in(User::getUsername, List.of(&quot;Scott&quot;, &quot;Lee&quot;))<br>                        .and().notNull(User::getEmail)))<br>        .sort(JpaSort.asc(User::getUsername))<br>        .selectThis().list();</pre><p>Output:</p><pre>select u1_0.id, u1_0.email, u1_0.password, u1_0.username, u1_0.vip<br>from example_user u1_0<br>where u1_0.vip = ? or u1_0.username in (?, ?) and u1_0.email is not null<br>order by u1_0.username</pre><h3>Example 2 — Four tables, two branches</h3><p>Input: order lines with their order, customer and product; only discounted products on paid or shipped orders; newest first.</p><p>Execution:</p><pre>orderProductDao.customPage()<br>        .join(OrderProduct::getOrder, &quot;o&quot;, null)      // OrderProduct -&gt; Order<br>        .join(Order::getUser, &quot;u&quot;, null)              // Order        -&gt; User<br>        .join(OrderProduct::getProduct, &quot;p&quot;, null)    // OrderProduct -&gt; Product, second branch<br>        .filter(new FilterList().notNull(Product::getDiscount).and()<br>                .in(Property.forName(&quot;o&quot;, &quot;status&quot;), List.of(OrderStatus.PAID, OrderStatus.SHIPPED)))<br>        .sort(JpaSort.desc(Order::getOrderDate), JpaSort.asc(Product::getName))<br>        .select(new ColumnList().addColumns(User::getUsername)<br>                .addColumns(Order::getId, Order::getOrderDate, Order::getStatus)<br>                .addColumns(Product::getName, Product::getPrice)<br>                .addColumns(OrderProduct::getAmount)<br>                .addColumns(Fields.multiply(Property.forName(Product::getPrice),<br>                        Property.forName(OrderProduct::getAmount)).as(&quot;subtotal&quot;)));</pre><p>Output:</p><pre>select u1_0.username, o1_0.id, o1_0.order_date, o1_0.status,<br>       p1_0.name, p1_0.price, op1_0.amount, (p1_0.price * op1_0.amount)<br>from example_order_product op1_0<br>join example_order   o1_0 on o1_0.id = op1_0.order_id<br>join example_user    u1_0 on u1_0.id = o1_0.user_id<br>join example_product p1_0 on p1_0.id = op1_0.product_id<br>where p1_0.discount is not null and o1_0.status in (?, ?)<br>order by o1_0.order_date desc, p1_0.name<br>offset ? rows fetch first ? rows only</pre><p>The third join branches back off OrderProduct rather than continuing from User: the lambda carries its own entity, so the tree comes out the way it reads.</p><h3>Example 3 — A grouping pagination that counts groups</h3><p>Input: best sellers — units sold, distinct orders and turnover per product, keeping products sold more than once. Then the total, for the page navigation.</p><p>Execution:</p><pre>JpaPageResultSet&lt;Tuple&gt; resultSet = orderProductDao.customPage()<br>        .join(OrderProduct::getProduct, &quot;p&quot;, null)<br>        .groupBy(new FieldList().addFields(Product::getName))<br>        .having(Restrictions.gt(Fields.count(OrderProduct::getId), 1L))<br>        .sort(JpaSort.desc(2))<br>        .select(new ColumnList().addColumns(Product::getName)<br>                .addColumns(Fields.sum(OrderProduct::getAmount).as(&quot;soldAmount&quot;),<br>                        Fields.countDistinct(Property.forName(&quot;this&quot;, &quot;order.id&quot;)).as(&quot;orderAmount&quot;),<br>                        Fields.sum(Fields.multiply(Property.forName(Product::getPrice),<br>                                Property.forName(OrderProduct::getAmount))).as(&quot;turnover&quot;)));<br><br>long groups = resultSet.rowCount();<br>List&lt;SalesVo&gt; rows = resultSet.setTransformer(Transformers.asBean(SalesVo.class)).list();</pre><p>Output — the listing query:</p><pre>select p1_0.name, sum(op1_0.amount), count(distinct op1_0.order_id),<br>       sum((p1_0.price * op1_0.amount))<br>from example_order_product op1_0<br>join example_product p1_0 on p1_0.id = op1_0.product_id<br>group by p1_0.name having count(op1_0.id) &gt; ?<br>order by 2 desc<br>offset ? rows</pre><p>Output — the counting query, derived from the same definition:</p><pre>select count(1) from (<br>    select 1 from example_order_product op1_0<br>    join example_product p1_0 on p1_0.id = op1_0.product_id<br>    group by p1_0.name having count(op1_0.id) &gt; ?<br>) derived1_0(c)</pre><p>One statement, having honoured, no groups fetched. This is the bug you do not have to find.</p><h3>Example 4 — A correlated subquery</h3><p>Input: the users who have ever ordered.</p><p>Execution:</p><pre>JpaQuery&lt;User, User&gt; query = userDao.query();<br>JpaSubQuery&lt;Order, Long&gt; subQuery = query.subQuery(Order.class, &quot;o&quot;, Long.class)<br>        .filter(Restrictions.eq(Order::getUser, User::getId))<br>        .select(Order::getId);<br><br>List&lt;User&gt; customers = query.filter(Restrictions.exists(subQuery)).selectThis().list();</pre><p>Output:</p><pre>select u1_0.id, u1_0.email, u1_0.password, u1_0.username, u1_0.vip<br>from example_user u1_0<br>where exists (select o1_0.id from example_order o1_0 where o1_0.user_id = u1_0.id)</pre><p>The subquery is created from the outer query, which is what correlates the two. Negate it with .not() for not exists; swap exists for in and it becomes an in subquery.</p><h3>Example 5 — Joining an aggregate as a derived table</h3><p>Input: every product with its sales, without a correlated subquery per row.</p><p>Execution:</p><pre>JpaQuery&lt;Product, Tuple&gt; query = productDao.customQuery();<br>JpaSubQuery&lt;OrderProduct, Tuple&gt; sales = query.subQuery(OrderProduct.class, &quot;op&quot;, Tuple.class);<br>sales.groupBy(new FieldList().addFields(Property.forName(&quot;op&quot;, &quot;product.id&quot;)))<br>     .select(new ColumnList()<br>             .addColumns(Property.forName(&quot;op&quot;, &quot;product.id&quot;).as(&quot;productId&quot;))<br>             .addColumns(Fields.sum(&quot;op&quot;, &quot;amount&quot;, Integer.class).as(&quot;soldAmount&quot;),<br>                         Fields.count(&quot;op&quot;, &quot;id&quot;).as(&quot;orderAmount&quot;)));<br><br>query.joinSubQuery(sales, &quot;s&quot;, Restrictions.eq(Property.forName(&quot;s&quot;, &quot;productId&quot;),<br>                                               Property.forName(&quot;this&quot;, &quot;id&quot;)))<br>     .sort(JpaSort.desc(Property.forName(&quot;s&quot;, &quot;soldAmount&quot;)))<br>     .select(new ColumnList().addColumns(Product::getName, Product::getPrice)<br>             .addColumns(Property.forName(&quot;s&quot;, &quot;soldAmount&quot;).as(&quot;soldAmount&quot;)))<br>     .setTransformer(Transformers.asCaseInsensitiveMap()).list();</pre><p>Output:</p><pre>select p1_0.name, p1_0.price, s1_0.soldAmount, s1_0.orderAmount<br>from example_product p1_0<br>join (select op1_0.product_id c0, sum(op1_0.amount) c1, count(op1_0.id) c2<br>      from example_order_product op1_0<br>      group by c0) s1_0(productId, soldAmount, orderAmount)<br>  on s1_0.productId = p1_0.id<br>order by 3 desc</pre><h3>Example 6 — Update by an expression, delete by a subquery</h3><p>Input: shout a username; decrement a stock; drop the users who never ordered.</p><p>Execution:</p><pre>userDao.update().setField(User::getUsername, Fields.concat(Fields.upper(User::getUsername), &quot;!&quot;))<br>       .filter(Restrictions.eq(User::getUsername, &quot;Lee&quot;)).execute();<br><br>stockDao.update().setField(Stock::getAmount, Fields.minusValue(Stock::getAmount, 1L))<br>        .filter(Restrictions.gt(Stock::getAmount, 0L)).execute();<br><br>JpaDelete&lt;User&gt; delete = userDao.delete();<br>JpaSubQuery&lt;Order, Order&gt; subQuery = delete.subQuery(Order.class)<br>        .filter(Restrictions.eq(Order::getUser, User::getId));<br>delete.filter(Restrictions.exists(subQuery).not()).execute();</pre><p>Output:</p><pre>update example_user  u1_0 set username = (upper(u1_0.username)||?) where u1_0.username = ?<br>update example_stock s1_0 set amount = (s1_0.amount - cast(? as bigint)) where s1_0.amount &gt; ?<br><br>delete from example_user u1_0<br>where not exists (select o1_0.id from example_order o1_0 where o1_0.user_id = u1_0.id)</pre><h3>The rest of the surface</h3><p>The entities — dao.query()</p><p>Columns into a VO — dao.query(Vo.class)</p><p>Any columns — dao.customQuery(), returning a Tuple</p><p>The same, with a total count — dao.page() or dao.customPage(), returning a JpaPageResultSet</p><p>Fetch an association — fetch(Order::getUser), or leftFetch(User::getOrders).distinct() for a collection</p><p>CASE WHEN — new IfExpression&lt;&gt;(attr).when(a, b).otherwise(c)</p><p>Date parts — Fields.year(...), Fields.month(...), Fields.day(...)</p><p>Any database function — Function.build(&quot;LOWER&quot;, String.class, Product::getName)</p><p>A result shape — Transformers.asBean(...), asMap(), asCaseInsensitiveMap(), asList(), noop()</p><p>Native SQL — dao.queryForMap(sql, args), which keeps rowCount() and paginate()</p><h3>Configuration</h3><p>EasyJPA has no properties of its own — there is no spring.easyjpa.* namespace. What you configure is the provider and, for three databases, the driver.</p><p>The provider goes on @EnableJpaRepositories(repositoryFactoryBeanClass = …): HibernateEntityDaoFactoryBean is the default, EclipseLinkEntityDaoFactoryBean and StandardEntityDaoFactoryBean are the alternatives.</p><p>On EclipseLink, add spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration, since Spring Boot autoconfigures Hibernate alone, and build the entity manager yourself.</p><p>On SQL Server, put calcBigDecimalPrecision=true in the JDBC url. Without it the driver binds every BigDecimal as decimal(38,0), a coalesce against such a parameter drops the decimals, and 0.90 reads back as 1.</p><p>On SQLite, map entity ids as GenerationType.IDENTITY. AUTO falls back to a sequence table that Hibernate writes from a transaction of its own, which SQLite locks against.</p><p>On Oracle 23 and later with Hibernate, subclass OracleDialect with DatabaseVersion.make(21).</p><h3>Comparison</h3><p>Feature-level. No timings are claimed — nothing here is a benchmark.</p><p>Type-safe attributes. EasyJPA uses method references. The raw Criteria API and Spring Data Specification use the metamodel or plain strings. QueryDSL uses generated Q classes.</p><p>Code generation. Only QueryDSL needs an annotation processor and a generated-sources step. EasyJPA, Criteria and Specification need none.</p><p>Dynamic composition. EasyJPA, Specification and QueryDSL all compose. The raw Criteria API means assembling a Predicate[] by hand.</p><p>Multi-branch joins. One call each in EasyJPA and QueryDSL. By hand in Criteria and Specification, tracking Join&lt;?, ?&gt; variables yourself.</p><p>Correlated subqueries. EasyJPA builds them from the outer query, so they correlate themselves. QueryDSL handles them well. Criteria is manual; Specification is awkward.</p><p>Pagination count over a group by. EasyJPA wraps the grouping in a derived table and counts it in one statement. Everywhere else you write that query yourself.</p><p>Derived-table join. joinSubQuery(...) in EasyJPA. Manual in Criteria, unavailable in Specification, limited in QueryDSL.</p><p>Native SQL with pagination kept. EasyJPA only; QueryDSL has a separate API for it.</p><p>Provider gaps discoverable at runtime. EasyJPA only, through JpaProvider flags.</p><h3>What EclipseLink does not reach</h3><p>Hibernate is the default and reaches every feature. On EclipseLink these are reported through JpaProvider rather than failed over silently:</p><p>A subquery as a selected column — supportsSubQueryAsSelection(). Joining a subquery as a derived table — supportsDerivedTable(). A right join — supportsRightJoin(). Selecting an entity column by column — supportsPartialEntity(); take a Tuple or a bean instead. Filling a bean by its properties — supportsBeanProjection(); its constructor is used instead. Sorting by a column position — supportsOrdinalSort(). Passing a database function through — supportsPassThroughFunction(). year(), month(), day() — supportsDatePart(); the Criteria API defines none.</p><p>A subquery as one side of a comparison works, but has to be quantified by Fields.all/any/some. Pagination counting falls back from a derived table to count(distinct …).</p><h3>Database coverage testing</h3><p>The same 202 tests run on three providers against six databases.</p><p><strong>H2 </strong>, PostgreSQL 16+ and MySQL 8+ : everything passes, on all three providers.</p><p><strong>SQL Server 16+ : </strong>one failure on Hibernate and on the plain Criteria API: + is read by its first operand, so a max(price) asked for as text is taken for arithmetic. Cast it first. EclipseLink passes.</p><p><strong>SQLite: </strong>three failures: no quantified subquery, no repeat, and no date type. EclipseLink ships no SQLite platform at all.</p><p><strong>Oracle 23+: </strong>four derived-table failures on Hibernate 7.4, which writes group by c0, its internal alias rather than the select item&#39;s; the same tests pass once the dialect is told it is Oracle 21. Two paginated-Tuple failures on EclipseLink, which writes no aliases and then wraps as select a.* from (…) a, and Oracle rejects two columns of the same name. The plain Criteria API passes.</p><p>Every one of those belongs to the database or to the provider on that database, never to a query EasyJPA built.</p><h3>Limitations &amp; Trade-offs</h3><p>Aliases are strings — a typo in &quot;op&quot; is a runtime error, not a compile error.</p><p>It is not auto-configured. You name the factory bean yourself; there is no @AutoConfiguration.</p><p>The Criteria API is the ceiling. Window functions, CTEs and UNION need native SQL.</p><p>EclipseLink reaches less than Hibernate, as listed above.</p><p>A fetched collection paginates in memory, so keep paginated fetches to to-one associations.</p><p>No reactive support — blocking JDBC only.</p><p>It is not a good fit for a project that writes mostly static queries. Spring Data method names are shorter for those.</p><h3>Summary</h3><p>One chain instead of ten lines. query().filter().sort().select().list() replaces the whole CriteriaBuilder / Root / Predicate[] ritual.</p><p>Method references throughout, so a renamed field is a compile error rather than a runtime surprise. No code generation, no annotation processor.</p><p>Joins branch by themselves. A lambda carries its own entity, so a four-table, two-branch join is four calls that read in order.</p><p>A pagination is one definition, two statements. The counting query is derived from the listing query and cannot drift from it.</p><p>A grouping pagination counts groups, through a derived table, in one statement, having included — the count most hand-written pagination gets wrong.</p><p>Subqueries correlate themselves because they are created from the query that uses them. Filter, column, comparison or nested — all the same way.</p><p>Aggregates join once, as a derived table, instead of running a correlated subquery per row.</p><p>Provider gaps are a boolean, not a stack trace. JpaProviders.getProvider().supportsXxx() answers before you build.</p><p>Tested where it matters — 202 tests, three providers, six databases, and every documented failure attributed to the database or the provider rather than hidden.</p><p>Zero configuration. No properties; name one factory bean and extend EntityDao.</p><p>GitHub: <a href="https://github.com/paganini2008/easyjpa">paganini2008/easyjpa</a>, MIT licensed. Spring Boot 4 takes the 2.0.x line, Spring Boot 3 the 1.0.x line.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2aad7e4e8f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cronsmith: Stop hand-writing cron and build and parse it in Java]]></title>
            <link>https://medium.com/@paganini.fy/cronsmith-stop-hand-writing-cron-build-and-parse-it-in-java-bd1eee545121?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd1eee545121</guid>
            <category><![CDATA[java17]]></category>
            <category><![CDATA[task-scheduler]]></category>
            <category><![CDATA[antlr]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[crontab]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 07:46:36 GMT</pubDate>
            <atom:updated>2026-10-01T07:48:59.071Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Cronsmith</em></strong> is a Java library that builds, parses and runs cron through a fluent, object-oriented API instead of hand-typed strings. You describe <em>when</em> in plain method calls and get back a cron expression, the exact date-times it fires at, or a running task, which across Quartz, Spring, AWS EventBridge and crontab.</p><pre>// &quot;the last Friday of every month at 18:00&quot; — readable, and it tells you the next fire<br>CronExpression cron = new CronBuilder().everyMonth().lastDayOfWeek(DayOfWeek.FRIDAY.getValue()).at(18, 0);<br>cron.toString();              // 0 0 18 ? * FRIL<br>cron.getNextFiredDateTime();  // the next last-Friday-of-the-month, 18:00</pre><h3>What problem does it solve?</h3><p>Quick, what does 0 0 18 ? * FRIL fire at? Cron strings are write-only: easy to fumble, hard to review, and impossible to unit-test by eye. Worse, &quot;cron&quot; isn&#39;t one language — Quartz, Spring, AWS EventBridge and Unix crontab disagree on fields, on whether 1 is Sunday or Monday, and on which of L / # / W they even support. Copy a Quartz line into crontab and it can silently fire at a different time.</p><p>Cronsmith makes the schedule code you can read, iterate and port: build it with method calls, print it for the target you deploy to, and ask it for its next fire times before you trust it.</p><h3>Quick start</h3><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;com.github.paganini2008&lt;/groupId&gt;<br>    &lt;artifactId&gt;cronsmith&lt;/artifactId&gt;<br>    &lt;version&gt;1.0.0-SNAPSHOT&lt;/version&gt;<br>&lt;/dependency&gt;</pre><pre>CronExpression weekdays9 = new CronBuilder().everyWeek().Mon().toFri().at(9, 0);<br>weekdays9.toString();              // 0 0 9 ? * MON-FRI<br>weekdays9.getNextFiredDateTime();  // next weekday at 09:00</pre><h3>Requirements</h3><ul><li>JDK 17+</li><li>The core parser’s only runtime dependency is ANTLR — no Spring, HTTP or database.</li></ul><h3>How it works</h3><p>A CronBuilder builds a CronExpression (an iterable, serializable model of the schedule). From there you render it for a dialect, or walk its fire times:</p><pre>CronBuilder  ──build──▶  CronExpression  ──┬── toString / toQuartz / toAws / toUnix  (render)<br>   or CRON.parse(str) ───────────────▶     ├── consume(..) / getNextFiredDateTime()  (iterate)<br>                                            └── serialize() / deserialize(..)         (store)</pre><h3>Code examples</h3><h3>Build — describe it, don’t assemble strings</h3><p>Input → Output:</p><pre>new CronBuilder().everySecond(5);                                       // */5 * * * * ?<br>new CronBuilder().everyMonth().day(10).andDay(15).andLastDay().everyHour(2).everyMinute(5);<br>// 0 */5 */2 10,15,L * ?<br>new CronBuilder().everyMonth().dayOfWeek(3, DayOfWeek.SATURDAY).everyHour(2);  // 0 0 */2 ? * SAT#3 (3rd Sat)<br>new CronBuilder().everyMonth().lastDay(3).at(23, 30);                   // 0 30 23 L-3 * ?<br>new CronBuilder().everyMonth().latestWeekday(15).at(9, 0);              // 0 0 9 15W * ? (nearest weekday to 15th)</pre><h3>Look at a schedule instead of trusting it</h3><pre>new CronBuilder().setStartTime(LocalDate.of(2027, 1, 1).atStartOfDay())<br>        .everyMonth().latestWeekday(15).at(9, 0)<br>        .consume(System.out::println, 5);<br>// 2027-01-15T09:00 … 2027-05-14T09:00  &lt;- the 15th is a Saturday, so it moves to the nearest weekday</pre><h3>Parse a string back into a model</h3><pre>CRON.parse(&quot;*/5 * * * *&quot;);    // 0 */5 * * * ?      a crontab line, seconds filled in<br>CRON.parse(&quot;0 9 * * 1-5&quot;);    // 0 0 9 ? * MON-FRI  crontab: 1 is Monday<br>CRON.parse(&quot;0 0 12 ? * 1&quot;);   // 0 0 12 ? * SUN     Quartz:  1 is Sunday</pre><p>The field count picks the reading: 5 = crontab, 6 = Quartz without a year, 7 = Quartz with a year.</p><h3>YCRON — schedules month-based cron can’t say</h3><p>Seven fields: &lt;sec&gt; &lt;min&gt; &lt;hour&gt; &lt;day-of-week&gt; &lt;week-of-year&gt; &lt;day-of-year&gt; [&lt;year&gt;].</p><pre>new CronBuilder().year(2026).day(100).at(12, 0, 0);        // 0 0 12 ? ? 100 2026  (100th day of the year)<br>new CronBuilder().year(2026).week(20).Mon().at(9, 0, 0);   // 0 0 9 MON 20 ? 2026  (Mon of ISO week 20)<br>YCRON.parse(&quot;0 0 12 ? ? 100 2026&quot;).getCronType();          // CronType.YCRON (its own grammar)</pre><h3>Turn a plain interval into cron, or run it in-process</h3><pre>CRON.setInterval(&quot;PT30S&quot;);                    // every 30s (PT1H30M / sub-second are rejected, not rounded)<br><br>ScheduledExecutorService exec = Executors.newScheduledThreadPool(4);<br>CronFuture f = new CronBuilder().everySecond(5).scheduler(exec).runTask(() -&gt; doWork(), 10); // run 10x</pre><h3>Configuration — syntax reference</h3><pre> sec min hour day-of-month month day-of-week [year]<br>  *   *   *       *           *        ?        *<br> day-of-month: 1-31, L, LW, L-n, nW, ?     day-of-week: 1-7 (SUN=1) or SUN-SAT, nL, n#m, ?<br> YCRON: &lt;sec&gt; &lt;min&gt; &lt;hour&gt; &lt;day-of-week&gt; &lt;week-of-year&gt; &lt;day-of-year&gt; [&lt;year&gt;]</pre><ul><li>L / L-n / LW: last day / n days before last / last weekday of the month</li><li>nW: weekday nearest the nth, without leaving the month</li><li>&lt;dow&gt;L / &lt;dow&gt;#n: last &lt;dow&gt; (FRIL) / nth &lt;dow&gt; (TUE#2, skipped in months without an nth)</li><li>a-b / a/n / a,b: range / step / list (entries may themselves be ranges or steps)</li></ul><h3>How it compares across schedulers</h3><p>One schedule, printed for whatever you deploy to. Unsupported features are reported, not silently rewritten (Quartz | Spring | AWS EventBridge | Unix crontab):</p><ul><li>weekdays 09:00: 0 0 9 ? * MON-FRI | 0 0 9 ? * MON-FRI | 0 9 ? * MON-FRI * | 0 9 * * MON-FRI</li><li>every 15 seconds: */15 * * * * ? | */15 * * * * ? | no seconds field | no seconds field</li><li>last day of month: 0 59 23 L * ? | 0 59 23 L * ? | 59 23 L * ? * | no L</li><li>2nd Tuesday: 0 0 10 ? * TUE#2 | 0 0 10 ? * TUE#2 | 0 10 ? * TUE#2 * | no #</li><li>restricted to 2027–2029: … 2027-2029 | no year field | … 2027-2029 | no year field</li></ul><pre>CronExpression daily = new CronBuilder().everyDay().at(9, 30);<br>CRON.toUnixString(daily);   // 30 9 * * *   (throws if the schedule needs L/#/seconds crontab lacks)</pre><h3>Limitations &amp; trade-offs</h3><ul><li>It is a cron builder / parser / in-process runner, not a distributed job system. For persistent, clustered scheduling with retries and a console, use cronsmith-spring-boot-starter in <a href="https://github.com/paganini2008/cronflower">cronflower</a>.</li><li>Rendering is honest, not lossy: a schedule a target can’t express throws instead of quietly changing when it fires.</li><li>ISO-8601 intervals must be an exact multiple of one unit (PT1H30M, PT90M, sub-second are rejected).</li></ul><h3>Summary</h3><ul><li>Build cron with a fluent, readable API; toString() for the string.</li><li>Parse any dialect back into an inspectable, iterable CronExpression.</li><li>Port one schedule to Quartz / Spring / AWS / crontab; unsupported features are reported.</li><li>YCRON expresses year-, week- and day-of-year schedules classic cron can’t.</li><li>Iterate the real fire times (consume / getNextFiredDateTime) before you trust a schedule.</li><li>Serializable, with an in-process timing-wheel scheduler for simple runs.</li></ul><p>Get it: <a href="https://github.com/paganini2008/cronflower">cronsmith</a> · JDK 17+, Apache 2.0.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd1eee545121" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cronflower: Turn Spring Boot apps into a distributed cron cluster]]></title>
            <link>https://medium.com/@paganini.fy/cronflower-turn-spring-boot-apps-into-a-distributed-cron-cluster-ef30811e853d?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef30811e853d</guid>
            <category><![CDATA[quartz]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[spring-schedulling-task]]></category>
            <category><![CDATA[crontab]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 06:53:33 GMT</pubDate>
            <atom:updated>2026-10-01T07:52:47.571Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Cronflower</strong> is an open-source, distributed cron scheduler and DAG platform for the JVM with a web console. It forms its own cluster and needs no external database, broker, or coordinator. This story focuses on distributed task scheduling with CronFlower. For DAG workflows, see <a href="https://medium.com/@paganini.fy/cronflower-run-a-dag-workflow-across-your-cluster-instead-of-chaining-cron-jobs-b979459b9ff3?sharedUserId=paganini.fy">Run a DAG Workflow Across Your Cluster Instead of Chaining Cron Jobs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eYuRPn8qTZTWUMVK.jpg" /></figure><h3>What problem does it solve?</h3><p>@Scheduled runs in one JVM, so the moment you scale to two instances the job fires twice. It has no retry, no timeout, no history, and no view of what ran, if the box reboots at 02:00 the nightly rollup just quietly doesn&#39;t happen.</p><p>The usual fix is to bolt on Quartz, a database, a lock table, and a dashboard you wrote yourself. cronflower is that whole stack behind two dependencies: you annotate a method, run the scheduler alongside it, and get a clustered, persistent, retrying scheduler with a console.</p><h3>Quick start</h3><pre>git clone https://github.com/paganini2008/cronflower<br>cd cronflower/deploy<br>./run-local.sh -e 1          # scheduler + console + 1 executor  (embedded H2)</pre><p>Open <a href="http://localhost:7200/">http://localhost:7200</a>, sign in admin / admin123, and the example tasks are already running. Scale into a real cluster with ./run-local.sh -n 3 -e 2.</p><h3>Requirements</h3><ul><li>JDK: 17+ (builds via the bundled Maven Wrapper)</li><li>Node: 20+ (builds the console)</li><li>Docker: optional (container path only)</li><li>Database: optional: none → embedded H2, MySQL / PostgreSQL for a shared store</li></ul><h3>How it works</h3><p>Two kinds of process: a scheduler owns the schedule and durable state, an executor is your app, which declares tasks and runs the code on callback. Run several schedulers and they elect a leader over gossip, state lives in the store, so a restart or failover loses nothing.</p><pre>   your app (executor, @Task beans)<br>          |  register on boot<br>          v<br>   +----------------------------------------+<br>   |   scheduler cluster (gossip)           |<br>   |   leader  &lt;-&gt;  follower  &lt;-&gt;  follower |<br>   +----------------------------------------+<br>          |                        ^<br>          v                        |  dispatch when due (HTTP callback)<br>     store:  H2  /  MySQL  /  PostgreSQL</pre><p>The leader loads only the tasks due in the next few minutes into a timing wheel and claims the rest from the store as they come due, so a cluster with hundreds of thousands of tasks still starts instantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1RhenksiTz6ps-0.jpg" /></figure><h3>Code examples</h3><h3>A task is a @Task bean method</h3><p>Input: annotate a method on an executor (0 args, or a single String parameter):</p><pre>@Component<br>public class DemoTasks {<br><br>    @Task(cron = &quot;*/5 * * * * ?&quot;, group = &quot;showcase&quot;, name = &quot;sayHello&quot;, initialParameter = &quot;world&quot;)<br>    public String sayHello(String who) { return &quot;hello, &quot; + who; }<br><br>    // Fixed interval, no cron. Or iso = &quot;PT1H30M&quot; for an ISO-8601 duration.<br>    @Task(interval = 10, intervalUnit = TimeUnit.SECONDS, group = &quot;showcase&quot;, name = &quot;every10s&quot;)<br>    public void every10s() { /* ... */ }<br><br>    // YCRON: noon on the 200th day of the year — no classic cron can say this.<br>    @Task(cron = &quot;0 0 12 ? ? 200&quot;, parser = &quot;ycron&quot;, group = &quot;showcase&quot;, name = &quot;dayOfYear200&quot;)<br>    public void dayOfYear200() { /* ... */ }<br>}</pre><p>Execution: discovered on boot, registered with the scheduler, which owns the schedule from then on.</p><p>Output: each task shows up in the console with its schedule, run count, and next fire time.</p><h3>Reliability is configured, not coded</h3><p>Input: add attributes, the scheduler enforces them, so every executor behaves the same:</p><pre>@Task(cron = &quot;*/20 * * * * ?&quot;, group = &quot;reliability&quot;, name = &quot;flaky&quot;,<br>      maxRetryCount = 2, retryInterval = 1000,   // retry with back-off<br>      timeout = 2000,                            // fail a run that overruns 2s<br>      misfirePolicy = &quot;SKIP&quot;,                    // FIRE_ONCE_NOW (default) / FIRE_ALL / SKIP<br>      repeatCount = 10)                          // finish after 10 fires<br>public String flaky(String parameter) { /* fails twice, then succeeds */ }</pre><p>Output: in the execution history a single fire is three rows (attempt #0 fail, #1 fail, #2 success), each tagged with the scheduler that dispatched it and the executor that ran it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lQUMarLcHhRKseeU.jpg" /></figure><h3>Build a schedule in code, or skip the executor</h3><pre>// Fluent, self-validating schedule instead of a hand-typed string:<br>@Bean CronExpressionBuilder mondayMornings() {<br>    return () -&gt; new CronBuilder().everyWeek().Mon().at(9, 0).toString();<br>}<br>@Task(builder = &quot;mondayMornings&quot;, description = &quot;weekly report&quot;)<br>public void weeklyReport() { /* ... */ }</pre><p>For a job that is just “call this URL on a schedule”, create an HTTP-API task from the console’s <em>New task</em> form or the REST API: the scheduler makes the call itself, no executor involved, and operators add or edit tasks with no redeploy.</p><p>The @Task attributes, for reference:</p><ul><li>cron / parser: a cron schedule, or YCRON with parser = &quot;ycron&quot;</li><li>interval + intervalUnit / iso: a fixed interval, or an ISO-8601 duration</li><li>builder: a CronExpressionBuilder bean (wins over cron, the only way to set a deadline)</li><li>initialParameter: a constant, or a SpEL template evaluated on each fire</li><li>maxRetryCount / retryInterval: retry a failed run, with back-off</li><li>timeout: fail a run that overruns, in milliseconds</li><li>misfirePolicy: FIRE_ONCE_NOW (default) / FIRE_ALL / SKIP</li><li>repeatCount: total fires before the task finishes</li></ul><h3>Configuration</h3><ul><li>cronsmith.client.server-urls: on the executor, the scheduler seed URL(s)</li><li>cronsmith.server.scheduler.zone (default UTC): fire-time zone, must match cluster-wide</li><li>cronsmith.server.scheduler.window-minutes (default 5): windowed-loading horizon</li><li>cronsmith.server.scheduler.sharding (default true): group sharding over a shared store (else leader-only)</li><li>cronsmith.server.dispatch.routing (default ROUND_ROBIN): WEIGHTED / CONSISTENT_HASH / RANDOM / FIRST / LAST</li><li>spring.datasource.url (default H2 file): point at MySQL/PostgreSQL for a shared, sharding-capable store</li></ul><h3>Limitations &amp; trade-offs</h3><ul><li>It is a platform to run (its own gossip cluster + store), not a tiny in-process library. For one JVM with a couple of fixed jobs, plain @Scheduled is lighter.</li><li>Group sharding needs a shared database (MySQL/PostgreSQL), on node-local H2/SQLite it safely degrades to leader-only.</li><li>Fire-time zone must match across nodes, everything is computed and stored in UTC.</li><li>Bean tasks run on executors, so the scheduler reaches them over HTTP: they must be reachable from it.</li></ul><h3>Summary</h3><ul><li>A distributed, stateful cron scheduler for Spring Boot, two dependencies, no external infra.</li><li>Nodes gossip and elect a leader, a restart or failover loses nothing, and no cron fires twice.</li><li>@Task gives you cron / YCRON / interval / ISO-8601, plus retry, timeout, misfire, repeat: declarative.</li><li>HTTP-API tasks need no executor, operators add or edit tasks with no redeploy.</li><li>Every run is recorded with result, timing, attempt, and the node that ran it.</li><li>Group sharding + weighted dispatch scale it horizontally, a timing wheel + windowed loading keep startup instant.</li><li>A console shows tasks, cluster, executors, health, and live settings from one endpoint.</li></ul><p>Run it: <a href="https://github.com/paganini2008/cronflower">cronflower on GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef30811e853d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cronflower: Run a DAG workflow across SpringBoot cluster, instead of chaining cron jobs]]></title>
            <link>https://medium.com/@paganini.fy/cronflower-run-a-dag-workflow-across-your-cluster-instead-of-chaining-cron-jobs-b979459b9ff3?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b979459b9ff3</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 06:37:11 GMT</pubDate>
            <atom:updated>2026-10-01T06:57:06.995Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Cronflower</strong> is an open-source, distributed scheduler and DAG platform for the JVM with a web console. Declare the steps and how they depend on each other as a graph, and the same cluster runs the graph node by node, with data flowing between nodes over typed channels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b6LyXoYs5AQRD23I.jpg" /></figure><h3>What problem does it solve?</h3><p>Teams end up chaining cron jobs: one at 02:00 charges the orders, another at 02:15 ships them, set fifteen minutes later because that is <em>usually</em> long enough for the first to finish. There is no real dependency, no shared data, and no way to tell afterwards whether step two ran because step one succeeded or just because the clock moved.</p><p>That is the wall plain cron hits: it schedules single jobs, it cannot orchestrate a flow of them. A DAG makes the dependency real, lets data flow between steps, and records every run.</p><h3>Quick start</h3><pre>git clone https://github.com/paganini2008/cronflower<br>cd cronflower/deploy<br>./run-local.sh -e 1          # scheduler + console + 1 executor</pre><p>Open <a href="http://localhost:7200/">http://localhost:7200</a> (admin / admin123) and the example DAGs are registered under DAG &gt; Workflows, ready to trigger.</p><h3>Requirements</h3><ul><li><strong>JDK</strong>: 17+ (builds via the bundled Maven Wrapper)</li><li><strong>Node</strong>: 20+ (builds the console)</li><li><strong>cronflow add-on</strong>: cronflow-spring-boot-starter on the scheduler, cronflow-executor-spring-boot-starter on the executor</li><li><strong>Database</strong>: optional — none → embedded H2; MySQL / PostgreSQL for a shared store</li></ul><h3>How it works</h3><p>A DAG is a Spring bean: @Dag names it, each @DagNode method is a step, and the to list is the edges. A node returns a Map of named channel writes; downstream nodes read them from a DagState. Each @Channel says how concurrent writes to it merge, via a reducer. The engine drives the graph across the whole cluster, dispatching each node to a live executor.</p><pre>                 input<br>                   |<br>                 intake<br>              /    |    \<br>        credit  income  collateral<br>              \    |    /<br>                 decide<br>              /          \<br>       score &gt;= 70       else<br>       APPROVED          REJECTED</pre><h3>Code examples</h3><h3>Declare a workflow — @Dag</h3><p>Input — a bean with @Dag + @DagNode methods and typed channels:</p><pre>@Dag(name = &quot;scoring-flow&quot;, inputs = {&quot;input&quot;}, channels = {<br>        @Channel(name = &quot;score&quot;,   reducer = ChannelReducer.SUM_INT),<br>        @Channel(name = &quot;factors&quot;, reducer = ChannelReducer.JOIN_CSV)})<br>@Component<br>public class ScoringFlow {<br><br>    @DagNode(entry = true, to = {&quot;credit&quot;, &quot;income&quot;, &quot;collateral&quot;})     // fan out to 3 parallel scorers<br>    public Map&lt;String, Object&gt; intake(DagState state) {<br>        return Map.of(&quot;applicant&quot;, state.getString(&quot;input&quot;));<br>    }<br><br>    @DagNode(to = {&quot;decide&quot;})<br>    public Map&lt;String, Object&gt; credit(DagState state) {<br>        return Map.of(&quot;score&quot;, 40, &quot;factors&quot;, &quot;credit&quot;);<br>    }<br>    // income(), collateral() … same shape, writing to the same channels<br><br>    @DagNode(trigger = &quot;ALL&quot;)                                           // join: wait for all three<br>    public Map&lt;String, Object&gt; decide(DagState state) {<br>        long total = state.getLong(&quot;score&quot;);                           // reducer already summed them<br>        return Map.of(&quot;decision&quot;, total &gt;= 70 ? &quot;APPROVED&quot; : &quot;REJECTED&quot;);<br>    }<br>}</pre><p><strong>Execution</strong>: three scorers run in parallel, each writing score; the reducer merges the concurrent writes (summed to 90) so decide reads one combined value. No shared mutable state, no ordering assumptions.</p><p><strong>Output: </strong>the console renders the graph; a run lights up node by node, each node showing which executor ran it, so a wide fan-out really runs in parallel on different machines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dCcrxgrYSJB5F-OE.jpg" /></figure><h3>Edges beyond straight lines</h3><pre>@DagNode(when = @When(expr = &quot;#risk &gt; 80&quot;, to = &quot;humanReview&quot;), otherwise = {&quot;fulfilment&quot;})<br>public Map&lt;String, Object&gt; riskScore(DagState state) { /* writes risk */ }<br><br>@DagNode(subgraph = &quot;fulfilment-flow&quot;, to = {&quot;notify&quot;})                // nest a whole other DAG<br>public Map&lt;String, Object&gt; fulfilment(DagState state) { /* ... */ }<br><br>@DagNode(shard = @Shard(input = &quot;ids&quot;, output = &quot;squares&quot;), to = {&quot;report&quot;})<br>public Map&lt;String, Object&gt; square(DagState state) { /* runs once per id, at run time */ }</pre><ul><li>when + otherwise: route by a SpEL expression over the channels.</li><li>trigger: ALL waits for every upstream edge; ANY fires on the first arrival.</li><li>subgraph: a node is a whole other DAG, so you compose instead of copy.</li><li>@Shard: fan out dynamically, once per element of a list known only at run time.</li></ul><h3>Three ways to start it</h3><ul><li>By hand from the console’s <em>Trigger</em> button, with an optional JSON seed for the input channels.</li><li>From a finished @Task in the same bean: its return value becomes the DAG input.</li><li>On a schedule directly: pick <em>Trigger DAG</em> in the <em>Create Task</em> form.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*npb-1h2OX4YdaKi9.jpg" /></figure><h3>Configuration — channels &amp; reducers</h3><p>Each @Channel merges concurrent writes with a built-in reducer, or your own via customReducer:</p><ul><li>SUM_INT / SUM_LONG: add the numbers</li><li>MAX / MIN: keep the largest / smallest</li><li>AND / OR: boolean fold</li><li>CONCAT_LIST / UNION_SET: append / union collections</li><li>CONCAT_STRING / JOIN_CSV: join text</li><li>MERGE_MAP: merge maps</li><li>LAST_WINS / FIRST_WINS / WRITE_ONCE: pick one writer</li></ul><p>The executor points at the scheduler with cronflow.client.server-urls; the server prefix is cronflow.server.api-prefix (default /cronflow).</p><h3>Limitations &amp; trade-offs</h3><ul><li>The DAG engine rides on the cronflow add-on over the scheduler cluster; it is not a standalone workflow server. You run cronflower, then add @Dag beans.</li><li>Nodes are dispatched to executors over HTTP, so a node’s bean must be reachable and idempotent enough to be retried.</li><li>Channel values cross the wire between nodes, so keep them serializable and reasonably small.</li><li>It orchestrates <em>your</em> steps; it is not a data-pipeline engine for streaming large datasets.</li></ul><h3>Summary</h3><ul><li>A DAG is a Spring bean: @Dag + @DagNode, edges via to, data via typed channels.</li><li>Reducers merge concurrent writes, so joins need no locks or ordering assumptions.</li><li>Branching (when), join modes (ALL/ANY), subgraphs, and dynamic fan-out (@Shard) are annotation attributes.</li><li>The engine drives the graph across the cluster, dispatching each node to a live executor.</li><li>Trigger a flow by hand, from a finished task, or on a schedule.</li><li>Every run is recorded node by node, showing what ran where and what it produced.</li></ul><p>Run it: <a href="https://github.com/paganini2008/cronflower">cronflower on GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b979459b9ff3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Greenfinger 2.0: One url in, a searchable archive out]]></title>
            <link>https://medium.com/@paganini.fy/greenfinger-2-0-one-url-in-a-searchable-archive-out-11820f50de3e?source=rss-c95d1b8ea67b------2</link>
            <guid isPermaLink="false">https://medium.com/p/11820f50de3e</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[web-crawler]]></category>
            <category><![CDATA[java-nio]]></category>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Fred Feng]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 06:00:59 GMT</pubDate>
            <atom:updated>2026-10-01T08:25:11.815Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Crawl the whole site. Keep every page, every picture, every version. Search it by words, by meaning, or by describing a picture you remember. Add a node and it goes faster. Kill one and it keeps going. Nothing to install. Nothing to sign up for.</strong></p><p>A distributed web crawler for the JVM. One pass writes plain files, a full text index and a vector collection. Every node runs the same jar, there is no coordinator to deploy, and the first crawl needs no database, no search server and no API key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QxVTriJ3fFYordk5" /></figure><h3>What problem does it solve</h3><p>Most crawler code starts the same way. Someone needs the content of a site, writes a hundred lines around an http client and an html parser, and it works. Then reality arrives. One advert link and the crawler is downloading the rest of the web. The process is killed at page 40,000 and the queue was in memory. Half the site turns out to be pdfs. Navigation and cookie banners get indexed alongside the article, so every result looks the same.</p><p>And then storing it is a second project: files here, an index there, embeddings somewhere else, and three pieces of glue that each fail differently. Greenfinger answers all of that in the product rather than in your code, and the first crawl needs nothing running beside it.</p><h3><strong>Quick start</strong></h3><pre>git clone https://github.com/paganini2008/greenfinger.git<br>cd greenfinger/backend &amp;&amp; mvn clean install<br>cd ../deploy<br><br>./greenfinger-cli.sh --cluster=demo crawl \<br>    --url=https://books.toscrape.com</pre><h3><strong>Result:</strong></h3><pre>Crawling &#39;books.toscrape.com&#39; from https://books.toscrape.com<br><br>  pages kept    1,000      urls seen   28,411      1 in every 28<br>  images        3,204      indexed      1,000      elapsed  0h 4m 12s<br><br>Finished: reached maxFetchSize<br>Catalog 01a0c3d5-7d3d-7000-81f6-76057a403db8, version 0, now searchable.</pre><p>Prefer the console:</p><pre>./run-local.sh                # nodes plus the page on :9700<br>GF_NODES=3 ./run-local.sh     # three nodes sharing one crawl<br>./run-docker.sh               # the same, in containers<br>./run-local.sh stop</pre><p>Sign in with admin and the password in deploy/config/api/users.xml. Seven example catalogs ship with a fresh install, so there is something to crawl before you have picked anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yyadIaf5dSdnyF4f" /></figure><h3>Requirements</h3><ul><li>JDK 17 or later, to run anything.</li><li>Maven 3.9 or later, to build from source.</li><li>Node 20 or later, to build the web interface.</li><li>Docker, any current version, for run-docker.sh only.</li></ul><p>Everything else is optional and opt in one variable at a time: PostgreSQL, MySQL, SQL Server, Oracle or SQLite instead of the H2 file, Elasticsearch instead of the embedded Lucene index, Qdrant or Weaviate instead of the embedded vector store, MinIO or any S3 compatible store instead of local disk, and Ollama or OpenAI instead of the local ONNX models.</p><p>A browser is needed only for the playwright and selenium extractors. The default adaptive falls back to HtmlUnit, which is pure Java and needs nothing installed.</p><h3><strong>How it works</strong></h3><p>One pass, three outputs</p><pre>                        ┌─────────────┐<br>                        │  file       │  disk, MinIO, any S3<br> one crawl  ──────────► ├─────────────┤<br>                        │  index      │  Lucene, Elasticsearch<br>                        ├─────────────┤<br>                        │  vector     │  Lucene, Qdrant,<br>                        │             │  Weaviate, ES<br>                        └─────────────┘<br>                              │<br>             replay ──────────┘  rebuild index or vectors<br>                                 from the files, without<br>                                 fetching the site again</pre><ul><li>file holds the page as fetched, the article text and every image.</li><li>index holds the full text, searchable by words.</li><li>vector holds text chunks and image embeddings.</li></ul><p>The file layer is always on, because the database keeps metadata only and the other two rebuild from what it wrote. Search never reads the database, and that single constraint is where the rest of the design comes from.</p><h3>What one page goes through</h3><pre>url from the frontier<br>     │<br>     ├─ UrlPathAcceptor chain<br>     │    domain, start url prefix, assets, robots.txt,<br>     │    depth, path patterns. First refusal wins<br>     │<br>     ├─ ExistingUrlPathFilter     seen before? RocksDB<br>     │<br>     ├─ Extractor                 plain http, a browser only<br>     │                            for unrendered shells<br>     │<br>     ├─ ContentExtractor          the article, not the navigation<br>     │    └─ DocumentContentParser   when it is not html<br>     │<br>     ├─ ContentDedupFilter        SHA-256 or SimHash<br>     │<br>     ├─ OutputChannels            file, index, vector<br>     │<br>     └─ new links ──► back to the frontier, to whichever<br>                      node owns them</pre><p>Every drop is counted and named. That is why the monitor can say where 22,000 urls went rather than only that 140 pages were kept.</p><h3>How the cluster shares one crawl</h3><pre>node A ── finds /a/b ──► node C owns it<br>                            │<br>                    fetches, finds /a/b/c<br>                            │<br>                     ──► node A owns it ──► ...<br><br>no central queue      no leader in the fetch path<br>join or leave mid crawl</pre><p>A crawl is a recursive function, and the only thing distribution changes is that the recursive call crosses a process. There is no join, because a parent page does not care what its children found.</p><p>Completion is decided by everyone. Every node checks the shared counters against maxFetchSize and fetchDuration, and the first to notice writes the reason. A leader that dies mid crawl cannot leave a crawl that never ends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FPd9w4dnbGv2yaaF" /></figure><h3>Code examples</h3><h3>Crawl, update, rebuild, replay</h3><p>Input: A catalog id, and a verb.</p><pre>./greenfinger-cli.sh --cluster=nightly crawl   --id=&lt;id&gt;<br>./greenfinger-cli.sh --cluster=nightly update  --id=&lt;id&gt;<br>./greenfinger-cli.sh --cluster=nightly merge   --id=&lt;id&gt;<br>./greenfinger-cli.sh --cluster=nightly rebuild --id=&lt;id&gt;<br>./greenfinger-cli.sh --cluster=nightly resume  --id=&lt;id&gt;<br>./greenfinger-cli.sh --cluster=nightly replay  --id=&lt;id&gt; \<br>    --layers=index+vector</pre><p>Output:</p><ul><li>crawl goes from the start url and writes everything it saves.</li><li>update fetches only urls not seen before, and writes new pages only.</li><li>merge does that and revisits what is held, writing nothing for pages that came back unchanged.</li><li>rebuild opens a new version, crawls the whole site again, and the old version keeps serving.</li><li>resume continues from what is left on the frontier.</li><li>replay fetches nothing and rebuilds an output from what is already stored.</li></ul><p>replay is the one to remember. Your Elasticsearch was down for an hour, you changed the analyzer, or you decided six months in that you want embeddings after all. None of those need the site to be polite to you a second time.</p><h3>Search three ways</h3><p>Input: One box, three modes, from the console or the prompt.</p><pre>greenfinger:&gt; search --query=&quot;bread&quot;<br>greenfinger:&gt; search --mode=meaning \<br>        --query=&quot;what happens when a star runs out of fuel&quot;<br>greenfinger:&gt; search --mode=pictures \<br>        --query=&quot;a bright spiral galaxy against black sky&quot;</pre><p>Output: Words goes to the index, so this is exact terms with the matches highlighted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9AY10TnZEu6L3y36" /></figure><p>Meaning goes to the text vectors. The top answer below is a supernova remnants page that never contains the sentence that was typed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4xXqNo9KeEz9xLo1" /></figure><p>Pictures goes to the image vectors, matched against the picture itself rather than the filename or the alt text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kxjkv0KtlWyksooH" /></figure><p>The same thing at the prompt:</p><pre>Score   Title<br>0.9356  APOD: 2008 June 4 - Chasing the ISS<br>0.9317  APOD Index - Nebulae: Supernova Remnants<br>0.9289  APOD Index - Stars: Binary Stars</pre><p>The embedding models run locally and need no account: multilingual-e5-small for text and SigLIP 2 for images, both ONNX, both preloaded at startup on a background thread.</p><h3>Add a pdf parser</h3><p>Input: A site that links pdfs.</p><p>Code. Every default component is @ConditionalOnMissingBean, so publishing a bean is the whole registration. There is no plugin registry and no ordering property.</p><pre>@Bean<br>DocumentContentParser pdfParser() {<br>    return new DocumentContentParser() {<br>        public Set&lt;String&gt; fileTypes() {<br>            return Set.of(&quot;pdf&quot;);<br>        }<br><br>        public String extractText(byte[] content, String url,<br>                Charset encoding) throws Exception {<br>            return new Tika()<br>                    .parseToString(new ByteArrayInputStream(content));<br>        }<br>    };<br>}</pre><p>Output: With GF_DOCUMENTS=true and GF_DOCUMENT_TYPES=pdf, pdfs linked from a crawled page now contribute text to the index and the vectors. Collecting the links was always happening and costs nothing. Fetching and reading them is what you just switched on.</p><p>The same shape works for every decision the crawler makes:</p><ul><li>Extractor decides how a page is fetched. Ships restclient, htmlunit, playwright, selenium and adaptive.</li><li>UrlPathAcceptor decides whether a link is followed. Ships domain, start url, assets, robots.txt, depth and path patterns.</li><li>ContentDedupFilter decides whether two urls are the same page. Ships sha256 and simhash.</li><li>ContentExtractor finds the text inside a page. Ships link density boilerplate removal.</li><li>DocumentContentParser gets text out of a non html file. Ships text, markdown and csv.</li><li>CompletionChecker decides when the crawl is over. Ships saved count and elapsed time.</li><li>OutputChannel decides where results are written. Ships file, index and vector.</li><li>BlobStore holds pages and images as bytes. Ships local and minio.</li><li>Searcher and VectorStore handle full text and meaning. Ships lucene, elasticsearch, qdrant and weaviate.</li><li>EmbeddingClient turns text into a vector. Ships local ONNX, ollama and openai.</li></ul><h3>Embed it in your own application</h3><p>Input: A Spring Boot application of your own.</p><pre>@EnableGreenfingerServer<br>@SpringBootApplication<br>public class MyApplication { }</pre><p>Output: The whole REST api, the login and the page, inside your process. It is explicit rather than auto configured, because sitting on a classpath is not a reason to open RocksDB and take a crawl permit. To drive a crawl without the server, CrawlerLauncher.crawl(catalogId, onReady) returns a CrawlerEngine.Result with the counters and the reason it ended.</p><h3>Configuration</h3><h3>Per catalog</h3><p>One url is required. Everything else has a default chosen to give a useful crawl of a site you know nothing about. Run options at the prompt for the live values.</p><ul><li><strong><em>url</em></strong> takes http:// or <a href="https://./">https://.</a> Required, and it is the identity and the outer boundary.</li><li><strong><em>name</em></strong> takes unique text. Defaults to the domain.</li><li><strong><em>cat</em></strong> is one of nine categories, used to filter and group. Defaults to other.</li><li><strong><em>start-url</em></strong> is where fetching begins and must sit under url. Defaults to url.</li><li><strong><em>sitemap-url</em></strong> takes a url. Empty discovers it from robots.txt.</li><li><strong><em>include</em></strong> takes an ant path pattern, comma for several. Defaults to **.&lt;domain&gt;/**.</li><li><strong><em>exclude</em></strong> takes an ant path pattern, comma for several. Defaults to empty.</li><li><strong><em>encoding</em></strong> is the page charset, for when the server is wrong about it. Defaults to UTF-8.</li><li><strong><em>extractor</em></strong> takes adaptive, restclient, htmlunit, playwright or selenium. Defaults to adaptive.</li><li><strong><em>max-size</em></strong> is saved pages before the crawl stops. Defaults to 10000.</li><li><strong><em>depth</em></strong> is link depth, with -1 for no limit. Defaults to -1.</li><li><strong><em>duration</em></strong> is minutes before the crawl stops. Defaults to 30.</li><li><strong><em>interval</em></strong> is milliseconds between fetches, per node. Defaults to 1000.</li><li><strong><em>retry</em></strong> is retries per url. Defaults to 1.</li><li><strong><em>images</em></strong> decides whether pictures are fetched at all. Defaults to true.</li><li><strong><em>output-types</em></strong> takes file+index+vector, and file is always on. Defaults to file.</li><li><strong><em>content</em></strong> decides what reaches the index and the vectors. Defaults to text+image.</li><li><strong><em>max-versions</em></strong> is how many versions to keep. Defaults to 10.</li></ul><h3>Node wide, in deploy/.env</h3><ul><li><strong>GF_DB_URL</strong> defaults to an H2 file. PostgreSQL, MySQL, SQL Server, Oracle and SQLite all work.</li><li><strong>GF_INDEX_PROVIDER</strong> defaults to lucene, or elasticsearch with GF_ES_URIS.</li><li><strong>GF_VECTOR_STORE</strong> defaults to lucene, or elasticsearch, qdrant, weaviate.</li><li><strong>GF_FILE_TARGET</strong> defaults to local, or minio. Object keys match the local paths exactly.</li><li><strong>GF_EMBEDDING_PROVIDER</strong> defaults to local, or ollama, openai.</li><li><strong>GF_LUCENE_ANALYZER</strong> defaults to standard, and takes smartcn or cjk.</li><li><strong>GF_ES_ANALYZER</strong> defaults to standard. ik_max_word needs the analysis-ik plugin.</li><li><strong>GF_DOCUMENTS</strong> defaults to false, and decides whether linked documents are fetched and read.</li><li><strong>GF_CLUSTER_TRANSPORT</strong> defaults to NETTY, or the built in NIO.</li><li><strong>GF_NODES</strong> defaults to 1, and is how many processes a launcher starts.</li><li><strong>GF_MEMORY</strong> defaults to 2g per container, and caps heap and off heap together.</li></ul><p>Analyzers matter more than they look. The standard analyzer cuts Chinese into single characters, and a Chinese analyzer drops French characters outright, so chaîne comes back as cha î ne.</p><h3>Performance Test</h3><p>Test environment:</p><p>Apple M2 Max, 12 cores, 32 GB, macOS 26.3.1, JDK 17.0.12.</p><p>Two nodes started by run-local.sh, each -Xms256m -Xmx2g. H2 file, embedded Lucene index, embedded Lucene vector store, pages and images on local disk, local ONNX embeddings.</p><p>Home broadband. Polite crawling, so fetchInterval is the floor on throughput rather than the hardware.</p><p><strong>Crawling</strong></p><pre>site                  nodes  kept        images  urls seen   elapsed<br>apod.nasa.gov           2     46 pages      16     3,773      1m 41s<br>simplefood.blog         2    140 pages   2,489    22,857 *    5m 01s<br>books.toscrape.com      1  1,000 pages   3,204    28,411      4m 12s<br><br>* already known</pre><p>The gap between urls seen and pages kept is the point rather than an inefficiency. On simplefood 22,340 urls were filtered out by the boundary rules and 561 were duplicate content, which is work the outputs never had to do.</p><p><strong>How evenly the work spreads.</strong></p><pre>two nodes, apod.nasa.gov<br>  node 18bbe738   76 handled   51%   26 pages   9 images<br>  node 96c82b18   74 handled   49%   19 pages   7 images<br><br>three nodes, a 61 page site, one shared PostgreSQL<br>  dispatched 61      handled 66      saved 61<br>  node-1: 18 fetches   node-2: 18   node-3: 25<br>  each page exactly once</pre><p><strong>Everything else we measured.</strong></p><ul><li>Word search, embedded Lucene over 151 documents: 38 matches in 16 ms.</li><li>Cluster wire format, one CrawlTask: JSON 327 bytes, encode and decode 1,394 ns.</li><li>Both ONNX models preloaded: 4.47 s, on a background thread after the app is ready.</li><li>Idle RSS with both models loaded: 1.77 to 1.88 GB.</li><li>Image vector duplication, before and after the fix: 43 times, then 1.00 times.</li><li>Container memory: 1g is OOMKilled with either vectors or a browser, 2g passes, and the two want separate runs.</li><li>Tests: 133 classes, 1,098 methods, with an 80% line coverage gate on four modules.</li></ul><h3>Limitations and trade-offs</h3><ul><li>The embedded index and vector store cost the extracted text once per node. Fine for one or two nodes. For a real cluster, point it at Elasticsearch or Qdrant, which the startup report recommends out loud.</li><li>Replication is asynchronous. Immediately after a crawl, one node can answer a search before another has caught up. Seconds, not minutes.</li><li>A node killed mid crawl leaves runningState set. The registry says nothing is running, the row says otherwise, and interrupt has nothing to interrupt. On the list to fix.</li><li>Article extraction leaves some template text in the chunks. Share buttons and footer credits turn up in search snippets.</li><li>Politeness is the throughput ceiling. fetchInterval defaults to a second per node, and that is deliberate. This is not the tool for hammering one host as fast as it will answer.</li><li>One crawl at a time per cluster. Two crawls means two clusters, which is a name and a port.</li><li>Pdf, Word and Excel need a bean. Each is another dependency with its own licence and its own appetite for memory, so the choice belongs to the application.</li></ul><h3>Summary</h3><ol><li>One url in, and you get a whole site on disk, a full text index and vectors, from one pass.</li><li>Nothing to provision. H2, an embedded Lucene index and local ONNX models are the defaults. No database, no search server, no API key, no model download by hand.</li><li>Decentralised by default. Every node runs the same jar. No coordinator, no central queue, no leader in the fetch path.</li><li>Scaling is starting another process and pointing it at the same cluster name, including joining a crawl already running.</li><li>Replay rebuilds an output from the files, so a lost index or a changed analyzer never means crawling the site again.</li><li>Versions make a rebuild safe. The old version keeps answering searches, and an interrupted rebuild publishes nothing.</li><li>Three ways to search, including finding a picture by describing it, with the models running locally.</li><li>Every decision the crawler makes is an interface with a shipped default that a bean replaces.</li><li>The crawl cannot leave the site. Two boundary rules, neither of which can be switched off.</li><li>Apache 2.0, JDK 17, Spring Boot 4.1.</li></ol><p><strong>Source, issues and full documentation:</strong> <a href="https://github.com/paganini2008/greenfinger">github.com/paganini2008/greenfinger</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11820f50de3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>