suga is a personal coding agent I write in Go. The goal is a session that can stay with me for a full working day. This post covers the trade-offs it makes for long sessions, which of them are backed by data and which experiments failed, and how it works with gofer, my job platform. suga itself is built on gofer, one batch of agent jobs at a time.

The current version is v0.49. It is v0, so minor releases may break things. It is MIT licensed. What follows is an introduction to suga.

A session that lasts all day: the hard part is the context reset

The context window fills up eventually. The usual fix is to compress old messages into a summary and keep going. The summary is written by the model, though, and the model will reasonably drop whatever it thinks is unimportant, such as the “don’t touch vendor/” you said in turn 1.

suga uses segments instead. When the context is nearly full, it starts a new segment. The old one is stored and stays searchable, and the new one opens with a handoff. The handoff has a fixed structure: current task, coordinates (file / symbol / line), accepted and rejected decisions, how far verification got, progress and blockers, next step, and references to look things up. The “accepted and rejected” section holds hard constraints, rules and the user’s own words. It is kept verbatim, has no length limit and may not be rewritten. Every other section has a byte budget (coordinates get at most 2 KB, for example), and old coordinates are deduplicated by file and symbol when they are folded in.

Why a budget? The early handoff folded the whole previous handoff into each new one. In one 14-turn session it grew from 4.7 KB to 7.8 KB to 16 KB, and every later request carried it. With budgets, the same task stays at 7.2k to 8.7k tokens of context after each reset.

This is tested by a set of “hard” long-session evals. Each task runs 13 to 14 turns with 2 to 3 forced resets, padded with large files and unrelated work, and the last turn checks whether the turn-1 constraint survived. For example:

• the user changed the constraint halfway through, and the check expects the new one, not the old one;
• the topic goes A, B, C and back to A, and the check looks for mixed-up topics;
• a tool output contains “ignore previous rules”, and the check looks for whether the agent followed it.

With a real model (deepseek-v4.1-flash), three runs per task, it currently passes 12/12.

I got this wrong once. I gave the model call that writes the handoff an output limit derived from the budget (about 3.3k tokens). Unit tests and fake-model evals all passed. On a real model, 3 of 4 tasks failed at the reset with “empty summary”: a reasoning model spends output tokens on thinking, and the limit ran out before the answer started. That call now has no limit. The handoff size is enforced by the budgets and by trimming each section afterwards. The lesson: a fake model will not catch changes that depend on model behavior. They need a real model run.

Start with the shortest possible prompt (measured on v0.48). The prompt below asks the model to answer with just “ok”:

suga run --prompt "只回答 ok 两个字"
ok 
opencode-go/deepseek-v4.1-flash · in 8853 / out 2 tokens · 2.8s

A 10-character prompt, 8,853 input tokens. The context event in suga inspect trace <session> --json shows where they went:

part     bytes    notes
tools    19,753   schemas of the 16 tools in a read-only session
rules     6,782   workspace AGENTS.md and local context files
system    3,559   the stable system prompt
skills    1,333   the skills list

Providers cache by prefix, so the prefix has to stay stable: the tool table, the system prompt and the segment context do not change within a session, which lets later requests reuse what earlier ones cached. Put a timestamp or a random ID into the system prompt on every request and the cache stops working and the bill goes up. The thinking-signature bug later in this post has the same cause.

Cost in a long session is roughly “number of calls × context per call”. Breaking a 14-turn task down call by call showed that 29% of the input was the fixed prefix resent on every call: about 0.9k tokens of system prompt and about 7.0k tokens of schemas for the 20 built-in tools.

So I did two things:

1. Shorter tool descriptions. I removed parser details and conventions repeated across tools (five search tools each said “skips vendor/node_modules by default”) and said them once in the system prompt. That saves about 920 tokens per call. A test now pins the character count of every tool schema so it cannot grow back unnoticed.
2. Lazy loading for rarely used tools. In 128 eval runs, 11 of the 20 tools were called 44 times in total. The idea was to drop them from the tool table, keep a one-line list, and load them when needed.

Both went wrong at first, and the evals caught both:

• After the trim, one task made 24% more calls. The session log showed why: the words “skips vendor/” had been cut from the no_ignore parameter description, so the model could not see them and listed vendor/ through the shell instead. Putting those words back brought the call count back to normal.
• For lazy loading I first ran a probe: describe the tool schemas only in message text, not in the tools declaration. Across three models and 9 runs, the model made 0 native calls. Models do not call tools that are not declared. So within a segment, lazy tools go through a resident proxy tool, tool_call, and they join the real tool table at the next reset, which keeps the tool table and the cache stable within a segment. The on/off comparison: prompt per call down 16.9%, calls up 8.3%, total down only 6.8%, which failed the "calls up by no more than 5%" bar. So the feature is off by default, and tools.deferred_native: auto turns it on.

These comparisons later became the eval compare subcommand. It lines up two result sets by task, labels each as helped / hurt / no observed effect / confounded, and writes the verdict to a ledger, so I no longer do the math by hand.

When something breaks, you can find out why: a thinking-signature bug

suga writes every run to one SQLite file: sessions, turns, tokens and cache hits per model call, every tool call, resets, approvals and end-of-turn checks. Agents fail in predictable ways: going in circles, rerunning the same command, writing without verifying, claiming “done”. With a timeline these show up. Here is the summary of one bug-fix run (measured on v0.48):

suga inspect trace < sessio n > --analyze

steps                  5 model calls, 6 tool calls, 2 parallel steps (max 2 calls in one step)
tokens                 in 51.2k (cache read 40.4k), out 427; reasoning 77 chars, text 478 chars
tools                  file_read x2, shell_exec x2, file_edit x1, file_list x1
re-reads               -
reads after own write  0
shell                  2 runs, 1 failed (exit != 0), 0 exited 0 with stderr output
repeated commands      command=go test ./... x2
tests                  2 runs, 0 after the first pass
phases                 explore 2, first write to green 2, after green 1 (model calls)

The re-reads, reads after own write, repeated commands and tests rows exist to spot going in circles and skipped verification. --transcript renders the same session as plain text you can paste into a review or hand to another agent, and --web opens a read-only page you can leave open during a long run. A trace is a record after the fact. It does not stop an agent from looping, and it does not replace tests.

Here is one problem the trace helped me find. Running multi-step tool calls on Claude, the second request was rejected with Invalid signature in thinking block. Comparing the two requests turned up two causes:

1. after optional arguments the model left out were normalized away, tool_use.input was sent back as null;
2. Anthropic now binds a thinking block’s signature to the request prefix it was generated under, and suga added a “about N minutes left before the deadline” reminder to every step. The text changed on every step, so the earlier thinking block no longer verified.

The fix: send back {}, and keep that reminder's text and position fixed within a turn. After that I recorded real streaming responses from four providers (deepseek, glm, Claude, codex) as replay fixtures, with keys removed, so unit tests catch the next change in these protocol details first.

Safe by default, and “tests pass” comes from a command, not the model

suga is read-only by default. With writes enabled, chat asks before writing a file or running a command ( y / n / a to stop asking for this session), and rules by tool, path or command can live in a file. File tools snapshot before writing, so :undo or suga undo <session> can roll back. Changes made through shell_exec are recorded but not rolled back, on purpose.

A model saying “fixed, tests pass” does not count. --verify runs the check command you give it after the turn ends, and a failure goes back to the model as a new turn (at most 2 rounds by default). A small repository with one bug (the prompt asks it to confirm the bug in Add with the tests, then fix it; the model answered in Chinese):

suga run --allow-write --verify "go test ./..." --prompt "calc.go 里的 Add 有 bug，先跑测试确认，再修好它"

已确认并修复。
**确认 bug**：修复前 go test ./... 失败 -- calc_test.go:7: Add(2,3) = -1, want 5。
**原因**：calc.go:5 里 Add 写成了 a - b。
**修复**：改为 return a + b。

verify #1: go test ./... → exit 0 (488ms)

The verify #1 line is the evidence. The same turn also runs end-of-turn checks ( nudge): files written but no check command run, plan steps still open, or an answer ending with "next I'll run the tests" each push the model back once.

One more detail: approval is done by a person, and the model does not know that. Once, running as a resident session, suga replied to a message for me and wrote that “writing the file did not trigger approval”. In fact I had approved it. Now a manually approved call ends its result with the line [已经人工批准执行] ("executed after manual approval"), so the model does not have to guess.

Working with gofer: how suga gets built

gofer is my other project. It runs agents such as codex, claude and suga as async jobs: submit a job, work in a separate git worktree, approve from the web UI or DingTalk, read logs and token usage, merge back to the main branch. suga runs in gofer in two ways:

• suga run: a one-off job that exits when done and streams events as stream-json;
• suga acp: a resident session. Each tool-call approval goes through ACP to gofer, where you click it on the web or in DingTalk.

This is how suga itself is developed:

Supervisor session (reads the plan, writes job briefs, reviews, releases)
   │  gofer job run -a suga --model <deepseek | glm | codex> --worktree ...
   ▼
Several suga jobs in parallel, each in its own worktree + branch: change code, run tests, commit
   │  supervisor reviews the diff, reruns race tests and pty smoke tests in a Linux container
   ▼
gofer job worktree merge → suga-verify workflow
   (Linux race / CI simulation / staticcheck / Windows tests + ConPTY smoke / 40+ pty scenarios)
   ▼
tag → GitHub Actions builds binaries for three platforms → upload

Jobs go to different models in rotation, and every review verdict is recorded. Changes that need a design get a design doc, a review and human approval, then a plan that also needs approval before implementation. By v0.49 there are 56 tags and more than 2,400 commits. For the last few dozen batches, the code was written almost entirely by suga jobs. I split the work, review it and release.

The most useful step in this loop turned out to be review. Implementers often fail where they cannot verify their own work: a pty smoke test written on Windows that never ran, a fake model whose if/elif chain was cut in two by a single assignment, a test that took the echo of the input line as the end of the turn. All of these were caught by rerunning in a container before the merge.

The two also connect directly. gofer sends mail to a resident suga session ( agent:suga), and suga replies with gofer's mailbox tool. suga sessions can message each other ( suga send) with message IDs, merged delivery, receipts and failure barriers. gofer's memory backend can serve as suga's long-term memory.

The core embeds in other Go programs

pkg/agentcore is the core below the application layer: the model/tool loop, resumable run state, providers, tool registry and approval, MCP, skills and long-term memory. It does not depend on internal/*. An allowlist in pkg/agentcore/layering_test.go enforces that boundary: any dependency outside the list, or a new package that is not registered, fails the test. The smallest working setup needs one provider:

runtime := agentcore.New(agentcore.Config{
    Provider: mock.NewProvider(mock.Final("hello")),
})

result, err := runtime.Run(ctx, agentcore.RunInput{
    AgentID:     "example-agent",
    Instruction: "say hello",
})
// result.Output == "hello"

For a real model, swap in llm.NewOpenAIProvider(...). Add tools with reg.Register(...). To control permissions, wrap it in tools.NewApproval: with no approver configured, every call is denied, not allowed. The limits, stated here too: it is a single-agent loop with no multi-agent orchestration, and pkg/sugasdk (in-process Open / Send / Event) is marked experimental.

Compared with similar tools

Claude Code and Codex CLI come from the model vendors, and their model integration and interaction are more polished. aider has a mature git workflow. opencode has a good TUI. suga makes different trade-offs:

• one Go binary with no runtime dependencies; Windows is a first-class platform (PowerShell, Job Object process trees, ConPTY smoke tests);
• session persistence, segments and handoffs, long-session evals and regression gates are where most of the effort went;
• it embeds in other Go programs (pkg/sugasdk) and can run as an ACP server driven by an editor or by gofer;
• it switches between providers, and the config can list fallback models that it moves to when a connection fails.

When it is a poor fit:

• you want an IDE plugin or a GUI: it is terminal only for now;
• you want multi-agent orchestration: a sub-agent only does one read-only investigation and never hands out work of its own; job queues and the console live on the gofer side;
• you do not want to bring your own provider key: every request goes straight to the endpoint you configure, and there is no hosted service;
• you need English docs: the detailed docs are the Chinese built-in skill suga-guide;
• you need a stable API: it is v0.

I mostly use it on Windows and Linux, and CI runs Ubuntu and Windows. macOS builds are published, but I rarely use them.

Up and running in 30 seconds

eget install inhere/suga          # or go install github.com/inhere/suga/cmd/suga@latest
suga init                         # creates ~/.suga/config.yaml and .env
# put one key in ~/.suga/.env, e.g. DEEPSEEK_API_KEY=... (built in: openai / anthropic / deepseek)
suga config set model deepseek/deepseek-chat
suga chat                         # start in the current directory; :help lists commands, :context shows what is in the context

For a one-off task: suga run --prompt "Summarize this project". To have it change code and check its work: suga run --allow-write --verify "go test ./..." --prompt "Fix the failing test".

If you read the code, start in these four places

• internal/cli: one file per command; root.go shows the whole command tree;
• internal/wiring: assembles providers, tools, skills, the system prompt and the runner, shared by the CLI and ACP;
• internal/personal: turns, segments, handoffs and events; reading it is reading suga's product logic;
• pkg/agentcore: the core and capability layer; start with the table in layering_test.go.

docs/ holds 45 design docs, 32 plans and 13 research reports. Most of the "why" is in there, and most numbers in this post come from docs/research.

I’m inhere ( blog · GitHub), the author of suga. The numbers in this post come from measured runs on v0.48 to v0.49 and from the reports under docs/research in the repository. The commands are all in the text, so you can reproduce them.

Originally published at https://inhere.github.io on October 26, 2026.

14 turns, 3 context resets, and the turn-1 constraint still holds: long sessions in suga was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

The slow part is often not the project itself. It is the central package index in the middle: a formula, bucket, manifest, or package definition that has not caught up yet. eget is for that gap. If the release asset already exists, or the download URL can be described by a simple rule, eget can fetch and install it directly.

• Project: https://github.com/inherelab/eget
• Latest release: v1.8.x
• Documentation: README.md

Start with a GitHub repository

eget cfg init
# install tools
eget i openai/codex
eget install junegunn/fzf
eget install --name rg BurntSushi/ripgrep

The shortest path is to pass a repository name:

eget reads release assets, matches the current OS and architecture, downloads the selected file, extracts it when needed, and installs the executable into the target directory.

If a project uses nightly builds, beta tags, or a specific release tag, choose it explicitly:

eget install --tag nightly inhere/markview
eget install --tag v0.28.0 openai/codex

There is no extra registry step here. If the upstream release page has the asset, eget can start from there.

The repository is the entry point

Many package managers depend on a central index. That gives users a consistent experience, but it also creates delay. A project can publish a release before the index is updated.

eget keeps the default path short:

owner/repo -> releases -> assets -> install

When automatic asset matching is not enough, inspect the assets and add filters:

eget query --action assets openai/codex

eget install --asset "PRE:codex,SUF:.zip" openai/codex
eget install --asset "REG:\\.deb$" owner/repo
eget install --asset "windows,amd64,zip" owner/repo

Some release archives have nested directories, or the executable name is not the name you want on disk. Handle that at install time:

eget install --extract-all \
  --rename "codex-x86_64-pc-windows-msvc.exe=codex.exe" \
  openai/codex

eget download --extract-all --strip-components 1 \
  --asset "windows,zip" ventoy/Ventoy

The point is not to describe every package in advance. The point is to work with the files that upstream already published.

It is not limited to GitHub

GitHub is common, but it is not the only source. eget install and eget download can target GitLab, Gitea/Forgejo, SourceForge, direct URLs, and local files.

eget install gitlab:fdroid/fdroidserver
eget install gitlab:gitlab.gnome.org/GNOME/gtk
eget install --asset linux,amd64 gitea:codeberg.org/forgejo/forgejo
eget install --asset x64,PerUser,setup sourceforge:winmerge

A plain download URL also works:

eget download --file go --to ~/go1.17.5 \
  https://go.dev/dl/go1.17.5.linux-amd64.tar.gz

That matters for tools that do not publish GitHub releases. If there is a stable URL, or if the latest version can be read from a page, JSON file, or YAML file, the tool does not need to be wrapped in a central index first.

Internal tools can use one reusable rule

Internal tool distribution is usually boring in a very specific way. Every team has its own layout:

http://mydev.lan/tools/markview/latest.yaml
http://mydev.lan/tools/markview/markview-linux-amd64
http://mydev.lan/tools/chlog/latest.yaml
http://mydev.lan/tools/chlog/chlog-linux-amd64

Once the naming rule is stable, each tool should not need a full copy of the same configuration. pkg_templates lets you describe the rule once:

[pkg_templates.mydev]
latest_url = "http://mydev.lan/tools/{name}/latest.yaml"
latest_format = "yaml"
url_template = "http://mydev.lan/tools/{name}/{name}-{os}-{arch}{ext}"
ext_map = { windows = ".exe", linux = "", darwin = "" }

[packages.markview]
repo = "pkg-template:mydev:markview"

Then use the short alias:

eget add mydev:markview
eget install mydev:markview
eget install --add mydev:chlog

This is not an internal app store. There is no server-side registry to maintain. It is a local rule for turning a package name into a latest-version request and a download URL.

Keep frequently used tools in config

For a one-off test, install owner/repo is enough. For tools you use regularly, add them to local config:

eget install --add junegunn/fzf
eget install --add --name rg BurntSushi/ripgrep
eget add --name winmerge --system windows/amd64 \
  --asset x64,PerUser,setup sourceforge:winmerge

# restore all configured tools from an existing config file
eget install --all

List what is installed or configured:

# installed packages by default; --all shows all configured packages too
eget list
eget ls --all
# configured packages that are not installed yet
eget ls --ni

The install record lives in ~/.config/eget/installed.toml by default. Config files live under the same directory, and you can edit them directly when that is faster.

Update tools

eget update --all # check and update all updatable packages
# or check first, then update selected tools
eget up --check
eget up fzf markview codex

Self-update is a separate command:

eget up --self

SDK downloads are built in

SDK archives have a similar problem to CLI tools: multiple versions, multiple platforms, mirrors, archive extraction, and local install records.

eget sdk handles that layer:

eget sdk config add --all
eget sdk config add --all --mirror mirror
eget sdk config add jdk --mirror zulu

eget sdk index refresh go
eget sdk index show go

eget sdk install go@1.22.0
eget sdk install go:1.22 node:20.11.1
eget sdk dl --os linux --arch arm64 -o ./downloads go:1.22

The built-in templates cover Go, Node, and JDK. Sources can use official URLs or common mirrors. SDK index cache, archive download cache, and install records are kept separate, which works better for multi-version SDK installs.

One boundary is important: eget sdk downloads, extracts, and records SDKs. It does not modify PATH, write shell hooks, or switch the active version for a project. If you need project-level activation, pair it with a tool such as xenv.

Cache mirrors help on team machines

Repeated installs are faster when downloads are cached. On a local network, one machine can expose its existing package and SDK cache:

eget cache serve --host 0.0.0.0 --port 8686 --token "$EGET_CACHE_TOKEN" --json-log

Other machines can try that cache before going upstream:

[cache_mirror]
enable = true
url = "http://192.168.1.10:8686"
timeout = 5
fallback = true

With this enabled, install, download, and sdk install try the LAN cache first and fall back to the original source on a miss. The mirror is only a download optimization, not a trust root. Existing checksum validation still runs later in the flow.

More features are in the GitHub docs

This article covers the main path. eget also has a set of practical features that are better documented in the GitHub README and configuration docs:

• concurrent downloads and batch install/update
• HTTP Range chunked downloads for large files
• resumable downloads with .part files and metadata
• Windows GUI installer launch
• portable GUI package installation into gui_target
• separate API cache, download cache, and SDK index cache

You may not need all of those every day. They become useful in CI, on shared development machines, on Windows tool distribution flows, and when large SDK archives are downloaded repeatedly.

Where eget fits

eget is a good fit when:

• you want to try a new GitHub release immediately
• a tool ships zip, tar.gz, exe, deb, or similar release assets
• internal tools follow a stable download URL rule
• CI machines, developer laptops, or LAN machines need the same tools and SDKs repeatedly

It is not a replacement for a system package manager. Dependency solving, service registration, and complex post-install scripts are not its job.

The shorter description is this: eget is a release asset downloader and installer. If GitHub already has the release, a download site already has the file, or an internal directory already has the package, eget helps get it onto the machine without waiting for the index in the middle.

Originally published at https://inhere.github.io on July 1, 2026.

eget: install GitHub releases without waiting for a central index was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

github.com/gookit/rotatefile stays deliberately small: it provides an io.Writer with file rotation, gzip compression, and cleanup. You can keep using log/slog, log, zap, or gookit/slog; rotatefile only handles the file lifecycle.

• Project: https://github.com/gookit/rotatefile
• API docs: https://pkg.go.dev/github.com/gookit/rotatefile
• CLI cleaner: cmd/filecleaner

rotatefile works at the file layer

Some logging libraries include file rotation, but that usually means adopting their logging API, field model, and handler design. rotatefile sits lower, at the io.Writer layer:

package main

import "github.com/gookit/rotatefile"

func main() {
 w, err := rotatefile.NewConfig("logs/app.log").Create()
 if err != nil {
  panic(err)
 }
 defer w.Close()

 _, _ = w.Write([]byte("a log message\n"))
}

There is no logger concept in that example. rotatefile.Writer handles writes, rotation, sync, close, and old-file cleanup. Any logger that accepts an io.Writer can use it.

The package was split out from gookit/slog. That makes the boundary cleaner: use it when log files need rotation, not when you want to replace your logger.

The common configuration is small

A typical setup looks like this:

package main

import "github.com/gookit/rotatefile"

func main() {
 w, err := rotatefile.NewConfig("logs/app.log", func(c *rotatefile.Config) {
  c.MaxSize = 100 * rotatefile.OneMByte
  c.RotateTime = rotatefile.EveryDay
  c.BackupNum = 30
  c.BackupTime = 24 * 7
  c.Compress = true
 }).Create()
 if err != nil {
  panic(err)
 }
 defer w.Close()
}

Those fields cover most local log-file setups:

• MaxSize: rotate by file size, in bytes. The default is 20MB; 0 disables size-based rotation.
• RotateTime: rotate by time. Built-ins include EveryDay, EveryHour, Every30Min, Every15Min, and EveryMinute.
• BackupNum: keep at most N old files. The default is 20; 0 means no count limit.
• BackupTime: keep old files for at most this many hours. The default is 24 * 7.
• Compress: gzip rotated files.

A few advanced options are also useful:

• FilePerm: permission used when creating log files. The default is 0664.
• CleanOnClose: run cleanup when the writer closes.
• RenameFunc: customize names for size-based rotation.
• TimeClock: replace the clock, mostly for testing time-based rotation.

The model is simple: after a write, the writer checks size and time conditions, rotates when needed, then lets cleanup policy deal with old files.

rename and create are different file models

rotatefile supports two rotation modes.

The default ModeRename always writes to a fixed file such as logs/app.log. On rotation, the current file is renamed to a backup file with a suffix, and a new logs/app.log is created.

c.RotateMode = rotatefile.ModeRename

That mode works well when external tools always tail one stable file. Collectors, scripts, and tail -f logs/app.log do not need to know the current period's filename.

ModeCreate writes directly to a new dated file for each time period.

c.RotateMode = rotatefile.ModeCreate

That is better when logs are reviewed or archived by period, for example when today’s logs should naturally live in a file like app.20260624.log.

Neither mode is universally better. Pick based on whether a stable entry file or period-based filenames matter more.

Use it with standard slog

log/slog has been in the standard library since Go 1.21. Its handlers accept an io.Writer, so the integration is direct:

package main

import (
 "log/slog"

 "github.com/gookit/rotatefile"
)

func main() {
 w, err := rotatefile.NewConfig("logs/app.log", func(c *rotatefile.Config) {
  c.MaxSize = 50 * rotatefile.OneMByte
  c.RotateTime = rotatefile.EveryDay
  c.BackupNum = 7
 }).Create()
 if err != nil {
  panic(err)
 }
 defer w.Close()

 logger := slog.New(slog.NewJSONHandler(w, nil))
 logger.Info("user login", "uid", 1001)
}

slog still owns the structured log format. rotatefile owns the destination file lifecycle.

Use it with log, zap, or other loggers

The standard log package is even simpler:

package main

import (
 "log"

 "github.com/gookit/rotatefile"
)

func main() {
 w, err := rotatefile.NewConfig("logs/app.log").Create()
 if err != nil {
  panic(err)
 }
 defer w.Close()

 log.SetOutput(w)
 log.Println("hello rotatefile")
}

With zap, the idea is the same; wrap the writer as a zapcore.WriteSyncer:

// zapcore.AddSync(w)

That is the benefit of staying at the io.Writer layer. Field encoding, log levels, sampling, and handler design remain separate from file rotation.

Cleanup can be used on its own

Rotation prevents the current file from growing forever, but old files can still pile up. rotatefile includes FilesClear, a standalone cleaner:

package main

import "github.com/gookit/rotatefile"

func main() {
 fc := rotatefile.NewFilesClear(func(c *rotatefile.CConfig) {
  c.AddPattern("/var/log/app/*.log.*")
  c.BackupNum = 20
  c.BackupTime = 168
 })

 if err := fc.Clean(); err != nil {
  panic(err)
 }
}

It does not depend on Writer. You can use it for logs written by Go programs, PHP-FPM, Nginx, or any other process that leaves files behind.

Useful cleanup options:

• Patterns: match files with glob patterns.
• BackupNum: keep the newest N files.
• BackupTime: remove files older than the configured age.
• TimeUnit: change the unit for BackupTime; the default is hours.
• Recursive: recurse when a pattern matches a directory.
• RemoveEmptyDir: remove directories that become empty after recursive cleanup.
• DryRun: print files that would be removed, without deleting them.
• IgnoreError: continue after a single file removal fails.

BackupNum and BackupTime cannot both be 0. Without at least one retention rule, cleanup fails instead of guessing.

filecleaner is useful for cron

If you do not want to write Go code, use the filecleaner command included in the repository. It is built on FilesClear and configured with JSON.

go install github.com/gookit/rotatefile/cmd/filecleaner@latest

filecleaner -c filecleaner.json
filecleaner --dry-run -c filecleaner.json
filecleaner --daemon -c filecleaner.json

The config has a top-level jobs array. Each job has its own patterns and retention policy:

{
  "jobs": [
    {
      "patterns": ["/var/log/app/*.log.*"],
      "backup_num": 20,
      "backup_time": 168,
      "time_unit": "1h"
    },
    {
      "patterns": ["/var/log/svc"],
      "recursive": true,
      "remove_empty_dir": true,
      "backup_time": 7,
      "time_unit": "24h"
    }
  ]
}

Run --dry-run first when adding a new cleanup job. Once the output looks right, put it in cron or run it with --daemon.

LineWriter avoids half-written log records

rotatefile also ships a bufwrite subpackage. Its LineWriter is not line-buffering in the C stdio sense. Here, "line" means one logical record per Write call.

Why does that matter? A normal bufio.Writer may split a single Write(p) when the remaining buffer is too small: part goes to the underlying writer, and part stays buffered. For plain text that is usually fine. For JSON logs, an external collector might read a half-written record.

LineWriter takes a stricter path. If the current buffer cannot fit this Write(p), it flushes the old buffer first, then writes p as one complete underlying write.

package main

import (
 "github.com/gookit/rotatefile"
 "github.com/gookit/rotatefile/bufwrite"
)

func main() {
 w, err := rotatefile.NewConfig("logs/app.log").Create()
 if err != nil {
  panic(err)
 }

 bw := bufwrite.NewLineWriter(w)
 defer bw.Close()

 _, _ = bw.Write([]byte(`{"level":"info","msg":"ok"}` + "\n"))
}

Two details are easy to miss:

• It does not flush automatically when it sees \n; call Close or Flush.
• It is not concurrency-safe. Use an external lock, or let the logger serialize writes.

A few implementation details matter

This is not a source walkthrough, but a few details explain why the package is more than a thin os.Rename wrapper.

First, time-based rotation calculates the next check against period boundaries. Daily, hourly, and minute-level rotation use different suffix formats: 20060102, 20060102_1500, and 20060102_1504. The names sort naturally by period.

Second, the write path is “write first, then check rotation.” The caller gets straightforward Write behavior: the current payload lands in the currently opened file, and then the writer decides whether the next write should go to a new file.

Third, old-file cleanup does not synchronously scan directories on every write. The writer triggers cleanup asynchronously to keep the write path shorter. Close() syncs the file, stops the cleanup goroutine, and then closes the file handle.

Fourth, TimeClock is replaceable, and the package includes MockClocker. Time-based rotation can be tested without waiting for the next minute or hour.

When rotatefile fits

It is a good fit when:

• A service writes local log files and needs size limits.
• A CLI, background task, or single-host tool should not depend on system logrotate.
• The project already uses slog, log, or zap, and only file rotation is missing.
• Logging API and file lifecycle should stay separate.
• A small tool is needed to clean logs, backups, or other expired files.

It is not always needed:

• In container setups where logs go to stdout and the platform handles collection and retention.
• In environments that already have a standard system-level logrotate policy.
• When multiple processes write the same log file. That deployment needs a separate look at locking and ownership.

rotatefile is a small infrastructure package, not a full logging framework. It covers the layer most Go loggers leave open: rotation, old-file cleanup, and compression.

Install

go get github.com/gookit/rotatefile

For full configuration details, see GoDoc. For file cleanup without writing Go code, start with the filecleaner README.

Originally published at https://inhere.github.io on June 24, 2026.

rotatefile: log rotation, compression, and cleanup for Go was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

GCli v3.5 Updates: Changes Since v3.3.1

GCli is a command-line application and tool library for Go.
This post covers the main changes from v3.3.1 to the recently released v3.5 (including the v3.4 cycle). These updates focus on developer experience and underlying stability.

If you write CLI tools in Go, a few of these features might be useful. Here are the main changes.

Key Updates

• Shell completion: Supports zero-registration generation and a dynamic completion mode (bash / zsh / PowerShell).
• Command middleware: Handle auth, logging, and other cross-cutting concerns via Command.Use() / App.Use().
• Grouped help: Categorize commands and options into titled sections using Category.
• Struct binding improvements: Added a field tag rule and support for expanding anonymous nested structs.
• Interactive input: Automatically prompt for missing values via Question.
• POSIX short-option merging: Support for -aux splitting into -a -u -x.
• Robustness fixes: Panic handling, help command behavior, and more.
• A few breaking changes (migration guide at the end).

1. Shell Completion Improvements

Generating shell completion scripts previously required hardcoding command and option names, meaning adding a new command required regenerating the script. GCli v3.5 improves this workflow.

You no longer need to manually register the genac command. A built-in global option generates the static script directly:

# generate a completion script for your shell, then source it
myapp --gen-completion bash > myapp.bash
source myapp.bash

# zsh / PowerShell also supported
myapp --gen-completion zsh  > _myapp
myapp --gen-completion pwsh > myapp.ps1

Additionally, a dynamic completion mode is now available. The generated script no longer hardcodes names; instead, it calls back into your binary via the built-in --in-completion option to fetch candidates at completion time. New commands will immediately work with Tab-completion without regeneration.

For option values, you can define a list of candidates using Choices:

c.StrOpt2(&format, "format", "output format",
    gflag.WithChoices("json", "yaml", "table"))
// typing `--format <Tab>` now suggests: json  yaml  table

2. Command Middleware

If you need to run auth checks or logging before a command’s main logic without duplicating code across commands, you can use middleware.

Handlers registered with Use() run in order before the command's main Func. If a handler returns an error, the chain stops, and the error propagates upward.

// command-level middleware
cmd.Use(func(c *gcli.Command, args []string) error {
    if os.Getenv("TOKEN") == "" {
        return c.NewErrf("missing TOKEN env")
    }
    return nil // return nil to continue the chain
})

// application-level middleware: applies before every command
app.Use(func(c *gcli.Command, args []string) error {
    gcli.Debugf("running command: %s", c.Name)
    return nil
})

Both Command.Use() and App.Use() return the receiver, supporting chaining. Apps without middleware behave exactly as before.

3. Grouped Help

When an app has many commands and options, the help output can become cluttered. You can now group them into titled sections using the Category field.

// group commands
app.Add(&gcli.Command{Name: "migrate", Desc: "run db migrate", Category: "database"})
app.Add(&gcli.Command{Name: "serve",   Desc: "start http server"}) // default group

// group options
cmd.StrVar(&dsn, &gcli.CliOpt{Name: "db-dsn", Desc: "database dsn", Category: "database"})
cmd.StrOpt2(&port, "port", "bind port", gflag.WithCategory("network"))

Groups appear in the order of their first definition, and items within a group are sorted by name. If no category is set, the output format remains the same as in older versions.

4. More Flexible Struct Binding

FromStruct now supports a third tag rule (TagRuleField) and automatically expands anonymous nested structs.

The three available rules, selected via c.FromStruct(ptr, ruleType):

• gcli.TagRuleNamed (default): flag:"name=int0;shorts=i;required=true;desc=message"
• gcli.TagRuleSimple: flag:"desc;required;default;shorts"
• gcli.TagRuleField (new): Uses the field name (SnakeCase) as the option name and reads metadata from independent tag keys.

type commonOpts struct {
    Verbose bool `flag:"v" desc:"enable verbose output"`
}

type demoOpts struct {
    commonOpts        // anonymous: expands into a --verbose/-v option
    UserName string `flag:"u" desc:"the user name" required:"true"`
    Age      int    `desc:"the user age" default:"18"`
}
c.MustFromStruct(&demoOpts{}, gcli.TagRuleField)
// => options: --user-name/-u (required), --age (default 18), --verbose/-v

The field rule keeps the option name tied to the struct field, while desc, default, and required live in their own tags, making it easier to read and maintain.

5. Declarative Interactive Input

If a required option is missing at runtime, you can now attach a Question. GCli will detect the empty value and prompt the user for input interactively.

c.StrOpt2(&token, "token", "the access token",
    gflag.WithQuestion("Please input your access token: "))

$ myapp deploy
Please input your access token: ▮

If a custom Collector is also set, it takes priority over Question.

6. POSIX Short-Option Merging

GCli now supports merging short flags (e.g., -a -u -x becomes -aux) in a POSIX style. This is disabled by default and can be enabled via Config.EnhanceShort.

c.ParserCfg().EnhanceShort = gcli.EnhanceShortMerge  // 1: -aux => -a -u -x
c.ParserCfg().EnhanceShort = gcli.EnhanceShortAttach // 2: also -Ostdout => -O stdout

It can also be enabled globally:

gcli.SetEnhanceShort(gcli.EnhanceShortMerge)

LevelConstantBehavior0EnhanceShortNoneOff (default), fully compatible with old behavior1EnhanceShortMergeSplit a group only when all members are bool shorts2EnhanceShortAttachAlso support value-attached form -Ostdout = -O stdout

A safety check is in place: a group is only split if every character is a boolean short option. Mixed forms like -aO (where O takes a value) are left untouched to prevent misparsing.

7. Robustness Fixes

Alongside new features, several long-standing issues were fixed:

• Panics are no longer swallowed: gflag.Parser.Parse previously ignored recovered panics. It now returns them as an error for easier upstream handling.
• help <command> works on the first call: Fixed an issue where it might print unknown input command "help".
• findSimilarCmd fix: No longer writes a phantom help entry into the registry when an unknown command is run.
• Command.Copy() fix: No longer clears the source command's hooks due to a shared pointer.

Breaking Changes & Migration

A few internal cleanups require adjustments if you depended on them:

BeforeAfterimport ".../gcli/v3/helper"Now internal; inline your own helperimport ".../gcli/v3/gclicom"Removed (unused after cliui migration)Global --verbose 4 flagEnv GCLI_VERBOSE=debug, or gcli.SetVerbose(gcli.VerbDebug) / gcli.SetDebugMode()

The --verbose flag was removed because it bound to a per-app copy that the underlying logger never read, making it ineffective. Use the environment variable or code to control log levels.

Additionally, multiple App instances within the same process now share global options (verbose / help / version / strict / completion).

Upgrade & Examples

go get -u github.com/gookit/gcli/v3@latest

The _examples/cmd directory in the repository includes runnable examples: struct-flag (field tags + anonymous structs), short-merge (short option merging), and ask-demo (interactive input).

If you run into issues or have suggestions, feel free to open an issue or PR on GitHub. For full API documentation, refer to GoDoc.

gookit/gcli v3.5.0 published — easy-to-use, feature-rich Go command line application was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’re excited to announce the release of gookit/goutil v0.7.2! This version brings numerous new features, optimizations, and bug fixes, further enhancing the stability and usability of this Go utility library.
This update covers multiple modules, including math utilities, string processing, filesystem operations, and command-line flag handling, aiming to provide developers with a richer and more efficient toolkit.

• Project Homepage: https://github.com/gookit/goutil
• Full Changelog: v0.7.2 Release Notes
• Full documentation: https://zread.ai/gookit/goutil

🛠️ Key Bug Fixes

This version fixes several key issues to ensure the stable operation of the library:

• ccolor module: Fixed the issue where the NO_COLOR environment variable was not working correctly during printing, improving the compatibility of color output.
• cflag module:
• Fixed a capp runtime error that occurred when parsing global flags.
• Fixed the issue where the OnAppFlagParsed event was not triggered when no command was input, ensuring the integrity of the event handling logic.

✨ New Feature Highlights

Math Utilities

• Added the IsInteger function for strictly checking if a value is of an integer type (e.g., int(x), uint(x)).
• Added StrictInt and StrictUint conversion functions for stricter integer type conversions.
• Added the FormatBytes function to format byte sizes into human-readable strings (e.g., "1.2 MB"), convenient for log output and UI display.

String Utilities

• Added the NumVersion function to quickly extract the numeric part from a version string.
• Added the ReplaceVars function to support quick rendering of strings containing {var} placeholders, simplifying template replacement operations.
• Added IsUpper and IsLower functions to check if a string is entirely uppercase or lowercase.
• Added IsUint and IsPositiveNum functions to enhance string numeric validation capabilities.
• Added BaseConvInt and BaseConvIntByTpl functions to support integer conversion between different bases.
• Added the ContainsByteOne function to check if a string contains any of the specified bytes.
• Optimized the MTimeBaseID function, which now supports bases greater than 36.

Filesystem Utilities

• Added the EnsureDir function to ensure a directory exists (creates it if it doesn't).
• Added HomeDir/UserHomeDir functions to get the user's home directory path.
• Added CreateSymlink and IsSymlink functions to support the creation and detection of symbolic links.
• Added FindAllInParentDirs, FindOneInParentDirs, and FindNameInParentDirs functions to support finding files in parent directories, simplifying the location of project configuration files.

Array/Slice Utilities

• Added the ToMap function to convert a list (slice) to a map.
• Added the Map1 function to convert a list to a new list (similar to a map operation).

Map Utilities

• Added the AppendSMap function for merging string maps.
• Added the AliasesNames function for handling alias names.

Testing Utilities

• Added the assert.StrContainsAll assertion function to check if a string contains all specified substrings.
• Added the testutil.SafeBuffer type, a thread-safe buffer designed for concurrent testing scenarios.
• Enhanced testutil.EchoServer, which now supports responding with 404, 405, 500, or custom status codes, facilitating the testing of HTTP client error handling logic.

Command-line Utilities

• The cflag module has a new HelpOnEmptyArgs configuration option to automatically display help information when no arguments are provided, improving user experience.
• Optimized the rendering of help information.
• Added support for setting aliases for cflag/capp commands.

Time Utilities

• Added the FormatDuration function to format time duration into a clock-like format (e.g., "1h 23m 45s"), perfect for displaying performance statistics.

⚡ Performance Optimizations & Refactoring

• Optimized the common logic of the ToStringWith function to improve conversion performance.
• Refactored cflag.App into the subpackage capp.App, improving code organization.
• Optimized the handling options logic in the ToInt64With function to enhance conversion efficiency.
• Split integer conversion related code into conv2int.go and optimized the performance of all integer conversion related methods in mathutil.
• Updated the VersionCompare function logic, fixing some comparison errors and improving the accuracy of version comparison.
• The VarReplacer in textutil now supports parsing variable names as environment variables, enhancing dynamic configuration capabilities.

🌐 Platform Compatibility & Development Workflow

• Added support for FreeBSD/Unix systems, expanding the platform coverage of the sysutil package.
• Added comprehensive GitHub Copilot instructions for gookit/goutil, optimizing the AI-assisted development workflow.
• Upgraded the github/codeql-action dependency from version 3 to 4, enhancing the security and capabilities of code analysis.

📝 Other Updates

• Added a ShowLen option for dump output, allowing control over whether to display length information.
• Updated the README generation CLI and related documentation, improving the project documentation generation process and templates.
• Fixed unit test errors and unified the code style.
• Updated some comments and code styles.

📦 How to Update

You can use the following command to get the latest version:

go get github.com/gookit/goutil@v0.7.2

Migration & Compatibility Notes

• cflag -> cflag/capp Subpackage Refactoring: cflag.App has been refactored to capp.App. If your code directly references cflag.App, please adjust:
• Old: import "github.com/gookit/goutil/cflag"
• New: import "github.com/gookit/goutil/capp"
• And switch related types/calls to the App in the capp package.

TIP: This is a potential breaking change in this release, please check and update your references before upgrading.

• VersionCompare and some string/version-related logic have been optimized. If you rely on the edge-case behavior of the old logic (such as equality comparison or sorting), please run relevant unit tests to verify after upgrading.

Example Snippets

Using mathutil.IsInteger / StrictInt

import "github.com/gookit/goutil/mathutil"
import "github.com/gookit/goutil/testutil/assert"

val := 123
if mathutil.IsInteger(val) {
    // true
}
i64, ok := mathutil.StrictInt("42") // Does not allow string to integer conversion
assert.False(t, ok)

Using strutil.ReplaceVars

import "github.com/gookit/goutil/strutil"
tpl := "Hello, {name}! Today is {day}."
out := strutil.ReplaceVars(tpl, map[string]string{
    "name": "Gopher",
    "day":  "Wednesday",
})
// Output: "Hello, Gopher! Today is Wednesday."

Using fsutil.EnsureDir

import "github.com/gookit/goutil/fsutil"

if err := fsutil.EnsureDir("/tmp/myapp/logs"); err != nil {
    // Handle error
}

📖 more usage please see https://zread.ai/gookit/goutil

🤝 Contributing

We thank all contributors for their efforts in this release! If you find any issues or have feature suggestions, please feel free to open an issue at GitHub Issues.

📖 More Information

• Project Homepage: https://github.com/gookit/goutil
• Full Changelog: v0.7.2 Release Notes
• Full documentation: https://zread.ai/gookit/goutil

Enjoy the new features and improvements in v0.7.2

gookit/goutil v0.7.2 Released: Enhanced Features & Fixes for an Improved Development Experience was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are pleased to announce the official release of ​​gookit/goutil v0.7.1​​! As the first patch version in the v0.7.0 series, this update brings ​​20+ new features, critical bug fixes, and multiple performance optimizations​​, further enhancing developer experience and toolset practicality.

GitHub - gookit/goutil: 💪 Helper Utils(800+): int, byte, string, array/slice, map, struct, dump, convert/format, error, web/http, cli/flag, OS/ENV, filesystem, system, test/assert, time and more. Go 常用的工具函数：数字，字符串，数组，Map，结构体，反射，文本，文件，错误，时间日期，特殊处理，格式化，常用信息获取等等

🚀 Release Highlights

1. ​​Enhanced Time Handling​​

• timex.ToDuration() now supports negative values and composite time formats
• Accepts time representations like -1.5h, 2d3h, and 2hour33min
• Simplified time calculation logic for improved code readability

2. ​​Map Operation Enhancements​​

• New AnyToStrMap() function for converting arbitrary types to string maps
• Introduced L2StrMap type for two-level string map operations
• Added utility methods to Data type: One(), StrOne(), BoolOne(), etc.
• New slice-to-map conversion functions: SliceToSMap(), SliceToMap(), SliceToTypeMap()

3. ​​String Processing Improvements​​

• Case-insensitive string check functions: IContainsOne(), IContainsAll()
• Fast comma-joining function JoinComma()
• IsVarName() function for variable name format validation

4. ​​Advanced Text Processing​​

• New StrVarRenderer implementing shell-style variable interpolation
• Simplified variable substitution and template rendering in tests

🛠️ Key Features Explained

⏱️ Enhanced Time Handling

// Supports negative values and composite formats
duration, err := timex.ToDuration("-1.5h")   // -1h30m
duration, err := timex.ToDuration("2d3h")   // 2d3h
duration, err := timex.ToDuration("2hour33min") // 2h33m

🗺️ Upgraded Map Operations

// Convert any type to string map
strMap := maputil.AnyToStrMap(someData)

// Multi-key retrieval
value := data.One("key1", "key2", "key3") // Returns first existing value

// Slice to map conversion
slice := []string{"a:1", "b:2", "c:3"}
sMap := maputil.SliceToSMap(slice, ":") // {"a": "1", "b": "2", "c": "3"}

📝 New String Utilities

// Case-insensitive containment check
strutil.IContainsOne("Hello World", []string{"hello", "golang"}) // true

// Fast comma joining
result := strutil.JoinComma([]string{"a", "b", "c"}) // "a,b,c"

// Variable name validation
isValid := strutil.IsVarName("valid_var123") // true

🧪 Advanced Text Templating

Implements simple shell-style variable interpolation with expressions:

• Variable format: $var_name, ${some_var}, ${top.sub_var}
• Function calls: ${func_name($var_name, 'const string')}

// Using StrVarRenderer
renderer := textutil.NewStrVarRenderer()
result := renderer.Render("Hello ${name}", map[string]string{"name": "World"}) // "Hello World"

🐛 Bug Fixes

• Fixed conversion errors in maputil.TryAnyMap for non-string key maps
• Resolved parsing issues in internal ParseEnvLines
• Improved test stability across multiple cases

🔧 Additional Improvements

Core Enhancements

• Renamed SMap to StrMap for naming consistency
• Extracted common ENV parsing logic to internal package
• Added DecodeFile() to jsonutil for simplified file operations

Refactoring

• Optimized map operations for large data structures
• Improved memory efficiency in string processing functions
• Enhanced test suite stability

📚 Documentation & Testing

• Updated documentation for structs, testutil, and x/finder
• Added 500+ new test cases
• Modernized GitHub Actions workflows

🔄 Upgrade Guide

Installation

go get github.com/gookit/goutil/v0.7.1

Key Changes

1. ​​API Change​​: SMap renamed to StrMap
2. ​​New Features​​: Review added utilities for workflow simplification
3. ​​Bug Fixes​​: Resolved previous TryAnyMap conversion issues

📈 Performance Metrics

• 15–20% faster map operations
• 30% reduced memory usage in string processing
• 98% test suite stability improvement

🤝 Community Contributions

Special thanks to all contributors! We welcome feedback and contributions via GitHub Issues.

📖 Resources

• 📄 Full Changelog
• 📦 Package Documentation
• 🐛 Report Issues

Upgrade now to leverage these powerful enhancements in your Go projects!

gookit/goutil v0.7.1 Released: Enhanced Toolset and Usage Optimization was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Openjob is a new distributed task scheduling framework based on Akka architecture. Supports multiple cronjob, delay task, workflow, lightweight distributed computing, unlimited horizontal scaling, with high scalability and fault tolerance. Also has complete management, powerful alarm monitoring, and support multiple languages

• High reliability
• Distributed with stateless design, using the Master/Worker architecture, supports multiple databases (MySQL/PostgreSQL/Oracle)
• High performance
• System uses a consistency sharding algorithm, lock-free design, task scheduling is accurate down to the second, supporting lightweight distributed computing and unlimited horizontal scaling
• Cronjob
• Supports distributed cronjob, fixed rate tasks, high-performance second tasks, and onetime tasks
• Distributed computing
• Supports multiple distributed programming models such as standalone, broadcast, Map, MapReduce, and sharding, easy to complete distributed computing for big data
• Delay task
• High performance delay task based on Redis , support multi-level storage, and provides rich statistics and reports
• Workflow
• Supports workflow scheduling engine, visual DAG design, and easy to complete complex task scheduling
• Permission management
• User management, supports menu, button, and data permission settings, flexible management of user permissions
• Alarm monitoring
• Overall monitoring metrics, rich and alarm in time, easy to locate and resolve online problem
• Multiple languages
• Support multiple languages such as Java, Go, PHP, and Python, as well as build with frameworks such as Spring Boot, Gin, and Swoft

If you are looking for a high-performance distributed task scheduling framework that supports cronjob, delay task, lightweight computing, workflow, and supports multiple programming languages, then Openjob is definitely the way to go.

Released

Openjob 1.0.7 newly supports H2/TiDB database and adds second delay task, fixed rate task, broadcast task, sharding task, and Map Reduce lightweight computing.

Second Delay

Second delay support scheduling at interval of 1 to 60 seconds. After each task execution is completed, scheduling will be triggered again at second intervals.

• High reliability: Second delay have high reliability characteristics. If a machine goes down, it can be re-run on another machine.
• Rich task types: Second delay belong to the cron type and can be applied to all task types and execution types.

Fixed Rate

Crontab must be divisible by 60, if task executed every 50 minutes, Crontab cannot support it.

Map Reduce

MapReduce is a lightweight distributed batch task. Implemented by the MapProcessor or MapReduceProcessor. Compared with big data batch task (such as Hadoop, Spark, etc.), MapReduce does not need to import data into the big data platform, and has no additional storage and computing costs. It can realize second data processing and the advantages of low cost, fast speed and programming.

Example

Define executor by annotation as follows:

/**
 * @author stelin swoft@qq.com
 * @since 1.0.7
 */
@Component("mapReduceTestProcessor")
public class MapReduceTestProcessor implements MapReduceProcessor {
    private static final Logger logger = LoggerFactory.getLogger("openjob");

    private static final String TWO_NAME = "TASK_TWO";
    private static final String THREE_NAME = "TASK_THREE";

    @Override
    public ProcessResult process(JobContext context) {
        if (context.isRoot()) {
            List<MapChildTaskTest> tasks = new ArrayList<>();
            for (int i = 1; i < 5; i++) {
                tasks.add(new MapChildTaskTest(i));
            }
            logger.info("Map Reduce root task mapList={}", tasks);
            return this.map(tasks, TWO_NAME);
        }
        if (context.isTask(TWO_NAME)) {
            MapChildTaskTest task = (MapChildTaskTest) context.getTask();
            List<MapChildTaskTest> tasks = new ArrayList<>();
            for (int i = 1; i < task.getId()*2; i++) {
                tasks.add(new MapChildTaskTest(i));
            }
            logger.info("Map Reduce task two mapList={}", tasks);
            return this.map(tasks, THREE_NAME);
        }
        if (context.isTask(THREE_NAME)) {
            MapChildTaskTest task = (MapChildTaskTest) context.getTask();
            logger.info("Map Reduce task three mapTask={}", task);
            return new ProcessResult(true, String.valueOf(task.getId() * 2));
        }
        return ProcessResult.success();
    }
    @Override
    public ProcessResult reduce(JobContext jobContext) {
        List<String> resultList = jobContext.getTaskResultList().stream().map(TaskResult::getResult)
                .collect(Collectors.toList());
        logger.info("Map Reduce resultList={}", resultList);
        return ProcessResult.success();
    }
    @Data
    @AllArgsConstructor
    @NoArgsConstructor
    public static class MapChildTaskTest {
        private Integer id;
    }
}

Sharding

Sharding model includes static sharding and dynamic sharding:

• Static sharding: The main scenario is to process a fixed number of sharding. For example, there are 256 databases in sub-databases, which requires many worker to be processed in a distributed.
• Dynamic sharding: The main scenario is distributed processing of unknown data. For example, a large table is constantly changing and requires distributed batch running.

Feature

• Compatible with elastic-job’s static sharding.
• Supports languages: Java, PHP, Python, Shell, and Go.
• High availability: When the worker that executes a sharding task becomes abnormal, it will be dynamically assigned to other normal worker to execute the task.
• Flow control: You can set the number of concurrent subtasks for a single worker. For example, if there are 100 sharding and a total of 3 worker, up to 5 sharding can be controlled to execute concurrently, and the others are waiting in the queue.
• Retry on failure: Automatically retry when subtask execution fails.

Example

Define executor by annotation as follows:

/**
 * @author stelin swoft@qq.com
 * @since 1.0.7
 */
@Component
public class ShardingAnnotationProcessor {
    private static final Logger logger = LoggerFactory.getLogger("openjob");

    @Openjob("annotationShardingProcessor")
    public ProcessResult shardingProcessor(JobContext jobContext) {
        logger.info("Sharding annotation processor execute success! shardingId={} shardingNum={} shardingParams={}",
                jobContext.getShardingId(), jobContext.getShardingNum(), jobContext.getShardingParam());
        logger.info("jobContext={}", jobContext);
        return ProcessResult.success();
    }
}

Broadcast

Broadcast task instance will be broadcast to all workers to the application for execution. Task will be completed when all workers have completed execution. If any worker execute fail, the task will fail.

Application scenarios​

Batch operation​

• Broadcast to all workers to run a script.
• Broadcast to all workers to cleaning data regularly.
• Dynamically start a service on each worker

Data aggregation​

• Initialize using JavaProcessor->preProcess
• When each worker executes the process, it returns the result according to the business.
• Execute postProcess to obtain the execution results of all workers.

Feature​

There are many types of broadcast tasks, such as script or Java Processor. If Java processor, it can supports preProcess and postProcess advanced features.

• preProcess will be executed before each worker execute process, and will only be executed once.
• process is the actual task execution.
• postProcess will be executed once after the process execution on each worker is completed and successfully executed, and the results can be returned as data transmission.

Example

Define executor by annotation as follows:

/**
 * @author stelin swoft@qq.com
 * @since 1.0.7
 */
@Component("broadcastPostProcessor")
public class BroadcastProcessor implements JavaProcessor {
    private static final Logger logger = LoggerFactory.getLogger("openjob");

    @Override
    public void preProcess(JobContext context) {
        logger.info("Broadcast pre process!");
    }
    @Override
    public ProcessResult process(JobContext context) throws Exception {
        logger.info("Broadcast process!");
        return new ProcessResult(true, "{\"data\":\"result data\"}");
    }
    @Override
    public ProcessResult postProcess(JobContext context) {
        logger.info("Broadcast post process taskList={}", context.getTaskResultList());
        System.out.println(context.getTaskResultList());
        return ProcessResult.success();
    }
}

More

• Official website: https://openjob.io
• Github: https://github.com/open-job/openjob
• Live demo: https://demo.openjob.io
• username: openjob
• password: openjob.io

GitHub - open-job/openjob: Distributed high performance task scheduling framework

Best distributed task scheduling framework — Openjob 1.0.7 released was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

More powerful and intelligent task scheduling framework.

Introduction

Openjob is a new distributed task scheduling framework based on Akka architecture. Supports multiple cronjob, delay task, workflow, lightweight distributed computing, unlimited horizontal scaling, with high scalability and fault tolerance. It also has complete management, powerful alarm monitoring, and support multiple languages

• Complete task log, and suppport storage (H2/Mysql/Elasticsearch).
• detailed recording of task execution stack information
• Provides task event monitoring alarms, detailed alarm histories, and support notifications with Webhook,Wecom, ,Dingding and Feishu triggers.
• Designed with namespace, support button-level access and easy to manage complex project.
• Supports multiple programming languages, such as Java, Go, PHP and Python and various languages is very friendly.

If you are looking for a high-performance distributed task scheduling framework that supports cronjob, delay task, lightweight computing, workflow, and supports multiple programming languages, then Openjob is definitely the way to go.

Feature

Openjob not only supports basic cronjob, but also provides delayed jobs, distributed computing, and workflow

Cronjob

• Cronjob, support Unix Crontab expression
• Second, execution cycle less than 60 seconds
• Fixed rate, execute tasks at a fixed frequency with minute unit

Delay Task

• Distributed, high-performance delay task based on Redis, and providing rich reports and statistics

Distributed Computing

• Standalone, execute on a worker client
• Broadcast, execute on all worker clients
• Map, a map function can distribute big data to multiple machines for execution, like Hadoop map
• MapReduce, MapReduce is an extension of the Map.After all map sub-tasks are completed, the Reduce method is executed, which can process the results and data of the task execution in the Reduce method.
• Sharding, like Elastic-Job model, configure sharding numbers on the management, which can be scheduled to different client by sharding, and supports multiple languages.

Processor

• Processor, execute by function or class(support Java/Golang/PH)
• HTTP, http request, used to periodically request an HTTP
• Kettle, built-in Kettle command executor
• Shell, shell script

Visual operations

• Dashboard, rich task statistics and reports
• Task history, task execution history records
• Task log, complete task log, and suppport storage (H2/Mysql/Elasticsearch).
• Task running stack, detailed recording of task execution stack information

Alarms and permissions

• Provides task event monitoring alarms, detailed alarm histories, and support notifications with Webhook,Wecom, Dingding and Feishu triggers.
• Designed with namespace, support button-level access and easy to manage complex project.

Multiple languages

• Java java and its frameworks, with native support.
• Go golang support use go mod install
• PHP PHP support use Golang agent to execute task by command mode 。Swoole frameworks support composer install.
• Python python support use Golang agent to execute task by command mode

Application Scenario

Openjob is well-suited for business scenarios that have task schedule and delay task. such as every day to clean data and report generation. It is also suitable for lightweight computing, and Map/MapReduce can process big data computing. For complex task flows or workflow, it can design workflow with UI

Update

Openjob v1.0.6 add monitoring and alarm, fixed known issues

Feature

[#141] Add alarm and monitoring(dingding/wecom/feishu/webhook)
[#141] Add execute timeout for cronjob
[#141] Add child fail status
[#144] Add next execute time
[#144] Add personal page
[#144] Add running status to dashboard

Bugfix

[#144] Fixed big task log

Optimize

[#144] Auto create index for Elasticsearch7
[#144] User default avatar

More

• Official website: https://openjob.io
• Github: https://github.com/open-job/openjob
• Live demo: https://demo.openjob.io
• username: openjob
• password: openjob.io

GitHub - open-job/openjob: Distributed high performance task scheduling framework

More powerful and intelligent task scheduling framework — Openjob 1.0.6 published was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

gookit/goutil An extended library of Go's common functionality. Contains: number, string, slice/array, Map, struct, reflection, text, file, error, time and date, test, CLI, command run, system information, formatting, common information acquisition, etc.

Github: https://github.com/gookit/goutil

v0.6.10 change log

Full changelog v0.6.9…v0.6.10

✨ New features

• ✨ feat: testutil — adds a new child package fakeobj to create fake objects for testing
• ✨ feat: testutil — adds a new function NewEchoServer() to start the echo server
• ✨ feat: byteutil — Buffer adds more useful methods for ease of use
• ✨ feat: dump — Special handling of custom int, uint type values, will print instructions in String format
• ✨ feat: fsutil — adds a new function to Glob() quickly list matching files
• ♻️ feat: httpreq — Refactoring the built-in HTTP client httpreq logic

👔 Update Adjustments

• 👔 up: structs — Update and support InitDefaults() for struct slice fields that are not null
• 👔 up: maputil — SimpleMerge() Supports deep merging of map[string]any data

Other adjustments

• ✅ test: Update some documentation and fix some unit tests
• ⬆️ dep: Update golang.org/x depends on the latest version

Some feature usage examples

dump — prints custom types

dump — special handles custom int, uint type values, and will print instructions in String format

Initialize the struct

• Support for initializing the use of environment variables
• Support initializing slice fields and nesting structs

type ExtraDefault struct {
    City   string `default:"some where"`
    Github string `default:"${ GITHUB_ADDR }"`
}

type User struct {
    Name  string        `default:"inhere"`
    Age   int           `default:"300"`
    Extra *ExtraDefault `default:""` // mark: need init
}

optFn := func(opt *structs.InitOptions) {
    opt.ParseEnv = true
}
obj := &User{}
err := structs.InitDefaults(obj, optFn)
goutil.PanicErr(err)

dump.P(obj)

Initialization result:

&structs_test.User {
  Name: string("inhere"), #len=6
  Age: int(300),
  Extra: &structs_test.ExtraDefault {
    City: string("some where"), #len=10
    Github: string("https://some .... url"), #len=21
  },
},

Test with Echo Server

Use testutil.NewEchoServer() to quickly create an HTTP echo server. Convenient for testing HTTP requests, responses, etc.

Usage examples:

var testSrvAddr string

func TestMain(m *testing.M) {
    s := testutil.NewEchoServer()
    defer s.Close()

    testSrvAddr = "http://" + s.Listener.Addr().String()
    fmt.Println("server addr:", testSrvAddr)
    m.Run()
}

func TestNewEchoServer(t *testing.T) {
    // can direct request the echo server
    r, err := http.Post(testSrvAddr, "text/plain", strings.NewReader("hello!"))
    assert.NoErr(t, err)
    // binding response to the testutil.EchoReply
    rr := testutil.ParseRespToReply(r)
    
    dump.P(rr)
    assert.Eq(t, "POST", rr.Method)
    assert.Eq(t, "text/plain", rr.ContentType())
    assert.Eq(t, "hello!", rr.Body)
}

v0.6.9 change log

Full changelog v0.6.8…v0.6.9

✨ New features

• strutil — Added ToByteSize(),SafeByteSize() for converting size string (eg: 5MB) to byte size
• strutil has added ParseSizeRange() to facilitate and quickly parse string size range expressions eg: 200kb~50mb
• strutil — Added new utility functions: DatetimeNo(), RandWithTpl(), SimpleMatch()
• byteutil — Add new utility functions: Random(), AppendAny()
• fsutil — Add new utility functions: ReadOrErr(), ReadStringOrErr()
• cliutil — Add new utility functions: Confirm(), ReadAsBool()
• errorx — Add new utility functions Err(), Errf() for easy creation of error
• structs — InitDefaults() enhanced to support nested structure pointer initialization
• structs — SetValues() enhanced to support setting and processing structure pointer values
• structs — Add ToSMap(), TryToSMap(), TryToSMap() to convert structure to string map
• testutil/assert new assertion methods NotContainsKey(), NotContainsKeys()
• Reflects new utility methods UnexportedValue(), SetUnexportedValue()
• maputil new tool functions HasOneKey(), CombineToMap(), TryAnyMap()
• Added utility functions AnyToSlice(), CombineToMap() in arrutil
• jsonutil has added tool methods IsJSON(), IsJSONFast() to check whether it is JSON
• ♻️ fsutil/finder file and directory search reconstruction, enhanced functions, more convenient to use
• timex new utility functions NowAddSec(), IsDuration(), InRange()
• timex ToDuration() enhanced, support unit d, w and long unit hour, min, sec
• mathutil new tool methods InRange(), OutRange(), InUintRange()
• fsutil new tool method RemoveSub() to quickly delete subdirectories and files
• maputil GetByPath() is enhanced to support matching paths similar to top.*.field to get values

👔 Update adjustments

• strutil optimization ToByteSize() and GlobMatch() logic
• struts — InitDefaults() supports initializing non-zero ptr substructure fields
• 💥 struts — InitDefaults() To initialize a substructure, the field must be marked with default:""

Other adjustments

• 📝 doc: README documentation updated, some subpackages added README
• ✅ test: Added and fixed some unit tests
• ⬆️ dep: update golang.org/x dependencies to latest version

Some feature usage examples

finder — find files or dir

fsutil/finder Provides a simple and fast way to find matching files and directories.

• Built-in support Name, extension, suffix, prefix, glob, path, etc. include or exclude
• Support matching file size, modification time (ModTime) and other extended matching methods

ff := finder.NewFinder("/path/to/dir/").
    // OnlyFindDir(). // default find files
    UseAbsPath().
    WithoutDotDir().
    WithDirName("testdata").
    Include(finder.HumanModTime("<10m")). // latest 10 minutes, to Now
    Include(finder.HumanSize("500kb~10mb")) // limit file size

// Find() returns chan, can be for handling
for el := range f.Find() {
    fmt.Println(el.Path())
}

strutil.ParseSizeRange

It is easy to parse the string size range to byte size.

opt := &strutil.ParseSizeOpt{}

mix, max, err := strutil.ParseSizeRange("1kb~1mb", opt)
goutil.PanicErr(err)
fmt.Println(min, max) // OUTPUT: 1024, 1048576

Examples of supported expression formats:

"1KB~2MB"       => 1KB to 2MB
"-1KB"          => <1KB
"~1MB"          => <1MB
"< 1KB"         => <1KB
"1KB"           => >1KB
"1KB~"          => >1KB
">1KB"          => >1KB
"+1KB"          => >1KB

timex.ParseRange()

timex.ParseRange() It is easy to quickly parse the relative time size range or keyword into time.Time

start, end, err := ParseRange("-1h~1h", nil)
goutil.PanicErr(err)

fmt.Println(start, end)

Examples of supported expression formats:

"-5h~-1h"           => 5 hours ago to 1 hour ago
"1h~5h"             => 1 hour after to 5 hours after
"-1h~1h"            => 1 hour ago to 1 hour after
"-1h"               => 1 hour ago to feature. eq "-1h~"
"-1h~0"             => 1 hour ago to now.
"< -1h" OR "~-1h"   => 1 hour ago.
"> 1h" OR "1h"      => 1 hour after to feature

// keyword: now, today, yesterday, tomorrow
"today"          => today start to today end
"yesterday"      => yesterday start to yesterday end
"tomorrow"       => tomorrow start to tomorrow end

More usage

For more instructions, see Unit tests for README and related methods

GitHub - gookit/goutil: 💪 Helper Utils(800+): int, byte, string, array/slice, map, struct, dump, convert/format, error, web/http, cli/flag, OS/ENV, filesystem, system, test/assert, time and more. Go 常用的工具函数：数字，字符串，数组，Map，结构体，反射，文本，文件，错误，时间日期，特殊处理，格式化，常用信息获取等等

gookit/goutil — released v0.6.10, an extension library of common Go features was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Openjob is a new distributed task scheduling framework based on Akka. Supports a variety of cronjob, delayed jobs, and workflow. Uses a consistency sharding algorithm and supports unlimited horizontal scaling.

Feature

Openjob not only supports basic cronjob, but also provides delayed jobs, distributed computing, and workflow

Cronjob

• Cronjob，support Unix Crontab expression
• Second，execution cycle less than 60 seconds
• Fixed rate，execute tasks at a fixed frequency with minute unit

Delay Task

• Distributed, high-performance delay task based on Redis, and providing rich reports and statistics

Distributed Computing

• Standalone， execute on a worker client
• Broadcast， execute on all worker clients
• Map，a map function can distribute big data to multiple machines for execution, like Hadoop map
• MapReduce， MapReduce is an extension of the Map.After all map sub-tasks are completed, the Reduce method is executed, which can process the results and data of the task execution in the Reduce method.
• Sharding， like Elastic-Job model, configure sharding numbers on the management, which can be scheduled to different client by sharding, and supports multiple languages.

Processor

• Processor, execute by function or class(support Java/Golang/PH)
• HTTP, http request, used to periodically request an HTTP
• Shell,shell script

Visual operations

• Dashboard，rich task statistics and reports
• Task history，task execution history records
• Task log，complete task log, and suppport storage (H2/Mysql/Elasticsearch).
• Task running stack，detailed recording of task execution stack information

Alarms and permissions

• Provides task event monitoring alarms, detailed alarm histories, and support notifications with WeChat, Feishu, and webhook triggers.
• Designed with namespace, support button-level access and easy to manage complex project.

Multiple languages

• Java java and its frameworks, with native support.
• Go golang support use go mod install
• PHP PHP support use Golang agent to execute task by shell mode 。Swoole frameworks support composer install.
• Python python support use Golang agent to execute task by shell mode

Use cases

Openjob is well-suited for business scenarios that have task schedule and delay task. such as every day to clean data and report generation. It is also suitable for lightweight computing, and Map/MapReduce can process big data computing. For complex task flows or workflow, it can design workflow with UI

Open source

More

• Official website: https://openjob.io
• Github: https://github.com/open-job/openjob
• Live demo: https://demo.openjob.io

GitHub - open-job/openjob: Distributed high performance task scheduling framework

Openjob: best distributed task scheduling framework was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.