[FEATURE] Loki datasource plugin

[abelyakin]

Description

This PR introduces a new Loki datasource plugin for Perses, enabling log querying and visualization capabilities. The implementation provides a foundation for log-based observability within Perses dashboards.
perses/perses#2999

Key Features

• Loki Datasource Plugin: Complete implementation supporting Loki API integration
• Loki Query Plugin: Enables LogQL query execution with results displayed via Perses table component
• Basic LogQL Editor: Streamlined editor using @uiw/react-codemirror and @grafana/lezer-logql for syntax highlighting and validation. Autocomplete version is WIP and will be released in future PR's.
• Logs panel: Panel specialized for logs display. UI heavily inspired from other observability platforms. (Screenshots above)
  • Display logs stream
  • Level color indication
  • Expand / collapse row for details
  • Virtualized table for performance
  • Basic panel configs: Wrap lines, Disable expandable rows, show/hide timestamp.

LogQL Editor will be improved as well as separate Logs panel will be introduced in next PRs.

Screenshots

Checklist

• Pull request has a descriptive title and context useful to a reviewer.
• Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
  following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
• All commits have DCO signoffs.

UI Changes

• Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
• Code follows the UI guidelines.

[Nexucis]

so here I guess we need to provide another super type of Query. Like LogQuery

[abelyakin]

That's what I was thinking about as well. While Loki is primarily a log query datasource, it can also return matrix response data that's consumable by other panels which support time series.

So my idea is to use it as a subclass of TimeSeriesQuery.

[jgbernalp]

This plugin will probably support both as we can have metrics from logs which match the time series query format, and specific Logs data that was added perses/perses#3104

[abelyakin]

Thank you for the suggestion and PR! I’ve added a separate LokiLogQuery type of LogQuery kind, as introduced in your PR. Now we have distinct query kinds for the two main Loki query types:

• Streams (actual logs): LokiLogQuery (of LogQuery kind)
• Matrix (timeseries): LokiTimeSeriesQuery (of TimeSeriesQuery kind)

This separation should make the usage of each query type clearer and also allows panels to explicitly define which query types they support

[Nexucis]

Does it make sense this panel is an independent plugin instead of being related to the Loki datasource ?

[AntoineThebaud]

The question is, how much (are we confident that) this panel is agnostic to Loki specificities & could be compatible with other future log data sources?

[abelyakin]

Yes, it makes sense to keep this panel datasource-agnostic. We already have a PR with ClickHouse support #241 , but it currently duplicates current LogsTable code and defines a separate log interface. The panel should consume a small Common Log Envelope, and each datasource (Loki, ClickHouse) maps into that envelope.
ClickHouse itself doesn’t enforce a schema, but since the docs recommend OTEL logs, we should explicitly state that the ClickHouse adapter only supports OTEL and reflect that in the shared interface.

This avoids duplication and lets us add others later (Elasticsearch/OpenSearch etc.) by implementing adapters that emit the same envelope.

Aligning both datasources would require adjusting two PRs at once, which feels like too much for now. I think we should properly discuss the direction first, and I’m open for feedback.

[Nexucis]

Aligning both datasources would require adjusting two PRs at once, which feels like too much for now. I think we should properly discuss the direction first, and I’m open for feedback.

Well I think having in another PR the TableLogs panel would not hurt and it will open the discussion. Then afterwards we can align both PRs or plugins if the PRs are already merged.

[Nexucis]

Are you willing to lead the change @abelyakin ? Or at least opening the PR with the single panel, and probably the PR with the small common log envelope ?

[abelyakin]

I’m willing to lead the change 👍
What I’m wondering is whether it makes sense to approach this in two steps: first merge this panel as is to unblock progress, and then start working on a separate datasource agnostic version. That way we can align adapters without blocking the current PR.

What do you think would be the best sequence of steps here?

[Nexucis]

yes that would be acceptable as long as the new panel is named LogsTable to avoid a breaking change when the panel contained in this plugin will be removed.

[Nexucis]

and this is a really cool news you are willing to lead it. Thank you @abelyakin !

[Nexucis]

to fix the CI job ci/lint you need to run go mod tidy in the loki folder that should update the file go.sum.

To fix the ci/build, you need to add the dependencies @grafana/lezer-logql to the file loki/package.json as a dependency.

Also as we bumped again a new version of perses/perses (0.52.0-beta.5) you will need to update again the dependencies.

[Nexucis]

excepting my last comment, I think it's in a good shape. @jgbernalp do you see other things to update in this PR ? Perhaps it should follow also what you are doing in #266 ?

[jgbernalp]

excepting my last comment, I think it's in a good shape. @jgbernalp do you see other things to update in this PR ? Perhaps it should follow also what you are doing in #266 ?

Yes, there was some fixes to add. I took the liberty to also fix the dependencies and add one suggestion regarding the default values of the proxy. PTAL @abelyakin I created a PR on your fork abelyakin#1 that will fix the ci

[abelyakin]

excepting my last comment, I think it's in a good shape. @jgbernalp do you see other things to update in this PR ? Perhaps it should follow also what you are doing in #266 ?

Yes, there was some fixes to add. I took the liberty to also fix the dependencies and add one suggestion regarding the default values of the proxy. PTAL @abelyakin I created a PR on your fork abelyakin#1 that will fix the ci

Great! thanks, merged.

[abelyakin]

added basic tests since npm-test ci action requires tests for plugin

[jgbernalp]

Awesome contribution!

[Nexucis]

This is amazing. Thank you for this incredible contribution @abelyakin ! Looking forward for the next one around the common LogsTable!